Elastic Beanstalk Security Certificates

Overview

Instructions on installing and maintaining SSL for AWS Elastic Beanstalk web apps.

Amazon offers its own security certificates for load balanced EC2 instances, but not for smaller stand-alone instances.

Let's Encrypt offers free security certificates.

The instructions for installing Let's Encrypt change depending on the platform. E.g.

• Amazon Linux AMI vs Amazon Linux 2
• nginx server vs Apache server
• Web server for python vs web server for PHP.

It is important to determine which platform is needed, what its limitations are in terms of supporting Let's Encrypt certificates before proceeding.

Prerequisites

Enable SSH on the server

It is useful, evenessential, to be able to issue commands on the server from the command line,

See Enabling SSH Connections to an Elastic Beanstalk Environment

Open HTTPS port (443)

See Open HTTPS Port For Elastic Beanstalk Environments

Enable SSL on Elastic Beanstalk environment

There are (at least) two major types of EBS platforms Amazon Linux AMI vs Amazon Linux or Amazon Linux 2.

• TODO: Confirm if PHP is limited to one of these two platforms.
• TODO: Confirm if Python is limited to one of these two platforms.

Amazon Linux 2

• SSL library is mod_ssl.
• Default web server is nginx? (<-- TODO: confirm)
• PHP platform options appear to be limited to PHP for EBS. (<-- TODO: Confirm)

Amazon Linux AMI

• SSL library is mod24_ssl.

Generating LE Certificates

Unresolved questions

• Does uploading a new application version for an Elastic Beanstalk environment cause the security certificates and/or HTTPS configurations for that environment to be destroyed?
• Does updating the platform for an Elastic Beanstalk environment cause the security certificates and/or HTTPS configurations for that environment to be destroyed?

Process

Manually run certbot to install Let's Encrypt security certificates for the first time.

Prepare to install

Download the Extra Packages for Enterprise Linux (EPEL) 7 repository packages. These are required to supply dependencies required by Certbot.

$ cd ~
$ sudo wget -r --no-parent -A 'epel-release-*.rpm' https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/

Install the repository packages:

$ sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm

Enable EPEL:

$ sudo yum-config-manager --enable epel*

Now you can confirm that EPEL is enabled:

$ sudo yum repolist all

...
epel/x86_64                          Extra Packages for Enterprise Linux 7 - x86_64                               enabled: 12949+175
epel-debuginfo/x86_64                Extra Packages for Enterprise Linux 7 - x86_64 - Debug                       enabled:      2890
epel-source/x86_64                   Extra Packages for Enterprise Linux 7 - x86_64 - Source                      enabled:         0
epel-testing/x86_64                  Extra Packages for Enterprise Linux 7 - Testing - x86_64                     enabled:    778+12
epel-testing-debuginfo/x86_64        Extra Packages for Enterprise Linux 7 - Testing - x86_64 - Debug             enabled:       107
epel-testing-source/x86_64           Extra Packages for Enterprise Linux 7 - Testing - x86_64 - Source            enabled:         0
...

Install and run Certbot

1. Install Certbot packages and dependencies:

$ sudo yum install -y certbot python2-certbot-nginx

2. Run Certbot

$ sudo certbot

3. Follow the instructions and enter the appropriate responses at the prompts.

Configure automated certificate renewal

1. Edit /etc/crontab to add the following line:

39      1,13    *       *       *       root    certbot renew --no-self-upgrade

• 39 is the minutes after the hour to run the command
• 1,13 are the hours at which to run the command
• root is the user to use to run the command
• The rest is the command to execute.
• With these settings, the renewal will run twice daily.

2. Restart the cron deamon.

$ sudo systemctl restart crond

Other Platforms

Apache

See Converting Amazon Linux nginx to Apache

Python

Set Up Let's Encrypt SSL Certificate With AWS Elastic Beanstalk Single Instance has good clear instructions on how to accomplish this. Also see Terminating HTTPS on EC2 Instances Running Python in the AWS documentation, which is the basis for that blog post.

Create a config file in .ebextensions for SSL and use eb deploy to update the environment (this should also work for PHP environments).

With this method the domain names and admin email address are specified with environment variables. This is nice because staging and production can have different secure domain names while sharing the same code base.

To set the values for these variables: AWS Management Console > Elastic Beanstalk > choose application > choose environment > Configuration > Software > Modify

• LE_DOMAIN_ARGS - List of all domains to be included in the certificate. Each domain name should be preceded by the -d flag, e.g. -d mydomain.com -d www.mydomain.com
• LETSENCRYPT_DOMAIN - Primary domain for the purposes of creating a symlink between /etc/letsencrypt/live/ebcert/ and the directory where Let's Encrypt actually places the certificate files.
• LETSENCRYPT_EMAIL - Contact email

Reference the north-rose project for a working example of this configuration.

Troubleshooting

Installation

eb deploy failure

• Run eb logs in the local distribution of the environment to view the latest commands run in the EBS environment.
  • If eb logs generates a "permission denied" message, try eb logs --instance [EBS-INSTANCE-ID] --verbose
• View the tail of the file /var/log/cfn-init-cmd.log. This file will list all commands in .ebextensions and whether they executed successfully or not.
• Check the .ebextensions config files on the server in /var/app/staging/
• Confirm the domain settings in AWS Route 53.
• Confirm that the web files have been uploaded to /var/www/html/.
• Confirm the EBS raw URL to the site: AWS Console > Elastic Beanstalk > Environments > click on the URL for the environment in question
• Scale back the application and incrementally deploy.
  • Remove all web files except for a boilerplate "Hello, World" index file.
  • Comment out all commands in .ebextensions configuration files.
  • Remove composer.json
  • Run eb deploy to see if it can successfully upload this most basic environment.
  • Incrementally add back components of the application and redeploy to isolate problematic components.

Directives in .ebextensions config files aren't executed

AWS eb cli uses git HEAD to create zip file to upload to the server.

Confirm that zip files have been added and committed to the repo.

Or to deploy changes before they are committed:

$ eb deploy --staged

Make sure to stage any edits with git (e.g. git add) before running eb deploy with the --staged option!

Cannot create files in /etc/nginx/conf.d with .ebextensions

According to this Stackoverflow thread, WebSockets on Elastic Beanstalk with Docker, it seems that when EBS creates an application it basically clears out the nginx configuration after the .ebextensions commands are run. So any custom ngnix configuration done through .ebextensions would be overwritten.

I have confirmed this insofar as I put my nginx configuration in a file and uploaded it successfully to the ec2-user home directory. I put in another command to move that file to the nginx configuration directory, and after the application was successfully deployed, the custom nginx configuration file was gone.

There were some solutions offered on the Stackoverflow thread above. They involved moving Python scripts to an EBS “hooks” directory which would be executed after the application is deployed. There is no “hooks” directory in that location on my EBS server. For the time being, I am manually creating the nginx config file on the command line on the server after the application is deployed. This will allow the server to use the Let’s Encrypt certificates to serve https requests, and should stay in place through LE certificate renewals until the next application deployment.

The AWS documentation assumes that you generate the certificates manually and insert the contents of the certificate in the .ebextensions config file. The alternative example above places the command to generate the certificates in the .ebextensions config file. There is a flag that is incompatible with production environments: --staging. This will cause the Let's Encrypt staging server to issue the certificates. The server address will also get stored in a local config file, so subsequent attempts to reissue the certificates without the --staging flag will still invoke that staging server.

systemctl command not found

The AWS documentation uses systemctl to restart the Apache server. If this command is not available use the service command instead.

Cannot find SSLCertificateFile directive

When running letsencrypt-auto or certbot-auto

Cannot find an SSLCertificateFile directive in /files/etc/httpd/conf/httpd-le-ssl.conf/IfModule/VirtualHost. VirtualHost was not modified
Unable to find an SSLCertificateFile directive

This was fixed by successfully installing mod_ssl

Re-installing certificates after upgrading an Elastic Beanstalk instance platform

When updating the platform of an Elastic Beanstalk instance (e.g. Linux 2.0.x > Linux 2.9.x), the /opt/letsencript/ directory is not copied over to the new EB instance.

It is necessary to go through all the installation steps for Let's Encrypt and a new Let's Encrypt certificate.

Connectivity

ERR_CONNECTION_REFUSED in Chrome

Attempting to load the site using https protocol in Chrome results in ERR_CONNECTION_REFUSED error.

Check the security certificates in /etc/letsencrypt/live/. There should be a directory with the name of the domain, and another directory named ebcert that is a symbolic link to /etc/letsencrypt/live/securedomainname.com

Check that the server is configured to accept requests on port 443, e.g. in /etc/nginx/conf.d/https_custom.conf

nginx configuration is set back to defaults during eb deploy, meaning the certificates configuration is removed from the server. Also, haven't figured out how to insert custom configuration on the server via .ebextensions configuration directives. It may be necessary to copy this https configuration file manually after running eb deploy.

Certificates

URLs for evaluating a domain's SSL

• Qualys SSL Labs
• SSL Shopper
• High-Tech Bridge
• digicert

Let's Encrypt security certificates

It may be necessary to reinstall any Let's Encrypt security certificates if the upgrade process has not copied the /opt/letsencript/ directory.

Error running certbot: "Sorry, I don't know how to bootstrap Certbot on your operating system!"

There can be an issue running the certbot script (and certbot-auto) on Amazon Linux 2 where a line in the script does not correctly identify the OS.

Edit /opt/certbot/certbot-auto to replace this line

elif [ -f /etc/redhat-release ]; then

with

elif [ -f /etc/redhat-release ] || grep 'cpe:.*:amazon_linux:2' /etc/os-release > /dev/null 2>&1; then

On the command line this can be achieved with:

$ sed -i '/elif \[ -f \/etc\/redhat-release \];[[:space:]]*then/c\elif [ -f /etc/redhat-release ] || grep '"'"'cpe:.*:amazon_linux:2'"'"' /etc/os-release > /dev/null 2>&1; then' /opt/certbot/certbot-auto

In yaml this can be achieved with:

30_certbot_os_test_fix:
        command: "sed -i '/elif \[ -f \/etc\/redhat-release \];[[:space:]]*then/c\elif [ -f /etc/redhat-release ] || grep '\"'\"'cpe:.*:amazon_linux:2'\"'\"' /etc/os-release > /dev/null 2>&1; then' /opt/certbot/certbot-auto"

Digicert reports certificate issuer as "Fake LE Intermediate X1"

One symptom of this situation are testing the domain's SSL with digicert will result in a report stating that "Certificate Name matches domaininquestion.com", however the issuer will be listed as "Fake LE Intermediate X1". A valid production certificate will have "Let's Encrypt Authority X3" as the issuer.

Another symptom is that the browser will state that the domain's certificate is not from a trusted source, even though everything will look as expected in the ssl.conf file and in /etc/letsencrypt/live/ebcert/.

Certificates are not renewed automatically

• SSH to EC2 server hosting the domain covered by the certifcate.
• Confirm that certbot is installed at /opt/certbot/certbot-auto.
• Confirm that the renewal task has been added as a cron tab. View the contents of /etc/crontab. See Renewing Certificates for the certbot renew command.
• Confirm that nginx configuration has not been removed by an application platform update. See Installing LE Certificates with nginx.

Reference

See also

• Let's Encrypt Integration Guide
• Force HTTPS requests - Wiki