Elastic Beanstalk Security Certificates

Overview

Instructions on installing and maintaining SSL for AWS Elastic Beanstalk web apps.

Amazon offers its own security certificates for load balanced EC2 instances, but not for smaller stand-alone instances.

Let's Encrypt offers free security certificates.

The instructions for installing Let's Encrypt change depending on the platform. E.g.

• Amazon Linux AMI vs Amazon Linux 2
• nginx server vs Apache server
• Web server for python vs web server for PHP.

It is important to determine which platform is needed, what its limitations are in terms of supporting Let's Encrypt certificates before proceeding.

Prerequisites

Enable SSH on the server

It is useful, evenessential, to be able to issue commands on the server from the command line,

See Enabling SSH Connections to an Elastic Beanstalk Environment

Open HTTPS port (443)

See Open HTTPS Port For Elastic Beanstalk Environments

Enable SSL on Elastic Beanstalk environment

There are (at least) two major types of EBS platforms Amazon Linux AMI vs Amazon Linux or Amazon Linux 2.

• TODO: Confirm if PHP is limited to one of these two platforms.
• TODO: Confirm if Python is limited to one of these two platforms.

Amazon Linux 2

• SSL library is mod_ssl.
• Default web server is nginx? (<-- TODO: confirm)
• PHP platform options appear to be limited to PHP for EBS. (<-- TODO: Confirm)

Amazon Linux AMI

• SSL library is mod24_ssl.

Installing LE Certificates Manually

Unresolved questions

• Does uploading a new application version for an Elastic Beanstalk environment cause the security certificates and/or HTTPS configurations for that environment to be destroyed?
• Does updating the platform for an Elastic Beanstalk environment cause the security certificates and/or HTTPS configurations for that environment to be destroyed?

Process

Manually run certbot to install Let's Encrypt security certificates for the first time.

eb deploy commands will fail if .ebextensions commands contain any reference to the certificates and the security certificates have not been installed with Certbot.

Prepare to install

Download the Extra Packages for Enterprise Linux (EPEL) 7 repository packages. These are required to supply dependencies required by Certbot.

$ cd ~
$ sudo wget -r --no-parent -A 'epel-release-*.rpm' https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/

Install the repository packages:

$ sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm

Enable EPEL:

$ sudo yum-config-manager --enable epel*

Now you can confirm that EPEL is enabled:

$ sudo yum repolist all

...
epel/x86_64                          Extra Packages for Enterprise Linux 7 - x86_64                               enabled: 12949+175
epel-debuginfo/x86_64                Extra Packages for Enterprise Linux 7 - x86_64 - Debug                       enabled:      2890
epel-source/x86_64                   Extra Packages for Enterprise Linux 7 - x86_64 - Source                      enabled:         0
epel-testing/x86_64                  Extra Packages for Enterprise Linux 7 - Testing - x86_64                     enabled:    778+12
epel-testing-debuginfo/x86_64        Extra Packages for Enterprise Linux 7 - Testing - x86_64 - Debug             enabled:       107
epel-testing-source/x86_64           Extra Packages for Enterprise Linux 7 - Testing - x86_64 - Source            enabled:         0
...

Install and run Certbot

1. Install Certbot packages and dependencies:

$ sudo yum install -y certbot python2-certbot-nginx

2. Run Certbot

$ sudo certbot

3. Follow the instructions and enter the appropriate responses at the prompts.

Configure automated certificate renewal

systemctl start crond
    systemctl enable crond
    
    LINE="15 1 */29 * * certbot -q renew; service nginx reload; service nginx restart"
    
    (
        crontab -u root -l
        echo "$LINE"
    ) | crontab -u root -

• 15 1 */29 defines the times at which to run the command, e.g. 1:15 am every 29 days
• The tail end uses Certbot to renew the existing cerificates and restarts nginx

2. Restart the cron deamon.

$ sudo systemctl restart crond

Installing LE certificates with eb deploy

Overview

Previously, these configuration files worked to install certificates, but it relies on certbot-auto which is no longer supported on Amazon Linux 2. An updated installation repo is linked to in the thread which uses only certbot.

It executes a shell script that contains the commands described in the AWS documentation for Certbot.

The shell script is located in [PROJECT_ROOT]/.platform/hooks/postdeploy/ so that it can execute after the web files have been placed in the final web directory.

The certbot command can be configured so that it edits the nginx configuration files with the necessary SSL directives to apply the Let's Encrypt certificates.

As part of the application deployment Elastic Beanstalk replaces /etc/nginx/nginx.conf with its base configuration. This causes the SSL configuration to be overwritten on subsequent deployments of the application because certbot is not run again after its certificates have been installed.

To preserve SSL configuration, open up /etc/nginx/nginx.conf after the deploying the app the first time when certbot installs the certificates. The inserted certbot directives have associated comments identifying them. Copy these directives and copy them into a new nginx config file located locally at [PROJECT_ROOT]/.platform/nginx/conf.d/ssl.conf.

Files in [PROJECT_ROOT]/.platform/nginx/ will be copied to corresponding locations in /etc/nginx/ on the server.

Make sure to include the following at the end of server {} blocks if needed. Otherwise elastic beanstalk configurations could be ignored.

    # Include the Elastic Beanstalk generated locations
    include conf.d/elasticbeanstalk/*.conf;

Other Platforms

Apache

See Converting Amazon Linux nginx to Apache

Python

Set Up Let's Encrypt SSL Certificate With AWS Elastic Beanstalk Single Instance has good clear instructions on how to accomplish this. Also see Terminating HTTPS on EC2 Instances Running Python in the AWS documentation, which is the basis for that blog post.

Create a config file in .ebextensions for SSL and use eb deploy to update the environment (this should also work for PHP environments).

With this method the domain names and admin email address are specified with environment variables. This is nice because staging and production can have different secure domain names while sharing the same code base.

To set the values for these variables: AWS Management Console > Elastic Beanstalk > choose application > choose environment > Configuration > Software > Modify

• LE_DOMAIN_ARGS - List of all domains to be included in the certificate. Each domain name should be preceded by the -d flag, e.g. -d mydomain.com -d www.mydomain.com
• LETSENCRYPT_DOMAIN - Primary domain for the purposes of creating a symlink between /etc/letsencrypt/live/ebcert/ and the directory where Let's Encrypt actually places the certificate files.
• LETSENCRYPT_EMAIL - Contact email

Reference the north-rose project for a working example of this configuration.

Troubleshooting

Installation

See Troubleshooting Elastic Beanstalk Deployment

Certificates

URLs for evaluating a domain's SSL

• Qualys SSL Labs
• SSL Shopper
• High-Tech Bridge
• digicert

Command to confirm certificate expiration date

$ echo | openssl s_client -connect yourdomain.tld:443 -servername yourdomain.tld 2>/dev/null | openssl x509 -noout -dates

Let's Encrypt security certificates

It may be necessary to reinstall any Let's Encrypt security certificates if the upgrade process has not copied the /opt/letsencript/ directory.

Error running certbot: "Sorry, I don't know how to bootstrap Certbot on your operating system!"

There can be an issue running the certbot script (and certbot-auto) on Amazon Linux 2 where a line in the script does not correctly identify the OS.

Edit /opt/certbot/certbot-auto to replace this line

elif [ -f /etc/redhat-release ]; then

with

elif [ -f /etc/redhat-release ] || grep 'cpe:.*:amazon_linux:2' /etc/os-release > /dev/null 2>&1; then

On the command line this can be achieved with:

$ sed -i '/elif \[ -f \/etc\/redhat-release \];[[:space:]]*then/c\elif [ -f /etc/redhat-release ] || grep '"'"'cpe:.*:amazon_linux:2'"'"' /etc/os-release > /dev/null 2>&1; then' /opt/certbot/certbot-auto

In yaml this can be achieved with:

30_certbot_os_test_fix:
        command: "sed -i '/elif \[ -f \/etc\/redhat-release \];[[:space:]]*then/c\elif [ -f /etc/redhat-release ] || grep '\"'\"'cpe:.*:amazon_linux:2'\"'\"' /etc/os-release > /dev/null 2>&1; then' /opt/certbot/certbot-auto"

Digicert reports certificate issuer as "Fake LE Intermediate X1"

One symptom of this situation are testing the domain's SSL with digicert will result in a report stating that "Certificate Name matches domaininquestion.com", however the issuer will be listed as "Fake LE Intermediate X1". A valid production certificate will have "Let's Encrypt Authority X3" as the issuer.

Another symptom is that the browser will state that the domain's certificate is not from a trusted source, even though everything will look as expected in the ssl.conf file and in /etc/letsencrypt/live/ebcert/.

Certificates are not renewed automatically

• SSH to EC2 server hosting the domain covered by the certificate.
• Confirm that certbot with which certbot on the command line.
• Confirm that the renewal task has been added as a cron tab. View the contents of /etc/crontab. See Renewing Certificates for the certbot renew command.
• Confirm that nginx configuration has not been removed by an application platform update. See Installing LE Certificates with nginx.

Reference

See also

• Let's Encrypt Integration Guide
• Force HTTPS requests - Wiki