Elastic Beanstalk Security Certificates

Overview

Instructions on installing and maintaining SSL for AWS Elastic Beanstalk web apps.

Amazon offers its own security certificates for load balanced EC2 instances, but not for smaller stand-alone instances.

Let's Encrypt offers free security certificates.

The instructions for installing Let's Encrypt change depending on the platform. E.g.

• Amazon Linux AMI vs Amazon Linux 2
• nginx server vs Apache server
• Web server for python vs web server for PHP.

It is important to determine which platform is needed, what its limitations are in terms of supporting Let's Encrypt certificates before proceeding.

Prerequisites

Open SSH port (22)

It is useful, evenessential, to be able to issue commands on the server from the command line,

• AWS Management Console > EC2 > EC2 instance corresponding to the EBS environment > click for details > Security Group > click for details
• Inbound Rules tab > Edit button
• Add Rule button
  • Type: SSL
  • Protocol: TCP
  • Port Range: 22
  • Source: XX.XX.XX.XX/32 where XX.XX.XX.XX is the public IP address of the local development LAN.

This port should NEVER be publicly accessible. This setting should NEVER be specified in any config files for the web application.

For convenience, in the AWS management console view details fro the EC2 instance corresponding to the EBS environment. From there, click the Connect button at the top for details on how to connect to the instance.

Open HTTPS port (443)

• AWS Management Console > EC2 > EC2 instance corresponding to the EBS environment > click for details > Security Group > click for details
• Inbound Rules tab > Edit button
• Add Rule button
  • Type: HTTPS
  • Protocol: TCP
  • Port Range: 443
  • Source: 0.0.0.0/0, ::/0

This can also be accomplished with a directive in an .ebextensions config file. See the bottom of Terminating HTTPS on EC2 Instances Running Python in the AWS documentation for an example.

Key Pairs

• AWS Management Console > Elastic Beanstalk > web app environment > Configuration > Security
• EC2 Key Pair > choose from dropdown menu

Enable SSL on EC2 instance

There are (at least) two major types of EBS platforms Amazon Linux AMI vs Amazon Linux or Amazon Linux 2. The library is mod_ssl on Amazon Linux 2 and mod24_ssl on Amazon Linux AMI. PHP platform options in EBS seem to be Amazon Linux 2.

• TODO: Confirm that PHP is limited to that platform.
• TODO: Confirm if Python is limited to one of the platforms.

Apache server

The Amazon documentation instructs you to install mod_ssl with the following command which should create a file /etc/httpd/conf.d/ssl.conf when it completes.

Note that applies to Apache servers. Confirm that Apache is serving web requests with

$ sudo systemctl is-enabled httpd

If the command returns disabled then another server, most likely nginx, is already serving web requests.

$ sudo yum install mod_ssl

This did not work for me for dbarchowsky.com which was on a t1.micro instance, Amazon Linux AMI version 2018.03. What worked instead was:

$ sudo yum install mod24_ssl

These commands don't need to be entered manually. Instead they should be put into an .ebextensions config file so that the site can be deployed without manual configuration.

packages:
  yum:
    mod_ssl : []

Installing certbot

certbot is a command line script that will install Let's Encrypt certificates and configure the current web server to recognize the certificates in order to serve https requests.

Let's encrypt certificates typically are stored in .pem files located in /etc/letsencrypt/live/mydomain.com.

Also the server's config file must be updated in order store the location of the certificate files. A typical path for this file is /etc/httpd/conf.d/ssl.conf.

Installing certbot:

$ mkdir -p /opt/certbot 
$ wget https://dl.eff.org/certbot-auto -O /opt/certbot/certbot-auto 
$ chmod a+x /opt/certbot/certbot-auto

The above commands likely would require sudo on the command line. The best location for this configuration is in a .ebextensions config file with something like this:

20_install_certbot:
    command: "mkdir -p /opt/certbot && wget https://dl.eff.org/certbot-auto -O /opt/certbot/certbot-auto && chmod a+x /opt/certbot/certbot-auto"

Installing certificates

PHP

Use Let's Encrypt to install security certificates.

The source instructions gave this command:

$ /opt/letsencrypt/letsencrypt-auto --debug

But that returned the following error:

PluginError: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

This is caused by not having the typical VirtualHost running on port 80 in the Apache configuration, so Certbot needs an alternative method for authenticating the doamin. It's possible to manually specify the web root of the website: (Set /var/www/webroot to a path appropriate to the environment.)

$ /opt/letsencrypt/letsencrypt-auto --debug --authenticator webroot --installer apache -w /var/www/webroot -d mydomain.com -d www.mydomain.com

Note that the path to webroot can be different from the example above, e.g. /var/www/html

Preserving Let's Encrypt configuration

Create a configuration file at /etc/letsencrypt/config.ini.

$ echo "rsa-key-size = 4096" >> /etc/letsencrypt/config.ini
$ echo "email = ________@____.com" >> /etc/letsencrypt/config.ini

Python

Set Up Let's Encrypt SSL Certificate With AWS Elastic Beanstalk Single Instance has good clear instructions on how to accomplish this. Also see Terminating HTTPS on EC2 Instances Running Python in the AWS documentation, which is the basis for that blog post.

Create a config file in .ebextensions for SSL and use eb deploy to update the environment (this should also work for PHP environments).

With this method the domain names and admin email address are specified with environment variables. This is nice because staging and production can have different secure domain names while sharing the same code base.

To set the values for these variables: AWS Management Console > Elastic Beanstalk > choose application > choose environment > Configuration > Software > Modify

• LE_DOMAIN_ARGS - List of all domains to be included in the certificate. Each domain name should be preceded by the -d flag, e.g. -d mydomain.com -d www.mydomain.com
• LETSENCRYPT_DOMAIN - Primary domain for the purposes of creating a symlink between /etc/letsencrypt/live/ebcert/ and the directory where Let's Encrypt actually places the certificate files.
• LETSENCRYPT_EMAIL - Contact email

Reference the north-rose project for a working example of this configuration.

Troubleshooting

The AWS documentation assumes that you generate the certificates manually and insert the contents of the certificate in the .ebextensions config file. The alternative example above places the command to generate the certificates in the .ebextensions config file. There is a flag that is incompatible with production environments: --staging. This will cause the Let's Encrypt staging server to issue the certificates. The server address will also get stored in a local config file, so subsequent attempts to reissue the certificates without the --staging flag will still invoke that staging server.

One symptom of this situation are testing the domain's SSL with digicert will result in a report stating that "Certificate Name matches domaininquestion.com", however the issuer will be listed as "Fake LE Intermediate X1". A valid production certificate will have "Let's Encrypt Authority X3" as the issuer.

Another symptom is that the browser will state that the domain's certificate is not from a trusted source, even though everything will look as expected in the ssl.conf file and in /etc/letsencrypt/live/ebcert/.

Renewing certificates

Cron job

Let's Encrypt certificates expire every 90 days. Create a cron job (in /etc/crontab) that will run once per day to check and renew the certificates as necessary.

Follow the renewal of the certificate with a restart of the Apache server in order to have the server recognize any renewed certificates.

# Renew SSL Certs
0  1  *  *  *  ec2-user  /opt/letsencrypt/letsencrypt-auto --no-bootstrap renew

# Refresh Server
10 1  *  *  *  root      apachectl -k restart > /dev/null 2>&1

Manually

Certificates can be manually forced to renew with

$ /opt/letsencrypt/certbot-auto renew --force-renewal

Troubleshooting

Installation

systemctl command not found

The AWS documentation uses systemctl to restart the Apache server. If this command is not available use the service command instead.

Cannot find SSLCertificateFile directive

When running letsencrypt-auto or certbot-auto

Cannot find an SSLCertificateFile directive in /files/etc/httpd/conf/httpd-le-ssl.conf/IfModule/VirtualHost. VirtualHost was not modified
Unable to find an SSLCertificateFile directive

This was fixed by successfully installing mod_ssl

Re-installing certificates after upgrading an Elastic Beanstalk instance platform

When updating the platform of an Elastic Beanstalk instance (e.g. Linux 2.0.x > Linux 2.9.x), the /opt/letsencript/ directory is not copied over to the new EB instance.

It is necessary to go through all the installation steps for Let's Encrypt and a new Let's Encrypt certificate.

Web Application

URLs for evaluating a domain's SSL

• Qualys SSL Labs
• SSL Shopper
• High-Tech Bridge
• digicert

Let's Encrypt security certificates

It may be necessary to reinstall any Let's Encrypt security certificates if the upgrade process has not copied the /opt/letsencript/ directory.

ERR_CONNECTION_REFUSED in Chrome

Attempting to load the site using https protocol in Chrome results in ERR_CONNECTION_REFUSED error.

This was fixed after the certificate was installed (creating the httpd-le-ssl.conf file with correct SSL directives).

Reference

See also

• Let's Encrypt Integration Guide