Elastic Beanstalk Security Certificates

Overview

Instructions on installing and maintaining SSL for AWS Elastic Beanstalk web apps.

Amazon offers its own security certificates for load balanced EC2 instances, but not for smaller stand-alone instances.

Let's Encrypt offers free security certificates.

The instructions for installing Let's Encrypt change depending on the platform. E.g.

• Amazon Linux AMI vs Amazon Linux 2
• nginx server vs Apache server
• Web server for python vs web server for PHP.

It is important to determine which platform is needed, what its limitations are in terms of supporting Let's Encrypt certificates before proceeding.

Prerequisites

Enable SSH on the server

It is useful, evenessential, to be able to issue commands on the server from the command line,

See Enabling SSH Connections to an Elastic Beanstalk Environment

Open HTTPS port (443)

See Open HTTPS Port For Elastic Beanstalk Environments

Enable SSL on EC2 instance

There are (at least) two major types of EBS platforms Amazon Linux AMI vs Amazon Linux or Amazon Linux 2.

• TODO: Confirm if PHP is limited to one of these two platforms.
• TODO: Confirm if Python is limited to one of these two platforms.

Amazon Linux 2

• SSL library is mod_ssl.
• Default web server is nginx? (<-- TODO: confirm)
• PHP platform options appear to be limited to PHP for EBS. (<-- TODO: Confirm)

Amazon Linux AMI

• SSL library is mod24_ssl.

Apache server

The Amazon documentation instructs you to install mod_ssl with the following command which should create a file /etc/httpd/conf.d/ssl.conf when it completes.

Note that applies to Apache servers. Confirm that Apache is serving web requests with

$ sudo systemctl is-enabled httpd

If the command returns disabled then another server, most likely nginx, is already serving web requests.

$ sudo yum install mod_ssl

This did not work for me for dbarchowsky.com which was on a t1.micro instance, Amazon Linux AMI version 2018.03. What worked instead was:

$ sudo yum install mod24_ssl

These commands don't need to be entered manually. Instead they should be put into an .ebextensions config file so that the site can be deployed without manual configuration.

packages:
  yum:
    mod_ssl : []

Installing certbot

Certbot Commands documentation

certbot is a command line script that will install Let's Encrypt certificates and configure the current web server to recognize the certificates in order to serve https requests.

Let's encrypt certificates typically are stored in .pem files located in /etc/letsencrypt/live/mydomain.com.

Also the server's config file must be updated in order store the location of the certificate files. A typical path for this file is /etc/httpd/conf.d/ssl.conf.

Installing certbot from the command line

$ mkdir -p /opt/certbot 
$ wget https://dl.eff.org/certbot-auto -O /opt/certbot/certbot-auto 
$ chmod a+x /opt/certbot/certbot-auto

Installing certbot in .ebextensinos

The above commands likely would require sudo on the command line. The best location for this configuration is in a .ebextensions config file with something like this:

20_install_certbot:
    command: "mkdir -p /opt/certbot && wget https://dl.eff.org/certbot-auto -O /opt/certbot/certbot-auto && chmod a+x /opt/certbot/certbot-auto"

Installing LE Certificates with nginx

The following instructions originate from Let's Encrypt with AWS Elastic Beanstalk by “PirateFaché”. This installs LE certificates on nginx.

The following directives in the .ebextensions file will NOT change the configuration of the nginx server. See #Troubleshooting for details.

Creating /etc/nginx/conf.d/https_custom.conf allows requests to port 443 to access the LE certificates

# HTTPS server
server {
    listen       443 default ssl;
    server_name  localhost;
    error_page  497 https://$host$request_uri;

    ssl_certificate      /etc/letsencrypt/live/ebcert/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/ebcert/privkey.pem;

    ssl_session_timeout  5m;
    ssl_protocols  TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_prefer_server_ciphers   on;

    # Include the Elastic Beanstalk generated locations
    include conf.d/elasticbeanstalk/*.conf;
}

It’s necessary to create a symbolic link from the “ebcert” directory specified in the file above with the actual directory containing the LE certificates:

$ sudo ln -sf /etc/letsencrypt/live/wiki.dbarchowsky.com /etc/letsencrypt/live/ebcert

Restart nginx

$ sudo systemctl restart nginx

At this point it should be possible to make https requests to the site:

$ curl https://mydomain.com

TLS 1.2

Note that this line in /etc/nginx/conf.d/https_custom.conf

ssl_protocols  TLSv1.1 TLSv1.2;

Allows for both TLS 1.1 and TLS 1.2. TLS is the most current protocol (as of mid-2020) and addresses security issues with TLS 1.1. When confirming the certificate with SSL Labs this will result in a “B” grade only because TLS 1.1 is allowed. If you scroll down on the report TLS 1.2 is still supported. It looks like requests should still be handled if the client is limited to TLS 1.2.

Other Reference

• Nginx Server SSL Setup on AWS EC2 Linux with Letsencrypt
  This covers some of the same territory as the link at the top of this section. Some configuration examples using Let’s Encrypt and Amazon Linux 2
• Tutorial: Configure SSL/TLS on Amazon Linux 2
  This steers you to using apache, which is not running by default on an Amazon Linux 2 server by default. There is some information that helps understand how these servers are set up to process security certificates and https requests.
• SSL On Single Instance Elastic Beanstalk Tutorial
  Ultimately wasn’t helpful as it was configuring an Apache server. The basic steps are covered with working examples.
• Setup (AWS) Amazon Elastic Beanstalk to work with Let's Encrypt
  Another Apache example]

Converting EBS nginx platform to apache

Tutorial: Configure SSL/TLS on Amazon Linux 2 - AWS Documentation

The page above links to Tutorial: Install a LAMP Web Server on Amazon Linux 2 which describes installing an Apache web server on Amazon Linux 2.

As a part of those instructions, this diagnostic was provided to confirm the state of the httpd service:

$ systemctl status -l httpd.service

Which returned these lines:

May 30 04:12:27 ip-172-31-39-99.us-west-2.compute.internal httpd[16139]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/ssl.conf:
May 30 04:12:27 ip-172-31-39-99.us-west-2.compute.internal httpd[16139]: SSLCertificateFile: file '/etc/letsencrypt/live/LETSENCRYPT_DOMAIN/fullchain.pem' does not exist or is empty

“LETSENCRYPT_DOMAIN” should be the actual name of the domain that was used to create the security certificate. See below, [#Accessing_EBS_environment_variables_from_the_command_line]

This page, Migrating your Elastic Beanstalk Linux application to Amazon Linux 2, contains information about the difference between Amazon Linux and Amazon Linux 2, and some of the properties of Amazon Linux 2 that impact SSL.

$ sudo systemctl stop nginx

Then try running Apache.

Ok, after stopping nginx and then starting apache, it’s possible to get a response making a https request to the server. However, now in the EBS console the server’s status is listed as “degraded”.

Next question: how to put all of this into .ebextensions config files.

Also: make sure that the Let’s Encrypt certificate renews automatically.

Installing certificates

PHP

Use Let's Encrypt to install security certificates.

The source instructions gave this command:

$ /opt/letsencrypt/letsencrypt-auto --debug

But that returned the following error:

PluginError: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

This is caused by not having the typical VirtualHost running on port 80 in the Apache configuration, so Certbot needs an alternative method for authenticating the doamin. It's possible to manually specify the web root of the website: (Set /var/www/webroot to a path appropriate to the environment.)

$ /opt/letsencrypt/letsencrypt-auto --debug --authenticator webroot --installer apache -w /var/www/webroot -d mydomain.com -d www.mydomain.com

Note that the path to webroot can be different from the example above, e.g. /var/www/html

Preserving Let's Encrypt configuration

Create a configuration file at /etc/letsencrypt/config.ini.

$ echo "rsa-key-size = 4096" >> /etc/letsencrypt/config.ini
$ echo "email = ________@____.com" >> /etc/letsencrypt/config.ini

Python

Set Up Let's Encrypt SSL Certificate With AWS Elastic Beanstalk Single Instance has good clear instructions on how to accomplish this. Also see Terminating HTTPS on EC2 Instances Running Python in the AWS documentation, which is the basis for that blog post.

Create a config file in .ebextensions for SSL and use eb deploy to update the environment (this should also work for PHP environments).

With this method the domain names and admin email address are specified with environment variables. This is nice because staging and production can have different secure domain names while sharing the same code base.

To set the values for these variables: AWS Management Console > Elastic Beanstalk > choose application > choose environment > Configuration > Software > Modify

• LE_DOMAIN_ARGS - List of all domains to be included in the certificate. Each domain name should be preceded by the -d flag, e.g. -d mydomain.com -d www.mydomain.com
• LETSENCRYPT_DOMAIN - Primary domain for the purposes of creating a symlink between /etc/letsencrypt/live/ebcert/ and the directory where Let's Encrypt actually places the certificate files.
• LETSENCRYPT_EMAIL - Contact email

Reference the north-rose project for a working example of this configuration.

Troubleshooting

The AWS documentation assumes that you generate the certificates manually and insert the contents of the certificate in the .ebextensions config file. The alternative example above places the command to generate the certificates in the .ebextensions config file. There is a flag that is incompatible with production environments: --staging. This will cause the Let's Encrypt staging server to issue the certificates. The server address will also get stored in a local config file, so subsequent attempts to reissue the certificates without the --staging flag will still invoke that staging server.

One symptom of this situation are testing the domain's SSL with digicert will result in a report stating that "Certificate Name matches domaininquestion.com", however the issuer will be listed as "Fake LE Intermediate X1". A valid production certificate will have "Let's Encrypt Authority X3" as the issuer.

Another symptom is that the browser will state that the domain's certificate is not from a trusted source, even though everything will look as expected in the ssl.conf file and in /etc/letsencrypt/live/ebcert/.

Renewing certificates

Cron job

Let's Encrypt certificates expire every 90 days. Create a cron job (in /etc/crontab) that will run once per day to check and renew the certificates as necessary.

Follow the renewal of the certificate with a restart of the Apache server in order to have the server recognize any renewed certificates.

# Renew SSL Certs
0  1  *  *  *  ec2-user  /opt/letsencrypt/letsencrypt-auto --no-bootstrap renew

# Refresh Server
10 1  *  *  *  root      apachectl -k restart > /dev/null 2>&1

Manually

Certificates can be manually forced to renew with

$ /opt/letsencrypt/certbot-auto renew --force-renewal

Troubleshooting

Installation

systemctl command not found

The AWS documentation uses systemctl to restart the Apache server. If this command is not available use the service command instead.

Cannot find SSLCertificateFile directive

When running letsencrypt-auto or certbot-auto

Cannot find an SSLCertificateFile directive in /files/etc/httpd/conf/httpd-le-ssl.conf/IfModule/VirtualHost. VirtualHost was not modified
Unable to find an SSLCertificateFile directive

This was fixed by successfully installing mod_ssl

Re-installing certificates after upgrading an Elastic Beanstalk instance platform

When updating the platform of an Elastic Beanstalk instance (e.g. Linux 2.0.x > Linux 2.9.x), the /opt/letsencript/ directory is not copied over to the new EB instance.

It is necessary to go through all the installation steps for Let's Encrypt and a new Let's Encrypt certificate.

Web Application

URLs for evaluating a domain's SSL

• Qualys SSL Labs
• SSL Shopper
• High-Tech Bridge
• digicert

Let's Encrypt security certificates

It may be necessary to reinstall any Let's Encrypt security certificates if the upgrade process has not copied the /opt/letsencript/ directory.

ERR_CONNECTION_REFUSED in Chrome

Attempting to load the site using https protocol in Chrome results in ERR_CONNECTION_REFUSED error.

This was fixed after the certificate was installed (creating the httpd-le-ssl.conf file with correct SSL directives).

Reference

See also

• Let's Encrypt Integration Guide