Generating Certificates for Local Development

SHARE ON

Mixmax is a rich communications client integrated directly into Gmail using iframes. As such, weneed to load all our resources over https – both for the security of our users, and to comply with
Gmail’s strict security policies. This creates issues during our normal development workflow, where
we configure the app to refer to local resources.

Overview

Architecture

Our platform is hosted across a set of microservices, each having a unique role. We have a
microservice for serving contact autocomplete, rendering out the compose modal, displaying the
dashboard, and more. Every service sits on its own fully-qualified domain
name

(FQDN) – contacts.mixmax.com, compose.mixmax.com, and app.mixmax.com, respective to the
aforementioned services. For development, however, we need to host our microservices locally. To
maintain consistency and functional parity with our production environment, we put a proxy server in
front of our microservices, which handles requests to specific domains, such as app.mixmax.com,
and forwards them to the appropriate server. This parity means we don’t need to hard-code a
microservice’s port to make a request to it. Up until this point, unlike our production environment,
our proxy only handled http connections.

Problem

A content security policy determines how the browser handles requests to origins different from the
page’s origin. A strict content security policy like Gmail’s is intended to keep the user safe, in
particular disallowing insecure requests on a secure page. When those resource requests are
rejected, our app fails to load, which makes it impossible to test during development. Read more
about content security policy on HTML5
ROCKS
. Gmail’s content
security policy makes it so that if you inject content that uses insecure resources, the requests
for those resources are rejected. From the Mozilla Developer
Network
:

When a user visits a page served over HTTPS, their connection with the web server is encrypted
with TLS and is therefore safeguarded from sniffers and man-in-the-middle attacks. If the HTTPS
page includes content retrieved through regular, cleartext HTTP, then the connection is only
partially encrypted; the unencrypted content is accessible to sniffers and can be modified by
man-in-the-middle attackers, so the connection is not safeguarded. When a web page exhibits this
behavior, it is called a mixed content page.

As a result, we’ve had to run Chrome in an insecure mode when testing our changes. This introduces
developer friction, and makes results from testing locally inconsistent with results post-deploy, so
we’d like to mitigate this problem.

We’d like to share our findings while reducing friction in our development workflow, specifically
regarding changes to our proxy server. We’ll also discuss the changes we made to support our
existing livereload mechanism. The straightforward solution
entails enabling our local proxy server to serve content over https so that the content security
policy allows our app to function.

Generating the Certificates

Any https connection requires a set of certificates, which serve to prove the identity of the
organization or entity on the other end of the connection, and to encrypt the data between the two parties.

Actual certificates should not be used during this process – they are not necessary, and
distributing them opens them up to unnecessary risk. Were any developer’s computer compromised, the
attacker could use the certificates to intercept legitimate traffic to the real servers, and cause
real damage to both users and the company. Since we don’t want to use real certificates, we’ll need
to generate our own. In our case, we have a number of different domains we’d like to handle,
including those on two separate second-level
domains
: mixmax.com and
mixmaxusercontent.com.

To support arbitrary servers and minimize the number of changes we’ll need to make if we add a new
subdomain, we’ll use wildcard certificates for *.mixmax.com and *.mixmaxusercontent.com. We’ll
also create a self-signed certificate authority with which to sign the certificates. Finally, we’ll
add the certificate as trusted to the root certificate store, which instructs Chrome to trust the
certificate. Here’s the script:


#!/bin/bash
KEY_SIZE=2048
SUBJECT="/C=US/ST=California/L=San Francisco/O=Mixmax/OU=Engineering"
DOMAINS="mixmax.com mixmaxusercontent.com"

# generate the certificate authority private key
openssl genrsa -out ca.key "$KEY_SIZE"

# self-sign the certificate authority
CA_SUBJECT="$SUBJECT/CN=Mixmax Engineering" # add a common name
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -out ca.pem -subj "$CA_SUBJECT"

# create a .key and a wildcard .pem for each domain
for domain in $DOMAINS; do
  # add the wildcard common name to the subject
  LOCAL_SUBJECT="$SUBJECT/CN=*.$domain"

  # generate the local certificate
  openssl genrsa -out "$domain.key" "$KEY_SIZE"

  # generate the certificate signing request for the local certificate
  openssl req -new -key "$domain.key" -out tmp-local.csr -subj "$LOCAL_SUBJ"

  # sign the local certificate with the certificate authority
  openssl x509 -req -in tmp-local.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out  -days 512 -sha256
done

# cleanup
rm -f tmp-local.csr

We run this script once for each developer, rather than create one set of self-signed wildcard
certificates and check them in to git. This mitigates the admittedly
small risk of the pre-shared certificates leaking, and keeps our developers marginally safer. To
make installing the certificates easier, the above script also prompts the developer to add them as
trusted certificates to the keychain:

sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" ca.pem

It seems that Chrome doesn’t acknowledge the certificate unless it’s in the admin cert store, hence
the -d flag.

Configuring the Proxy

Our existing proxy server uses http-proxy to
actually proxy the individual requests to the microservices, and includes a short mapping of
fully-qualified domain names (FQDNs) to local ports. This mapping mirrors our production setup with
different FQDNs for each app server, and means we don’t need to hardcode the ports for each
microservice into our codebase.

The original proxy server:

const httpProxy = require('http-proxy'),
  proxy = httpProxy.createServer({ws: true});

const domainsToPorts = {
  'app.mixmax.com': 3000,
  'compose.mixmax.com': 3001,
  // ...
};

// Handle requests to the proxy.
function proxyRequest(req, res) {
  const destinationPort = domainsToPorts[req.headers.host];

  // Only proxy if we have a destination.
  if (destinationPort) {
    req.target = `http://localhost:${destinationPort}`;
    proxy.web(req, res, {
      target: req.target
    });
  } else {
    res.writeHead(503); // Bad gateway
    res.end();
  }
}

// Handle request upgrades on the proxy.
function proxyWebsocket(req, res) {
  proxy.ws(req, res, {
    target: req.target
  });
}

http.createServer(proxyRequest)
  .on('upgrade', proxyWebsocket)
  .listen(80);

It may seem as simple as replacing http with https, but it’s not. Node uses and
maintains
its own set of trusted certificates. As such,
we can’t simply make all server-to-server requests run over https without also making changes to
every server-to-service request location. Moreover, we want to give our developers a chance to
migrate their code – the parts that don’t directly integrate with Gmail – so we’ll keep the http
proxy code.

The https proxy code will, similar to the http proxy code, need to map individual requests to the
corresponding FQDN. Given the requirement that we support both mixmax.com and
mixmaxusercontent.com, and given that we don’t want to specify a port with each request, we’ll
need to use a TLS extension called Server Name
Indication
(SNI).
With SNI, before the server responds with its certificate, the client signals the name of the server
it wants to communicate with, and the server uses this to select the appropriate certificate. Node
supports SNI out of the box in both the
tls and https modules,
which require a little effort to configure correctly.

const domainsToCertificates = {
  'mixmax.com': {/* cert and key */},
  'mixmaxusercontent.com': {/* cert and key */}
};

// Use the server name from the client to select the appropriate certificate.
function SNICallback(servername, callback) {
  // Grab the top-level and second-level domain names.
  const domainMatch = /(?:^|.)((?:[^.])+.(?:[^.])+)$/.exec(servername),
    topAndSecondLevelDomains = domainMatch[1];

  // Create the secure context for this request.
  const secureContext = tls.createSecureContext(domainsToCertificates[topAndSecondLevelDomains]);

  // And hand off the secure context to tls to complete the handshake.
  callback(null, secureContext);
}

https.createServer({SNICallback}, proxyRequest)
  .on('upgrade', proxyWebsocket)
  .listen(443);

Fixing Livereload

Our app now loads fine, but our livereload doesn’t work anymore. Livereload is super useful during
development, so let’s fix it. Chrome is rejecting the websocket connections due to Gmail’s content
security policy, which is causing mixed content issues. We’re using
gulp-livereload and
connect-livereload, and the gulp plugin provides its own development
certificates. We haven’t instructed Chrome to trust those certificates, so we’ll need to feed the
livereload connections through our proxy as well.

Each of our servers uses a different port for livereload. In the name of reducing the use of
hardcoded ports, we’ll use <server>-livereload.mixmax.com to connect the proxy server to the
upstream livereload server.

At the Proxy

On the proxy side, we already have a mechanism to handle different domains – namely, the
domainsToPorts map – so it makes sense to handle the livereload subdomains there:

const domainsToPorts = {
  // ...
  'app-livereload.mixmax.com': 35729,
  'compose-livereload.mixmax.com': 35730,
  // ...
};

That’s all the proxy needs, because the existing code supports both loading the livereload script,
and upgrading requests into websockets for livereload.

On the Client

On the client side, we can override the port by specifying the port option to
connect-livereload, but we’d like to change the domain as well, which means we’ll need to use the
src option in connect-livereload.

There’s a catch – by default, livereload will use the domain and protocol of the page, but uses the
port option and falls back to the port 35729. We’d like the script to be served over https
on port 443, but we’d also like the subsequent connections to be over port 443. When reading the
src attribute of script tags, Chrome omits the port if it corresponds to the protocol.
Therefore, we can’t just specify the port in the URL. The undocumented query parameter port solves
the problem by overriding the default port:
https://app-livereload.mixmax.com/livereload.js?snipver=1&port=443. Now the livereload script
makes the correct request to the correct protocol/hostname/port tuple.

if (environment.is(Environment.LOCAL)) {
  app.use(require('connect-livereload')({
    // This is necessary to prevent the mixed-content issues with the secure proxy. The port query
    // parameter is parsed by the livereload script to ensure that it connects to 443 instead of the
    // default livereload port (35729).
    src: 'https://app-livereload.mixmax.com/livereload.js?snipver=1&port=443'
  }));
}

Conclusions

Now our developers can launch Chrome as normal, without needing to include flags to disable web
security. We hope you’ve gained some insight into setting up your own secure proxy for development.

Interested in a low-friction development workflow? Come join us at Mixmax.

SHARE ON

Written By

Eli Skeggs

Eli Skeggs

From Your Friends At