Upgrading to Node 6 on Elastic Beanstalk

SHARE ON

This blog post is part of the Mixmax 2016 Advent Calendar. The previous post on December 1st was about Mixmax’s open-source culture.

In case you haven’t heard, Node 6 went LTS mid-October,
with AWS Elastic Beanstalk adding support at the
end of the month. Since Node 6
promised support for 99% of ES6 features as well as a host of
performance and security improvements, we moved quickly to adopt it. We found it to be very
easy to upgrade locally—we only had to upgrade a few native dependencies to their latest
version to pick up new bindings, and did not have to change any code. Kudos to the Node
Foundation for a stable release and the community for embracing Node 6 well in advance of LTS.

It was not as easy to upgrade Elastic Beanstalk,
however—upgrading the platform version
persistently resulted in stuck deploys and rollbacks. Debugging this required exploring
Elastic Beanstalk’s inner workings, but we ultimately made fixing it as simple as installing
a Node package. And not only did that package
enable us to upgrade Elastic Beanstalk to Node 6, but it also sped up npm install by 95%.
Here’s how we did it.

What went wrong

We initially tried to upgrade the platform version in place using
the upgrade button
on our application’s dashboard. But the configuration deploy never finished—boxes would
just time out. We then cloned the environment
with the latest platform. This initially succeeded, only for further deploys to fail.

Watching these deploys fail was agonizing. Elastic Beanstalk’s dashboard gives very little
insight into what’s going on during a deploy. But you can easily SSH into the EC2 instances
and tail the deployment logs. Using the
EB CLI tool:

eb ssh -i 
tail -f /var/log/eb-activity.log

This revealed that the boxes were getting stuck running npm install:

[2016-12-02T01:17:44.287Z] INFO  [27173] - [Application update app-91d6-161202_011650-stage-161202_011650@28/AppDeployStage0/AppDeployPreHook/50npm.sh] : Starting activity...

We were perplexed. EB’s docs said that platform 3.1.0 was using npm 2.15.5, same as the
previous platform. What was the difference?

We quickly suspected that EB’s docs were wrong, since Node 6.9.1 usually ships with npm 3.10.8. We confirmed this on an EC2 instance:

[ec2-user@ip-10-20-4-104 ~]$ export PATH=/opt/elasticbeanstalk/node-install/node-v6.9.1-linux-x64/bin:$PATH
[ec2-user@ip-10-20-4-104 ~]$ /opt/elasticbeanstalk/node-install/node-v6.9.1-linux-x64/bin/npm -v
3.10.8

(Note: we reported this to AWS on 11/11/2016,
but as of 12/01/2016 the docs are still wrong ?.)

We upgraded to npm 3 locally and timed npm install in several of our projects. We found that
npm 3.10.8 consistently takes about 2x longer to run npm install than npm 2.15.5. And on
the resource-constrained EC2 instances, it was taking even longer—much longer than the
command timeout.
Cloning the environment appeared to fix the problem only because EB uses a longer timeout when
creating an environment than when deploying configuration changes to an existing environment.

So the fix was going to involve downgrading npm 3 to npm 2… on every EC2 instance, across
all of our services, whenever Elastic Beanstalk deployed to a new instance. How could we
automate this?

ebextensions

Luckily, Elastic Beanstalk offers a way to hook into its deploy process: by adding
configuration files to a folder named .ebextensions,
you can add scripts for EB to run during deploy and even overwrite its default scripts.

This let us make an ebextension file that would install an npm-downgrading script. (Note:
these intermediate scripts are for illustration, not use, since the
ultimate set of scripts is way better.)

# EB runs deploy scripts in alphabetical order http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html,
# Node is installed using a script called "40install_node.sh", and `npm install` is
# run using a script called "50npm.sh", so we downgrade Node in a script called
# "45npm_downgrade.sh".


files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/45npm_downgrade.sh":
    mode: "000755"
    owner: root
    group: users
    content: |
      #!/usr/bin/env bash

      EB_NODE_VERSION=$(/opt/elasticbeanstalk/bin/get-config optionsettings -n aws:elasticbeanstalk:container:nodejs -o NodeVersion)

      # Make sure Node binaries can be found (required to run npm).
      # And this lets us invoke npm more simply too.
      export PATH=/opt/elasticbeanstalk/node-install/node-v$EB_NODE_VERSION-linux-x64/bin:$PATH

      if [ $(npm -v) != "2.15.9" ]; then
        echo "Downgrading npm to 2.15.9..."
        npm install npm@2.15.9 -g
      else
        echo "npm already at 2.15.9"
      fi

But now we had the challenge of distributing this file across our 14
microservices. Were we
going to copy-and-paste it? No way!

install-files

Awhile back, we made an npm package precisely to solve the problem of distributing files like
this ebextension. The package is called
install-files, and what it does is allow
another package to install files into its host package’s directory.

Let’s say that my-microservice installs the eb-fix-npm package. The eb-fix-npm package
can then call install-files source from an install script
to copy the contents of source into my-microservice:

This tool lets you share files between Node projects the same way you would share code, using
npm and declarative package names/versions. And with the ability to quickly distribute
changes to the script, we got ambitious.

Speeding up npm install

Simply by downgrading to npm 2, we were able to upgrade our Elastic Beanstalk environments to
Node 6. But, in the process of investigating Elastic Beanstalk’s npm script, we noticed
several inefficiencies.

First, it installed Node modules afresh on every deploy. So we introduced a cache:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/46cache_node_modules.sh":
    mode: "000755"
    owner: root
    group: users
    content: |
      #!/usr/bin/env bash
      # Cache Node modules in /var.

      if [ ! -d "/var/node_modules" ]; then
        mkdir /var/node_modules ;
      fi
      ln -s /var/node_modules /tmp/deployment/application/

By comparing timestamps when tailing EB’s activity log, we could see that EB’s npm script
went from taking ~4m to ~1m: a 75% speedup.

Then we noticed that EB was calling npm rebuild after installing. But modules are
automatically built for the appropriate architecture when installing! The only time you need
to rebuild is when the architecture changes—on configuration deploy. And on
configuration deploy, EB was trying to install new modules—even though package.json
doesn’t change on configuration deploy, only on application deploy.

So, no npm rebuild on application deploy, and no npm install on configuration deploy:

files:
  "/opt/elasticbeanstalk/env.vars":
    mode: "000775"
    owner: root
    group: users
    content: |
      # Exports variables for use by the other scripts below.

      EB_NODE_VERSION=$(/opt/elasticbeanstalk/bin/get-config optionsettings -n aws:elasticbeanstalk:container:nodejs -o NodeVersion)
      export PATH=/opt/elasticbeanstalk/node-install/node-v$EB_NODE_VERSION-linux-x64/bin:$PATH

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh":
    mode: "000755"
    owner: root
    group: users
    content: |
      #!/usr/bin/env bash
      #
      # Note that this *overwrites* Elastic Beanstalk's default 50npm.sh script.

      . /opt/elasticbeanstalk/env.vars

      cd /tmp/deployment/application && npm install --production

  "/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh":
    mode: "000755"
    owner: root
    group: users
    content: |
      #!/usr/bin/env bash
      #
      # Note that this *overwrites* Elastic Beanstalk's default 50npm.sh script.

      . /opt/elasticbeanstalk/env.vars

      cd /tmp/deployment/application && npm rebuild --production

During application deploy, our replacement npm script now took ~10 seconds: a 95% speedup compared to the initial duration.

Bonus: EB’s configuration deploy npm script doesn’t actually do anything—it uses the
wrong working directory. Our script actually rebuilds your modules if, for instance, you change your Node version.

One package to fix everything

So there you have it: you can unblock upgrading your Elastic Beanstalk environments to Node 6 and virtually eliminate npm install time by installing a single package,
eb-fix-npm. After installation, you’ll
effectively only have to npm install when Elastic Beanstalk spins up new EC2 instances,
without the cache. But we think we have a way to get rid of this hiccup too. Stay tuned…

Like working on the cutting edge of JavaScript devops? Join us!

SHARE ON

Written By

Jeff Wear

Jeff Wear

From Your Friends At