Yarn-ifying Mixmax

Last week, we finished transitioning all of our services and shared modules to Yarn, Facebook’s replacement for npm. Here’s how we decided to make the move, how we did it, and what we learned along the way.

We've published a module to help you convert your own projects if you want to get started right away!

Why'd we move?

We were initially drawn to Yarn for two reasons: speed and better dependency pinning.

Speed

Over the past several months, npm install had become very time-consuming for us, taking upward of 25 minutes in some projects. Since we needed to reinstall our dependencies before removing, adding, or upgrading a dependency to ensure the accuracy of our shrinkwrapping, npm was draining a lot of time that we would have preferred to spend developing new features.

When benchmarked, Yarn proved to install dependencies more than 20x faster, even with a cold cache.

Results of one sample:

time npm install - 18:16.91

time yarn (cold cache) - 48.463

Dependency pinning

We had been using Uber’s shrinkwrapping tool since we agreed that npm’s built-in shrinkwrap wasn’t great. Unfortunately, Uber’s tool came with its own bugs and less-than-ideal workflow. When we began to consider Yarn, we were excited by dependency pinning being a first-class feature.

Considerations

Before moving forward with Yarn, we needed to make sure that it satisfied some prerequisites:

• Could we use it to install our private modules from the npm registry? Yes (discovered experimentally).
• Did it pin transitive dependencies in addition to top-level dependencies? Yes.
• Could we use it in CI (ie Codeship, Travis)? Yes.
• How would our workflow change? We looked up the commands we’d be using most often and observed that our workflow would be almost exactly the same, but with yarn instead of npm in our commands.
• Most importantly, how could we minimize risk? With Yarn being a relatively new project, it was important to us to be able to revert to using npm if we discovered that Yarn was not a good fit for us. Our plan was simple: we’d start by only using Yarn in a few projects, and would go back to using shrinkwrap if we decided that Yarn wasn’t right for us.

Here's what our workflow changes look like:

Implementation

Minimizing risk

We began by using Yarn in one of our open-source modules, Erik, a node package for running client-side Jasmine tests headlessly. This allowed us to gain familiarity with Yarn before taking on any risk.

Ramping up

The next phase involved converting some of our services to use Yarn. We started with an internal service in order to minimize our risk, then moved on to other low-priority services. Along the way, we learned a few things:

• In some cases, yarn check, a command that “Verifies that versions of the package dependencies in the current project’s package.json matches that of yarn’s lock file”, would fail immediately after running yarn for the first time (which generates the yarn.lock file). We were able to resolve this issue most of the time by rm -rfing node_modules and running yarn again. Other times, the fix was more involved.
• Some packages unsafely depend on their dependencies being installed in a node_modules directory contained within the module directory. Unlike npm, Yarn tries to install all dependencies (transitive dependencies included) in your project’s root node_modules folder, so some packages break as a result. Luckily, we were able to resolve this issue by upgrading our dependencies since the authors of the underlying packages had fixed the issue.
• Not all npm environment variables are implemented by Yarn (eg npm_package_directories). We wound up circumventing this issue by ending our usage of the offending package (for reasons unrelated to Yarn), but some packages might require updates.
• The --mutex network option is handy for when you’re running multiple instances of yarn simultaneously.
• Rarely, we’d see 404s when attempting to install private packages from the npm registry. This was often resolved by re-running the install command. When that didn’t work, we were able to resolve the issue by running npm login to regenerate our npm authentication tokens.

Going all-in

We didn’t see any major issues running the initially converted services with Yarn after a couple of weeks, so we decided to move forward with updating our remaining services ad modules. To help make this process easier, we wrote a module that stripped out our shrinkwrap configuration, generated a yarn.lock file, ran the project’s tests as a sanity check, ran yarn check to ensure the validity of the generated lockfile, and output a list of manual steps to be taken afterward in order to finish the update. This script helped us ensure that we were updating our projects correctly throughout the 75-project sweep.

In this last phase, we ran into a couple of issues:

• We encountered an issue with installing phantomjs-prebuilt on Codeship (our CI service), consistent with this issue. There are several proposed workarounds in that thread; we've added node node_modules/phantomjs-prebuilt/install.js to our Codeship configuration.
• We also ran into some issues stemming from Codeship caching our node_modules (in accordance with our configuration). Resetting Codeship’s dependency cache fixed these issues for us.

Cast-off

Yarn isn’t without its own bugs, but its community momentum and energy as well as its prioritization of speed and reliability excite us and have already proven valuable. Since last week, we’ve saved hours of development time that without Yarn would have been spent on npm installs (and, frustratingly, repeated npm installs after consecutive failures). Yarn has also enabled us to resume our tradition of having new engineers ship code to production on their first day, which had previously stalled due to lengthy and error-prone npm installs.

Shoot us a message if you’re interested in joining a culture where engineering speed and autonomy are top priorities.