Why and how we moved from npm to Yarn
Monday, Apr 24th, 2017
Mixmax is a communications platform that brings professional communication & email into the 21st century.
Last week, we finished transitioning all of our services and shared modules to Yarn, Facebook’s replacement for
npm. Here’s how we decided to make the move, how we did it, and what we learned along the way.
We've published a module to help you convert your own projects if you want to get started right away!
Why'd we move?
We were initially drawn to Yarn for two reasons: speed and better dependency pinning.
Over the past several months,
npm install had become very time-consuming for us, taking upward of 25 minutes in some projects. Since we needed to reinstall our dependencies before removing, adding, or upgrading a dependency to ensure the accuracy of our shrinkwrapping,
npm was draining a lot of time that we would have preferred to spend developing new features.
When benchmarked, Yarn proved to install dependencies more than 20x faster, even with a cold cache.
Results of one sample:
time npm install- 18:16.91
time yarn(cold cache) - 48.463
We had been using Uber’s shrinkwrapping tool since we agreed that
shrinkwrap wasn’t great. Unfortunately, Uber’s tool came with its own bugs and less-than-ideal workflow. When we began to consider Yarn, we were excited by dependency pinning being a first-class feature.
Before moving forward with Yarn, we needed to make sure that it satisfied some prerequisites:
- Could we use it to install our private modules from the npm registry? Yes (discovered experimentally).
- Did it pin transitive dependencies in addition to top-level dependencies? Yes.
- Could we use it in CI (ie Codeship, Travis)? Yes.
- How would our workflow change? We looked up the commands we’d be using most often and observed that our workflow would be almost exactly the same, but with
npmin our commands.
- Most importantly, how could we minimize risk? With Yarn being a relatively new project, it was important to us to be able to revert to using
npmif we discovered that Yarn was not a good fit for us. Our plan was simple: we’d start by only using Yarn in a few projects, and would go back to using
shrinkwrapif we decided that Yarn wasn’t right for us.
Here's what our workflow changes look like:
|Adding a package||
|Upgrading a package||
|Installing a package globally||
|Linking a local module||
(in directory to be linked)
(in directory using local module)
We began by using Yarn in one of our open-source modules, Erik, a node package for running client-side Jasmine tests headlessly. This allowed us to gain familiarity with Yarn before taking on any risk.
The next phase involved converting some of our services to use Yarn. We started with an internal service in order to minimize our risk, then moved on to other low-priority services. Along the way, we learned a few things:
- In some cases,
yarn check, a command that “Verifies that versions of the package dependencies in the current project’s package.json matches that of yarn’s lock file”, would fail immediately after running
yarnfor the first time (which generates the
yarn.lockfile). We were able to resolve this issue most of the time by
yarnagain. Other times, the fix was more involved.
- Some packages unsafely depend on their dependencies being installed in a
node_modulesdirectory contained within the module directory. Unlike
npm, Yarn tries to install all dependencies (transitive dependencies included) in your project’s root
node_modulesfolder, so some packages break as a result. Luckily, we were able to resolve this issue by upgrading our dependencies since the authors of the underlying packages had fixed the issue.
- Not all
npmenvironment variables are implemented by Yarn (eg
npm_package_directories). We wound up circumventing this issue by ending our usage of the offending package (for reasons unrelated to Yarn), but some packages might require updates.
--mutex networkoption is handy for when you’re running multiple instances of
- Rarely, we’d see 404s when attempting to install private packages from the npm registry. This was often resolved by re-running the install command. When that didn’t work, we were able to resolve the issue by running
npm loginto regenerate our
We didn’t see any major issues running the initially converted services with Yarn after a couple of weeks, so we decided to move forward with updating our remaining services ad modules. To help make this process easier, we wrote a module that stripped out our
shrinkwrap configuration, generated a
yarn.lock file, ran the project’s tests as a sanity check, ran
yarn check to ensure the validity of the generated lockfile, and output a list of manual steps to be taken afterward in order to finish the update. This script helped us ensure that we were updating our projects correctly throughout the 75-project sweep.
In this last phase, we ran into a couple of issues:
- We encountered an issue with installing
phantomjs-prebuilton Codeship (our CI service), consistent with this issue. There are several proposed workarounds in that thread; we've added
node node_modules/phantomjs-prebuilt/install.jsto our Codeship configuration.
- We also ran into some issues stemming from Codeship caching our
node_modules(in accordance with our configuration). Resetting Codeship’s dependency cache fixed these issues for us.
Yarn isn’t without its own bugs, but its community momentum and energy as well as its prioritization of speed and reliability excite us and have already proven valuable. Since last week, we’ve saved hours of development time that without Yarn would have been spent on
npm installs (and, frustratingly, repeated
npm installs after consecutive failures). Yarn has also enabled us to resume our tradition of having new engineers ship code to production on their first day, which had previously stalled due to lengthy and error-prone
Shoot us a message if you’re interested in joining a culture where engineering speed and autonomy are top priorities.