Troubleshooting `npm link`

As of this writing, Mixmax runs 15 Node microservices. We keep this manageable by sharing a ton of code between services, in the form of npm packages both public and private.

To keep those packages manageable, we develop them in their own repositories. But this poses a challenge for local development — how do we quickly test a new version of a package inside another?

Luckily, npm provides a command called link, which symlinks one package into the node_modules directory of another. But developing modern JavaScript modules is at least two steps and often three:

1. Transpile/bundle your JS (optional)
2. Get that JS into the node_modules directory of your application
3. Get your application to load the new JS

and npm link only helps with step 2.

Here’s how to get steps 1 and 3 working. This process can be frustrating and seem magical, so two principles before we start:

1. If you don’t see code in your running application, it’s because it’s not there. :) This will show you how to determine exactly what code require and import are loading and why it might not be what you’d expect.
2. Avoid FUD! "Always rebuild/restart after making a change" is not necessary and if you blindly rely on doing that you will not only waste time, you may be disappointed. This article will help you take the minimum number of steps to get your changes live.

Troubleshooting server packages

This problem takes the form “my changes aren't visible when I require('my-package'). Follow this process:

1. Ask, what’s node_modules/my-package? If it’s a regular folder vs. a symlink, npm link your local module.
2. Ask, what does require('my-package') actually do? That is, check which file it will load, as specified as pkg.main; this could be a build artifact rather than source code (even for a server package, since some of our packages use Flow). At Mixmax, artifacts are usually written to a directory called ‘dist’. If the main file is such an artifact, build the package. At Mixmax, this is usually done via a package script called “build”, so run npm run build. Many packages also offer another script called “watch” that will automatically rebuild as you make changes.
3. Restart the server to reload the code. See the “Restart less” section below for the most efficient way of doing this.

Sometimes packages ship server and browser modules, where the browser module is bundled and transpiled and the server module is not. You only need to rebuild if the server uses bundled/transpiled JS, as specified by pkg.main: our convention is that bundled/transpiled files are written to a directory called dist.

Troubleshooting client packages

We use ES6 modules and Rollup client-side, so this problem takes the form “my changes aren't visible when I import foo from 'my-package'; Step 1 is the same as in "Troubleshooting server packages” above—make sure you’ve run npm link. Then:

2. Ask, what does import foo from ‘my-package‘ actually do? Once again, check which file it will load. This will usually be pkg.main, but may be pkg.browser instead if the package ships a different module for the client than the server. Either way, import almost certainly loads a build artifact, since all of our client-side packages are self-hosting (transpile themselves to ES5 as well as bundle themselves); so build the package. At Mixmax, this is usually done via a package script called “build”, so run npm run build. Many packages also offer another script called “watch” that will automatically rebuild as you make changes.
3. Rebundle your client application to reload the code. See the “Restart less” section below for the most efficient way of doing this.

In some rare cases, import foo from ‘my-package’ may not actually import foo from node_modules, either because we haven’t yet configured Rollup to do so, or because we’ve told Rollup to do something different for my-package for a special reason. But if my-package is in node_modules to begin with, you can generally trust that the project’s been configured to import it.

A diagram for troubleshooting

Restart less

At Mixmax, we use Gulp to transpile/bundle each microservice’s JS as well as launch the web server. You only ever need to restart this Gulp process (at Mixmax, supervisorctl restart …) if you changed something in the Gulpfile. Don’t do this otherwise! Rebuilding the entire application and starting the server can take many seconds.

The build process uses file watchers to restart the server and rebuild the client if their source code changes. But those watchers don’t monitor node_modules, for performance reasons and since we don’t usually expect those files to change.

This means that you have to manually reload the server or client when their dependencies change. But this has an easy fix. To pick up your change to a Node module, you just need to “poke” the file where you’re importing that module, by doing one of the following:

• inserting a new line and saving
• running touch /path/to/file at the command line
• some editors like Sublime Text will touch the file if you save it (e.g. Cmd-S), even if there are no pending changes

Updated 2018-07-26: See the last section of this post for a way to do this automatically!

Bonus: npm unlinking

npm unlink foo, unfortunately, does not reinstall the “real” foo (though this is going to change!). To fix this, unlink by doing npm unlink ../path/to/package rather than npm unlink <package name>.

You’ll once again need to poke the file where you import the module to get it to reload.

How could this be made simpler?

One answer might be to use a monorepo. That might require unifying all our packages’ build processes, whereas some of them use different sets of Babel and Rollup plugins at the moment. We’d also have to do research into the most efficient use of file watchers across the entire tree. And if we pulled our services into this monorepo as well, we’d have to refactor our CI/CD process to deploy individual directories rather than entire repos.

Using one repo per package makes it pretty easy to reason about, test, document, and publish those packages—especially if we’re going to share them with the public, which is important to our open-source culture. So we have an incentive to figure out how to optimize this npm link process.

My feeling is that having to link and build individual packages is not that bad. You do each of those steps once per change you have to make to a package, and many of our packages even have “watch” scripts to rebuild as you make changes. The thing to fix is having to manually reload the server or client after rebuilding your package. We might do this by conditionalizing our application file watchers to monitor node_modules/@mixmaxhq, since we make changes to our private packages more often than we do our public ones. Updated 2018-07-26: See the next section!

[Updated 2018-07-26] Automatically reloading after rebuilding

As discussed in the "Restart less" section, our application file watchers don't monitor the contents of node_modules, and so won't automatically reload the client and/or server when you rebuild your linked package. Our previous solution for this was to touch an application file after rebuilding the linked package, but we always had to do this manually.

I recently saw a coworker rebuilding his linked package by running npm run build && touch /path/to/application/file. This let him rebuild and reload in a single command — but he still had to run this manually after every change, whereas we can continuously rebuild (only) by running npm run watch (which delegates to rollup -cw under the hood). But seeing the commands in conjunction like that made me wonder — could we run the latter as part of the linked package's build process?

A little digging revealed rollup-plugin-execute, a Rollup plugin that can execute shell command(s) sequentially after Rollup finishes bundling. By registering that plugin with our build process like so

// rollup.config.js
import _ from 'underscore';
import execute from 'rollup-plugin-execute';

export default {
  input: 'src/index.jsx',
  plugins: _.compact([
    /* other plugins */

    // Touch the specified path (if any) after the bundle is generated, to trigger file watchers.
    // In practice: specify the path of the module that's importing this package. That way your
    // application will automatically reload to pick up the new package!
    process.env.TOUCH_PATH && execute(`touch ${process.env.TOUCH_PATH}`)
  ]),
  output: [{
    format: 'es',
    file: pkg['main']
  }]
};

We can now continuously rebuild and reload by doing env TOUCH_PATH="/path/to/application/file" npm run watch.

If you're interested in advancing the state of the art in open-source JS, drop us a line at careers@mixmax.com.