Troubleshooting `npm link`

SHARE ON

Mixmax is a communications platform that brings professional communication & email into the 21st century.

As of this writing, Mixmax runs 15 Node microservices. We keep this manageable by sharing a ton ofcode between services, in the form of npm packages both
public and private.

To keep those packages manageable, we develop them in their own repositories. But this poses a
challenge for local development—how do we quickly test a new version of a package inside another?

Luckily, npm provides a command called link, which symlinks one
package into the node_modules directory of another. But developing modern JavaScript modules is at
least two steps and often three:

  1. Transpile/bundle your JS (optional)
  2. Get that JS into the node_modules directory of your application
  3. Get your application to load the new JS

and npm link only helps with step 2.

Here’s how to get steps 1 and 3 working. This process can be frustrating and seem magical, so two
principles before we start:

  1. If you don’t see code in your running application, it’s because it’s not there. 🙂 This will show
    you how to determine exactly what code require and import are loading and why it might not be
    what you’d expect.
  2. Avoid FUD! “Always rebuild/restart after making a change” is not necessary and if you blindly
    rely on doing that you will not only waste time, you may be disappointed. This article will help
    you take the minimum number of steps to get your changes live.

Troubleshooting server packages

This problem takes the form “my changes aren’t visible when I require('my-package'). Follow this process:

  1. Ask, what’s node_modules/my-package? If it’s a regular folder vs. a symlink, npm link your
    local module.
  2. Ask, what does require('my-package') actually do? That is, check which file it will load, as
    specified as pkg.main; this could be a build
    artifact rather than source code (even for a server package, since some of our packages use
    Flow). At Mixmax, artifacts are usually written to
    a directory called ‘dist’. If the main file is such an artifact, build the package. At Mixmax,
    this is usually done via a package script called “build”, so run npm run build. Many packages
    also offer another script called “watch” that will automatically rebuild as you make changes.
  3. Restart the server to reload the code. See the “Restart less” section below for
    the most efficient way of doing this.

Sometimes packages ship server and browser modules, where the browser module is bundled and
transpiled and the server module is not. You only need to rebuild if the server uses
bundled/transpiled JS, as specified by pkg.main: our convention is that bundled/transpiled files
are written to a directory called dist.

Troubleshooting client packages

We use ES6 modules and Rollup client-side, so this problem takes the form “my changes aren’t visible
when I import foo from 'my-package'; Step 1 is the same as in “Troubleshooting server packages”
above—make sure you’ve run npm link. Then:

  1. Ask, what does import foo from ‘my-package‘ actually do? Once again, check which file it will
    load. This will usually be pkg.main, but may
    be
    pkg.browser
    instead if the package ships a different module for the client than the server. Either way,
    import almost certainly loads a build artifact, since all of our client-side packages are
    self-hosting (transpile themselves to ES5 as well as bundle themselves); so build the package. At
    Mixmax, this is usually done via a package script called “build”, so run npm run build. Many
    packages also offer another script called “watch” that will automatically rebuild as you make changes.
  2. Rebundle your client application to reload the code. See the “Restart less” section below for the
    most efficient way of doing this.

In some rare cases, import foo from ‘my-package’ may not actually import foo from
node_modules, either because we haven’t yet configured Rollup to do so, or because we’ve told
Rollup to do something different for my-package for a special reason. But if my-package is in
node_modules to begin with, you can generally trust that the project’s been configured to import it.

A diagram for troubleshooting

Restart less

At Mixmax, we use Gulp to transpile/bundle each microservice’s JS as well as
launch the web server. You only ever need to restart this Gulp process (at Mixmax, supervisorctl
restart …
) if you changed something in the Gulpfile. Don’t do this otherwise! Rebuilding the entire
application and starting the server can take many seconds.

The build process uses file watchers to restart the server and rebuild the client if their source
code changes. But those watchers don’t monitor node_modules, for performance reasons and since we
don’t usually expect those files to change.

This means that you have to manually reload the server or client when their dependencies change. But
this has an easy fix. To pick up your change to a Node module, you just need to “poke” the file
where you’re importing that module, by doing one of the following:

  • inserting a new line and saving
  • running touch /path/to/file at the command line
  • some editors like Sublime Text will touch the file if you save it (e.g. Cmd-S), even if there
    are no pending changes

Updated 2018-07-26: See the last section of this
post for a way to do this automatically!

Bonus: npm unlinking

npm unlink foo, unfortunately, does not reinstall the “real” foo
(though this is going to change!). To fix this, unlink by
doing npm unlink ../path/to/package rather than npm unlink <package name>.

You’ll once again need to poke the file where you import the module to get it to reload.

How could this be made simpler?

One answer might be to use a
monorepo. That might require
unifying all our packages’ build processes, whereas some of them use different sets of Babel and
Rollup plugins at the moment. We’d also have to do research into the most efficient use of file
watchers across the entire tree. And if we pulled our services into this monorepo as well, we’d have
to refactor our CI/CD process to deploy individual directories rather than entire repos.

Using one repo per package makes it pretty easy to reason about, test, document, and publish those
packages—especially if we’re going to share them with the public, which is important to our
open-source culture. So we have an incentive to
figure out how to optimize this npm link process.

My feeling is that having to link and build individual packages is not that bad. You do each of
those steps once per change you have to make to a package, and many of our packages even have
“watch” scripts to rebuild as you make changes. The thing to fix is having to manually reload the
server or client after rebuilding your package. We might do this by conditionalizing our application
file watchers to monitor node_modules/@mixmaxhq, since we make changes to our private packages
more often than we do our public ones. Updated 2018-07-26: See the next section!

[Updated 2018-07-26] Automatically reloading after rebuilding

As discussed in the “Restart less” section, our application file watchers don’t
monitor the contents of node_modules, and so won’t automatically reload the client and/or server
when you rebuild your linked package. Our previous solution for this was to touch an application
file after rebuilding the linked package, but we always had to do this manually.

I recently saw a coworker rebuilding his linked package by running npm run build && touch
/path/to/application/file
. This let him rebuild and reload in a single command—but he still
had to run this manually after every change, whereas we can continuously rebuild (only) by running
npm run watch (which delegates to rollup -cw under the hood). But seeing the commands in
conjunction like that made me wonder—could we run the latter as part of the linked package’s
build process?

A little digging revealed rollup-plugin-execute,
a Rollup plugin that can execute shell command(s) sequentially after Rollup finishes bundling. By
registering that plugin with our build process like so

// rollup.config.js
import _ from 'underscore';
import execute from 'rollup-plugin-execute';

export default {
  input: 'src/index.jsx',
  plugins: _.compact([
    /* other plugins */

    // Touch the specified path (if any) after the bundle is generated, to trigger file watchers.
    // In practice: specify the path of the module that's importing this package. That way your
    // application will automatically reload to pick up the new package!
    process.env.TOUCH_PATH && execute(`touch ${process.env.TOUCH_PATH}`)
  ]),
  output: [{
    format: 'es',
    file: pkg['main']
  }]
};

We can now continuously rebuild and reload by doing
env TOUCH_PATH="/path/to/application/file" npm run watch.

If you’re interested in advancing the state of the art in open-source JS, drop us a line at
careers@mixmax.com.

SHARE ON

Written By

Jeff Wear

Jeff Wear

From Your Friends At

Email tracking, instant scheduling, surveys, and more