How Non-Engineers Test New Features at Mixmax

Mixmax is essential to our customers’ workflows. Because of this, when we deploy new features to production, we want to avoid introducing any regressions that might disrupt our customers’ work.

Before deploying new code, we want to make sure that:

1. Our code works correctly.
2. The new code doesn’t introduce any usability problems.

Engineers ought to be able to verify #1, since that’s simply a matter of implementing the spec they developed. It's not always best for the person who wrote the code to verify #2, since they'll often have gotten used to the UX while building it. To reduce the risk of introducing features with usability issues, we ask non-engineers test out new features before releasing them to the public.

At Mixmax, we have 3 methods for letting non-engineers test new features:

1. Test on our staging environment, which mirrors our production environment.
2. Give testers access to the developer’s local environment using a technique we call "shared testing".
3. Ship the new feature to production under a feature flag, and test on production.

This post describes what these methods are, their advantages and disadvantages, and how we decide which methods to use when shipping a new feature.

1. Staging

Our earliest solution was to set up an environment that mirrored our production environment, except without using production data. We called this the staging environment.

First, we deploy our new feature to staging, which every Mixmax employee can access through a secure VPN connection. We then schedule a meeting with our QA team to test the feature. We call this meeting a test party. When the test party is complete and all issues have been resolved, we ship the code to production.

Advantage: Deploying to staging is almost the same as deploying to production, except no real users will be affected. This makes staging a high-quality, low-risk testing environment.

Disadvantage: We only have one staging environment, and it’s part of our deploy pipeline. This means testing on staging can block other engineers from deploying changes to production until the test party is complete.

2. Shared Testing

As our team grew and we were able to develop more features simultaneously, we noticed our staging environment and deploy pipeline were constantly being blocked by test parties. In response, we began using a technique we call shared testing.

In shared testing, the developer runs a command that:

1. Makes a copy of all their code.
2. Runs the code on different ports than the ports normally used during development.
3. Uses ngrok to give testers access to the shared testing ports.

Advantage: Shared testing is a low cost testing method because we don’t need to deploy any code. We also avoid blocking other engineers from shipping code to production while testing.

Disadvantage: Shared testing doesn’t mirror our production environment, and not every part of our product can be tested using shared testing (for example our Mixmax Chrome extension).

3. Feature Flags in Production

The third technique we use is to ship the new feature to production, but put it behind a feature flag. A feature flag is a few lines of code that decides whether to load the new feature or the old feature based on the user. Here’s an example:

if (User.isMixmaxStaff()) {
  // Load new code.
} else {
  // Load old code.
}

Advantage: Testing using a feature flag is the most accurate testing method because we’re actually testing on production servers. Testers can test by using the new feature in their everyday workflow, which means the tests more accurately reflect how users actually use the new feature.

Disadvantage: Some features cannot be placed under a flag because the old code has been removed or almost totally refactored. In addition, adding feature flags introduces tech debt into our code because we have to deploy a second time to remove all the feature flags.

Guidelines For Choosing a Testing Method

The goal of testing is to make sure our features are polished while minimizing risk for users and cost for developers. With this in mind, here are some guidelines we follow when deciding which testing methods to use for a given feature:

1. We use a feature flag when possible because it provides the highest quality of testing. However, even if we use a feature flag, a subset of our users will be affected if there are usability issues with the code changes.

   • If the cost of disrupting these users is high, we do a test party on shared testing (preferred) or staging before deploying to production.

   • If the cost is low, we can ship to production under a feature flag and do a test party on production if needed.

2. If we can’t use a feature flag, we do a test party on shared testing before deploying to production. This has the lowest cost for developers because we don't have to deploy our code to staging, which will block our deploy pipeline.

3. If we can’t test on shared testing, we block our staging environment and do a test party on staging.

Choosing the right testing method is about maximizing the benefits of testing while minimizing the costs. If you follow this basic principle, you can build a great testing process regardless of which testing tools your company uses.

Interested in working on the communication platform of the future? Email us at careers@mixmax.com and follow us @Mixmax