This blog post is part of the Mixmax 2016 Advent Calendar. The previous post on December 10th was about making a holiday guestbook using the Mixmax API.
Raise your hand if you’ve had an outage that could have been prevented by integration testing. And keep your hand raised if you’ve put off writing those tests for some very good reasons. What if there were another way?
At Mixmax, we’ve had a bunch of experience at prior companies writing integration tests for iOS and web, the latter using Selenium WebDriver. And that experience has taught us that such tests are enormously expensive to write and maintain. Selenium tests are coupled to all sorts of implementation details (e.g. CSS selectors) that a user would gloss right over; and can all-too-often work locally, yet crash or timeout in CI environments, due to the CI servers being starved for resources.
That said, integration tests fill a crucial niche in a testing ecosystem. Unit and functional tests only verify that individual subsystems are working — integration tests verify that those APIs mesh to create the desired user experience. Something’s got to execute such tests before shipping.
Something… or perhaps someone?
Integration Testing for Humans
What we realized was that, for now, it’d be a lot simpler if we ran the tests. All we really needed to automate was the reminder to do so. The result was integration-testing-for-humans (ITFH), an open-source CI service that reminds you, the developer, to test your staging environment before deploying to production.
The package defines Express middleware, as well as a standalone server, that responds to GitHub webhooks. Once you set up its hosting and decide what test you’d like to run e.g. “Send an email”:
var express = require('express');
var humans = require('integration-testing-for-humans');
var app = express();
var middleware = humans({
githubAccessToken: process.env.ITFH_GITHUB_ACCESS_TOKEN,
githubWebhookSecret: process.env.ITFH_GITHUB_WEBHOOK_SECRET,
location: 'https://example.com/humans',
defaultTest: 'Send an email',
});
middleware.on('error', (err) => console.error(err));
app.use('/humans', middleware);
app.listen(process.env.PORT);it will use the GitHub commit status API to prompt you to test when you open or update a PR:
When you click "Details", you are shown exactly what you should test.
After doing so, you click the button:
And can merge the PR:
Still… manual testing?
If you have the resources and need to write a fully-automated suite, by all means go for it. ITFH is for fast-moving teams like Mixmax that need to spend their time building the platform, and can afford to ship small bugs. As our lists of must-not-break features grows longer, we'll need to test more things before deploying, and then it will become more painful to run the tests ourselves than invest in Selenium — but for now, we just need a reminder to run the simplest and most important tests before shipping.
At Mixmax we believe in simple solutions to hard problems and just the right amount of process. If you’d like to move fast in that sort of environment, join us!
