Rewriting 30,000 lines of code in a friendly way

This blog post is part of the Mixmax 2016 Advent Calendar. The previous post on December 6th was about Improving Elasticsearch query times.

Sequences is one of our flagship features on Mixmax. Sequences enables users to create outbounds campaigns that can be configured to the last detail, with granular customizations per recipient.

Sequences evolved from the Mail Merge feature. Originally, Mail Merges were a subset of the functionality currently provided by Sequences. It allowed to create a single email campaign with user variables that were filled with a CSV uploaded by the user. Eventually, Mail Merges evolved to allow sending multiple emails in a campaign, and more recently, we allowed for adding recipients after the initial set of recipients, as well as customizing stages once they were sent and we plan to implement many more features in the coming months.

Everything that powered Mail Merge was rebuilt on a period of about 2 months. During this time, there was over 30,000 lines of code removed, reimplementing the whole features under the Sequences concept in a fraction of that code. Also, hundreds of thousands of documents were migrated and amended to match the new collection design and modules interfaces.

It was an incredibly daunting task. Sequences being a stable and well known feature for our valuable users, it had to remain stable during the whole process of the refactor. Mail Merges were actually becoming more unstable as we added more features due the high amount of complexity. We even had dedicated engineers to support the mail merge infrastructure while we were rewriting into Sequences. It was really a race against time that we eventually beat.

A quick look into Mail Merge

We use MongoDB as our database, as such, we designed our collection similar to this:

{
  name: 'My Campaign'
  subject: 'Introduction',
  body: 'Hello world!',
  variables: ['name'],
  scheduledAt: '2016-11-05T18:00:00.000Z',
  sentAt: '2016-11-05T18:00:00.000Z',
  messages: ['messageId_1', 'messageId_2', 'messageId_3'],
  recipients: [
    {
      email: 'foo@example.com',
      variables: {
        name: 'Foo'
      }
    },
    {
      email: 'bar@example.com',
      variables: {
        name: 'Bar'
      }
    }
  ]
}

The Mail Merge sending flow is specialized for its use case (hundreds of recipients in a campaign), a lot of code supported this specialized use case and, with time, organically grew to become a very daunting component in our system.

Once we decided to expand Mail Merges so it could send multiple stages at the time, we needed to do so in the least disruptive way and taking care not to break existing interfaces and components, as such, we created a new collection MailMergeStage that looks exactly like the MailMerge with a few more additions:

// The same as `MailMerge` but with the following added properties:
{
  trigger: 'notRead', // The stage will be sent as long as it hasn't been read
  offset: 259200000 // the stage will be sent after this amount of time (in millis) passes after the previous stage.
}

In addition, the MailMerge collection received a new field: stages which is an array of the stage ids belonging to the mail merge campaign. Additionally, message documents receive an additional metadata field: mailMergeId if the message belongs to a MailMerge or mailMergeStageId if the message was sent from a MailMergeStage document.

Since a MailMergeStage is almost exactly the same as a MailMerge document, all interfaces recognized these objects without any changes, since stages are sent conditionally, we did build some code that worked exclusively with MailMergeStages but overall, the impact on the existing code was minimal. However it already showed signs of code smell:

• Message documents would either have mailMergeId or mailMergeStageId to reference the parent campaign document. So we needed to query both fields because functions would receive either a MailMerge or a MailMergeStage document.
• As the codebase evolved, we needed to distinguish between a MailMergeStage and a MailMerge more and more often.
• The first stage of a mail merge campaign was always explicitly handled in a special way because the first stage happened to be the MailMerge document itself, while the remaining stages were MailMergeStages documents.
• From its conception, a mail merge was designed to run as a monolithic campaign that runs once. Adding recipients after the initial set users uploaded from the CSV was very difficult

Also, more often than not, we needed information specific to a Message, but we only kept the Message id field, so if we wanted to simply check the recipients in a given campaign, we needed to run a second query on the Message collection.

Processing a stage could become a slow process, mainly due the high amount of logic involved during a message creation.

We had many ideas to improve our Mail Merge product but we were more and more slowed down by the existing design of the Mail Merge collections, and the lack of modularity. While we had a powerful Inbox Syncing feature and scalable and fast infrastructure for our Live Feed, Mail Merges could not make use of these features due tech debt and the highly specific workflows that were in place for Mail Merge.

Refactoring Mail Merges into Sequences

The first step on our refactor effort was to determine what is the central actor in a Sequence. The more and more we analyzed the use cases, UX, mocks and future features, we saw how the "Recipient" is the central actor in the feature:

• A sequence is sent to a recipient
• A recipient has individual analytics, independent of other recipients in the campaign.
• Every recipient responds differently to the campaign.
• Campaigns should be totally customizable. While they have general configurations, such as message content, subject and other settings, it was clear that users want to have the option to tailor the campaign for each recipient if necessary.
• A campaign should be a living entity that can be modified, extended and that can have new recipients added to it at any point in time.

Under this line of thinking, a Sequence then became a sort of "blueprint" which is carried over for each recipient.

While with Mail Merge the central actor was the Stage, with Sequences the central actor is the recipient. The document structure for the recipient is similar to this:

// SequenceRecipient
{
  email: 'foo@example.com',
  sequence: 'sequence_id',
  stages: [
    {
      id: 'stage_id',
      scheduledAt: '2016-11-05T18:00:00.000Z',
      sentAt: '2016-11-05T18:00:00.000Z',
      body: 'customized body',
      message: {
        id: 'message_id'
      }
    }
  ],
  variables: {
    name: 'Foo'
  }
}

// SequenceStage
{
  body: 'Stage Body',
  subject: 'My stage subject',
  trigger: 'notRead'
}

// Sequence
{
  name: 'My campaign',
  stages: ['stage_1', 'stage_2']
}

What makes this design so flexible is that the stages array in SequenceRecipient is capable of overriding any setting in the SequenceStage document, allowing to make per-recipient customizations in a very trivial way. the Stage object is generated doing something like:

const stage = _.extend(referenceStage, recipientStage);

Where referenceStage is the SequenceStage document with the default body and subject content, and where recipientStage is an object in the stages property of the SequenceRecipient document that may optionally have its own body or subject or any other property of a SequenceStage document, thus making the recipient stage unique if so desired.

Also, given that each recipient has its stages scheduled independently of other recipients, it is also possible to send recipients at different times, something that was not possible under mail merges given that a stage was sent for every recipient at a given point in time.

Making use of the modular infrastructure

By making a recipient a central concept in Sequence, we can easily transform the recipient actor into a message actor, naturally, the Message is the central actor almost everywhere in our systems and it is certainly the actor in two very important modules to sequences: Send and Inbox Syncing.

Previously, with Mail Merges, we had limited possibility to integrate with other modules — this was one of the reasons why there was so much specialized code for sending messages in a mail merge campaign, evaluating sending triggers, capturing analytics and so on.

Now, given that a SequenceRecipient can quickly be "translated" into a Message actor, the Sequences feature can seamlessly integrate other modules with a Message interface.

Migration to the Sequences infrastructure

Despite the very high amount of code going in, we were making use of stable features (in particular Inbox Syncing), thus, by making use of these modules we were sure to a degree that the Sequences feature was stable as well.

Also, we spent a considerable amount of time working on a comprehensive set of unit tests for critical workflows, we also adapted existing tests for the new Sequences interfaces, which made it possible to ensure backwards compatibility.

Migrating the data over to the new sequences was also a challenge. The Mail Merge feature evolved over time becoming more and more complex, this meant that we had some discrepancies on data that didn't match with current logic, specially for really old data. Also the upgrade to Sequences was a complete update over to the new infrastructure, it was impossible to make incremental updates, making this upgrade specially difficult.

We approached this by first making minimal changes in the frontend. The UI remained the same, we built new views that used the same templates. The Sequences backend code lived alongside Mail Merge, so that we deploy our services constantly; this meant that Sequences code was constantly pushed to production in a daily basis which made pull requests smaller and easier to review.

When the Sequences feature was ready to be used under the new infrastructure, we opened it for internal use first, so that, besides our regular QA process, people in the office started using the new Sequences feature live on production before we made it publicly available. This allowed to test it in real-life situations and helped to iron out the rough edges before our users hit them.

We started our migration process on Friday afternoon, migrating the data on the database took around 11 hours (we underestimated the amount of time needed for the data migration). During the time we stopped the sequences feature entirely: we disabled new sequences creation and sequences processing to preserve data integrity after the data was moved to the new collections. We faced a few issues during the process, our database went down for a few stressful minutes when we did the switch over due an unfortunate coincidence and actually not related to the switch-over at all, we had some performance issues when the code was finally working at full production scale, but overall, minor issues that were solved in a matter of hours.

Conclusion

We made the code vastly less complex by using modules and services being used by other features, such as the live feed, and also streamlining the sending process so sequence messages get treated as regular email messages.

By keeping the SequenceRecipient as the central concept in sequences, we now make it much easier to implement new features as well as explore new ideas quickly. Since we keep the recipient data in a single document, we've made writes safer through extensive use of MongoDB findAndModify. We also reduced the amount of queries used in most of the code thanks to the collections redesign which opened up the option to make better use of MongoDB aggregations, in particular the $lookup pipeline has become a staple in most of our aggregation queries powering the sequences features.

We tackled this problem in the right time, mail merges/sequences was becoming more and more important, we had many unimplemented ideas at the time and the limiting factor was the existing design and implementation that was locked in into its own logic making it hard to be hooked up with other services and modules.

Overall, the greatest success metric of the refactor was that users barely noticed about the change. This was a huge success on our book. Since the refactor went in, we've implemented powerful features like a completely revamped sequences dashboard view and creation process, analytics directly in the dashboard, adding stages to existing sequences, customizing campaigns per recipient, integration with SalesForce and that was just in the past few weeks.

Would you like to help us scale our features for new ideas and for a fast growing user base? Come join us!.