Rewriting 30,000 lines of code in a friendly way

SHARE ON

How we evolved the Mail Merge into Sequences

This blog post is part of the
Mixmax 2016 Advent Calendar. The previous post on
December 6th was about
Improving Elasticsearch query times.

Sequences is one of our flagship features on Mixmax. Sequences
enables users to create outbounds campaigns that can be configured to the last detail, with granular
customizations per recipient.

Sequences evolved from the Mail Merge feature. Originally, Mail Merges were a subset of the
functionality currently provided by Sequences. It allowed to create a single email campaign with
user variables that were filled with a CSV uploaded by the user. Eventually, Mail Merges evolved
to allow sending multiple emails in a campaign, and more recently, we allowed for adding recipients
after the initial set of recipients, as well as customizing stages once they were sent and we plan
to implement many more features in the coming months.

Everything that powered Mail Merge was rebuilt on a period of about 2 months. During this time,
there was over 30,000 lines of code removed, reimplementing the whole features under the Sequences
concept in a fraction of that code. Also, hundreds of thousands of documents were migrated and
amended to match the new collection design and modules interfaces.

It was an incredibly daunting task. Sequences being a stable and well known feature for our valuable
users, it had to remain stable during the whole process of the refactor. Mail Merges were actually
becoming more unstable as we added more features due the high amount of complexity. We even had
dedicated engineers to support the mail merge infrastructure while we were rewriting into Sequences.
It was really a race against time that we eventually beat.

A quick look into Mail Merge

We use MongoDB as our database, as such, we designed our collection similar to this:

{
  name: 'My Campaign'
  subject: 'Introduction',
  body: 'Hello world!',
  variables: ['name'],
  scheduledAt: '2016-11-05T18:00:00.000Z',
  sentAt: '2016-11-05T18:00:00.000Z',
  messages: ['messageId_1', 'messageId_2', 'messageId_3'],
  recipients: [
    {
      email: 'foo@example.com',
      variables: {
        name: 'Foo'
      }
    },
    {
      email: 'bar@example.com',
      variables: {
        name: 'Bar'
      }
    }
  ]
}

The Mail Merge sending flow is specialized for its use case (hundreds of recipients in a campaign),
a lot of code supported this specialized use case and, with time, organically grew to become a very
daunting component in our system.

Once we decided to expand Mail Merges so it could send multiple stages at the time, we needed to do
so in the least disruptive way and taking care not to break existing interfaces and components, as
such, we created a new collection MailMergeStage that looks exactly like the MailMerge with a
few more additions:

// The same as `MailMerge` but with the following added properties:
{
  trigger: 'notRead', // The stage will be sent as long as it hasn't been read
  offset: 259200000 // the stage will be sent after this amount of time (in millis) passes after the previous stage.
}

In addition, the MailMerge collection received a new field: stages which is an array of the
stage ids belonging to the mail merge campaign. Additionally, message documents receive an
additional metadata field: mailMergeId if the message belongs to a MailMerge or
mailMergeStageId if the message was sent from a MailMergeStage document.

Since a MailMergeStage is almost exactly the same as a MailMerge document, all interfaces
recognized these objects without any changes, since stages are sent conditionally, we did build some
code that worked exclusively with MailMergeStages but overall, the impact on the existing code was
minimal. However it already showed signs of code smell:

  • Message documents would either have mailMergeId or mailMergeStageId to reference the parent
    campaign document. So we needed to query both fields because functions would receive either a
    MailMerge or a MailMergeStage document.
  • As the codebase evolved, we needed to distinguish between a MailMergeStage and a MailMerge
    more and more often.
  • The first stage of a mail merge campaign was always explicitly handled in a special way because
    the first stage happened to be the MailMerge document itself, while the remaining stages were
    MailMergeStages documents.
  • From its conception, a mail merge was designed to run as a monolithic campaign that runs once.
    Adding recipients after the initial set users uploaded from the CSV was very difficult

Also, more often than not, we needed information specific to a Message, but we only kept the
Message id field, so if we wanted to simply check the recipients in a given campaign, we needed
to run a second query on the Message collection.

Processing a stage could become a slow process, mainly due the high amount of logic involved during
a message creation.

We had many ideas to improve our Mail Merge product but we were more and more slowed down by the
existing design of the Mail Merge collections, and the lack of modularity. While we had a powerful
Inbox Syncing feature and scalable and fast infrastructure for our Live Feed, Mail Merges could not
make use of these features due tech debt and the highly specific workflows that were in place for
Mail Merge.

Refactoring Mail Merges into Sequences

The first step on our refactor effort was to determine what is the central actor in a Sequence.
The more and more we analyzed the use cases, UX, mocks and future features, we saw how the
“Recipient” is the central actor in the feature:

  • A sequence is sent to a recipient
  • A recipient has individual analytics, independent of other recipients in the campaign.
  • Every recipient responds differently to the campaign.
  • Campaigns should be totally customizable. While they have general configurations, such as message
    content, subject and other settings, it was clear that users want to have the option to tailor the
    campaign for each recipient if necessary.
  • A campaign should be a living entity that can be modified, extended and that can have new
    recipients added to it at any point in time.

Under this line of thinking, a Sequence then became a sort of “blueprint” which is carried over for
each recipient.

While with Mail Merge the central actor was the Stage, with Sequences the central actor is the
recipient. The document structure for the recipient is similar to this:

// SequenceRecipient
{
  email: 'foo@example.com',
  sequence: 'sequence_id',
  stages: [
    {
      id: 'stage_id',
      scheduledAt: '2016-11-05T18:00:00.000Z',
      sentAt: '2016-11-05T18:00:00.000Z',
      body: 'customized body',
      message: {
        id: 'message_id'
      }
    }
  ],
  variables: {
    name: 'Foo'
  }
}

// SequenceStage
{
  body: 'Stage Body',
  subject: 'My stage subject',
  trigger: 'notRead'
}

// Sequence
{
  name: 'My campaign',
  stages: ['stage_1', 'stage_2']
}

What makes this design so flexible is that the stages array in SequenceRecipient is capable of
overriding any setting in the SequenceStage document, allowing to make per-recipient
customizations in a very trivial way.
the Stage object is generated doing something like:

const stage = _.extend(referenceStage, recipientStage);

Where referenceStage is the SequenceStage document with the default body and subject content,
and where recipientStage is an object in the stages property of the SequenceRecipient document
that may optionally have its own body or subject or any other property of a SequenceStage
document, thus making the recipient stage unique if so desired.

Also, given that each recipient has its stages scheduled independently of other recipients, it is
also possible to send recipients at different times, something that was not possible under mail
merges given that a stage was sent for every recipient at a given point in time.

Making use of the modular infrastructure

By making a recipient a central concept in Sequence, we can easily transform the recipient actor
into a message actor, naturally, the Message is the central actor almost everywhere in our systems
and it is certainly the actor in two very important modules to sequences: Send and
Inbox Syncing.

Previously, with Mail Merges, we had limited possibility to integrate with other modules—this
was one of the reasons why there was so much specialized code for sending messages in a mail merge
campaign, evaluating sending triggers, capturing analytics and so on.

Now, given that a SequenceRecipient can quickly be “translated” into a Message actor, the
Sequences feature can seamlessly integrate other modules with a Message interface.

Migration to the Sequences infrastructure

Despite the very high amount of code going in, we were making use of stable features (in particular
Inbox Syncing), thus, by making use of these modules we were sure to a degree that the Sequences
feature was stable as well.

Also, we spent a considerable amount of time working on a comprehensive set of unit tests for
critical workflows, we also adapted existing tests for the new Sequences interfaces, which made it
possible to ensure backwards compatibility.

Migrating the data over to the new sequences was also a challenge. The Mail Merge feature evolved
over time becoming more and more complex, this meant that we had some discrepancies on data that
didn’t match with current logic, specially for really old data. Also the upgrade to Sequences was
a complete update over to the new infrastructure, it was impossible to make incremental updates,
making this upgrade specially difficult.

We approached this by first making minimal changes in the frontend. The UI remained the same, we
built new views that used the same templates. The Sequences backend code lived alongside Mail Merge,
so that we deploy our services constantly; this meant that Sequences code was constantly pushed to
production in a daily basis which made pull requests smaller and easier to review.

When the Sequences feature was ready to be used under the new infrastructure, we opened it for
internal use first, so that, besides our regular QA process, people in the office started using the
new Sequences feature live on production before we made it publicly available. This allowed to test
it in real-life situations and helped to iron out the rough edges before our users hit them.

We started our migration process on Friday afternoon, migrating the data on the database took around
11 hours (we underestimated the amount of time needed for the data migration). During the time we
stopped the sequences feature entirely: we disabled new sequences creation and sequences processing
to preserve data integrity after the data was moved to the new collections.
We faced a few issues during the process, our database went down for a few stressful minutes when we
did the switch over due an unfortunate coincidence and actually not related to the switch-over at
all, we had some performance issues when the code was finally working at full production scale, but
overall, minor issues that were solved in a matter of hours.

Conclusion

We made the code vastly less complex by using modules and services being used by other features,
such as the live feed, and also streamlining the sending process so sequence messages get treated
as regular email messages.

By keeping the SequenceRecipient as the central concept in sequences, we now make it much easier
to implement new features as well as explore new ideas quickly. Since we keep the recipient data in
a single document, we’ve made writes safer through extensive use of MongoDB findAndModify. We also
reduced the amount of queries used in most of the code thanks to the collections redesign which
opened up the option to make better use of MongoDB aggregations, in particular the $lookup
pipeline has become a staple in most of our aggregation queries powering the sequences features.

We tackled this problem in the right time, mail merges/sequences was becoming more and more
important, we had many unimplemented ideas at the time and the limiting factor was the existing
design and implementation that was locked in into its own logic making it hard to be hooked up with
other services and modules.

Overall, the greatest success metric of the refactor was that users barely noticed about the change.
This was a huge success on our book. Since the refactor went in, we’ve implemented
powerful features like a completely revamped sequences dashboard view and creation process,
analytics directly in the dashboard, adding stages to existing sequences, customizing campaigns per
recipient, integration with SalesForce and that was just in the past few weeks.

Would you like to help us scale our features for new ideas and for a fast growing user base?
Come join us!.

SHARE ON

Written By

Chuy Martinez

Chuy Martinez

From Your Friends At