We recently rolled out a substantial improvement to our email-sending infrastructure by migrating from Google’s SMTP server to Google’s Gmail API. In this post, we’ll share about our motivation to migrate and how we overcame challenges presented by the differences between Google’s two methods of email delivery.
Overview
First, let’s discuss a bit of history. SMTP, or simple mail transfer protocol, was first introduced in 1981 and became the internet standard for sending email. Even today, whether you use Gmail, Outlook, or AOL, they all transfer email externally using the SMTP protocol.
Fast forward to 2014, when Gmail introduced a REST API as a proprietary layer on top of their email infrastructure. In contrast to SMTP (and its email-fetching cousin IMAP), the Gmail API was recognized as being simpler to integrate with, allowing for more granular access to data using OAuth, and also offering a performance boost.
Motivation
Earlier this year, we heard reports from our customers that some of their Mixmax emails were going to their recipient’s spam folders. Because email deliverability is our number one job, we immediately launched an investigation.
We first set up a controlled test environment to isolate possible causes of our emails being sent to spam. We discovered that emails sent using SMTP—our exclusive method of email delivery at the time—had a higher likelihood of being labeled as spam. The same emails sent using the Gmail API, which we had previously only experimented with, did not show the same higher likelihood. After circulating this finding within the email community, we learned that Google’s spam classifier had indeed shipped a bug that caused it to unfairly target SMTP-sent emails.
We needed to act quickly to restore our users’ trust. Since we couldn’t wait for Google, we began our project to move our email-sending infrastructure to the Gmail API.
Benefits of the Gmail API
As a higher-level abstraction on top of SMTP, the Gmail API offers several key benefits.
Sending from Aliases
As anyone with multiple email addresses can attest, Gmail’s popular alias feature simplifies things by offering a way to send emails from multiple addresses while logged into a single Google account. Gmail’s SMTP server has no way to use this feature. In SMTP, setting the “From” email header to the alias address does nothing more than spoof the sender and direct the email to a nearly certain death by spam filter.
The Gmail API, on the other hand, allows us to simply set the “From” field to our preferred Google account address. Then, Gmail API correctly authenticates and DKIM signs the email.
Error Codes
Gmail’s SMTP sent us a plethora of ambiguous error codes. The SMTP error codes could represent anything from a Gmail server issue to the user being rate-limited because they’re sending too much email. Here’s a sample of some of the codes we’ve received:
Message failed: 451 4.3.0 Mail server temporarily rejected message
Data command failed: 421 4.3.0 Temporary System Problem
Connection timeout
invalid_grant
Invalid login: 454 4.7.0 Cannot authenticate due to temporary system problem
By comparison, the Gmail API has well-documented error codes that leverage HTTP’s error code structure.
Performance
The SMTP protocol requires at least ten back-and-forth request-responses to send an email. Here’s an example:
Resolved smtp.gmail.com as 173.194.68.108 [cache miss]Secure connection established to 173.194.68.108:465S: 220 smtp.gmail.com ESMTP c23sm7215939qkk.50 - gsmtpC: EHLO ip-127-1-1-1.ec2.internalS: 250-smtp.gmail.com at your service, [24.49.6.253]}S: 250-SIZE 35882577S: 250-8BITMIMES: 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTHS: 250-ENHANCEDSTATUSCODESS: 250-PIPELININGS: 250-CHUNKINGS: 250 SMTPUTF8SMTP handshake finishedC: AUTH XOAUTH2 dXNlcj1lbmd0gDN0dXNlcjFAbWl4bWFmNvbQFhdXRoPUJlYXJlciAvliBzZWNyZXQgs8BAQ==S: 235 2.7.0 AcceptedUser "engtestuser@mixmax.com" authenticated[2021-09-14 12:36:11] INFO Sending message <a32fd24e-43f5-3205-29dc-c74e4b0ba178@mixmax.com> to <engtestuser@mixmax.com>C: MAIL FROM:<brad@mixmax.com>S: 250 2.1.0 OK c23sm7215939qkk.50 - gsmtpC: RCPT TO:<engtestuser@mixmax.com>S: 250 2.1.5 OK c23sm7215939qkk.50 - gsmtpC: DATAS: 354 Go ahead c23sm7215939qkk.50 - gsmtp(...)<3844 bytes encoded mime message (source size 3841 bytes)>S: 250 2.0.0 OK 1631622972 c23sm7215939qkk.50 - gsmtp
By comparison, the Gmail API is only a single HTTP call, which we’ve measured to be 50% faster. While it might seem like a small difference, all of those milliseconds add up when you’re sending millions of emails!
Drawbacks of the Gmail API
Along with the advantages offered by the Gmail API, a few drawbacks did require significant technical work to overcome.
Lack of Rich HTML Support
SMTP allows us to send any HTML content in the email body.
By comparison, the Gmail API will rewrite email content by stripping away any HTML that isn’t on its allowlist of only basic tags.
For example, here’s an HTML email that conditionalizes content for Outlook:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"> <body> <!--[if mso]> <p>This is Microsoft Outlook.</p> <![endif]--> <!--[if !mso]><!--> <p>This is NOT Microsoft Outlook</p> <!--<![endif]--> </body></html>
\When sending using SMTP, this email will be sent as-is. However, sending using the Gmail API will strip it down to this:
<html>
<body>
<p>This is NOT Microsoft Outlook</p>
</body>
</html></html>
Recipients who open the email on Outlook will see the non-Outlook content. Not good! This issue is filed in Google’s email tracker here.
This flaw in the Gmail API ended up being a big problem for us, since our Mixmax Enhancements depend on a custom HTML doctype and HTML comments to ensure they render well in all email clients. Our solution was to rebuild our Enhancements to only use basic HTML. This also ended up being quite a challenge (and deserving of its own blog post later ;)).
Can’t Set a Custom Message-ID
At Mixmax, we’ve architected our system to use an email’s Message-ID header—a globally unique identifier for emails—to track its lifecycle. We track each email from the moment we start sending it to when we get a notification that the email is in the user’s “Sent” folder via a Gmail webhook.
When sending using SMTP, we can set our own Message-ID. This guarantees that at the time the webhook notification comes in, we’ll be able to relate it to email metadata in our database.
By comparison, the Gmail API doesn’t allow setting a custom Message-ID. Instead, we need to call message.get after sending an email to fetch Gmail’s generated value for it. This introduces a problematic race condition: a Gmail webhook notification can let us know about the sent email before the message.get call returns, causing us to not be able to associate the two.
Our solution was to use a semaphore lock to ensure we waited to process incoming Gmail webhooks while there was an outstanding call to fetch a Message-ID. If you’re using Node, here’s a nice library to implement locking.
Conclusion
As with all feature rollouts at Mixmax, the Gmail API was rolled out carefully and incrementally, with a watchful eye on delivery and error rates. It’s now rolled out 100% and we’re proud to continue to deliver on our number one job to ensure our customers’ emails are successfully delivered.
We preserved our SMTP sending codepath should the need ever arise to switch back due to another unfortunate Gmail spam engine bug.
Interested in working on cool email tech like this? Join us!
