An email odyssey: how we tamed the bit of the app no one wants to work on
This post was written by Jack Weeden. Jack's an engineer at Lost My Name and was the Tech Lead on this project.
We send a lot of email. Between transactional and behavioural emails we send around 60,000 emails per month (plus our regular newsletter). These emails tell customers vital information about their orders, and are an important part of our sales and marketing operation.
Email is clearly important. Unfortunately, email is also hard. Hard to work with as a developer and hard to understand and manage as a non-technical person, especially for an internationalised business like ours that operates in several languages.
Anyone working in a SaaS or Ecommerce company will probably be able to relate the challenge of email. Because its hard, it rarely gets solved well, and all round the world people are struggling with interlinked, gerry rigged, hard to visualise, hard to understand email templating / sending / reporting systems.
We had one of these at Lost My Name until recently. In this post I'm going to explain our journey from there to where we are today - operating a simple, easy to manage, easy to test custom email system that handles internationalised content easily.
In this post I'll outline:
The key types of email, transactional, behavioural, newsletters and internal
Our initial setup that came with our Ruby on Rails Ecommerce platform Spree and what was broken about it
Our experiment with 3rd party providers especially customer.io and what was broken about them
How we came to design our own system, Koala, how it works and why our email system is no longer broken
Hopefully this post will be useful to you in your work. We'd love to get your feedback via twitter @lostmynameHQ
The email landscape
There are many SaaS providers offering email delivery solutions, from direct SMTP delivery like Mandrill to more developer-focused API approaches like Mailgun, leaving you as a developer to concentrate on more domain-specific - and hopefully more interesting - problems.We group our emails into four categories: transactional, behavioural, newsletter and internal.
Emails are broadly defined by their delivery triggers and intent.
Transactional emails are event-triggered, consequences of a change in status of an order or a customer performing a specific action. Currently we send the following:
- Order success: containing a summary of a specific order along with estimated delivery time
- Book printing: we have a short grace period between order completion and the book being printed. This allows any mistakes to be rectified before committing the book to print
- Order dispatched: letting the customer know their book has been printed and is on its way. This contains a link to track their order with an external delivery service
- Password reset: a standard password recovery email for when a user forgets their password
Transactional emails should typically have a very high open rate because they contain information directly relating to an order a customer has placed or they are delivered in response to an action the customer has performed. Furthermore, a customer shouldn't be able to unsubscribe from these emails and they should be delivered immediately following the trigger event.
Behavioural emails are a mixture of event-triggered and time-based emails. They are primarily related to our referral program and in general are typically more marketing-driven deliveries aimed at conversion and customer retention.
- Referral introduction: information about our referral program. We aim to deliver this just after the customer has received their order. Depending on their country of origin this differs from customer to customer
- Referral updates: when a customer has referred a friend and that friend has bought a book, we let the original customer with an email
- Referral success: when a customer has referred three friends, they are eligible a free book not available to purchase through the site. This email allows them to claim the book
- Abandoned basket: if a new customer abandons their order part way through, we send a prompt containing the order information and a direct link to pick up from where they left off
We send out a montly newsletter to our customers with news, events, competitions and so on. These are authored by our marketing team and can be sent to a certain segment of customers depending on the content.###
Like many tech companies, we have background tasks which are run periodically. Certain people in the company need to be notified when a task has run successfully or has failed for some reason, so we have a number of jobs which trigger internal emails. These are typically plain text or sometimes are totally blank, only containing an attachment.
Our email journey begins...
We rebuilt our main app about a year ago around the open-source e-commerce platform Spree. We weighed up a few different options and settled on Spree as it's a fairly mature open source platform. Although we don't conform to your typical e-commerce company (our products are personalised so we don't stock any inventory), Spree takes care of a lot of things we would have had to build ourselves, allowing us to focus on making our product and user experience as great as possible.
Our initial transactional setup
While Spree is an open source project, it does have commercial backing, funded primarily by the sale of Wombat, its "operating system for commerce". This is essentially a queueing system connecting your Spree store front to any number of third-party integrations. The idea is you push your store data (orders, customers, shipments) to Wombat and it'll pipe that data to any integrations you have set up. In order for the data to be in the correct format for the integration, you define a transform in which you manipulate the data before you pass it off. One of Wombat's primary selling points is that its point-and-click interface allows non-developers to make changes, and changes to be made without app deployments. Need the subject line of an email changing? Just log in and edit the transform.
This was an attractive feature for us so we began sending our transactional emails using Wombat as it was quick to set up and relatively easy to maintain. The fact it's designed primarily to be used with Spree was also a contributing factor. Using Wombat, the flow for sending transactional emails was as follows:
- 1. Serlialise the order, shipment or user object and send it to Wombat
- 2. Transform the order data into a format Mandrill understands
- 3. Deliver the email using the appropriate email template in Mandrill, injecting in data from step 2
This was quite a simple approach and worked well to begin with, however problems arose as our product, and therefore main app, got more complex. The first issue was the transform, which quickly spiralled out of control. Changing it evolved from a job a non-developer could do to a mess of code incomprehensible to most of our developers.
The second issue was internationalisation. We had a nice HTML template set up in Mandrill for sending these emails, but when we launched the book in our first non-English language, we had to create another template. This in turn meant adding conditional logic in the transform in order to select the correct template based on the customer's locale.
Creating a new order confirmation email for a new locale therefore involved copying the entire HTML content from an existing email, gathering the new translations, overwriting the existing copy, and pasting it into a new template. Anyone who has dealt with email template HTML before knows what a mess the markup looks like:
Furthermore, a small change in the template meant changing it in every other template. This had the effect of developers being reluctant to make small copy changes. When we came to launch our fourth language, we'd had enough of this approach so we started to think about how we could change it.
In the end we settled on what would seem to be an obvious solution: send emails straight from our main Rails app. This comes with many advantages: * Emails templates are just normal ERB templates, so common parts can be refactored into partials and used across multiple templates. Additionally it means we can abstract away all of the verbose HTML markup into nice Rails helpers. * Translating the copy can be done in the same place as the copy on the rest of the site. Our team of translators use an external service. They're familiar with it, and it makes sense to keep all translations in the same place. * Rails is good at email. We chose to use ActionMailer but similarly we could have farmed off the actual sending mechanism to any third party service. * Emails are sent asychronously by pushing jobs onto our Sidekiq queue. This has two great upsides: * We already have mechanisms set up to detect when Sidekiq jobs aren't being processed, or they raise exceptions, so we get all of this for free by using Sidekiq for our email delivery. By using a third party service, we don't get visibility of this kind of stuff. * We can assign priorities to our queues. The reset password email should be delivered to a user as soon as possible after they request it, so this gets a higher priority over our referral program emails.
Using our main app does have its downsides though, too.
- We're adding complexity to an already quite large monolithic system. Using Wombat, we could just ship off the serialized object and forget about it. Now we've got this logic embedded in our e-commerce app.
- Third party services built around sending email typically have guards against multiple redelivery, ensuring people don't get sent the same email over and over. We had to build this functionality ourselves.
- Editing templates now requires the work of a developer and an app deploy to take effect. This is quite a big disadvantage, but we tried to minimise its imact as much as possible. Our international translators are able to change the wording of translations and deploy them using Slack. This means copy changes are quick and easy to implement. Changing the actual styling or structure of a template still requires developer resource though.
- Images are a bit of a pain to work with. By default, the Rails asset pipeline appends a unique hash to an image filename based on its contents. So if an image changes, its filename changes as well, meaning that older emails sent out before the file changed would no longer be able to render any of the images. We have plans to get around this in the future but for now email images are uploaded manually to S3 and referenced directly in the templates.
Overall we decided the pros far outweighed the cons and all of our transactional emails are now sent from our main app. ActionMailer is set up to perform the actual delivery using the Mandrill SMTP server.
Our first attempt at behavioural email
Just like with transactional emails, we wanted a system that could be used by non-developers. This was especially important with behavioural emails where we are trying to work towards some sort of conversion goal, not just presenting information relating to an order or shipment. This meant we wanted a solution where A/B tests could be implemented and measured, conversion goals tracked and customers could be segmented to target customers on a granular level.We shopped around a bit and finally settled on using Customer IO, which seemed to meet the majority of our needs.
The premise was similar to using Wombat for our transactional email: we would send relevant data to Customer IO throughout a customer's purchase journey and beyond, then let it take care of the rest. We could go into Customer IO, set up triggered campaigns based on specific customer attributes and we could set up our email campaigns that way.
We trialled Customer IO for our referral program and it worked well. We knew what data we needed to send in order to segment users the way we wanted, so we quickly implemented that in our main app.
Soon enough we had populated Customer IO with enough data about new customers to start sending our campaign emails. Members of our marketing team were able to set up campaigns, target them at specific users - e.g. a certain locale, number of previous orders, etc. - and could get a nice overview of the send, open and click rates as well as conversions.
This was very successful to start with and ran smoothly for several weeks; we trialled A/B testing email subject lines and set conversion goals. Everything was working nicely so we decided to set up another campaign directed at a subset of users we hadn't originally anticipated. This raised the issue that we didn't have the required data stored in Customer IO, so we had to write a small script to pick out data from our main app's database and update all users in Customer IO using their API. The API was rate limited so updating thousands of customer records took a significant amount of time.
We realised that unless we essentially mirrored our customer table in Customer IO, we'd continually be having to push up new attributes using the API. Even then, we might want to segment customers based on a combination of data points, e.g. customers who bought one book for a boy and one for a girl, or customers who bought 3 books within a certain date range. We'd have to push new data up for every customer for each new campaign. Unfortunately it became obvious that this wouldn't be a viable options either.
So, just like with our transactional emails, we opted to roll these emails into our main app. This gave us most of the advantages as before: we could use the same templates, internationalisation came for free and more visibility around errors and failed sends.
What about Newsletters?!
Mailchimp is a very popular email marketing service and we've been using it to send our monthly newsletters for some time now. It provides a nice WYSIWYG interface for composing a newsletter and delivering it to a customer base and, after initial template creation, it's very easy for anybody to use. Because of the complex authoring tools you get with Mailchimp, we decided not to roll our own version just yet.
The major downside of switching to our own system for email delivery is that we lost out on the nice analytics that third party systems typically provide. Analytics, and especially conversion rates, are very important to Lost My Name; we like to know what's going on and, even moreso, being able to dig around in our datasets. We were still using Mandrill under the hood to actually deliver the emails and Mandrill does a nice job of showing analytics for a short period of time, but we decided we could do a bit more, especially around conversion goals. So we made a small Rails service, Koala, to collect and analyse our email data.
Mandrill provides a nice webhook which it'll ping each time an even occurs - sends, opens, clicks, bounces, etc. We built a very simple app which listens to these webhook events and populates a database. While this part mimicks Mandrill's own analytics, we can store data indefinitely and mine it however we want.
Because Koala is being Slack authentication, anyone in the company is able to go in and get an overview of how our emails are performing. Additionally, because the data is persisted to a PostgreSQL database, we can hook it up Periscope, which means anyone can run queries against the data. If we want know how many users have unsubscribed in the past month, we can simply write an SQL query and Periscope will chart it.
Being fully in control of the data and how we store it also means we can augment it with data from other sources. In our main app we can make an API call to Koala, for example, to add converstion data to an email. We then get more visibility of the effectiveness of a certain campaign.
A bonus feature is that because Mailchimp sends its emails using Mandrill under the hood, we get all the data around our newsletter emails pumped straight into Koala for free.
An email history
We often get requests from our customer support team to regenerate an email for a customer if they've lost or deleted it. This used to be a bit of an involved process - we'd have to open up a console on Heroku, paste in a few commands, copy the HTML generated, paste it into a new document, take a screenshot and send it back to the customer. With Koala, we decided fairly early on that we'd like this process to either be automated or, even better, the customer support team would just be able to view all sent emails. Now every time an email is delivered, we store the HTML content in Amazon S3 and make this available through Koala, making the process of troubleshooting any past email deliveries a breeze. As a bonus, we made a Zendesk integration allowing our customer service team to see which emails have been delivered to a particular customer from within Zendesk.
A simplified architecture
Overall, we've greatly simplified our email architecture and in the process of doing so created a simple but powerful analytics platform. It was important for us to shop around and try out various third party services before jumping straight in an rolling our own solutions - we're big proponents of using tried and tested alternatives before reinventing the wheel. Having so many moving parts and our data duplicated in many different places meant we had to stop and evaluate our approach. In doing so we were able to simplify our existing system, which looked like this:
We ended up with a system which looks like the following:
It is now easier to understand, easier to change and easier to test email than ever before and generally it makes everyone in the business much more confident about the performance of our email systems.
It was a long journey to get here, but we're confident we've now built a foundation that will enable our email ambitions to grow with our company over the next 12 months.
As I mentioned before, I'd love to get your feedback and hear about how you tame the email beast at your startup on Twitter - ping us @lostmynameHQ