An Unusual Use of SendGrid's Inbound Parse

January 31, 2022 8 minutes Sysadmin

Many people who have their own domains will set up an email address for that domain. For me, I have [email protected] (though any @evanzhang.ca address also works).

For the longest time, I've had an unfavourable email situation for my domain. Recently, I completely revamped my email situation in an unusual way which may be a useful option to consider for anyone else setting up emails.

For the most part, there's many ways to set up emails for your own domain.

  1. For inbound mail, a common method is to use an email forwarder (e.g. Cloudflare email routing) to forward these emails to another mailbox (e.g. Gmail, Outlook). For outbound mail, surprisingly, a lot of people don't set this up. Granted, this is somewhat more difficult than setting up inbound emails, and most of the time you won't ever need to send emails anyways. For those that do set it up this way, many use email services such as SendGrid or Mailgun.
  2. Another common method is to use an all-in-one email provider (e.g. Google Workspace or ProtonMail), but these usually have a flat fee so you'll be paying a hefty amount (> $5/month) even if you send zero emails.
  3. A slightly less common method is to set up your own email server on a VPS, but this is usually discouraged given the notoriety of maintaining email sender IP reputation and managing spam.

Back in May 2019, I tried setting up my own email server. It was a horrible experience and I couldn't get any IPs unblocked by Outlook. Eventually, I gave up and settled with option 1 — using a forwarder for inbound email and SendGrid for outbound email.

Recently, I became frustrated at how many of my emails were going into spam despite them not being spam. I use Microsoft's Hotmail (no particular reason, I created the account more than 10 years ago and didn't bother to switch), and looking through forums online, there's apparently no way to disable the spam email filter (who decided this was a good idea?).

So, I set out to find a solution. I didn't find paying a flat monthly fee particularly appealing, and besides, I was comfortable with using SendGrid for my outbound emails. I also wanted to store my emails on my own server instead of using a large provider that may suddenly start messing with my emails (i.e. what Hotmail did). This requires a server, but I already have a VPS up for hosting this website, so no additional costs are incurred.

I already knew I could use Dovecot for email storage and access via IMAP, which had worked relatively well during my initial attempt more than two years ago. However, I still needed a way to receive inbound emails from other mail servers. Setting up an inbound mail server with Postfix was an option, but there's a few issues with it — notably maintaining availability to ensure no emails are lost and the fact that port 25 is blocked by many VPSs. To resolve the first issue, I most likely would need to run multiple servers and deal with syncing emails between these servers, which I don't trust myself to maintain properly given the importance of some of the emails that'll be routed through these systems. Running multiple servers would also cost more than simply using an all-in-one provider.

I stumbled upon the solution I went with quite randomly one day — SendGrid's Inbound Parse Webhook. I had known about this for quite some time, but I hadn't looked very closely at how it was implemented. It turns out that it solves both issues described above.

SendGrid's Inbound Parse supports auto-retry for up to three days if the backend responds with a 5XX HTTP code. Three days is plenty of time for me to fix any issues if they arise. Also, Inbound Parse runs on HTTP/HTTPS, which means I can put the backend behind Cloudflare. This is perfect, as Cloudflare returns a 521 if the backend is down, which'll cause Sendgrid to retry at a later date. Cloudflare does have a 100MB upload limit for the free/pro plan though, but this isn't a problem as emails shouldn't be more than a few megabytes in size anyways.

Albeit this isn't the intended use for the Inbound Parse, but it works well enough for my case. It also opens up the possibility for me to use Inbound Parse's intended use (parsing the emails in code and routing them accordingly) in the future if need be.

Proof of Concept

I threw together a small Flask app to forward emails to Dovecot, which you might find useful. This app forward all emails for a domain to one mailbox, which works for my case, but you may need to expand on this app to separate emails based on recipient addresses.

# app.py
import imaplib
import os
import time
import traceback

from flask import Flask, abort, request

app = Flask(__name__)
app.config.from_object('config')

IMAP = imaplib.IMAP4_SSL if app.config['IMAP_SSL'] else imaplib.IMAP4


@app.route('/parse', methods=['POST'])
def parse():
    email = request.form.get('email')
    sender = request.form.get('from')
    subject = request.form.get('subject')
    sender_ip = request.form.get('sender_ip')

    if any(x is None for x in (email, sender, subject, sender_ip)):
        abort(503)

    try:
        conn = IMAP(host=app.config['IMAP_HOST'], port=app.config['IMAP_PORT'])
        conn.login(app.config['IMAP_USERNAME'], app.config['IMAP_PASSWORD'])
        conn.append(None, None, None, email.encode('utf-8'))
        conn.logout()
    except: # noqa
        failed_dir = app.config['FAILED_EMAILS_DIRECTORY']
        if failed_dir is not None:
            t = str(int(time.time() * 10000))
            with open(os.path.join(failed_dir, t + '.err'), 'w') as f:
                f.write(traceback.format_exc())
            with open(os.path.join(failed_dir, t + '.email'), 'w') as f:
                f.write(email)
        raise

    return '', 200

# config.py

# Set this to a somewhat large number to account for large emails with attachments (in bytes).
# Sendgrid accepts emails up to 20 MiB, so it's recommended to set this value to something larger.
MAX_CONTENT_LENGTH = 32 * 1024 * 1024

# Dovecot login info.
IMAP_SSL = False
IMAP_HOST = '<host>'
IMAP_PORT = 143
IMAP_USERNAME = '<username>'
IMAP_PASSWORD = '<password>'

# If for some reason, any emails fail to be saved to Dovecot, you can save them here for debugging purposes.
FAILED_EMAILS_DIRECTORY = None

Note: you probably want to secure this endpoint so trolls can't spam your inbox. There's a few ways to do this:

  1. Use Basic Authentication. Instead of domain.tld/parse, you would use username:[email protected]/parse in the POST URL.
  2. Randomize the POST URL. Replace /parse with some un-guessable string so trolls won't be able to fake emails. Note that the URL will appear in any logs, which may be a problem depending on your use case.

Also note that you must check "Send Raw" in the SendGrid configuration for this app to work correctly.

Final Thoughts

I've been using the Flask app for well over a month now, and it's been working flawlessly. With this approach, I have complete control over my emails, while still utilizing SendGrid's infrastructure to its full potential. I won't have to deal with maintaining high availability or any of the other intricacies with email server hosting, and can off source it to SendGrid, which they would handle better anyways.

As for the other parts of setting up an email server, they are already documented extensively. You can look at Integrating Sendgrid with Postfix if you want to send emails with smtp.domain.tld instead of smtp.sendgrid.net or add more authorization checks, and the Dovecot documentation for setting up mail storage and access.

Also, this blog was mainly focused on SendGrid, as that is what I'm using. From a quick skim, a similar approach would work with Mailgun as well.