Two months ago, I wrote a blog post using SendGrid's Inbound Parse to forward emails to Dovecot (IMAP). The reasons behind this decision were discussed in detail there, but to summarize:

• Auto-retry meant I would not have to worry about high availability on my end.
• Port 25 (for inbound emails) is blocked on many VPSs so unblocking it would be a challenge.
• I don't need to expose my backend IP as Inbound Parse can sit in front of my server.

I also briefly noted that this wasn't the intended use of Inbound Parse, but that it did open up the possibility of using it for its intended purpose (parsing the emails in code and routing them accordingly) in the future.

There's been a few annoyances with directly forwarding emails to Dovecot that I wanted to address, which has finally lead me to using Inbound Parse for its intended purpose. In the process, I've had to create a library to properly resolve these annoyances.

Part 1: Background

There were 3 main issues I wanted to address.

1. Github doesn't allow account-wide webhooks. The only options are email and their mobile app for notifications. I've been growing increasingly irritated by getting spammed an email for every comment reply just because I commented on an issue/pull request once. Of course, I do want some notification, just not so many that it immediately clutters my inbox.
2. LinkedIn sends an email every time I log in, and from what I've seen, there's no option to turn it off. It's gotten to the point where I question if I really need to sign in and have to clean up the email spam afterwards.
3. For my DMOJ rating predictor, there are Request Tokens that users can sign up for to automate custom contest requests. These require emailing a specified email address. Right now, this is hackily added into the current Inbound Parse setup.

For the first two issues, my goal is to redirect the emails to a Discord channel and then archive the emails automatically. For issue #3, I need to direct the email to a custom script to create a request token.

Of course, this could all be hacked into the current Inbound Parse setup with little effort, but expandability is nice so more routes could be added in the future as needed.

Part 2: What about Mailgun routes?

I'll admit, Mailgun routes are pretty powerful for most cases. Unfortunately, they're just not powerful enough to cover what I wanted to do. Issue #3 requires interacting with a custom script which can't be accomplished directly with Mailgun routes without POSTing to a custom server. At that point, it would be identical to just using SendGrid's Inbound Parse and routing everything myself.

Part 3: Email Router Library

I searched for any Python libraries that implemented what I needed. The closest I could find was salmon, but it's a full-fledged email server and over-engineered for my purposes.

I decided to create my own library to route emails, uncreatively named email-router. It's a small Python package and is available on PyPI:

$ python3 -m pip install email-router

Part 4: Library Usage

You can route emails with:

from emailrouter import Email, Router

raw_email: str = ''  # The raw (MIME) email
config: str = 'config.yml'  # The routing configuration file

router = Router.from_yaml_file(config)
router.execute(Email(raw_email))

This will run the email through a series of routes, which have filters and handlers attached to determine which emails to process and how to process them. All configuration is defined in a YAML file for simplicity.

Part 5: Routing Configuration

If you're familiar with Python's logging framework, this format is very similar. There are some minor differences though.

Filters

Filters determine which routes are run for each email. Some common fields and comparison operators are available for use directly in the YAML configuration. For more advanced configuration, Python functions can be arbitrarily loaded via the config file.

Example filter:

type: all
conditions:
 - type: equal
   field: subject
   value: Test
 - type: isubstring
   field: body
   value: Hello
 - type: contains
   field: recipients
   value: [email protected]
   condition: iequal
 - type: any
   conditions:
    - type: listany
      field: senders
      value: "@evanzhang.ca"
      condition: iendswith 
    - type: listany
      field: senders
      value: ".*@[^.].com"
      condition: regex
 - type: python
   file: custom-filter.py

Most of the example should be self explanatory. A prefix of i on a type simply means that the comparison is case insensitive.

The following fields are available:

• message_id: str
• body: str
• subject: str
• bcc: list[str]
• bcc_names: list[str]
• cc: list[str]
• cc_names: list[str]
• recipient_names: list[str]
• recipients: list[str]
• reply_to: list[str]
• reply_to_names: list[str]
• sender_names: list[str]
• senders: list[str]

Note 1: Only the above fields are available in the YAML configuration. However, more fields are available via a Python function (i.e. headers).

The following types are available:

• python
• any
• all
• not
• listany
• listall
• true / false
• equal / iequal
• regex / iregex
• substring / isubstring
• startswith / istartswith
• endswith / iendswith
• length

Note 1: For list fields, you must use a listany or listall type followed by a condition: <actual type>.
Note 2: all, any, and not types are non-leaf types, meaning they require sub-conditions. all and any take a list of sub-conditions under conditions: while not takes a single sub-condition under condition:.
Note 3: python is for advanced filters which require complex code that can't be represented with this basic YAML configuration. In the example configuration, custom-filter.py should contain a function with the signature filter(email: Email) -> bool where a return of True means it passes the filter and False otherwise.

Handlers

Handlers process any emails that passes through the filters.

Example handler:

type: imap
kwargs:
  url: imap://username:[email protected]
  ssl: true
  flags: "(\\Seen)"
  mailbox: "Archives"

The following handlers are available at the time of writing:

• imap: for storing emails into an IMAP server.
• placeholder: temporary placeholder handler that doesn't do anything.
• discord: sends the email to a Discord channel via a Discord webhook.
• python: custom Python handler with the signature handle(email: Email) -> None.

Routes

Routes combine filters and handlers together to create a powerful route. They are processed in the order defined. By default, all routes will be attempted regardless of if previous routes have ran. If you want to stop once a route runs (passes filters and handlers have processed the email), you can set propagate: false. This will stop processing any further routes. Note that this means routes defined earlier have higher priority, so order them accordingly.

Example route:

name: Github Notifications
propagate: true
handlers: ...
condition: ...

• name: for human reference and isn't used by the library at all (besides in debug logs).
• propagate: whether future routes should be processed if this route is run (i.e. short circuit).
• handlers: list of handlers to run if the filters pass.
• condition: a filter with potential sub-conditions.

Named Filters and Handlers

Filters and handlers can be "named" in the configuration. For convenience, you can inline filters and handlers into the routes. However, there is a dedicated section in the configuration for naming them if you want to use them for multiple routes to avoid duplication.

For example:

filters:
  Named filter:
    ...
handlers:
  Named handler:
    ...
routes:
  name: Named route
  propagate: false
  handlers:
   - type: named
     name: Named handler
  condition:
    type: named
    name: Named filter

type: named filters can be used in sub-conditions, so you can do some pretty interesting configurations. Using previously defined named filters as sub-conditions in a new named filter works.

Part 6: Putting It All Together

To resolve the issues I outlined earlier, I'll be using this configuration for my inbound emails (with some information redacted). Feel free to take inspiration from this configuration.

handlers:
  Archives IMAP:
   type: imap
   kwargs:
     url: <redacted>
     ssl: <redacted>
     flags: "(\\Seen)"
     mailbox: "Archives"
  Request Token:
    type: python
    file: create-token.py
  Main Discord:
    type: discord
    kwargs:
      webhook_url: <redacted>
      colour: 0x44ff99
      payload:
        username: Mail
  IMAP:
    type: imap
    kwargs:
      url: <redacted>
      ssl: <redacted>

filters: {}

routes:
 - name: Linkedin Notifications
   propagate: false
   handlers:
     - type: named
       name: Archives IMAP
   condition:
     type: all
     conditions:
       - type: listany
         field: senders
         value: <redacted>
         condition: iendswith
       - type: isubstring
         field: body
         value: "signing in"
 - name: Github Notifications
   propagate: false
   handlers:
     - type: named
       name: Archives IMAP
     - type: named
       name: Main Discord
   condition:
     type: listany
     field: senders
     value: <redacted>
     condition: iequal
 - name: Request Token
   propagate: true
   handlers:
     - type: named
       name: Request Token
   condition:
     type: all
     conditions:
       - type: listany
         field: recipients
         value: <redacted>
         condition: iequal
       - type: iregex
         field: subject
         value: <redacted>
 - name: Catch All
   handlers:
     - type: named
       name: IMAP
   condition:
     type: "true"

Part 7: Final Thoughts

I hope some of you will find this library useful and give it a shot. As usual, I've been running this new inbound parse server for over a week, so any glaring issues have been fleshed out. If you do find any bugs or have additional feature requests (e.g. different handlers), please file an issue.