Why sending notifications is a nightmare for engineers

Source: notifkit.dev
33 points by coo1estguy a day ago on hackernews | 34 comments
Engineering

Sending one email is one line of code. Sending them properly is a whole other thing. Here's how that one line grows, one support ticket at a time, and seven times it went wrong for real companies.

Every notification system starts the same way. Someone files a ticket: send a receipt email when someone places an order. Say you work at Halden, an online shop that sells lamps. You've got an email provider and an API key, and ten minutes later checkout does this:

on order placed:
    send receipt email

Ana in Lisbon orders a lamp (order #8812), the receipt shows up in her inbox, you close the ticket and go get lunch. Easy.

The 'This is fine' dog sitting in a burning room. Text: Duplicate receipts. 2am pushes. Provider down. / Your notification system

Every team that wrote sendEmail() and called it done.

It's not easy. I ran into a bunch of these problems myself, which is why I built notifkit. I looked at the hosted options first. Courier, Knock and Novu Cloud all charge per message, so they get stupid expensive once you're sending real volume. Self-hosting Novu fixes the bill but not the server: it's a handful of services plus MongoDB, it needs a big box to run, and it was slow when I tested it.

So I can promise you that one line is going to grow. Let's follow it for three months. Every step below is a problem you hit once you send enough notifications, and most of them come with a public incident from a company that hit it first: Salesloft, Zoho, Asana, Buttondown, Starbucks, HBO Max and Everbridge.

Quick note before we start. I don't know how any of these companies built their systems, and I'm not calling anyone careless. Several of them published honest write-ups about what broke, which takes guts. The code below is my version, not a guess at what they should have done. My point is just that this stuff is hard, and careful teams still get bitten.

01Week one · Support: "A customer says her receipt never arrived."

The provider timed out, so you add retries

You check the logs. The provider took too long to answer, the call threw, and nothing went out. Networks do that. Fine, retries:

send receipt:
    try 3 times:
        send the email
        if it worked: done
        wait a bit longer each time          # 1s, 2s, 3s
    give up: "receipt not sent"

Done. Except that loop can take ten seconds, and it's running inside the checkout request, so Ana is staring at a spinner while you wait on an email. So you move the send into a background job, and now you've got a job queue to look after. For one email.

02Two days later · Support: "Now she has two receipts."

The retry sent it twice, so you give each message an ID

Here's what happened. A timeout only means you didn't get an answer. It doesn't mean the provider didn't send the email. It did, and then your retry sent it again.

Two identical Spider-Men pointing at each other. Text: Receipt #8812 / Receipt #8812 (the retry)

Ana's inbox after one timeout and one retry.

What you need is an ID for each message that stays the same every time you retry it, like order-8812-confirmed. (A random ID won't work, because every retry gets a fresh one.) Before you send, you check whether that ID already went out.

Without an ID. The retry looks like a new email.

Checkout

Provider

Ana's inbox12

timeout → retry

With order-8812-confirmed. The retry is recognised.

Checkout

ID check

Provider

Ana's inbox1

timeout → retry

seen it, skip

Figure 1. Same timeout, same retry. Green is the first attempt, amber is the retry.
send receipt:
    id = "order-8812-confirmed"              # the same on every retry

    if id was already sent: stop

    try 3 times:
        send the email, passing id to the provider
        mark id as sent                      # after the send, not before
        done

One thing in there is sneaky: the "sent" mark goes after the send. Write it before, and a crash in between means the receipt never goes out. Write it after, and a crash in between means the retry sends a second one. You don't get an option where neither can happen.

That second case is pretty much what Zoho described: the email went out, they couldn't mark it as processed, so it got retried. You can't fully fix that from your side. The provider has to accept your ID and refuse the repeat, which is why the ID gets passed to the provider too.

And honestly, none of this would have saved Salesloft. Gmail reported errors for emails it had actually delivered, so anyone would have retried. Only the provider can fix that one.

03The week after · Status page: "Email delivery: major outage."

The provider went down, so you add a second one

Retrying three times doesn't help when your provider is down for an hour. So you sign up for a backup and try them in order. If both are dead, you send a text instead.

email · primary✕ down→ email · backup✕ down→ sms✓ sent

Figure 2. First a backup provider on the same channel, then a different channel. If Ana has turned off texts, it stops before the SMS.
send receipt:
    ... id and sent check as before

    for provider in [primary, backup]:
        if provider failed 5 times in a row: skip it for 30s

        try 3 times:
            send the email through provider, passing id
            if it worked: mark id as sent, done

            if it's a bad address: stop, don't retry    # not an outage
            note that provider failed
            wait

    give up: "not sent by any provider"

The "skip it for 30s" line is there so that during an outage, every single message doesn't sit through three timeouts before moving on. If a provider keeps failing, you stop trying it for a bit.

The annoying part is deciding what counts as a failure. A provider rejecting a bad address is the provider working fine. Count that as a failure, and five dead addresses in a row switch off a perfectly healthy provider for all your customers.

Oh, and the ID from step 02 just sprang a leak. Only the provider that saw it knows about it. If the primary delivered the email and then timed out, the backup has never heard of that ID, and it sends the email again.

04A month in · Ana, 2:40 am: "Why are you emailing me in the middle of the night?"

Your users are asleep, so you add quiet hours

By now the receipt isn't alone. Product wanted a "your order has shipped" email, so "send receipt" is now a general notify(). The warehouse is in Ohio. They ship Ana's lamp at 9:40 pm, which is 2:40 am in Lisbon, and the email goes out the second the box gets scanned. Ana is not thrilled.

Drake meme. Rejecting: Send 'your order has shipped' the second the box is scanned. Approving: Hold it until 8:00 in Ana's time zone.

The warehouse works nights. Ana doesn't.

So the message has to wait until her morning, unless it's urgent:

"Your order has shipped"held, not dropped

z z z 02:40 arrives 08:00 sent

000608121823

"Card flagged for fraud" · criticalsent now

02:40 sent

000608121823

Figure 3. Ana's day in Lisbon time. The shaded area is her quiet hours, 23:00–08:00. They cross midnight, so the check is "after 23:00 or before 08:00".
notify(user, message):
    if not critical and it's night where the user is:
        save the message to the database
        send it at 8:00 their time
        stop

    ... sent check, providers and retries as before

Store the time zone as a name (Europe/Lisbon), not an offset like +01:00, because the offset changes with daylight saving. I'm skipping the "is it night for the user" check on purpose. Ana's quiet hours are 23:00 to 08:00, so it has to handle a window that crosses midnight, plus the two nights a year the clocks change. It's longer than you'd think.

The held message goes in a database table, because a setTimeout dies with your next deploy. Which means you now need something that wakes up every minute, finds what's due and sends it. At 08:00, before it sends, it has to check that Ana didn't unsubscribe overnight. And fraud alerts and login codes can't wait for morning, so that's what critical is for.

05Friday, 8:00 am · Provider: "429 Too Many Requests."

Everyone sends at once, so you add a rate limiter

You've added "out for delivery" too. All the vans leave the depot at 8:00, so thousands of those fire in the same minute. Your provider allows 100 a second and starts throwing 429s at you. Rate limiter time:

    for provider in [primary, backup]:
        skip provider if it keeps failing

        while we're over the provider's limit:
            wait a random 0.2 to 1 second       # so they don't all come back at once

        ... retries as before, now with a random wait too

The wait is random because if everyone backs off for exactly one second, everyone comes back at the exact same moment and you're right where you started.

Everyone retries at +1s

limit

The jam forms again every second.

Each picks a random time

limit

A queue that moves at the speed the provider allows.

Figure 4. 5,000 messages against a limit of 100 a second. Green got through, red was turned away. A message waiting for its turn hasn't failed, so it shouldn't count as a retry.

Here's the fun part: a rate limiter is new code, so it can have bugs too. Buttondown's did. A bug in their rate-limiting code sent duplicate emails for five newsletters over eight minutes.

Gru's plan meme. Add a rate limiter. Provider stops rejecting us. Rate limiter has a bug. Rate limiter has a bug.

Every fix in this article is more code that can break.

Two more things sneak in around here. A big send shouldn't hold up a password reset, so urgent messages need their own queue. And nobody should get fifty messages in an hour because your code got stuck in a loop, so you add a cap per user.

06Marketing: "Can we send a review request to everyone who ordered last month?"

Marketing wants a bulk send, so you add consent checks

A receipt is something Ana asked for. A review request isn't, and she's allowed to say no. Gmail and Yahoo require one-click unsubscribe on bulk email, and providers punish senders who keep mailing addresses that bounced or marked them as spam. So two checks go in front of everything:

notify(user, message):
    if the address bounced or reported spam: stop
    if not critical and user turned off this topic: stop

    ... quiet hours, sent check, providers, rate limit and retries as before

"How's the lamp?" going to four people who ordered last month

AnaLisbonturned off review emails✕ skipped

BenLeedshis last email bounced✕ skipped

KenjiTokyoit's the middle of the night there◷ waits until 8:00

MayaChicagonothing stops it✓ sent, with unsubscribe link

Figure 5. One send to the whole list, but each person gets a different answer. Only Maya gets the email right now.

Per-topic settings let Ana turn off review requests and still get receipts. A blocked list stops every send to an address that bounced or complained, critical ones included. And each of those drags in more stuff: an endpoint for the provider's bounce and complaint webhooks, an unsubscribe page, and a signed token in the link so nobody can unsubscribe someone else.

But all of those checks ask whether Ana wants the message. None of them asks whether you meant to send it. That's where the famous ones happen. HBO Max and Everbridge both had a test go out to real people.

A sweating man deciding between two identical red buttons. Text: Send test email / Send to all subscribers / Me, with the production API key

Both buttons look the same until one of them is in the news.

The cheapest fix I know is making it impossible for a test to reach a real inbox, whichever button you press:

if not production:
    use a provider that only writes to a log

You'll also want a log of who got what, and a big red stop button for when a bad send is already halfway out the door.

07Support: "A customer got 'Delivered' for an order she cancelled."

The messages don't know about each other, so you add a workflow

Last one. By now you're sending five messages per order, and each one is fired by a different bit of code:

11:26

Monday, Lisbon

HALDENTue 15:12Thanks for your order#8812 · Ceramic lamp

HALDENThu 08:00Your lamp has shippedTrack your parcel

HALDENFri 08:05Out for delivery todayArriving 10:00–14:00

HALDENFri 11:26DeliveredLeft at your front door

HALDENMon 11:26How's the lamp?Takes 30 seconds

  1. Receiptcalled from checkout
  2. Shippeda webhook from the warehouse
  3. Out for deliverya webhook from the courier
  4. Deliveredanother webhook from the courier
  5. Review requesta timer, three days later
Figure 6. Five messages for one order, each fired by a different piece of code.

None of those handlers know what the others did. "Delivered" can show up before "shipped". The review request goes out even though Ana sent the lamp back. And if the courier's webhook is wrong, you just repeat whatever it says.

'Is this a pigeon?' meme. Your app looks at order.delivered (for an order nobody placed) and asks: is this a real order?

Your code believes whatever the webhook says.

Your code can't tell a fake event from a real one. What it can do is make sure an event only moves its own order forward, and only in the right order. For that you need one record per order that knows what's been sent and what comes next:

on order event:
    flow = the saved record for this order

    if there's no record, or the order was cancelled: ignore it
    if it's not the step we're waiting for: hold it    # "delivered" before "shipped"

    notify(user, event)
    move flow on to the next step

Receipt email

wait for order.shipped

Shipped email

wait for order.out_for_delivery

Out for delivery push → sms

wait for order.delivered

Delivered push + email

wait 3 days, unless a return starts

Review request email

order.delivered #8813
✕ not this order, ignored

order.delivered #8812
✓ matched, continue

Figure 7. One workflow per order. Each step is saved before the next one runs, so after a deploy on Thursday it carries on from where it stopped.

And now you're building your own workflow system. It has to wait for days without losing its place, survive a deploy halfway through, and stop when the order gets cancelled. That's a whole project by itself.

Three months after the ticket

What one line turned into

Here's notify() with everything from above in one place:

notify(user, message, critical = false):
    id = order id + message type

    if the address is on the blocked list: stop
    if not critical and user opted out of this topic: stop

    if not critical and it's quiet hours for the user:
        save it, send at the end of quiet hours
        stop

    if id was already sent: stop

    for provider in [primary, backup]:         # log-only outside production
        skip provider if it keeps failing
        wait if we're over the provider's limit

        try 3 times:
            send through provider, passing id
            if it worked: mark id as sent, done

            if it's a bad address: stop, don't retry
            note that provider failed
            wait, plus a random extra

    give up: "not sent by any provider"

That's 25 lines of pseudocode, and the real thing is a lot longer. It also leaves out the quiet-hours check, the thing that sends held messages, the rate limiter, the "keeps failing" tracking, the order workflow, the unsubscribe page, the endpoint that fills the blocked list, the delivery log, and the tests for all of it.

Distracted boyfriend meme. Our engineers look back at 'building our own notification system' while 'the product customers pay for' looks on.

Halden sells lamps.

None of these pieces is that hard on its own. The hard part is that they all touch each other. Quiet hours have to respect the blocked list, retries have to respect the message ID, and the workflow has to survive a deploy in the middle of all of it.

Oftentimes things that seem like they should be simple end up being way more complicated when you get into the details.

Sophie Alpert, on Slack's notification logic · "Everyone is wrong about that Slack flowchart"

About notifkit

The same thing with notifkit

That's what notifkit is for. It's an open-source notification service that does everything in that listing, so none of it has to live in your codebase. You run it yourself, on your own Postgres and Redis. The receipt from the first ticket looks like this:

await notifkit.notify({
  user: "usr_ana",
  template: "receipt",
  channels: ["email", "sms"],
  fallback: true,                  // email first; SMS only if email fails
  data: { orderId: "8812" },
});

Ana's time zone, quiet hours and opt-outs live on her user record. The duplicate checks, the backup providers, the rate limits and the blocked list run on every send. And the five messages for an order are one workflow:

workflow("order", async ({ step, event }) => {
  const order = { orderId: event.orderId };

  await step.notify({ template: "receipt", channels: ["email"], data: order });

  const shipped = await step.waitForEvent("order.shipped", {
    match: order,
    timeout: "14d",
  });
  if (!shipped) return;
  await step.notify({ template: "shipped", channels: ["email"], data: order });

  // ... "out for delivery" and "delivered" wait for their events the same way

  await step.wait("3d");
  await step.notify({ template: "review", channels: ["email"], data: order });
});

Each step gets saved before the next one runs, so a deploy on Thursday picks up right where it left off. match stops one order's event from waking up another order's workflow. If Ana cancels or starts a return, you cancel the workflow and nothing else goes out.

To be upfront about what it doesn't do: it isn't hosted, so you run it. And it doesn't promise exactly-once delivery, because nobody can when a provider says an email failed after delivering it (hi, Salesloft). It catches the duplicates it can see.

Yes, this last section is a plug. I built the thing, so obviously I think you should use it. In my defence, it's MIT licensed, there's no paid plan, and nothing above needs notifkit. If you'd rather build it yourself, take the pseudocode and go for it. If you want to try it, the quickstart takes a few minutes. It's a young project and I'm actively working on it, so if something's missing or breaks, open an issue at github.com/devkitshq/notifkit and I'll take a look.