Why sending notifications is a nightmare for engineers

33 points by coo1estguy a day ago on hackernews | 34 comments

InsidiousWombat | 5 hours ago

I dunno, personally I find getting them to be a nightmare. I still winge when I hear the vibration sound from the first few years of the job when I was on call

edoceo | 5 hours ago

The email send problem seems made up, your MTA handles that. The rest feels sales-y.

subscribed | 4 hours ago

SMTP is slow and unreliable. Even after the recipient's system prints "queued" there's no guarantee it will be delivered to the mailbox.

edoceo | 4 hours ago

Sure, but TFA wrote that someone would do SMTP retry in their app-code, which is stupid. Your app should deliver to your very reliable MTA, which tries like heck to deliver. After that there is nothing to be done, this tool, Postmark or Google as the sender cannot address that issue.

Also, IME is pretty quick (<1m) and reliable. Some US government agencies use is as their sole communication method for regulatory information.

electric_mayhem | 5 hours ago

Email is not an acceptable mechanism for delivering critical alerts.

Change my mind.

thomascountz | 5 hours ago

I agree with you. See: exhibit [A].

[A]: https://news.ycombinator.com/item?id=50509411

ranger_danger | 4 hours ago

IMO It depends on your definition of "critical."

Our company has relied on email for decades and never had a major issue that warranted finding an alternative. Most of our customers would not be ok with using anything else anyway. Locked down machines limits your options.

neuroticnews25 | 5 hours ago

"04 Your users are asleep, so you add quiet hours"

Is this a real thing? Do people really expect to not get emails while sleeping? Doesn't that go against the very concept of an async communication?

ferngodfather | 5 hours ago

Believe it or not we actually had complaints about this from people with far too much time on their hands.

Nobody living a normal life cares though.

yubblegum | 5 hours ago

If filtered emails are high priority and devices start demanding attention of sleeping staff, various matters are being synchronized via asynch communication.

eastbound | 4 hours ago

My parents have a bell for every email they receive, this is probably the default in Android, to ensure you consciously turn them off if you desire.

manarth | 4 hours ago

How is it not dinging every 5 minutes with spam?

kzrdude | 4 hours ago

The android Gmail app does not treat all email equally w.r.t notifiations.

TrackerFF | 4 hours ago

In some countries (Japan comes to mind), they’ll straight up close down web services outside business hours. Not for maintenance purposes either.

[OP] coo1estguy | 2 hours ago

what? bruh this is crazy, if it's true

rendaw | 4 hours ago

Every sibling ATM is saying yes it's a problem... do your phones not have do not disturb mode? Sleep mode?

Roark66 | 4 hours ago

Yes, android phones have quiet hours. So you not use the feature?

Unless I'm "on call" and paid extra I'm not picking up my work phone after hours.

fusslo | 4 hours ago

I guess it depends on the user and notification type. I don't get notifications for email, I just check it a couple times a day

But, an anecdote: I worked on IoT devices. When we'd lose wifi connectivity ( which we did all the time because our wifi module was awful and this was like 2015 ), we'd blink the leds amber at 100% brightness.

Some people would install the devices in their bedroom. I did. I woke up at 2 am to the light blinking.

The next day I added an MR to set the alert brightness to 30% if the hours were between 10pm and 6am.

Wifi-down wasn't worth waking the user up.

sigseg1v | 4 hours ago

I just finished a notification subsystem redesign in a moderate sized SaaS and this was one of the top requirements from customers.

CalRobert | 4 hours ago

I think it’s generational. Phones, very rudely, notify people when they have emails. So some people learned to treat them like IM’s. On the other hand if email was something you had to check then it didn’t matter when you got it.

Loud email notifications were once so absurd that this was hilarious http://dontevenreply.com/view.php?post=99

“ Oh, so in your drunken stupor you decided it would be a good idea to wake up my entire family at 3:30 in the morning? I have my computer hooked up to a 7.1 surround sound system, and Outlook plays a sound every time I receive an e-mail. “

[OP] coo1estguy | 2 hours ago

Yeah, it's more common than you think. People get really angry if they forgot to turn on dnd mode at night lol

ferngodfather | 5 hours ago

Email alone is fine for like 90% of use cases, imo. This is neat, but to me just so obviously looks like an AI slop solution searching for a problem.

[OP] coo1estguy | 2 hours ago

Maybe it's not for you, My business deals with a lot of rural customers globally and in many countries people don't actively use email so I personally use it for sending many type of notifications like push, sms for scenarios like users have added items to cart but not ordered. SMS costs money so try sending push first, then sms. idk if you are just using emails then this is not for you. I also use whatsapp notifs alot. Expand globally you will find usecases

smallpipe | 5 hours ago

Thinly veiled sales pitch written by chatgpt, no thanks

trollbridge | 5 hours ago

It isn’t a nightmare. Use iOS and Android’s notification system and let the user have fine grained control to tweak how they get notifications. Include options for critical alerts.

A very good alternate channel is Telegram for distributing to a group; Slack and Discord work too.

drcongo | 4 hours ago

Notifications should never be over email anyway. Might as well send a pigeon.

ranger_danger | 4 hours ago

There can be other valid perspectives than your own.

We send all our notifications exclusively over email and never had a problem. Plus our customers (healthcare) use locked-down machines without unfettered internet access (and no signal for mobile devices), so notification options are very limited.

xeonax | 4 hours ago

Are engineers not competent enough?

ranger_danger | 4 hours ago

Have you seen gestures broadly lately?

cletus | 4 hours ago

My qualifications for talking about notifications. I wrote what I believe to still be the current notification system for Facebook Live, which peaked at >10M QPS and averaged somewhere in the 1-8M QPS range. This isn't exactly the same as notifying someone about their order and shipping but it shares a lot of the same problems.

Now none of this sent notifications directly using Google's or Apple's notification services. There were whole levels of infrastructure for that at (then) Facebook. Also, there were types of notification where an ML recommendation had to decide if a given notification was something you might be interested in ie it was a decision to send or not send.

Notifications are inherently asynchronous and you need to immediately get out of the mindset of verifying if they were sent. You don't actually care if they're sent immediately or an hour from now generally. There may be valid reasons for waiting that have nothing to do with your process. For example, when a new product goes live you'll get a flood of orders. That's going to slow things down and you'll absolutely need to throttle your notifications or you'll kill downstream services. Disclaimer: I had to deal with the ML Infra teams yelling at us for doing exactly this.

Retry logic isn't something you should concern yourself with. You need a service you can rely on to send a notification. That's it's responsibility, not yours. If you successfully pass it on to that service, your job is done. It's up to that service to schedule, send and retry if necessary.

I wouldn't concern myself with quiet hours. Users silence their phones. They'll figure it out. That's just unnecessary complexity and it's opaque. People will likely get confused why they're not getting an email or notification for something they expect. Even with a relatively small number of users this is guaranteed to happen. Don't try and get too clever.

The author brings up a good point about separating kinds of notifications. This is something I did. Example: FB at the time had a setting on a page where you could say that you absolutely wanted a notification when the page went live. I separated those into a completely different process to the "maybe you're interested" process. There were a lot fewer of those so they went out really quickly. News organizations and the like would use those to sanity-check that their live was working and it made them feel better to immediately see a notification rather than having to wait 5-10 minutes. Plus, the user has shown the intent that they want those notifications so we should prioritize them.

The talk about mass sends is what I was referring to earlier although in our case, it was a page going live that had 20M followers. And the spike here was news organizations tended to go live at the top of the hour. That was extremely noticeable. Users would have preferences (eg no live notifications) and some users were more likely to be interested than not (eg if they'd watched previous lives by that page or any lives at all).

I'm not sure "nightmare" is the word I'd use but it's definitely complicated. The first step is getting out of the mindset of getting a send result and doing retries yourself.

[OP] coo1estguy | 2 hours ago

Author here, I mean with current trend in tech, retry seems absolutely necessary (so many outages, server down incidents)

Quiet hours are only honored by low, medium priority messages, critical messages are delivered bypassing so many restrictions.

yeah I added "nightmare" so people would click it more, still this is one such system nobody bothers about, but if you got it wrong, it has potential to bring your reputation down.

Thanks for detailed comment, I learned a few stuff from this :)

ranger_danger | 4 hours ago

> Except that loop can take ten seconds, and it's running inside the checkout request

I just send emails through a local MTA, that way it's always accepted immediately, and it takes care of any external sending/retries in the background, separately from my application.

[OP] coo1estguy | 2 hours ago

Yeah you ended up adding your own background, retries and other logic which are not core to your business. Same thing you'll do for other channels too. That's the whole point of article, it's hard and easy to get wrong

jmull | 4 hours ago

How silly the AI future we live in is.

To get a taste, choose "docs" on the menu, go to "03 How It Works", and peruse. You may soon be laughing (or crying).