The main reason that domain names cost anything is because cheap and free DNS providers always get spammed to hell with junk and abuse, then they don’t have enough money to tackle the ensuing problems. Pricing is not a complete solution to spam, but it can cut out the worst of it. (OK, the other reason is ICANN’s feudal tax, which is why the clearest examples of disastrous spam have been ccTLDs that don’t have to pay ICANN; happily they learned from their mistakes and cleaned up their acts.) (OK OK, and the trademark lawyers must be appeased.)
I pay $4.20/month for the VM that hosts my outgoing mail server. It's used by me and a few other people for a handful of domains. I haven't had any problems reported with mail delivery for ages, but I've also been using the same domain for around 20 years and that domain has built reputation. Being an outgoing mail server uses less than 1% of the power of that VM, so having an IP address that's able to send a bunch of email is fine. My provider retains a fairly good reputation for the IP block because their policy is that outbound port 25 is blocked by default, you have to request that it is unblocked, and they'll block it again if you are reported for sending spam.
I suspect that people like SendGrid exist because of the problem that I recall hearing from someone who ran anti-spam things for Yahoo! Mail back in the day: a lot of users find it easier to hit 'spam' than 'unsubscribe'. If someone actually did sign up for a mailing list but forgot or doesn't want it anymore, they'll often just mark the messages as spam and not bother to unsubscribe. In a big mail provider, spam filtering is shared to a reasonable degree between users, so if 20% of people who ever subscribed to your mailing start marking it as spam then you'll start to find that your mails are marked as spam for everyone.
I am a bit skeptical about companies like SendGrid for this reason: they seem to exist to do reputation laundering. If you send your emails with them, they'll send them from a set of IPs that sends so many emails that it's hard for big mail providers to block them and so that their spam:ham ratio is maybe not too high.
I haven't had any problems reported with mail delivery for ages, but I've also been using the same domain for around 20 years and that domain has built reputation.
There are two kinds of people who self-host email. People who say the above, and people who no longer self-host email because their emails don't arrive. Which group you belong to is half skill (getting DMARC, DKIM, SPF and email headers right, trawling through all the blacklists to get yourself unblacklisted) and half luck. Tomorrow, Google might change their spam filter and your emails now go straight to junk. You may never realize that your answer to the life changing job offer never arrived and you missed the window of opportunity.
Anyway, the first group of self-hosters telling the second group "it works for me" does not help. It's not that easy. The risk of emails not arriving is significant and troubleshooting such issues is a tremendous time expenditure. I personally gave up, it's just not worth it.
That's a different question than the article is addressing, though. (Unless I've misunderstood you!) It sounds like you're running mail service for a small group of people you know. Where you and those people are sending "handwritten" emails to people that the senders (mostly) know personally.
TFA is talking about transactional emails from their website that gets real traffic from the general public.
One of those things carries more deliverability challenges than the other.
a lot of users find it easier to hit 'spam' than 'unsubscribe'.
To be fair, I'm unable to unsubscribe from three advertising lists from very real sellers (two also with physical shops) and of course, I have been subscribed to these lists without my consent.
I've also unsubscribed from some other lists but even over several years, not many more than three. This means that I'm unable to unsubscribe from a very large part of the lists I'm on and from the vast majority of marketing emails I get.
In the old days, if someone sent spam, we'd write their network operator, and they'd get booted or otherwise punished. Once corporations and profit became involved, that stopped.
Like everything else in life, if we want something to stay nice, we need to punish those who want to make a profit from shitting on it.
For all of its issues, i think the flexibility and now longevity of email is still quite impressive! Though, at the same time, I'll be the first person to agree that I wish we could create a new email system from scratch - of course, with properly applied learnings from all the decades that have been learned from conventional email.
Spam email is not something that can be fixed by only technical means.
Most of the problem arises from the economic structure of the protocol, in the broad sense of security economics: i.e. who is participating in the protocol and what their relationships are, in terms of contracts, payments, or the lack thereof. There’s also a macroeconomic version of Conway’s law (which I think is due to David Clark) which observes that an open protocol defines an interface between businesses that operate that protocol as a service, so when you are designing a protocol you need to keep that in mind because it determines who ultimately bears the costs.
The difficulty with spam is due to some of the basic attributes of email:
There’s no postage fee for last-mile delivery (unlike telephone or paper mail, where some of the money that the sender pays their provider is passed on to the recipient’s provider).
Like phones and post, you don’t need any prior contact with a recipient to send them a message: it’s enough to read their address from a business card or an ad on the side of a bus.
There’s no central authority that can enforce any rules on the system as a whole.
It’s easy to join in by getting an internet connection and a domain name.
It’s common for techies to want to fix spam by improving sender authentication (that’s the whole point of SPF and DKIM) but it’s still easy for spammers to mint new identities, so spam continues.
The upshot is that anti-spam is mostly about reputation management. Receivers keep track of sender reputations and use that as a major signal for classifying spam; and senders police their customers to avoid damaging their reputation. Senders for this purpose can be identified by IP address as well as domain name or email address. That basic structure has been the same since Paul Vixie introduced the RBL, years before SPF and DKIM came along.
Much of the cost of sending email is to pay for that policing effort. Some of the cost is also to discourage the bottom-feeding spammers who rely on nearly-free sending to cope with a very low response rate. But there’s a somewhat delicate balance on setting the cost of email: it isn’t possible to make it too expensive for spammers but still affordable for legitimate senders. (Observe the ratio of paper junk mail to legit post that comes through your letter box.)
I think basically all of the technical deficiencies of email can in principle be fixed by evolving the current system. I’ve ranted about this here before but the main reason for problems like shitty underspecified html is that the large email providers aren’t working with each other to improve things. (I suppose there are some things that have changed, such as DMARC and transport security, but they don’t have much positive effect on the way users perceive email.) And without a commitment to migrate the majority of email users to something better, all you have is yet another niche chat system, of which there have already been dozens.
Which is why it’s important to consider protocol economics when designing a replacement for email: there must be positive incentives in favour of adoption as well as negative incentives against abuse.
I just want to say thank you for your email comments, especially the linked rant. I share many of the same thoughts but am not as grounded in the history of it all. Your words help flesh things out.
And it's also nice hearing people beat the drum of the centrality of economics problems in protocols. Also the drum that email-qua-suite-of-protocols is Mostly Fine™ (I put DNS in a similar camp).
I guess I am not being spammed on Activitypub, but is that because of the protocol or because it is just not as popular?
It’s a combination of a small community and the fact that it isn’t funded by advertisers. The people running instances have a fairly strong motivation to keep the network clean. Roughly what email was like before 1995ish. This can change for the worse if it’s easy enough to fake up a new server instance that the benefit of spamming outweighs the cost of playing defect, and if legit servers don’t have easy ways to vet the reputation of new instances.
I tend to agree, too. It seems a no-brainer in hindsight to be able to do things like "pre-allow a given party to send me e-mail" (perhaps via revokable capability URLs)?
I would love a good way to revoke a sender's ability to send me email. There's a looooot of companies in the world where "unsubscribe" means "unsubscribe for the legally required 1 year or whatever, and then start spamming you again."
Give each company a different email address and revoke when you no longer want their emails. Of course, this is easier if 1) you have your own domain, and 2) you run your own email server, or use an email service that allows unlimited email addresses per domain.
I was thinking along these lines: instead of typing your e-mail address, your browser fills the field with a capability URL. This points indirectly to your inbox - kind of like mail forwarder services do today. Now that I think of it, it seems like it's achievable with no new protocols (with something like Firefox Relay), but service providers still treat temp-mail like a definite scam signal instead of a legitimate use case for normal people. Had this been the default since the early days, we might be able to enjoy revokable addresses, thanks to them being single-use only.
I'm signing up for a new service on-line, and it wants to send me an email. How do I pre-authorize it? How do I even know what address the email is coming from (a complaint I have now)? Why get the web involved at all? (I know, I know, it's the "All Singing, All Dancing Protocol For Everything From Now Until The End Of Time", much like FTP was on the very early Arpanet).
Or, I'm reading a blog post from Alice, and I want to send her an email about it. How can she "pre-allow" my email? She doesn't even know I want to send her one.
I don't use it (because DHH) but this is close to the idea of hey.com's email service. Literally nothing comes to your inbox unless you've allowed it. It means you end up with two inboxes -- in a way -- one to check in case there's something interesting in there, and the one where you know the emails that you want are in it. (Similar to GMail's priority inbox, but with less promise of magic).
I think this is a good model, perhaps even a better default model.
If I think of a real inbox -- I go and fetch the physical post, 80% is mass-mailed and goes straight into recycling and only letters from suppliers and government tend to end up in my actual in-tray. The idea that the postman could put stuff in your inbox is nuts!
with properly applied learnings from all the decades that have been learned from conventional email.
What are those learnings? I'd say the most important weakness of email is the spam problem; authentication is "solved" with TLS (non-TLS mailservers notwithstanding) and I personally don't consider encryption a very important core feature. So, spam: how to "fix" that without making email closed?
Learnings would include lots of stuff that apply elsewhere in tech, and tend to be something along the lines of trusting too much of other systems and devices back in the day, and yet nowadays knowing why we should not trust any other system or device on a network (either local or otherwise), and build up from there, etc. Now, as to your point about not considering encryption a very important core feature, i disagree...but that does not make you nor I correct or incorrect; merely that we differ in desired features...which i actually wish was easier to decide to turn on or off by a user. Because i know that we do indeed have encryption via email, but its implementation is not shall we say easy, etc.
It is not just „old and goofy“. There is some purpose and thought behind it.
The conversational character of the SMTP protocol makes perfect sense. You will not send e-mail data if the recipient mailbox does not exist, or if the attachment is larger than given server accepts. Why send several megabytes and then realize that it is over the limit? Why send message in an unsupported encoding? Semantics of a single request-response does not fit everywhere. (or you can see SMTP as a sequence of several requests and responses coupled together in a session).
And those „randomly“ disappearing e-mails that can not even be found in the spam folder? When I used to be a postmaster, it was a matter of honor to deliver every message and not miss anything. The suspect spam was just flagged and it was up to the user configurable filters to put it in the appropriate folder. We were essentially peer to peer and each of us was interested in making sure it worked reliably. Today, it is different. There are several global corporations and they would prefer it if everyone else would just disappear. These corporations can not simply nuke smaller providers or suddenly stop accepting their messages, but can make them look unreliable in the eyes of users.
From a technical perspective, [SMTP submission] is essentially the same as sending an HTTPS request.
It is not, and treating it this way will get your emails dropped. Email is fundamentally a store-and-forward system, and the protocol requires that you be able to do exponential backoff for delivery. That requires a persistent queue, for which there is no built-in analogue in HTTP, and it also makes the operation fundamentally asynchronous. HTTP, by contrast, is full of synchronous use-cases; most endpoints in most RESTish APIs are synchronous.
At minimum, this will get your emails dropped if the destination server goes down e.g. for maintenance or an outage and relies on senders doing standards-compliant retries. But some providers will always drop your mail as spam, because of greylisting.
gerikson | 19 hours ago
Title should be "Why is it so expensive to send a lot of emails without getting them marked as spam".
BenjaminRi | 17 hours ago
Next up in the series: "Why does a single table row cost $15 per year?", a riveting deep dive into DNS.
fanf | 10 hours ago
The main reason that domain names cost anything is because cheap and free DNS providers always get spammed to hell with junk and abuse, then they don’t have enough money to tackle the ensuing problems. Pricing is not a complete solution to spam, but it can cut out the worst of it. (OK, the other reason is ICANN’s feudal tax, which is why the clearest examples of disastrous spam have been ccTLDs that don’t have to pay ICANN; happily they learned from their mistakes and cleaned up their acts.) (OK OK, and the trademark lawyers must be appeased.)
david_chisnall | 16 hours ago
I pay $4.20/month for the VM that hosts my outgoing mail server. It's used by me and a few other people for a handful of domains. I haven't had any problems reported with mail delivery for ages, but I've also been using the same domain for around 20 years and that domain has built reputation. Being an outgoing mail server uses less than 1% of the power of that VM, so having an IP address that's able to send a bunch of email is fine. My provider retains a fairly good reputation for the IP block because their policy is that outbound port 25 is blocked by default, you have to request that it is unblocked, and they'll block it again if you are reported for sending spam.
I suspect that people like SendGrid exist because of the problem that I recall hearing from someone who ran anti-spam things for Yahoo! Mail back in the day: a lot of users find it easier to hit 'spam' than 'unsubscribe'. If someone actually did sign up for a mailing list but forgot or doesn't want it anymore, they'll often just mark the messages as spam and not bother to unsubscribe. In a big mail provider, spam filtering is shared to a reasonable degree between users, so if 20% of people who ever subscribed to your mailing start marking it as spam then you'll start to find that your mails are marked as spam for everyone.
I am a bit skeptical about companies like SendGrid for this reason: they seem to exist to do reputation laundering. If you send your emails with them, they'll send them from a set of IPs that sends so many emails that it's hard for big mail providers to block them and so that their spam:ham ratio is maybe not too high.
BenjaminRi | 16 hours ago
There are two kinds of people who self-host email. People who say the above, and people who no longer self-host email because their emails don't arrive. Which group you belong to is half skill (getting DMARC, DKIM, SPF and email headers right, trawling through all the blacklists to get yourself unblacklisted) and half luck. Tomorrow, Google might change their spam filter and your emails now go straight to junk. You may never realize that your answer to the life changing job offer never arrived and you missed the window of opportunity.
Anyway, the first group of self-hosters telling the second group "it works for me" does not help. It's not that easy. The risk of emails not arriving is significant and troubleshooting such issues is a tremendous time expenditure. I personally gave up, it's just not worth it.
altano | 13 hours ago
Jokes on you all: even the people paying for hosting don’t have reliable delivery. Half my emails from iCloud don’t reach Gmail.
pja | 8 hours ago
Yup. I've seen GMail classify their own mail servers as spam sources.
Mail delivery is hard & outsourcing your mail hosting to one of the big mail hosters does not guarantee anything like reliable delivery.
hoistbypetard | 15 hours ago
That's a different question than the article is addressing, though. (Unless I've misunderstood you!) It sounds like you're running mail service for a small group of people you know. Where you and those people are sending "handwritten" emails to people that the senders (mostly) know personally.
TFA is talking about transactional emails from their website that gets real traffic from the general public.
One of those things carries more deliverability challenges than the other.
adrien | 2 hours ago
To be fair, I'm unable to unsubscribe from three advertising lists from very real sellers (two also with physical shops) and of course, I have been subscribed to these lists without my consent.
I've also unsubscribed from some other lists but even over several years, not many more than three. This means that I'm unable to unsubscribe from a very large part of the lists I'm on and from the vast majority of marketing emails I get.
johnklos | 17 hours ago
In the old days, if someone sent spam, we'd write their network operator, and they'd get booted or otherwise punished. Once corporations and profit became involved, that stopped.
Like everything else in life, if we want something to stay nice, we need to punish those who want to make a profit from shitting on it.
mxuribe | 17 hours ago
For all of its issues, i think the flexibility and now longevity of email is still quite impressive! Though, at the same time, I'll be the first person to agree that I wish we could create a new email system from scratch - of course, with properly applied learnings from all the decades that have been learned from conventional email.
fanf | 16 hours ago
Spam email is not something that can be fixed by only technical means.
Most of the problem arises from the economic structure of the protocol, in the broad sense of security economics: i.e. who is participating in the protocol and what their relationships are, in terms of contracts, payments, or the lack thereof. There’s also a macroeconomic version of Conway’s law (which I think is due to David Clark) which observes that an open protocol defines an interface between businesses that operate that protocol as a service, so when you are designing a protocol you need to keep that in mind because it determines who ultimately bears the costs.
The difficulty with spam is due to some of the basic attributes of email:
It’s common for techies to want to fix spam by improving sender authentication (that’s the whole point of SPF and DKIM) but it’s still easy for spammers to mint new identities, so spam continues.
The upshot is that anti-spam is mostly about reputation management. Receivers keep track of sender reputations and use that as a major signal for classifying spam; and senders police their customers to avoid damaging their reputation. Senders for this purpose can be identified by IP address as well as domain name or email address. That basic structure has been the same since Paul Vixie introduced the RBL, years before SPF and DKIM came along.
Much of the cost of sending email is to pay for that policing effort. Some of the cost is also to discourage the bottom-feeding spammers who rely on nearly-free sending to cope with a very low response rate. But there’s a somewhat delicate balance on setting the cost of email: it isn’t possible to make it too expensive for spammers but still affordable for legitimate senders. (Observe the ratio of paper junk mail to legit post that comes through your letter box.)
quad | 15 hours ago
I lightly suspect that with the advent of AI bots and scrapers, we've finally brought this dynamic to HTTPS too.
mxuribe | 16 hours ago
I don't think spam is the only issue...but as it relates to spam: yep, i couldn't have said it better myself! :-)
fanf | 13 hours ago
I think basically all of the technical deficiencies of email can in principle be fixed by evolving the current system. I’ve ranted about this here before but the main reason for problems like shitty underspecified html is that the large email providers aren’t working with each other to improve things. (I suppose there are some things that have changed, such as DMARC and transport security, but they don’t have much positive effect on the way users perceive email.) And without a commitment to migrate the majority of email users to something better, all you have is yet another niche chat system, of which there have already been dozens.
Which is why it’s important to consider protocol economics when designing a replacement for email: there must be positive incentives in favour of adoption as well as negative incentives against abuse.
jcgl | 10 hours ago
I just want to say thank you for your email comments, especially the linked rant. I share many of the same thoughts but am not as grounded in the history of it all. Your words help flesh things out.
And it's also nice hearing people beat the drum of the centrality of economics problems in protocols. Also the drum that email-qua-suite-of-protocols is Mostly Fine™ (I put DNS in a similar camp).
mccd | 14 hours ago
Something I am curious about is if this is avoidable in any federated/decentralized social media?
I guess I am not being spammed on Activitypub, but is that because of the protocol or because it is just not as popular?
fanf | 14 hours ago
It’s a combination of a small community and the fact that it isn’t funded by advertisers. The people running instances have a fairly strong motivation to keep the network clean. Roughly what email was like before 1995ish. This can change for the worse if it’s easy enough to fake up a new server instance that the benefit of spamming outweighs the cost of playing defect, and if legit servers don’t have easy ways to vet the reputation of new instances.
rkaw92 | 17 hours ago
I tend to agree, too. It seems a no-brainer in hindsight to be able to do things like "pre-allow a given party to send me e-mail" (perhaps via revokable capability URLs)?
icefox | 14 hours ago
I would love a good way to revoke a sender's ability to send me email. There's a looooot of companies in the world where "unsubscribe" means "unsubscribe for the legally required 1 year or whatever, and then start spamming you again."
spc476 | 13 hours ago
Give each company a different email address and revoke when you no longer want their emails. Of course, this is easier if 1) you have your own domain, and 2) you run your own email server, or use an email service that allows unlimited email addresses per domain.
rkaw92 | 14 hours ago
I was thinking along these lines: instead of typing your e-mail address, your browser fills the field with a capability URL. This points indirectly to your inbox - kind of like mail forwarder services do today. Now that I think of it, it seems like it's achievable with no new protocols (with something like Firefox Relay), but service providers still treat temp-mail like a definite scam signal instead of a legitimate use case for normal people. Had this been the default since the early days, we might be able to enjoy revokable addresses, thanks to them being single-use only.
spc476 | 13 hours ago
I'm signing up for a new service on-line, and it wants to send me an email. How do I pre-authorize it? How do I even know what address the email is coming from (a complaint I have now)? Why get the web involved at all? (I know, I know, it's the "All Singing, All Dancing Protocol For Everything From Now Until The End Of Time", much like FTP was on the very early Arpanet).
Or, I'm reading a blog post from Alice, and I want to send her an email about it. How can she "pre-allow" my email? She doesn't even know I want to send her one.
insom | 10 hours ago
I don't use it (because DHH) but this is close to the idea of hey.com's email service. Literally nothing comes to your inbox unless you've allowed it. It means you end up with two inboxes -- in a way -- one to check in case there's something interesting in there, and the one where you know the emails that you want are in it. (Similar to GMail's priority inbox, but with less promise of magic).
I think this is a good model, perhaps even a better default model.
If I think of a real inbox -- I go and fetch the physical post, 80% is mass-mailed and goes straight into recycling and only letters from suppliers and government tend to end up in my actual in-tray. The idea that the postman could put stuff in your inbox is nuts!
tomsmeding | 16 hours ago
What are those learnings? I'd say the most important weakness of email is the spam problem; authentication is "solved" with TLS (non-TLS mailservers notwithstanding) and I personally don't consider encryption a very important core feature. So, spam: how to "fix" that without making email closed?
mxuribe | 16 hours ago
Learnings would include lots of stuff that apply elsewhere in tech, and tend to be something along the lines of trusting too much of other systems and devices back in the day, and yet nowadays knowing why we should not trust any other system or device on a network (either local or otherwise), and build up from there, etc. Now, as to your point about not considering encryption a very important core feature, i disagree...but that does not make you nor I correct or incorrect; merely that we differ in desired features...which i actually wish was easier to decide to turn on or off by a user. Because i know that we do indeed have encryption via email, but its implementation is not shall we say easy, etc.
franta | 9 hours ago
It is not just „old and goofy“. There is some purpose and thought behind it.
The conversational character of the SMTP protocol makes perfect sense. You will not send e-mail data if the recipient mailbox does not exist, or if the attachment is larger than given server accepts. Why send several megabytes and then realize that it is over the limit? Why send message in an unsupported encoding? Semantics of a single request-response does not fit everywhere. (or you can see SMTP as a sequence of several requests and responses coupled together in a session).
And those „randomly“ disappearing e-mails that can not even be found in the spam folder? When I used to be a postmaster, it was a matter of honor to deliver every message and not miss anything. The suspect spam was just flagged and it was up to the user configurable filters to put it in the appropriate folder. We were essentially peer to peer and each of us was interested in making sure it worked reliably. Today, it is different. There are several global corporations and they would prefer it if everyone else would just disappear. These corporations can not simply nuke smaller providers or suddenly stop accepting their messages, but can make them look unreliable in the eyes of users.
strugee | 3 hours ago
It is not, and treating it this way will get your emails dropped. Email is fundamentally a store-and-forward system, and the protocol requires that you be able to do exponential backoff for delivery. That requires a persistent queue, for which there is no built-in analogue in HTTP, and it also makes the operation fundamentally asynchronous. HTTP, by contrast, is full of synchronous use-cases; most endpoints in most RESTish APIs are synchronous.
At minimum, this will get your emails dropped if the destination server goes down e.g. for maintenance or an outage and relies on senders doing standards-compliant retries. But some providers will always drop your mail as spam, because of greylisting.
FRIGN | 15 hours ago
It's just a racket by large e-mail providers, both on the sending and receiving end.