email [deprecated]
September 5, 2022 10:26 AM Subscribe
Carlos Fenollosa has given up on self hosting email after 20 years. You might recognize Fenollosa from his handy list of Unix tricks. His argues that even emails from SDF don't work, noting that he's "positive that the beards of their admins are grayer than (his) and they will have tried to tweak every nook and cranny available."
Of course the SDF (previously) is older than the actual web and so old it refers to ARPANET emails.
Fenollosa's suggestions for the TLDR
Fenollosa's suggestions for the TLDR
- keep antispam measures
- exponential cooldown penalty instead of permeant blacklists
- don't blacklists whole IP blocks
- no blackholing
- recourse for legitimate servers
We started hosting email and web in 1998 (when we got our first DSL connection) in our house with my spouse as sysadmin. We gave up with our last move in 2018 for the same kind of reasons described in this article. Part of the reason we hung on so long is we run a PBEM (play by email) roleplaying game on an old school mailing list and those are super hard to find now.
Pretty much all our friends who used to run email servers locally for fun gave up years before we did. I feel for Fenollosa because I know for the people who have been desktop mail admins, it can be a huge part of their identity and there's definitely some grief involved in letting your local hosting go.
posted by gentlyepigrams at 10:46 AM on September 5, 2022 [5 favorites]
Pretty much all our friends who used to run email servers locally for fun gave up years before we did. I feel for Fenollosa because I know for the people who have been desktop mail admins, it can be a huge part of their identity and there's definitely some grief involved in letting your local hosting go.
posted by gentlyepigrams at 10:46 AM on September 5, 2022 [5 favorites]
Yeah, I too have let my own email serving systems expire. it seems like decentralized systems fall under the weight of abuse and exploitation of externalities, and the centralized systems survive thrive by promoting the abuse and exploitation, and then monetizing those views.
I keep thinking there's gotta be room for some private key decentralized messaging like Retroshare or that encrypted messaging system Brian Warner was working on back in the late nineties and early noughties, but without the political extremism there never seems to be enough uptake...
posted by straw at 10:50 AM on September 5, 2022 [3 favorites]
I keep thinking there's gotta be room for some private key decentralized messaging like Retroshare or that encrypted messaging system Brian Warner was working on back in the late nineties and early noughties, but without the political extremism there never seems to be enough uptake...
posted by straw at 10:50 AM on September 5, 2022 [3 favorites]
I hate email. The obligation it imposes to spend time try to sort through a endless stream of noise for the occasionally useful bit of information. You can never be done with it and missing the one or two messages that mattered is just another source of anxiety and frustration. We should just shut it down .
posted by interogative mood at 10:55 AM on September 5, 2022 [2 favorites]
posted by interogative mood at 10:55 AM on September 5, 2022 [2 favorites]
I am Gentlyepigram's spouse and I was really reluctant to give up self-hosting, but after 20 years, it was more work than joy.
I was (and were I still self-hosting, would continue to be) much harsher than Fenollosa was about what I banned or blocked, and never really cared about false positives. I had so many false negatives that I wasn't willing to relax my rules. Part of the reluctance I had was because I liked my rules. Part of the joy of moving to pro hosting is I can say to a support guy "set up SPF and DKIM for me" and he tells me "OK, I did it. Here's the magic..." (aside: Literally, the support dude said 'Here's the magic...' and went on to explain. He has the joy part going for him, and more power to him...).
And the technical detail that was the most fun about all of it was that we started self-hosting on Apple PowerPC 6100s with OS 9, back before it became Unix. We were ... not unique, but at least an odd corner of the mail-hosting universe.
posted by Mad_Carew at 11:00 AM on September 5, 2022 [14 favorites]
I was (and were I still self-hosting, would continue to be) much harsher than Fenollosa was about what I banned or blocked, and never really cared about false positives. I had so many false negatives that I wasn't willing to relax my rules. Part of the reluctance I had was because I liked my rules. Part of the joy of moving to pro hosting is I can say to a support guy "set up SPF and DKIM for me" and he tells me "OK, I did it. Here's the magic..." (aside: Literally, the support dude said 'Here's the magic...' and went on to explain. He has the joy part going for him, and more power to him...).
And the technical detail that was the most fun about all of it was that we started self-hosting on Apple PowerPC 6100s with OS 9, back before it became Unix. We were ... not unique, but at least an odd corner of the mail-hosting universe.
posted by Mad_Carew at 11:00 AM on September 5, 2022 [14 favorites]
I still self host a domain because I already have it set up, but whenever there's an email hosting question on Ask I always tell people not to self host. Deliverability is just too hard now. You can do everything right and still end up blocked outright or silently filtered. There was recently an ominous note from my VPS host saying SMTP would be disabled for all new accounts going forward, and indicating that they might turn it off for existing accounts at some point. I'm not sure I'd be sad if they blocked SMTP entirely, because it feels like a relic of another time. TBH I'm also a relic of another time and I actually like email, but I can see which way the wind blows.
posted by fedward at 11:03 AM on September 5, 2022 [6 favorites]
posted by fedward at 11:03 AM on September 5, 2022 [6 favorites]
I'm still self-hosting, postfix on CentOS 7 on a Linode VPS, for myself and a family member.
Both ends of the spam problem are an issue. But even without that, it's just gotten tedious. Twenty years ago when I set it up, it was new and shiny and interesting. Now it's just a boring utility, and I don't really remember how it works, so when it breaks I dread having to go back and dig through the documentation and my old notes and figure it out from scratch again.
So next time it needs a big upgrade or hits a major snag, I'll probably migrate elsewhere. (Recommendations? I'd like the option of keeping existing clients, so it probably needs reasonable imap support; last I heard, that ruled out gmail.)
I don't know what to think about what it all means for the market as a whole. It doesn't feel like a hill to die on. If I did feel like it was worth a fight--surely there are approaches with better impact/time ratio than maintaining my own mail server.
posted by bfields at 11:23 AM on September 5, 2022 [1 favorite]
Both ends of the spam problem are an issue. But even without that, it's just gotten tedious. Twenty years ago when I set it up, it was new and shiny and interesting. Now it's just a boring utility, and I don't really remember how it works, so when it breaks I dread having to go back and dig through the documentation and my old notes and figure it out from scratch again.
So next time it needs a big upgrade or hits a major snag, I'll probably migrate elsewhere. (Recommendations? I'd like the option of keeping existing clients, so it probably needs reasonable imap support; last I heard, that ruled out gmail.)
I don't know what to think about what it all means for the market as a whole. It doesn't feel like a hill to die on. If I did feel like it was worth a fight--surely there are approaches with better impact/time ratio than maintaining my own mail server.
posted by bfields at 11:23 AM on September 5, 2022 [1 favorite]
Hot take but you know it's true:
Email = Gmail
posted by alex_skazat at 11:30 AM on September 5, 2022 [3 favorites]
Email = Gmail
posted by alex_skazat at 11:30 AM on September 5, 2022 [3 favorites]
jwz tweeted something about some pain he's been having tryign to get his shit through SPF. If it's that bad on SDF, yeah, we must be doomed :(
posted by symbioid at 11:31 AM on September 5, 2022 [1 favorite]
posted by symbioid at 11:31 AM on September 5, 2022 [1 favorite]
I've never hosted email but looked into it a few years ago and decided against trying. It's just too fundamentally broken.
I miss the early days of the Internet before the word monetization ruined everything.
posted by Ickster at 11:33 AM on September 5, 2022 [6 favorites]
I miss the early days of the Internet before the word monetization ruined everything.
posted by Ickster at 11:33 AM on September 5, 2022 [6 favorites]
i had a automotive body shop client that was locally hosting* its own Exchange server right up until the moment they got bought out in 2020. that hardware is now sitting in my living room hosting my trueNAS install. no telling how hosed-up, goat-roped, exploited and compromised that box was, hence why i just completely blew it into the void once i decommissioned the hardware. i was never tasked with working on the Exchange part of it and it was so insane i just left it exactly as it was. like, we gotta pick our battles, ya know?
* if by "locally hosting" you mean "someone set it up years ago and it's just been left running un-administrated since then"
posted by glonous keming at 11:49 AM on September 5, 2022 [3 favorites]
* if by "locally hosting" you mean "someone set it up years ago and it's just been left running un-administrated since then"
posted by glonous keming at 11:49 AM on September 5, 2022 [3 favorites]
I take a hybrid approach. I have a cloud VM that runs CentOS which I host my own domain on with a few websites and SMTP mail, but all my mail to my address gets forwarded under the covers to Gmail for me, and other mail goes to Hotmail for one of my family members and yet another gets forwarded to their ISP email. That way I still retain control and ownership of the addresses, but can leverage Gmail (or Hotmail) for spam/phishing control and the webmail interface.
However, even that's getting harder to manage as there are challenges with properly forwarding mail and having it arrive at Gmail from my server instead of the original sender without Gmail canning it for spam due to the header mismatch.
posted by barc0001 at 11:58 AM on September 5, 2022 [1 favorite]
However, even that's getting harder to manage as there are challenges with properly forwarding mail and having it arrive at Gmail from my server instead of the original sender without Gmail canning it for spam due to the header mismatch.
posted by barc0001 at 11:58 AM on September 5, 2022 [1 favorite]
For people who are giving up on the self-hosting thing, I'd suggest giving fastmail or pobox (which is powered by fastmail) a look. I've been with pobox for years now and they've been very solid. Support for BYO domain, good spam filtering, good rules, good web UI, support for caldav/carddav so you can sync with your phone. And they're big enough that they have the resources to manage all the nuances of the ecosystem. I don't know if Fenollosa would consider them part of Big Email or not, but at least they're not Google.
posted by microscone at 12:03 PM on September 5, 2022 [13 favorites]
posted by microscone at 12:03 PM on September 5, 2022 [13 favorites]
I can recommend our outsourced provider. We went with Rackspace. The web hosting is ridiculous to the point of "no, thanks", but email for a family and our friends is pretty cheap and they often run specials for the first year for peanuts.
POP3 and IMAP, distribution lists, filtering, webmail, and gleefully technical support people on the other end of chat. They've been solid for the last 5 years for us.
posted by Mad_Carew at 12:14 PM on September 5, 2022 [1 favorite]
POP3 and IMAP, distribution lists, filtering, webmail, and gleefully technical support people on the other end of chat. They've been solid for the last 5 years for us.
posted by Mad_Carew at 12:14 PM on September 5, 2022 [1 favorite]
Having grown up in the era of sendmail, I was a little surprised last year to find that automating emails using self-hosted mechanisms on linux is almost impossible now. I thought with containerized services it was going to be a snap to set up! I ended up getting a sendgrid account and using their API, which was still trouble to get actually delivered.
posted by a robot made out of meat at 12:20 PM on September 5, 2022 [3 favorites]
posted by a robot made out of meat at 12:20 PM on September 5, 2022 [3 favorites]
I can recommend our outsourced provider. We went with Rackspace. The web hosting is ridiculous to the point of "no, thanks", but email for a family and our friends is pretty cheap and they often run specials for the first year for peanuts.
POP3 and IMAP, distribution lists, filtering, webmail, and gleefully technical support people on the other end of chat. They've been solid for the last 5 years for us.
16 years for us. Nary a complaint from me in all that time.
posted by BlueDuke at 12:48 PM on September 5, 2022 [2 favorites]
POP3 and IMAP, distribution lists, filtering, webmail, and gleefully technical support people on the other end of chat. They've been solid for the last 5 years for us.
16 years for us. Nary a complaint from me in all that time.
posted by BlueDuke at 12:48 PM on September 5, 2022 [2 favorites]
Hot take but you know it's true:
Email = Gmail
You may think so but I have a friend (about 10 years older and not very tech-savvy) who actually told me when I tried to get her email address for something (and very seriously) “I don’t have email. I have Gmail.” I’m still not sure she realizes the relationship between the two.
posted by TedW at 1:03 PM on September 5, 2022 [1 favorite]
Email = Gmail
You may think so but I have a friend (about 10 years older and not very tech-savvy) who actually told me when I tried to get her email address for something (and very seriously) “I don’t have email. I have Gmail.” I’m still not sure she realizes the relationship between the two.
posted by TedW at 1:03 PM on September 5, 2022 [1 favorite]
I miss the early days of the Internet before the word monetization ruined everything.
Monetization was a symptom, not the original sicknesses that birthed it: comfort and privilege. The early internet was born with no mechanisms for defence and accountability, because those existed elsewhere: university admissions, the HR department, tenure committees, and so forth. The original protocols were born in a world that didn’t understand why those things would need to exist, in its quiet little utopia
posted by mhoye at 1:06 PM on September 5, 2022 [18 favorites]
Monetization was a symptom, not the original sicknesses that birthed it: comfort and privilege. The early internet was born with no mechanisms for defence and accountability, because those existed elsewhere: university admissions, the HR department, tenure committees, and so forth. The original protocols were born in a world that didn’t understand why those things would need to exist, in its quiet little utopia
posted by mhoye at 1:06 PM on September 5, 2022 [18 favorites]
Well well,
I'm too maintaining my own smtp, I've seen similar problems emerging as well. My own try to fix (some) problems is http://katiska.org/classified-ads/ that lets you do exactly same things that SMTP allows but additionally
* Is distributed in P2P manner
* Has no central authority you need to ask permission about usage
* Has more privacy and data protection compared to smtp
* Is rather difficult to turn into business that makes users the actual product
* Has some additional features like public discussion forums, sharing of small files, distributed general purpose database
But it yet does not address well the nomimal problem called spam. I call it nominal because it is real problem for everyone, nomimal because big players use it as an excuse to blacklist small players (my smtp server usually serves around 5 users per day). Classified ads has made small effort in this direction, called trust-lists but my concern is that they just create echo-chambers and help in this area would really be needed, along android UI design + implementation as I have the back-end already running in android phone but UI needs complete re-write for that environment.
Don't know. SMTP RFC 821 is from year 1982. We could move on but it requires some kind of group movement to make any difference. My own feeling about most basic tools in computing (operating systems, compilers, text+image processing, guitar tabulature sw and all communications) need to be of good quality and available for everyone free of charge. Commercial companies don't fit into this picture very well.
--
Antti, profile id 6F69B225D632FE1566C5CA5351E59A8B65C64B0E in classified ads
posted by costello at 1:17 PM on September 5, 2022
I'm too maintaining my own smtp, I've seen similar problems emerging as well. My own try to fix (some) problems is http://katiska.org/classified-ads/ that lets you do exactly same things that SMTP allows but additionally
* Is distributed in P2P manner
* Has no central authority you need to ask permission about usage
* Has more privacy and data protection compared to smtp
* Is rather difficult to turn into business that makes users the actual product
* Has some additional features like public discussion forums, sharing of small files, distributed general purpose database
But it yet does not address well the nomimal problem called spam. I call it nominal because it is real problem for everyone, nomimal because big players use it as an excuse to blacklist small players (my smtp server usually serves around 5 users per day). Classified ads has made small effort in this direction, called trust-lists but my concern is that they just create echo-chambers and help in this area would really be needed, along android UI design + implementation as I have the back-end already running in android phone but UI needs complete re-write for that environment.
Don't know. SMTP RFC 821 is from year 1982. We could move on but it requires some kind of group movement to make any difference. My own feeling about most basic tools in computing (operating systems, compilers, text+image processing, guitar tabulature sw and all communications) need to be of good quality and available for everyone free of charge. Commercial companies don't fit into this picture very well.
--
Antti, profile id 6F69B225D632FE1566C5CA5351E59A8B65C64B0E in classified ads
posted by costello at 1:17 PM on September 5, 2022
jwz tweeted something about some pain he's been having tryign to get his shit through SPF. If it's that bad on SDF, yeah, we must be doomed :(
Yeah, I was also going to mention this, the comment section on his blog post about this is fascinating. (And definitely removed whatever latent desire I might have had to do any sort of email self-hosting..)
posted by advil at 1:25 PM on September 5, 2022
Yeah, I was also going to mention this, the comment section on his blog post about this is fascinating. (And definitely removed whatever latent desire I might have had to do any sort of email self-hosting..)
posted by advil at 1:25 PM on September 5, 2022
Costello, classified ads sounds cool but it is one of the least intuitive/most misleading names for a project I've ever heard. After reading your comment and visiting the site my brain still can't quite get past the idea that it's doesn't have anything to do with, well, classified advertising.
~
With my economics hat on the problem is pretty straightforward: "free" isn't really free, because bad actors abuse the system and impose costs on everyone. The solution a bit less so, of course. Systems that get the costs right (e.g. charging per email for unknown senders, with refunds for legit emails; you could use real or fake money) eliminate the problem, they just need to be built and adopted and maintained and all that other stuff.
The fediverse is the obvious other way forward here, though I don't get the impression the overlap between mail self hosters and fediverse self hosters is as big as you might think.
posted by ropeladder at 1:42 PM on September 5, 2022
~
With my economics hat on the problem is pretty straightforward: "free" isn't really free, because bad actors abuse the system and impose costs on everyone. The solution a bit less so, of course. Systems that get the costs right (e.g. charging per email for unknown senders, with refunds for legit emails; you could use real or fake money) eliminate the problem, they just need to be built and adopted and maintained and all that other stuff.
The fediverse is the obvious other way forward here, though I don't get the impression the overlap between mail self hosters and fediverse self hosters is as big as you might think.
posted by ropeladder at 1:42 PM on September 5, 2022
Hosting your own incoming email with Mail-in-a-Box is pretty straightforward. You can even encrypt messages as they come in if you need to temporarily store them in the cloud before downloading them to a secure device. Pointing your MX records to a Big Email Provider should not be necessary.
Outgoing email is the problem, as Fenollosa explains. The cartel is forcing small players out of the market by making it more and more difficult for independents to get through to customers of the cartel. Last month I had to tweak my reverse DNS and AAAA records to match what GOOG decided they should be. This month it will be something different, with no warning. The deliberate, conscious intent is to data-mine private communications.
The legitimate antispam measure Fenollosa does not mention is to whitelist signed, encrypted messages. The processing cost to encrypt each outgoing message with the recipient's certificate or public key is minimal for legitimate users and prohibitive for spammers. If the cartel was trying to get rid of spam, they could offer this recourse instead of obligating small-volume senders to file a form to beg off a blacklist and wait a few days to see if it worked. That's not what the cartel wants; they want to end privacy. Normalizing encryption is contrary to their goals. The necrosis of distributed email is one symptom of this systemic disorder.
posted by backwoods at 1:58 PM on September 5, 2022 [7 favorites]
Outgoing email is the problem, as Fenollosa explains. The cartel is forcing small players out of the market by making it more and more difficult for independents to get through to customers of the cartel. Last month I had to tweak my reverse DNS and AAAA records to match what GOOG decided they should be. This month it will be something different, with no warning. The deliberate, conscious intent is to data-mine private communications.
The legitimate antispam measure Fenollosa does not mention is to whitelist signed, encrypted messages. The processing cost to encrypt each outgoing message with the recipient's certificate or public key is minimal for legitimate users and prohibitive for spammers. If the cartel was trying to get rid of spam, they could offer this recourse instead of obligating small-volume senders to file a form to beg off a blacklist and wait a few days to see if it worked. That's not what the cartel wants; they want to end privacy. Normalizing encryption is contrary to their goals. The necrosis of distributed email is one symptom of this systemic disorder.
posted by backwoods at 1:58 PM on September 5, 2022 [7 favorites]
Anyone else fondly remember compiling Dan Bernstein's Qmail? Good times.
posted by mikelieman at 3:15 PM on September 5, 2022 [3 favorites]
posted by mikelieman at 3:15 PM on September 5, 2022 [3 favorites]
Hell, I ran qmail (and djbdns) until relatively recently!
posted by spacewrench at 3:16 PM on September 5, 2022 [4 favorites]
posted by spacewrench at 3:16 PM on September 5, 2022 [4 favorites]
I ran qmail for years but I wouldn’t say I was fond of compiling it. I had to learn Postfix for a job, and it was so much less persnickety that the next time I migrated my own domains to new hardware I also gave up qmail.
posted by fedward at 3:39 PM on September 5, 2022 [1 favorite]
posted by fedward at 3:39 PM on September 5, 2022 [1 favorite]
Well, for what extremely little it's worth, my emails from SDF are being received by gmail/&c just fine...
... for now. I have no faith whatsoever that this will continue indefinitely, and I suspect that my only indication will be the hollow silence of replies.
posted by Westringia F. at 3:57 PM on September 5, 2022 [1 favorite]
... for now. I have no faith whatsoever that this will continue indefinitely, and I suspect that my only indication will be the hollow silence of replies.
posted by Westringia F. at 3:57 PM on September 5, 2022 [1 favorite]
I totally feel this, as the author (and unintentional admin) of a Mailman-derived system.
This started back when Mailman 2 was widely installed and Mailman 3 was a set of (barely documented) launchpad.net pages. But since Mailman 2 was clearly headed for the dustbin (and some since-abandoned business priorities required features not directly implemented) we built a bunch of wrappers around the main functional pieces and have been running that system for many years now.
And from that perspective the security changes have been devastating. IIRC, to forward an email from an external user using DMARC, in theory you need to be able to produce an ARC-certified header, which requires a bunch of custom DNS headers and related header injection, but in practice what you actually need is something certified by a known DNS entity, at which point anyone just running a list on the side says "fml why did I agree to support this?" and starts working on migrating everything to Google Groups.
I think of it as, HTTPS+ represents a bunch of shared open standards for communication over TCP, but no open standards based on SMTP ever took hold -- basically it was always "wild west" and what Google/etc did was the first practical (if completely unstandardized) codification ever. So. Fair enough. And of course it benefited them, because T***p was president and they really wanted to shake that "do no evil" motto.
Huh, apparently I still have some feelings about this.
posted by bjrubble at 5:44 PM on September 5, 2022 [2 favorites]
This started back when Mailman 2 was widely installed and Mailman 3 was a set of (barely documented) launchpad.net pages. But since Mailman 2 was clearly headed for the dustbin (and some since-abandoned business priorities required features not directly implemented) we built a bunch of wrappers around the main functional pieces and have been running that system for many years now.
And from that perspective the security changes have been devastating. IIRC, to forward an email from an external user using DMARC, in theory you need to be able to produce an ARC-certified header, which requires a bunch of custom DNS headers and related header injection, but in practice what you actually need is something certified by a known DNS entity, at which point anyone just running a list on the side says "fml why did I agree to support this?" and starts working on migrating everything to Google Groups.
I think of it as, HTTPS+ represents a bunch of shared open standards for communication over TCP, but no open standards based on SMTP ever took hold -- basically it was always "wild west" and what Google/etc did was the first practical (if completely unstandardized) codification ever. So. Fair enough. And of course it benefited them, because T***p was president and they really wanted to shake that "do no evil" motto.
Huh, apparently I still have some feelings about this.
posted by bjrubble at 5:44 PM on September 5, 2022 [2 favorites]
I've been on SDF for almost two decades. I use their email as a .forward service and as a filter to my gmail account..
posted by jim in austin at 5:49 PM on September 5, 2022 [2 favorites]
posted by jim in austin at 5:49 PM on September 5, 2022 [2 favorites]
Mad_Carew: ...we started self-hosting on Apple PowerPC 6100s...
I would love to hear more about this whoooole story!
posted by wenestvedt at 6:51 PM on September 5, 2022 [2 favorites]
I would love to hear more about this whoooole story!
posted by wenestvedt at 6:51 PM on September 5, 2022 [2 favorites]
If you're trying to run a mailing list, pne good option is groups.io. The free tier is limited but not bad for small groups, but even at $20/mo it becomes very capable.
posted by Nelson at 7:50 PM on September 5, 2022 [3 favorites]
posted by Nelson at 7:50 PM on September 5, 2022 [3 favorites]
It’s been a long time since I have deal with managing email servers, but one thing I think helped derail things was an over-reliance on spam filtering and not enough spam blocking. As a legitimate sender, I would rather that if it’s problematic, my email get rejected outright instead of being shunted off to somewhere most end users never look.
posted by jimw at 7:57 PM on September 5, 2022 [1 favorite]
posted by jimw at 7:57 PM on September 5, 2022 [1 favorite]
I'm currently self-hosting a mail server, for a variety of reasons. I don't seem to have a lot of these issues...
...but I've realized that, over the last many years, I simply don't send that much email. I get a ton, and, so far, I don't seem to have issues receiving it. This includes order info from eCommerce, stuff from my bank, etc. Scrolling through my
(I have a gmail address for when I was looking for a new job, but I've had it since before I set up the server, and using a gmail address is a recommendation I've seen on job hunt sites.)
For good or for ill, most of the communication flow seems to be on other channels (social media, text messages, etc.).
I'm excluding work, where i basically live in email, but that's also mostly within my company in their system.
posted by MrGuilt at 9:00 PM on September 5, 2022 [1 favorite]
...but I've realized that, over the last many years, I simply don't send that much email. I get a ton, and, so far, I don't seem to have issues receiving it. This includes order info from eCommerce, stuff from my bank, etc. Scrolling through my
Sent Folder, I don't seem to have too many threads that I started--just a few where I reply to people. This is the email box I default too for most things.(I have a gmail address for when I was looking for a new job, but I've had it since before I set up the server, and using a gmail address is a recommendation I've seen on job hunt sites.)
For good or for ill, most of the communication flow seems to be on other channels (social media, text messages, etc.).
I'm excluding work, where i basically live in email, but that's also mostly within my company in their system.
posted by MrGuilt at 9:00 PM on September 5, 2022 [1 favorite]
I love how self-hosted email gets the entire IP block insta-banned if there is one suspicious message, but I can receive dozens of “buy-dick-pills@gmail.com” emails daily and it’s totally not an issue
I’ve been Gmail since 2004. It’s fine I guess as a basic email delivery system. But it’s getting progressively worse. I don’t use the web interface for my personal mail because I detest it with the burning fire of 1,000 suns. My Gmail-hosted university account was force-moved onto Gmail web interface for privacy reasons, no more option to use an external client. It’s a hot mess. Mail filters are essentially impossible if you want anything more complex than “tag email from this specific person with this specific tag”. There are hidden rules that just cause filters to fail if they are too long, but there’s no info when setting up the filters to tell you this. My inbox is full of crap all the time. I can’t find important mail because I can’t filter out the crap.
Even on my personal mail, where I can use real filters, it’s bad. Spam is out of control. Gmail helpfully tells me I can just “not look at my spam folder” but it also throws legitimate messages in there, and lets truly obvious garbage into my inbox. How the company can magically remove people from photos yet be unable to recognize scammy garbage email images is beyond me. If I could simply block senders, it would help. But Gmail doesn’t block senders, it just lets me mark them as spam. It’s ridiculous. They don’t even need to filter out spam, they could let users do that for them if they implemented a crowdsourced blocklist. Christ, put recaptcha to use - stop asking me to find cars in a photo, start asking me to mark the obvious spam in a set of 4 blocks of text.
This is our alternative now. Secure, sane, curated, self-hosted email is blocked, but the poorly-monitored garbagefire that is the source of 99% of the spam to begin with gets a pass, and we have no tools to make it better. “abuse@google.com” is a black hole, don’t bother reporting anything.
posted by caution live frogs at 5:40 AM on September 6, 2022 [4 favorites]
I’ve been Gmail since 2004. It’s fine I guess as a basic email delivery system. But it’s getting progressively worse. I don’t use the web interface for my personal mail because I detest it with the burning fire of 1,000 suns. My Gmail-hosted university account was force-moved onto Gmail web interface for privacy reasons, no more option to use an external client. It’s a hot mess. Mail filters are essentially impossible if you want anything more complex than “tag email from this specific person with this specific tag”. There are hidden rules that just cause filters to fail if they are too long, but there’s no info when setting up the filters to tell you this. My inbox is full of crap all the time. I can’t find important mail because I can’t filter out the crap.
Even on my personal mail, where I can use real filters, it’s bad. Spam is out of control. Gmail helpfully tells me I can just “not look at my spam folder” but it also throws legitimate messages in there, and lets truly obvious garbage into my inbox. How the company can magically remove people from photos yet be unable to recognize scammy garbage email images is beyond me. If I could simply block senders, it would help. But Gmail doesn’t block senders, it just lets me mark them as spam. It’s ridiculous. They don’t even need to filter out spam, they could let users do that for them if they implemented a crowdsourced blocklist. Christ, put recaptcha to use - stop asking me to find cars in a photo, start asking me to mark the obvious spam in a set of 4 blocks of text.
This is our alternative now. Secure, sane, curated, self-hosted email is blocked, but the poorly-monitored garbagefire that is the source of 99% of the spam to begin with gets a pass, and we have no tools to make it better. “abuse@google.com” is a black hole, don’t bother reporting anything.
posted by caution live frogs at 5:40 AM on September 6, 2022 [4 favorites]
Jeez. I've had my personal domain for over 30 years, was responsible for supporting the IP stack and related applications including the mail server for an OS vendor, and I couldn't get off of self-hosting my domain fast enough. It hasn't been worth the hassle for a long time.
posted by Runes at 6:22 AM on September 6, 2022 [2 favorites]
posted by Runes at 6:22 AM on September 6, 2022 [2 favorites]
I just passed 20 years self hosting web and email, and it has been a journey.
What I think mattered most:
*Keep my nose clean (small server with static web and postfix) always up to date on security.
*Static IP address I have had forever (an early adopter of Linode)
*Small steady correspondence with people on all the big exchanges, which weights my domain positively.
*When I have fallen under blanket IP blocks, the staff at Linode have the weight to get in touch with the bigger providers and get me unblocked.
posted by nickggully at 6:58 AM on September 6, 2022 [1 favorite]
What I think mattered most:
*Keep my nose clean (small server with static web and postfix) always up to date on security.
*Static IP address I have had forever (an early adopter of Linode)
*Small steady correspondence with people on all the big exchanges, which weights my domain positively.
*When I have fallen under blanket IP blocks, the staff at Linode have the weight to get in touch with the bigger providers and get me unblocked.
posted by nickggully at 6:58 AM on September 6, 2022 [1 favorite]
Spam used to be a big problem for me, even with Spamassasin and bulk email identification like Pyzor and its ilk. Then I turned on greylisting and the vast majority just went away. The other thing that worked was when one of my MXes went down for a while and I didn't notice. Apparently, spammers just don't retry.
The last 24 hours was a relative nightmare of spam. Two hit my inbox.
posted by wierdo at 7:35 AM on September 6, 2022 [1 favorite]
The last 24 hours was a relative nightmare of spam. Two hit my inbox.
posted by wierdo at 7:35 AM on September 6, 2022 [1 favorite]
The trouble I've had with greylisting is the delay it introduces.
A typical case is signing up for a new account with a system that requires email confirmation. I've often got a long wait for that first "click on this link to confirm your new account" mail.
Maybe I was doing it wrong, I don't know. It certainly did make a big difference to spam levels.
posted by bfields at 7:50 AM on September 6, 2022 [1 favorite]
A typical case is signing up for a new account with a system that requires email confirmation. I've often got a long wait for that first "click on this link to confirm your new account" mail.
Maybe I was doing it wrong, I don't know. It certainly did make a big difference to spam levels.
posted by bfields at 7:50 AM on September 6, 2022 [1 favorite]
Some delay is inevitable with greylisting when receiving mail from an unknown server, yes. Thankfully most senders will do their initial retry within 5-10 minutes. If something happens to make it fail twice it can get excessive, though.
posted by wierdo at 8:23 AM on September 6, 2022 [1 favorite]
posted by wierdo at 8:23 AM on September 6, 2022 [1 favorite]
Kind of like privacy, nobody cares about protecting it until you need it.
posted by filtergik at 10:11 AM on September 6, 2022 [1 favorite]
posted by filtergik at 10:11 AM on September 6, 2022 [1 favorite]
On a lighter note, community hosted servers might gain more of a foot hold. Let it model municipal reach, funding, and governance. Not Joe Alphabet Megacorp, "don't call us we don't even have a number."
posted by filtergik at 10:45 AM on September 6, 2022 [1 favorite]
posted by filtergik at 10:45 AM on September 6, 2022 [1 favorite]
I switched to HEY pretty much right when it launched, and it's astonishing how much nicer it's made email feel. Even spam has become virtually insignificant. Its design is polarizing, so you're as likely to hate it as love it, but I fall firmly in the "love" territory.
(Disclaimer/warning: the people who build HEY have a history of putting their foots in their mouths where politics is concerned—nothing alt-right but a little too much "Harper's letter" for my liking. Worth mentioning out the gate, in case your idea of ethical consumption precludes giving money to an annoying crank with a newsletter.)
posted by Tom Hanks Cannot Be Trusted at 11:52 AM on September 6, 2022 [2 favorites]
(Disclaimer/warning: the people who build HEY have a history of putting their foots in their mouths where politics is concerned—nothing alt-right but a little too much "Harper's letter" for my liking. Worth mentioning out the gate, in case your idea of ethical consumption precludes giving money to an annoying crank with a newsletter.)
posted by Tom Hanks Cannot Be Trusted at 11:52 AM on September 6, 2022 [2 favorites]
I self-hosted my website and email for a couple of years in the early 2000s. I had a computer sitting in my closet at home that was my server running NetBSD I think. At some point while I was living abroad my ISP decided to block port 25 and I decided to transition everything to hosting services before everything got blocked. Probably all for the best because going to an internet cafe to perform software updates probably wasn't the best use of time or money.
I was able to move that email to a free gmail for my domain service that they had back then and is still barely around. There was a scare earlier this summer when they were going to move everyone with a similar legacy account to Google Workspace but thankfully they listened to the outcry from the couple of people still using it and kept the service running as long as its for non-commercial purposes.
posted by any portmanteau in a storm at 12:17 PM on September 6, 2022 [1 favorite]
I was able to move that email to a free gmail for my domain service that they had back then and is still barely around. There was a scare earlier this summer when they were going to move everyone with a similar legacy account to Google Workspace but thankfully they listened to the outcry from the couple of people still using it and kept the service running as long as its for non-commercial purposes.
posted by any portmanteau in a storm at 12:17 PM on September 6, 2022 [1 favorite]
Hot take but you know it's true:
Email = Gmail
Didn't Google try to wall off Gmail at one point not too long ago, so that basic email and competitors would no longer be able to interoperate (or find it prohibitively difficult to do so), only backing away from the plan after it was leaked early to strong criticism?
posted by They sucked his brains out! at 4:59 PM on September 6, 2022
Email = Gmail
Didn't Google try to wall off Gmail at one point not too long ago, so that basic email and competitors would no longer be able to interoperate (or find it prohibitively difficult to do so), only backing away from the plan after it was leaked early to strong criticism?
posted by They sucked his brains out! at 4:59 PM on September 6, 2022
An accusation like that about Gmail really needs a citation.
posted by Nelson at 5:05 PM on September 6, 2022 [4 favorites]
posted by Nelson at 5:05 PM on September 6, 2022 [4 favorites]
I guess I still self-host on a few boxes or VPSes in a literal sense; but only minimal instances on server appliances, that do nothing but send notifications and etc. to another address under my control.
posted by snuffleupagus at 6:44 PM on September 6, 2022
posted by snuffleupagus at 6:44 PM on September 6, 2022
I have nothing but the greatest respect for jwz and have been on SDF for longer than I'd care to admit (and as I said to someone earlier today on Mastodon, SDF's one-time membership fee tier is the best deal going on the Internet)... but I am not seeing the sort of problems he's apparently seeing. And I both run an email server for a small organization, and send email from my SDF account all the time.
Yes, email is a pain in the ass. It's not a fire-and-forget service that you can just spin up and walk away from. I don't blame someone if "running a mail server like it's 1996" isn't their idea of a good time. (I'm just a masochist, I guess.)
Anyway: the problem jwz seems to be having is that he's doing something that modern email systems really don't like or approve of, because it's indistinguishable from spoofing. Sure, he's not spoofing or doing anything wrong, but it's the equivalent of walking into a bank with a ski mask on and wondering why you get such awful customer service.
The originator of the messages has said, very specifically: "don't forward our messages and make it look like it's still from us." So stop doing that. You just can't forward emails all over the place like that anymore. And that wasn't a decision Google made. Paul Vixie, who basically reports to nobody less than God, and even then probably only takes it as an advisement, was responsible for it, or at least popularizing the idea, back in circa 2002. SPF is older than Gmail. And it's in creating SPF that the decision was made—again, by the very pre-Gmail Internet community—that "uh, maybe letting anyone transmit email that appears to originate from someone else is a bad idea".
He's literally had 20 years to stop doing this and come up with a better solution, and the Internet, true to Postel's Law, was kind enough to let him keep getting away with it for all that time.
He even touches on some of the possible solutions in the post about the problem!
And a maintenance headache? Isn't that what he has right now, because he's basically sending out spoofed emails? Bite the bullet and get it over with, it's the right solution. If you want to provide people email addresses for incoming mail, provide the backend infrastructure to receive those messages and hold them for delivery to the recipient. That's the right way to do it.
But okay, maybe he's running this on, I dunno, a VAX in his basement, and those DEC DSSI drives are getting hard to come by. Fine, whatever. But he comes up with another viable solution!
This is the solution that I use for the mailinglist that I operate, because it's the solution that the much-smarter-than-me people who develop Mailman came up with to this very problem. Mailinglists (or listservs, or reflectors, depending on which corner of the Internet you spent your formative years in), in their traditional form, look just like spoofing too. And SPF looked for a while like it would be death for mailinglists. And there was much wailing and gnashing of teeth, and several solutions were devised.
The simple solution is called "From munging" in the mailinglist world, and it creates an outgoing message that looks like this:
To: bob@bigco.com
From: Anne Person via SomeIntermediateList
That works, because it's honest: it's not claiming (via the "From" line) that the message is from someone who it's not. The message is being sent from list@intermediate.co, to Bob, and we're telling Bob "if you want to reply to this message, send it directly back to Anne, here's her address".
Of course, there are more elegant ways to do this, courtesy of MIME, which is a hot new standard from... 1996. MIME specifies ways of wrapping one email message inside another email message, and if your mail program isn't totally stupid, it should be able to unwrap them for you—although there aren't any guarantees about how it's going to display the message in your Inbox, for the purposes of sorting it, etc. But IMO, this is the Technically Correct (the best kind of correct!) way to handle any sort of email-retransmission scenario where you're passing along a message that you didn't actually originate, be it mailinglist messages, forwards, bouncebacks, etc., and if we're going to pick a hill to die on, it's this: MUAs (mail client programs) should deal with MIME-encapsulated messages in a not-stupid way, e.g. by allowing the user to "bust" the outer wrapping and extract the inner message upon receipt. Because this is the only elegant way that you can do mailinglists and forwarding with encrypted or signed messages and not screw them all up.
Yes, this takes more than setting up a .forward file or a Postfix alias. Which sucks. But again, being able to forward email around like that without consequence was determined to be a bad idea somewhat before invading Iraq was.
posted by Kadin2048 at 2:58 AM on September 7, 2022 [15 favorites]
Yes, email is a pain in the ass. It's not a fire-and-forget service that you can just spin up and walk away from. I don't blame someone if "running a mail server like it's 1996" isn't their idea of a good time. (I'm just a masochist, I guess.)
Anyway: the problem jwz seems to be having is that he's doing something that modern email systems really don't like or approve of, because it's indistinguishable from spoofing. Sure, he's not spoofing or doing anything wrong, but it's the equivalent of walking into a bank with a ski mask on and wondering why you get such awful customer service.
To recap, my domain hosts its own SMTP server running Postfix, and /etc/postfix/virtual contains a bunch of entries forwarding "employee_name@dnalounge.com" to whatever their actual email address is, usually gmail.I know we all love to complain about Google, but Google really isn't the big bad here. Google is doing exactly what the owners of the email's originating domain have asked for, which is basically: "if you see an email that claims it's from us, and it's not coming from one of our mailservers that we control, at one of our IP addresses, it ain't from us—shitcan it". And so Google is doing that.
This has been mostly working fine for a decade or so, but lately there have been more bounces due to "strict SPF". For example, jksound.com's SPF record includes "-all" (dash instead of tilde) which means that when example@jksound.com tries to mail example@dnalounge.com, we forward that along to example@gmail.com, and then Google rejects it with 550 "SPF hard fail".
The originator of the messages has said, very specifically: "don't forward our messages and make it look like it's still from us." So stop doing that. You just can't forward emails all over the place like that anymore. And that wasn't a decision Google made. Paul Vixie, who basically reports to nobody less than God, and even then probably only takes it as an advisement, was responsible for it, or at least popularizing the idea, back in circa 2002. SPF is older than Gmail. And it's in creating SPF that the decision was made—again, by the very pre-Gmail Internet community—that "uh, maybe letting anyone transmit email that appears to originate from someone else is a bad idea".
He's literally had 20 years to stop doing this and come up with a better solution, and the Internet, true to Postel's Law, was kind enough to let him keep getting away with it for all that time.
He even touches on some of the possible solutions in the post about the problem!
"Provide an IMAP server for all of my employees" is a terrible answer, in terms of both maintenance headache and disk space.Again, much respect to jwz, but... my brother in Christ, it's 2022: how many employees do you have, that disk space is a problem for giving them IMAP? If you're really running your own hardware, just go buy a couple of disk drives and call it a day. That's literally one of the only reasons why it makes even a lick of sense to be running your own SMTP anymore—you can buy storage for a fraction of the cost of a cloud provider.
And a maintenance headache? Isn't that what he has right now, because he's basically sending out spoofed emails? Bite the bullet and get it over with, it's the right solution. If you want to provide people email addresses for incoming mail, provide the backend infrastructure to receive those messages and hold them for delivery to the recipient. That's the right way to do it.
But okay, maybe he's running this on, I dunno, a VAX in his basement, and those DEC DSSI drives are getting hard to come by. Fine, whatever. But he comes up with another viable solution!
Like, say, forward it as an attachment instead. (This would obviously be insane and terrible, and yet still better than bouncing.)That's not an insane or terrible idea at all. I mean, yes, it'd be stupid to make the message an "attachment" in the literal sense of something the user has to save to their Desktop and open manually, or something dumb like that, but yes, encapsulate the goddamn message before you retransmit it. Because the message you're sending isn't the same message as the one you received, because you're the one sending it. You need to re-wrap it in another envelope, basically, one that doesn't claim it's being sent by someone else.
This is the solution that I use for the mailinglist that I operate, because it's the solution that the much-smarter-than-me people who develop Mailman came up with to this very problem. Mailinglists (or listservs, or reflectors, depending on which corner of the Internet you spent your formative years in), in their traditional form, look just like spoofing too. And SPF looked for a while like it would be death for mailinglists. And there was much wailing and gnashing of teeth, and several solutions were devised.
The simple solution is called "From munging" in the mailinglist world, and it creates an outgoing message that looks like this:
To: bob@bigco.com
From: Anne Person via SomeIntermediateList
Reply-To: Anne Person
That works, because it's honest: it's not claiming (via the "From" line) that the message is from someone who it's not. The message is being sent from list@intermediate.co, to Bob, and we're telling Bob "if you want to reply to this message, send it directly back to Anne, here's her address".
Of course, there are more elegant ways to do this, courtesy of MIME, which is a hot new standard from... 1996. MIME specifies ways of wrapping one email message inside another email message, and if your mail program isn't totally stupid, it should be able to unwrap them for you—although there aren't any guarantees about how it's going to display the message in your Inbox, for the purposes of sorting it, etc. But IMO, this is the Technically Correct (the best kind of correct!) way to handle any sort of email-retransmission scenario where you're passing along a message that you didn't actually originate, be it mailinglist messages, forwards, bouncebacks, etc., and if we're going to pick a hill to die on, it's this: MUAs (mail client programs) should deal with MIME-encapsulated messages in a not-stupid way, e.g. by allowing the user to "bust" the outer wrapping and extract the inner message upon receipt. Because this is the only elegant way that you can do mailinglists and forwarding with encrypted or signed messages and not screw them all up.
Yes, this takes more than setting up a .forward file or a Postfix alias. Which sucks. But again, being able to forward email around like that without consequence was determined to be a bad idea somewhat before invading Iraq was.
posted by Kadin2048 at 2:58 AM on September 7, 2022 [15 favorites]
Just realized that MeFi stripped some key components of the sample message in my previous post, because they were inside < and > and I forgot they're not auto-escaped. My bad.
Anyway, this is what I was going for:
To: Bob <bob@bigco.com>
From: Anne Person via SomeIntermediateList <list@intermediate.co>
Reply-To: Anne Person <aperson@smallco.com>
This is done automatically by mailinglist software like GNU Mailman; in the context of forwarding messages like jwz is trying to do, apparently Sender Rewriting Scheme is the solution du jour. (It looks needlessly complex to me, but I guess there are some subtleties about keeping the original envelope From address accessible in a machine-readable way for the purposes of bounce messages and stuff.)
posted by Kadin2048 at 3:22 AM on September 7, 2022 [3 favorites]
Anyway, this is what I was going for:
To: Bob <bob@bigco.com>
From: Anne Person via SomeIntermediateList <list@intermediate.co>
Reply-To: Anne Person <aperson@smallco.com>
This is done automatically by mailinglist software like GNU Mailman; in the context of forwarding messages like jwz is trying to do, apparently Sender Rewriting Scheme is the solution du jour. (It looks needlessly complex to me, but I guess there are some subtleties about keeping the original envelope From address accessible in a machine-readable way for the purposes of bounce messages and stuff.)
posted by Kadin2048 at 3:22 AM on September 7, 2022 [3 favorites]
I guess I'm weird then. I've been self-hosting email since I had permanent broadband (instead of dial-up) which is roughly the last twenty years. Currently I base on Modoboa but previously I've installed Zimbra and even Lotus Notes. In part it's because I've a bad habit (though I'm in recovery) of registering tens of domains at a time and external services wanting a fee per domain, which I refuse to supply for something which may not even get any. Running my own (I've had a fixed IP block for many years too) has not been a problem except for the script kiddies attempting to gain access across my servers (I block them!) I send directly, not through any 'smarthost', and run my own DNS servers too (and NTP, etc etc). Maybe I've just been lucky, but while it continues to work fine I'm happy.
posted by Inanna at 10:02 AM on September 7, 2022 [2 favorites]
posted by Inanna at 10:02 AM on September 7, 2022 [2 favorites]
Huh. Interesting. This is the total opposite of my experience and seems to run counter to what looks to me like a growing trend of families and individuals moving to self-hosting and away from cloud services bought by consent to commercial surveillance.
I just evacuated my entire family from Google-hosted services because they saber-rattled at their entire early adopter base. Well, that and because having my mailboxes sitting in a Google datacenter instead of on rust I personally own was starting to get my goat.
There was a postfix/spamass-milter stack sitting on a mostly-stateless, kinda-ephemeral VPS container on a cheap host in an afternoon. I got the dovecot/radicale containers up and running on my own hardware after piddling around with it for a couple of weekends; dovecot sasl is slightly thornier than plain old passwd identities.
Since I wanted the relay out of the house and the mailboxes in the house -- necessitating LMTP across a WAN, which is pretty sketchy to do in the clear -- I took an entire couple of extra hours to throw everything we own on a slackhq/nebula overlay network so I could stop caring about perimeters. I spent the equivalent of a lazy Sunday sitting with family members to migrate their clients.
I ran my own mail for years before Google came along and now that they're unambiguously Evil I'll run my own mail for the rest of time.
posted by majick at 9:35 PM on September 7, 2022 [3 favorites]
I just evacuated my entire family from Google-hosted services because they saber-rattled at their entire early adopter base. Well, that and because having my mailboxes sitting in a Google datacenter instead of on rust I personally own was starting to get my goat.
There was a postfix/spamass-milter stack sitting on a mostly-stateless, kinda-ephemeral VPS container on a cheap host in an afternoon. I got the dovecot/radicale containers up and running on my own hardware after piddling around with it for a couple of weekends; dovecot sasl is slightly thornier than plain old passwd identities.
Since I wanted the relay out of the house and the mailboxes in the house -- necessitating LMTP across a WAN, which is pretty sketchy to do in the clear -- I took an entire couple of extra hours to throw everything we own on a slackhq/nebula overlay network so I could stop caring about perimeters. I spent the equivalent of a lazy Sunday sitting with family members to migrate their clients.
I ran my own mail for years before Google came along and now that they're unambiguously Evil I'll run my own mail for the rest of time.
posted by majick at 9:35 PM on September 7, 2022 [3 favorites]
« Older The 2022 Hugo Awards | Far out!! Wow!! YES!! Lynda Barry #1!! Newer »
This thread has been archived and is closed to new comments
posted by Nelson at 10:35 AM on September 5, 2022 [10 favorites]