DMARD: Domain-based Message Authentication, Reporting & Conformance

January 31, 2012 7:12 AM

For the past 18 months, engineers at PayPal, Google, Facebook, Yahoo, AOL, Microsoft and nine other technology companies have spent their off-hours (and some on-hours) working hand in hand to tackle the problem that plagues them all: e-mail phishing. The result is DMARC, or, "Domain-based Message Authentication, Reporting & Conformance". It's not new, but puts SPF and DKIM to work in a new way.
posted by Blake (45 comments total)

Why the frick isn't all email done on a white list basis anyway? It'd be trivial to build some kind of sender approval mechanism into email clients, and then it'd be fairly easy to keep spammers and phishers out of your inbox or to contain damage if some bad actor slipped through. Sure, it'd take an extra step to start receiving emails from a particular user, but it would put the power to control the inbox in the hands of the user and possibly even make email useful again.
posted by saulgoodman at 7:20 AM on January 31 [1 favorite]

Why the frick isn't all email done on a white list basis anyway?

And who would manage it, and who would pay for it? You? That's the frick why.
posted by furtive at 7:25 AM on January 31 [6 favorites]

If there is a simple step-by-step implementation guide for people managing domains and mail servers, a like would be appreciated.
posted by vanar sena at 7:31 AM on January 31

A link would be appreciated too.
posted by vanar sena at 7:32 AM on January 31

DMARD: Domain-based Message Authentication, Reporting & Donformance
posted by etc. at 7:36 AM on January 31 [4 favorites]

Well, how would legimate third-parties contact you to get on your whitelist? What if I want to send a message to a MeFite who probably wouldn't mind an email from me, but who wouldn't know who I was otherwise?

Whatever that approval mechanism is, that will then start getting spammed.
posted by Phire at 7:41 AM on January 31 [4 favorites]

The front page of dmarc.org atutely observes that "Simply inserting the logo of a well known brand into an email gives it instant legitimacy with many users." then goes on to try, yet again, to secure envelope from and/or the from: header, which does nothing to stop messages from showing a paypal logo above malicious content. Then it adds crazy reporting requirements (from mail receivers to mail senders) that are unlikely to be supported in shipping MTAs for years, if ever.
posted by jepler at 7:42 AM on January 31 [1 favorite]

Who manages Facebook accounts that effectively white list contacts? The end user of the email account. And the white lists would be account specific. A user would get a notice in their email client that someone not on the approved list wants to send them an email--until it's approved, the email wouldn't be delivered to the inbox but held in quarantine.

Business users could bulk import white lists of safe senders or make it the responsibility of users to accept add requests in accordance with company policies.

This approach would work best for personal email, but I can't imagine it would be all that difficult to offer different levels of protection for different kinds of email accounts, so that public relations oriented email addresses that take all comers wouldn't require senders to be on a white list, while individual accounts maintained separate white lists. It wouldn't be that hard to work out.

But white-list based email would effectively kill (or severely undermine) mass marketing via email, so for that reason alone, it's probably a nonstarter.
posted by saulgoodman at 7:44 AM on January 31 [2 favorites]

A user would get a notice in their email client that someone not on the approved list wants to send them an email--until it's approved, the email wouldn't be delivered to the inbox but held in quarantine.

This would work for phishing, but doesn't this just shift spam to approval requests instead?

spamdomain DOT com would like you to add them to your approved senders list!
posted by ODiV at 7:48 AM on January 31

Whatever that approval mechanism is, that will then start getting spammed.

Sure, but that can be designed so it doesn't also deliver malicious code, tracking cookies, etc. Also, it would be a simple list you could browse and select/approve in bulk.

And if a sender you approved misbehaved, you could quickly remove them from the white list and minimize the impact of their mischief.

You wouldn't get spammed for sender approvals anymore often than you get spammed with Facebook friend requests, for example, because the approval request mechanism couldn't be used to deliver any marketing content. It would just display the sender's email address/name and give you the option to approve or reject the sender. Subsequent attempts from the same sender would be ignored if they were rejected. Filling random people's sender approval ques with a bunch of pointless requests wouldn't really have any profit in it for anyone, so I doubt it would be an issue except maybe in malicious attacks, but that's always going to be possible in some form.
posted by saulgoodman at 7:52 AM on January 31

spamdomain DOT com would like you to add them to your approved senders list!

Sure--then reject them or ignore them. At the very least it would be a huge improvement over how it works now, from where I sit...
posted by saulgoodman at 7:53 AM on January 31 [1 favorite]

We need effective communication more than spam and phishing protection, saulgoodman, that's why email must not depend upon whitelists.

You could possibly handle spam by requiring senders "pay computational postage" by solving a difficult math problem, but presumably making botnets consume more CPU cycles. You cannot solve the authentication problem that way, i.e. protect against phishing. Instead, you'll require some public-key infrastructure system to authenticates senders, but our current CA infrastructure has proven sloppy and vulnerable.

There are much better non-authoritative key exchange protocols employed by off-the-record messaging and ZRTP that provide "historical" authentication and encryption, afaik nothing similar exists for email.

I haven't read anything about DMARC yet, I'd presume it's yet another top down system ripe with vulnerabilities, but maybe it'll reduce phishing scams.
posted by jeffburdges at 7:56 AM on January 31 [2 favorites]

Isn't this easily solved by turning off htmail? i.e. rely on the user to filter out phishing emails by content. of course the user would have to be able to parse urls or maybe even copy and paste to follow a link and then googlebookpaybaymicrosooaol (not to mention vericasatt) wouldn't be able to phish their own user base...
posted by ennui.bz at 7:59 AM on January 31

Are the Bayesian filtering techniques that were so in vogue in the early 2000s no longer doing it for normal people? They work just fine for me, using a version of spambayes from circa 2005 and a training database that is probably just about as old (though I do ongoing training with 10-20 quarantined mails every month). I don't feel a need for some new technology just to get legitimate mails from (say) Paypal into my inbox and mails that merely purport to be from Paypal into a quarantine folder or into the bit bucket.

Of course, I'm also the neckbeard type who stubbornly rejects any mailreader which has the capability to display an image inline with the text of an e-mail, so I literally have no idea about how similar a fake paypal message and a legitimate paypal message look. But when a message that purports to be from paypal links to 'afniuqjzk.com', well, enough said.
posted by jepler at 8:03 AM on January 31 [2 favorites]

So Facebook, Google, et al are going to become email sending licensing agencies?

No thanks.
posted by DU at 8:08 AM on January 31 [1 favorite]

Mail Receivers SHOULD also implement reporting instructions of DMARC in place of any extensions to SPF or DKIM that might enable such reporting. {R10}

It's nice that they didn't make it a "MUST" that you not fully implement future versions or extensions of other mail anti-forgery standards! (they only forbid you from completely complying with the future versions or extensions to SPF or DKIM when complying with DMARC; that one's a MUST)
posted by jepler at 8:08 AM on January 31

I can't believe I'm going to make an argument like this, but: anti-phishing is not for folks who already know to look at URLs or turn off HTML mail. Or good god, know what Bayesian filtering is.
posted by kmz at 8:10 AM on January 31 [6 favorites]

sure kmz (I tried to get out in front with my own neckbeard credentials), but administrators who can grow hair on their necks should do this stuff on the server side and be done with it. it's kind of like DMARC except there are already working implementations (pushing ten years old, I don't doubt), you don't have to cooperate with anybody outside your organization, and it actually seems to solve the spam problem in practice.
posted by jepler at 8:16 AM on January 31

Why the frick isn't all email done on a white list basis anyway?

Note that client-side whitelisting with user classification of incoming mail is trivial to implement, and could be rolled out tomorrow if somebody wanted it. Doing it in a manner that is efficient, non-intrusive and secure is not.

The problem is that the people who are most at risk from spamming/phishing (i.e. clueless users) probably are not up to the task of maintaining a secure whitelist - you will either need to have someone babysitting their e-mail, or people will just add the scammers to their whitelist to get to the screensavers/bill gates' forwarding fee or whatever.
posted by Dr Dracator at 8:19 AM on January 31

Jepler, one of the main reasons Bayesian approaches don't work as well on phishing attacks is because phishing attacks look just like real emails. In some cases, the criminals literally copy a real email (say from eBay) and just change where the link goes to.

The basic idea you're pointing out, about looking for unusual links, is one that we took in automated detection of phishing emails. Here's a link to one of our papers published in WWW 2007, entitled Learning to Detect Phishing Emails. We've also commercialized this filter too, it's in use in lots of places now (memail me if you want a link to the product page).
posted by jasonhong at 8:19 AM on January 31 [2 favorites]

I can't believe I'm going to make an argument like this, but: anti-phishing is not for folks who already know to look at URLs or turn off HTML mail.

But what's the point of html mail? big corporations get to try to get their own users to clicky clicky into some money sucking promotion i.e. phish their own userbase.

so basically, we want a system of email where some people/corps get to phish and some people/corps don't
posted by ennui.bz at 8:21 AM on January 31

We need effective communication more than spam and phishing protection, saulgoodman, that's why email must not depend upon whitelists

I would normally agree jeffburdges, but there have been criminal cases in my state (for example) in which email users were charged and prosecuted with child pornography crimes simply for having had illegal pornographic content spammed to their email inboxes unsolicited. Since the law is so utterly incompetent when it comes to dealing with those kinds of issues, it seems to me most email users could stand to have better tools to protect themselves. And, well, hell, I've for all purposes abandoned email myself because there's invariably far more noise than signal in my inboxes. For me, having the power to turn the spiggot on and off is probably the only thing that would make me seriously consider using email on a daily basis again in my personal life. But I've taken this derail too far already.
posted by saulgoodman at 8:21 AM on January 31

Also, if you're interested in learning more about phishing attacks in general, the economics of phishing, and current defenses against phishing attacks, I recently wrote up an article in the Communications of the ACM entitled The State of Phishing Attacks.
posted by jasonhong at 8:22 AM on January 31 [2 favorites]

Is spam really still a big problem for a lot of people?

I can count on one hand the number of spam messages that make it through the spam blocker per month and I run my own domain. At this point, it's down to the same level of annoyance as those physical junk mail envelopes that look like real bills.

I'd have figured that the big players like gmail or hotmail had similarly licked the problem, since they have a much bigger corpus to work with.
Are any of you really getting an unmanageable amount of spam dropped in your inbox?
posted by madajb at 8:24 AM on January 31 [1 favorite]

And just how would I maintain contact with my nigerian friends? For some of us phishing is all we have.
posted by mattoxic at 8:27 AM on January 31 [3 favorites]

But what's the point of html mail? big corporations get to try to get their own users to clicky clicky into some money sucking promotion i.e. phish their own userbase.

so basically, we want a system of email where some people/corps get to phish and some people/corps don't

As much as corporate spam can be annoying, it is in no way in the same category as actual phishing. Unless Google regularly tries to get you to tell them your Paypal password or something.
posted by kmz at 8:35 AM on January 31

Is spam really a problem for some people? I probably get upwards of 5000 spam emails a day. 98% of these or more are caught by my 2 layers of filtering. That leaves 100 or so for me to sift through. I'm atypical because my email(s) are the administrative contacts for websites and as such are big targets for spam.

What I want more than anything else, and what I think email needs, is the absolute certainty that email is from who it says it's from. With this, a lot of spam will disappear I think, and phishing may be reduced or much easier to track back to it's source.
posted by RustyBrooks at 8:37 AM on January 31 [1 favorite]

As much as corporate spam can be annoying, it is in no way in the same category as actual phishing.

Well, Verizon wants to take thousands and thousands of dollars out of my bank account in small enough amounts monthly that I don't notice the theft and sends me email daily trying to get me to click into websites which will increase that amount by offering various dubious services...

I just don't see a compelling reason why email has to include html, except to sell people stuff.
posted by ennui.bz at 8:59 AM on January 31

1. The corporate world like their fancy (intrabusiness) emails with bold headlines and underline and Comic Sans and half page signatures with 10 logos.

2. Lots of "regular" email users like being able to embed images, mess with fonts, etc.

I'm not saying these are good or ideal situations. If I could make everybody use PINE, I'd go for it in a heartbeat. (And hell, the latest versions of (AL)PINE even have some HTML support.) But that's just not the world we live in anymore. September's never ending.
posted by kmz at 9:06 AM on January 31

and possibly even make email useful again

What did I miss? Email works perfectly for me and is my main form of electronic communication. With Gmail I see maybe a couple fake emails per quarter.
posted by Meatbomb at 9:12 AM on January 31

Most of what I consider spam in my inbox is technically "legitimate" email marketing I either carelessly failed to "opt-out" of, or got auto-enrolled for because I made the mistake of supporting a political candidate or buying something online. I could unsubscribe, but who has time, and why should I have to?

Anyway, it sounds like this tech might offer some potential for allowing the producers of Grade-A quality spam to keep the inferior products off the market, but it doesn't improve email in the ways that would matter most to end-users in my opinion. At least not this end-user.

What did I miss? Email works perfectly for me and is my main form of electronic communication.

You probably have more time to invest in defending your inbox than I do. I'm not exaggerating. I've given up. The only time I use email anymore is for some immediate communication I've coordinated in advance with a user in meat-space. If I need to contact someone out of the blue, Facebook is usually how I do it (and I don't even like FB).
posted by saulgoodman at 9:17 AM on January 31

(And of course, I use email when I have to for work.)
posted by saulgoodman at 9:18 AM on January 31

Friends don't let friends use SPF: it breaks forwarding of the sort which is very common, namely, I own example.com but forward all my email to Gmail, say (see here and here). The SPF folk would love it if everyone could just adopt SRS, but this is one of those "everything would be great if everyone just did it my way" scenarios: forwarding has been around for a lot longer than SPF, so you'll struggle to get interoperability with everything that's already deployed.

The DMARC spec wibbles a bit about identifying trusted forwarders but doesn't seem to address this. To the extend that DMARC might encourage people to start using SPF to actually reject mail, it's a bad thing.
posted by pw201 at 9:24 AM on January 31

I wish email was backed by money. You send me spam I click the button that charges you a nickel. If it's not backed by money I don't see it. If I don't mind what you sent me I don't push the button and you're not charged.

It would end spam overnight.
posted by cjorgensen at 10:00 AM on January 31 [1 favorite]

If it's not backed by money I don't see it.

In this 'nickel and dimed' utopia - how much are the gatekeepers going to keep? 5%? 10% 30% 70%?

Why should my email be subject to yet another corporate payout?
posted by rough ashlar at 10:03 AM on January 31 [1 favorite]

I wish email was backed by money.

How about this: You pay someone to pre-read your email and they reject the spam for you.
posted by rough ashlar at 10:08 AM on January 31

Why the frick isn't all email done on a white list basis anyway? It'd be trivial to build some kind of sender approval mechanism into email clients, and then it'd be fairly easy to keep spammers and phishers out of your inbox or to contain damage if some bad actor slipped through. Sure, it'd take an extra step to start receiving emails from a particular user, but it would put the power to control the inbox in the hands of the user and possibly even make email useful again.

There's nothing preventing you from setting up an email whitelist today if you want.

Whatever that approval mechanism is, that will then start getting spammed.

Yeah, like getting a million of those popups that say "so and so wants to chat with you" on some IM services. I feel like, sometimes if I sign into MSN I get a million of those, although I never use it anymore so I have no idea if they've fixed that or what.

Who manages Facebook accounts that effectively white list contacts? The end user of the email account. And the white lists would be account specific. A user would get a notice in their email client that someone not on the approved list wants to send them an email--until it's approved, the email wouldn't be delivered to the inbox but held in quarantine.

But Facebook also actively works to prevent spammers from ever getting on their system in the first place. We've all seen accounts with misc hot girl profile pictures that seem to exist entirely for spamming people. But you can't setup an automated system to request every single Facebook user. SMTP is P2P, essentially. You can't remove abusers from the entire network.

The irony is, spammers have made email as a true P2P system impossible. I used to use my own domain for email. I figured I had a great way to prevent spam: Every time I had to enter an email I'd use a different address @mydomain. Then some spammer started Joe Jobbing me, meaning they sent out spam claiming to be from my domain. That means I got a ton of bounce messages.

Anyway, now I just use Gmail. While theoretically everyone is equal in email, the spammers have made it impossible to actually use it that way.

You wouldn't get spammed for sender approvals anymore often than you get spammed with Facebook friend requests, for example, because the approval request mechanism couldn't be used to deliver any marketing content. It would just display the sender's email address/name and give you the option to approve or reject the sender.

Yeah but what happens when the person's name is "Cheapvigra Dotru"

You could possibly handle spam by requiring senders "pay computational postage" by solving a difficult math problem, but presumably making botnets consume more CPU cycles.

Bitcoins! You could have a system that required bitcoin postage, which can theoretically be generated by pure computation, for now. You cold charge the equivalent of five minutes of 'mining time' to accept a message. Something like that.

Are the Bayesian filtering techniques that were so in vogue in the early 2000s no longer doing it for normal people?

Normal people just use gmail/hotmail/yahoo. I have no idea what they do on the backend, but they have enough resources to really put a damper in spam. Looking now I apparently got 65 spams in the past 30 days put in the spam folder. I would imagine a lot just get blocked from even being delivered. The email address I use for signing up for crap got 150, but pretty much everything in there is useless marketing BS.
posted by delmoi at 10:16 AM on January 31 [1 favorite]

You probably have more time to invest in defending your inbox than I do. I'm not exaggerating. I've given up. The only time I use email anymore is for some immediate communication I've coordinated in advance with a user in meat-space. If I need to contact someone out of the blue, Facebook is usually how I do it (and I don't even like FB).

Even though I don't get a lot of spam using gmail, I've never really liked email. I'd rather call someone and talk to them on the phone. Text messages also work well. My android phone has dictation software now so I don't even have to type text messages anymore. It's pretty awesome.
posted by delmoi at 10:21 AM on January 31

Your post advocates a

(x) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(x) Mailing lists and other legitimate email uses would be affected
(x) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(x) It will stop spam for two weeks and then we'll be stuck with it
(x) Users of email will not put up with it
(x) Microsoft will not put up with it
(x) The police will not put up with it
( ) Requires too much cooperation from spammers
(x) Requires immediate total cooperation from everybody at once
(x) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
(x) Lack of centrally controlling authority for email
(x) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(x) Asshats
(x) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
(x) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
(x) Eternal arms race involved in all filtering approaches
(x) Extreme profitability of spam
(x) Joe jobs and/or identity theft
(x) Technically illiterate politicians
(x) Extreme stupidity on the part of people who do business with spammers
(x) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

(x) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
(x) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
(x) Countermeasures must work if phased in gradually
( ) Sending email should be free
(x) Why should we have to trust you and your servers?
(x) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
(x) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(x) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!
posted by lohmannn at 10:34 AM on January 31 [18 favorites]

And who would manage it, and who would pay for it? You? That's the frick why.

To clarify: this is not why whitelist-only email hasn't been created, this is why whitelist-only email, options for which have been cheap and plentiful for decades, doesn't get used.
posted by roystgnr at 11:15 AM on January 31 [1 favorite]

Hah, lohman -- I haven't seen that in a while. :7)

As for this: A user would get a notice in their email client that someone not on the approved list wants to send them an email--until it's approved, the email wouldn't be delivered to the inbox but held in quarantine.

This is too clunky for users. When I get a message that someone has placed an email for me in a secure web site and I need to go there, log in, click some other damn thing, and then maybe read it...well, more often that not I just delete it. Have fun in your Fortress Of Solitude, man.

My best friend's home phone asks me to press zero if I am not a solicitor, and I have nearly hung up on him several times before he even knows I am calling.

And I say this as a bit of a privacy/security axe-grinder myself.
posted by wenestvedt at 1:19 PM on January 31

My best friend's home phone asks me to press zero if I am not a solicitor

Does the MeFi lawyer brigade know of this? What if a lawyer wants to talk to him in a non-lawyerly capacity? Do they have to chant I AM A LAWYER I AM NOT YOUR LAWYER THIS IS NOT LEGAL ADVICE IE IE CTHULHU FHTAGN before being put through?
posted by Dr Dracator at 1:56 PM on January 31

saulgoodman wrote: For me, having the power to turn the spiggot on and off is probably the only thing that would make me seriously consider using email on a daily basis again in my personal life. But I've taken this derail too far already.

This seems nutty to me. Spamassassin, which uses rule-based filters and bayesian filtering has been essentially eliminating spam from my inbox for the better part of a decade now. Greylisting eliminated much of the rest, to the point where less spam hits my filter than real mail. Looking at the stats, I get 128.7 non-spam mails a day and about 46.3 spams/suspected spams a day.

That's not to say I appreciate the amount of time and money I and my clients spent in the middle part of the last decade getting to the point where spam wasn't something we had to think about much, though.
posted by wierdo at 3:25 PM on January 31

It would just display the sender's email address/name and give you the option to approve or reject the sender.

Which doesn't solve the problem of unsolicited but still desired mail. IE: if amberglow decides he needs to fix his fridge and decides to drop me an email I'd like that email to go through even though I don't know his email address or his real name but I probably don't want to accept email from random [e-mail] email addresses because let's face it that amberglow is probably never going to email me.
posted by Mitheral at 3:25 PM on January 31

How about this: You pay someone to pre-read your email and they reject the spam for you.

Not a bad idea. I hate email for the most part. I am currently outsourcing some of my tasks to Malaysia as it is. I should see if I can add this to the list.
posted by cjorgensen at 8:08 PM on January 31