On Sun, May 25, 2025 at 11:20:16AM +0200, Tom Ivar Helbekkmo via NANOG wrote:
First: SPF/DKIM/DMARC are not about spam, so that part is irrelevant.
Perhaps you don't remember this, but when SPF was announced, its home page read: "Spam as a technical problem is solved by SPF." Shortly thereafter, spammers became far and away the largest early adopters of SPF. Of course, that claim was eventually disappeared down the memory hole. Now onto the long, and I do mean REALLY LONG part of this. Sorry, it's a complex issue and it takes time to explain all the pieces. Here's a one-sentence summary: All these alleged anti-email-forgery technologies have accomplished is to make the email forgery problem MUCH worse than before they existed. Introduction ------------ I've never considered email forgery to be a significant problem -- not when compared to the other problems we face. But let's put my opinion aside for a moment, and let's presume that email forgery really is a significant problem -- one so serious that it's worth adding an enormous amount of fragile complexity to an ecosystem already under serious stress from spam and other attacks/abuse. Let's assume that it's worth breaking email forwarding (working fine for decades) and mailing lists (working fine for decades, and clearly the best mass collaboration/communication mechanism we have) and adding enormous cost, effort, and complexity to every email system. There's a problem with that: email forgery can't be solved. Even if if these byzantine hacks were perfectly designed (they're not) and globally deployed (not happening) and correctly deployed everywhere (nope) and always worked perfectly (also no) and weren't attacked (they are) and so on, even if everything worked *exactly* as intended...the overall impact of all this effort and expense and brittle complexity is to make the email forgery problem MUCH worse. I've explained this before (briefly) but let me add a lot more detail this time around. Part A: contributory factors ---------------------------- A1. We've had a problem with fake/junk domains for decades. It was bad enough when there were only 7 gTLDs, but now there are now ~1000 new gTLDs that nobody needed or wanted except (a) registrars, who of course want to make the cash registers ring and (b) abusers/attackers, who buy domains in bulk, burn through them, and then buy more. This symbiotic relationship works beautifully for both parties involved but not for anyone else. And of course registrars are performing no due diligence whatsoever (why should they?) so it's trivial for abusers/attackers to register example.xyz or example.lol or example-com.xyz or example-site.red or whatever, where "example" is a name explicitly designed to suggest that they're some other Internet operation -- e.g., Amazon, Paypal, Google, Joe's Donuts, etc. I've been studying large samples of domain data for multiple decades, and there are enormous numbers of these fake/junk domains. Many are abandoned once they've served their purpose, but new ones are being registered constantly. And every time a new TLD is put into production, there's a land rush as abusers/attackers swarm it in an attempt to be the first to register the optimal fake domains. I see no signs that this problem will be solved, because the people responsible for creating it don't want it to be solved. If these registrars performed any kind of due diligence, it would (a) cost them money and (b) reduce their sales. There's no reason in the world for them to do that. Bottom line: it's cheap, simple, and easy to register an arbitrary number of domains whose names correspond to any operation that abusers/attackers wish to target. You want a real-world example? Okay. Here's an example: Infrastructure Laundering: Blending in with the Cloud https://krebsonsecurity.com/2025/01/infrastructure-laundering-blending-in-wi... U.S. Sanctions Cloud Provider "Funnull" as Top Source of "Pig Butchering" Scams https://krebsonsecurity.com/2025/05/u-s-sanctions-cloud-provider-funnull-as-... Complete List of Domains Attributed to Funnull https://www.ic3.gov/CSA/2025/Complete_List_of_Domains_Attributed_to_Funnull.... There are 332,696 lines in that list. Well over a quarter million fake/junk domains used by just one scamming operation. And as large as this is, it's the tip of the tip of the tip of the iceberg. Let me also ask a very pointed question: if someone shows up at your hosting or cloud operation, and they want to deploy, let's say, 52,782 domains: do you think that maybe, just *maybe*, you ought to ask a few questions about what the heck they're up to? If you're Microsoft or Amazon (see Krebs articles above): nope. Which brings me to: A2. There are all kinds of hosting/cloud operations out there which will happily take example.xyz's money because they perform the same level of due diligence as registrars in (1): none. These operations know full well that example.xyz is not Example Inc, and they know full well what they're doing, but because example.xyz pays the bills and they're repeat/bulk customers, the hosts/clouds just take the money and look the other way. This isn't a small problem. It's massive. And it includes a lot of hosting/cloud operations that should know better. And probably *do* know better, but: due diligence costs money and takes effort, it's cheaper and easier to just blow it off and only take action belatedly -- long after the damage is done -- if there's any bad press. Bottom line: even the most obvious scam/fraud operations have no problem finding hosting. A3. The combination of (A1) and (A2) mean that you can go out today and register 1000, 10000, 100000 fake domains impersonating the Example, Inc's of the world and you can find hosting for them, and you can do it all at a bulk rate discount. Your only real difficulty will be competing with the other people doing the same thing. And unless you do something that generates a sufficient amount of sufficiently bad press, it's very unlikely that a registrar or host will take any action to shut down anything you're during, and even if they do that, they'll only shut down the small part directly involved -- everything else you have in place will be just fine. And then, on top of all this, they'll slow-walk whatever they do in order to keep the income coming as long as they possibly can. A4. User interfaces in email clients -- whether standalone pieces of software, or webmail interfaces that run in a web browser -- are increasingly being modified to obfuscate sender addresses. In other words, rather than displaying: From: Joe Smith <joe@example.com> they display: From: Joe Smith and some kind of user action is required to actually see the address. The aggregate effect of this, across all UIs across all clients across all users, is to train users to ignore actual email addresses in favor of the text strings that purport to be someone. (Did YOU check the actual email address in this message to see if it's really from me?) A5. One consequence of (A4) is that users are increasingly unable and/or unwilling to understand email addresses. They can't distinguish @example.com from @example.xyz or @example-com.lol or @exammple.com or @exammple.somethingsomething.fun. (Yes, yes, there are *some* users that can do this. A few clueful ones. As in: "A person is smart. People are..." C'mon, you know the rest of this.) This loss of comprehension/function resulting from (A4) and (A5) works beautifully for the operators of domains in (A1) and (A2): they don't even have to try hard. Even sloppy half-ass typosquatted domains (and subdomains) work just fine. [ Let me interject that while I was drafting this I read about the plans by Mozilla to completely screw up the address bar in Firefox -- because of course leaving the simplest, most fundamental, and most important thing in the browser alone was never a choice for these incompetent clowns. If I understand what they're going to do correctly, and I really hope I don't, they're going to make life MUCH easier for forgers/scammers/et.al. who are deploying fake domains. ] A6. User interfaces in email clients are being modified to provide signaling to users that a given message has been "authenticated" or "verified" or "signed" or "certified" or some similar designation. This is of course the result of a successful check of whichever anti-forgery mechanism(s) the receiving email server chose to utilize. And just as users being trained to accept (A4), they're being trained to accept this. They believe what's in front of them on the screen. But of course this means *nothing*: a message from joe@example.xyz will be dutifully marked (provided the keepers of example.xyz set things up correctly and everything's working) as authentic even though example.xyz is a fake domain that has nothing to do with the real Example Inc. and Joe is a fake person. In other words, obviously fake messages from obviously fake domains are treated the same as authentic messages from authentic domains. ("But couldn't...". No. It's not a tractable problem, because it would require enumerating the authentic domain(s) of every operation on the Internet, i.e. determining that example.com is the "real Example Inc. -- for millions of example.com's, example.edu's, example.net's, etc. And not only would this be incredibly expensive, it would break badly as companies and domains change ownership or go out of business, it would greatly favor large operations over small ones, it would quickly be corrupted by the people with the biggest wallets, it would... So don't even go there.) This is analogous to the wry saying about https:// -- yeah, it means you have a (putatively) secure connection, but you could have a secure connection to the worst people in the world. A7. Many Example Inc's of the world don't send plaintext email messages; they mark them up with HTML and graphics and so on, which means that they're teaching their users that any message which looks like it's from them is really from them. They're training their users to fall for forgeries/phishes. And years (decades now) of this training have paid off: it's much, MUCH easier to trick users now than it was 20 years ago. A8. Similarly, many Example Inc's of the world include URLs in their messages. They've spent years (decades now) training users to utilize those URLs, and of course users have dutifully learned to do this, and once again: They're training their users to fall for forgeries/phishes. And years (decades now) of *this* training have also paid off: again, it's much easier to trick users now than it was 20 years ago. A9. The combination of (A7) and (A8) means that there's no help coming from the Example Inc's of the world. They'll continue to train their users to be victims while exhorting them to not be victims. The relevance to my point here? These email worst practices distract users from the small pieces of critical information that *might* give them a slim chance of spotting a forgery. They don't cause the problem, but they certainly exacerbate it. A10. The aggregate impact of (A1)-(A9) is: users see a message that looks as they expect it to -- it has the pretty graphics and typography of Example Inc. It claims to be from Example Inc. It's marked as "verified" in their email UI. It has a domain that looks kinda like Example Inc's (if they even bother to check, which almost none of them will). What do you *think* they're going to do? I can tell you what they won't do: they won't take the time to actually study the email address and see that it's @example-com-completely-fake.xyz, or to study the embedded URLs, and even if they do, a substantial fraction of them Will Not Get It. If you think I'm wrong about this, then I must ask: have you actually spent any time with a significant number of users? Part B: large email operations ------------------------------ It's not always necessary to go through the trouble and expense of registering domains, arranging for hosting, etc. It's easy to set up an arbitrary number of accounts that claim to be arbitrary people/companies and spam/phish/whatever from them at will, because there are some very large mail operations that make this easy. And because those same operations have implemented all this anti-forgery tech, those messages will dutifully pass every check that can be done on them. And before someone trots out the "...but they're so large and it's SOOOO hard..." nonsense as an excuse for the malfeasance of these very large email operations: no. If you can't run an enormous operation properly, you shouldn't *build* an enormous operation. It's irresponsible. It's unprofessional. It's unethical. Especially when it would be trivial for these operations to throw 1,000 people and a billion dollars at the problem tomorrow. (We *know* they have those resources, because they're already wasting them on bogus "AI" development.) Let me note, in fairness, that there are also some other rather large operations that -- as far as I can tell by measuring across hundreds of domains over several decades -- do a very good job. The problem is that they're the exception. Recap of part A and part B: --------------------------- The combination of (A) and (B) means that we have an environment where it's cheap, easy, and simple to register an arbitrary number of domains masquerading as Example Inc. It's also cheap, easy, and simple to find hosting for them. And then -- thanks to very poor choices in user interfaces combined with lack of user knowledge/effort, it's cheap, easy, and simple to convince lots of people that completely fake messages from completely fake domains are authentic. And there's even a pretty good chance that forgeries which don't even use a fake domain will work: there are certainly unlimited opportunities to try -- also cheap, easy, and simple. The sad reality is that while it's true that thanks to this anti-forgery tech, abusers will have a hard time successfully forging messages from the real example.com and getting them delivered: they don't have to. Part C: bots ------------ And then it gets even worse, because of a problem that still exists but doesn't garner the headlines that it did years ago. The problem is bots. Remember this? Vint Cerf: one quarter of all computers part of a botnet http://arstechnica.com/news.ars/post/20070125-8707.html That was 18 years ago. And in the ensuing time, *nothing* has happened to seriously address that problem. (Yes, yes, I'm aware of all the headline-grabbing when someone somewhere dismantles a botnet, but those are momentary disruptions at best.) On the other hand, lots of things have happened to make the bot problem worse, including (1) more computers (2) more users (3) vastly more sophisticated bot-creating malware (4) bot-creating malware targeting non-Windows systems (5) major improvements in botnet C&C (6) the IOT -- remember, the "S" in "IOT" stands for security. And (7) thanks to Microsoft's "Recall", things will get worse yet again. Oh, and let's (8) toss in AI, because the people frantically developing those systems want to spend lots of time and money hyping them but can't be bothered to put in any guardrails or do any real testing or even concern themselves for a moment about how this tech will be misused by some of the worst people on the planet. I'll bet even money that there's already a botnet being run by a commercial AI. It's an obvious move, don't you think? Note that: C1. Anyone in control of a bot can utilize any email credentials stored on or transiting that bot at will. If Joe has an address @example.org and one at @aol.com, then the new owner of Joe's system can send outbound messages via those at will. C2. They can also rummage through all email stored on that system or in the accounts from (1), which means that -- if they want -- they can easily make some first-order guesses at who's likely to believe that a message which claims to be from Joe AND that their email system's UI says is from Joe, really is from Joe. C3. This is already bad if Joe is just some guy and Joe's system is just some random laptop somewhere. But what if it's not? What if Joe works at Example Inc. and this is his desktop system and it's loaded with customer correspondence and it's connected to his work email account, also loaded with customer correspondence? *Now* the new owner of Joe's system is in an ideal situation to phish or scam all of those people, because such a message really will come from Joe's email address via Joe's work email server, it'll pass all the anti-forgery checks anyone cares to run, and it'll dutifully be marked as "authentic" in most/all those email UIs used by its recipients. So even if the content is a little sketchy -- maybe enough to raise an eyebrow or two among recipients -- they'll look at their screen, see that the message is marked as real, see that the fancy graphics and typography look real, AND THEY'LL BELIEVE IT. Of course they will, they've been trained to, see A7 and A8 and above C4. And what about the tiny fraction of people who will catch on that think that's something amiss? What, EXACTLY, do you propose that they do about it? Example Inc has probably done its very best to turn itself into an impenetrable corporation where it's nearly impossible to reach anyone in a position to understand a problem like this and take prompt action to fix it. Do they have working postmaster@, security@, abuse@ addresses -- something that every novice sysadmin on this planet should know about? Probably not. Do they have a published security contact? Probably not. Do they have any mechanism whereby someone who calls in can reach anyone useful? Probably not. So even if one of those rare, clueful, diligent people spots this and tries to report it, they're probably going to give up in frustration long before they achieve anything. C5. And even if -- by some miracle of science -- someone in (C4) manages to reach a contact inside Example Inc who understands the problem...the most likely outcome is that Example Inc will ignore it and pretend it doesn't exist. The next most likely outcome is that they'll quietly fix it but say nothing. Why should they inform or warn anyone? That's bad for business. And since there are no significant penalties for anyone involved, their best bet is to disappear the problem and claim it never happened. (If caught? Deny everything. If REALLY caught? Blame a miscommunication and throw a third-level manager under the bus.) Sadly, (C4) and (C5) are not hypothetical. They're why I don't even bother any more. Why should I invest my time and unpaid labor when these operations can't even be bothered to do the simplest, easiest, most obvious things like supporting RFC 2142 addresses and reading what shows up there, acting on it promptly, and publicly acknowledging what's happened? How many email accounts are affected by bot proliferation? A first-order estimate may be found by multiplying (a) an estimate of the worldwide population of bots by (b) an estimate of the average number of email accounts used/present on those bots. I think the floor for (a) is 750M and 1.5B is much more likely. (Go back to Cerf's estimate from 18 years ago and extrapolate.) As to (b) I think it's probably between 1 and 2 -- let's go with 1.5, probably low but that may also account for duplicate email accounts across bots. This gives us a quite conservative back-of-the-envelope guesstimate somewhere north of 1B. One billion compromised email accounts. And you want to tell me it's possible to do something truly meaningful about email forgery? Okay. Fine. *Fix this problem first*, then we'll talk. I'll wait. Let me know when you're done. Note that not only do you have to (1) remediate all those accounts, you'll also need to (2) remediate all the bot'd systems and then (3) keep them that way -- which means discovering the root cause(s) of their compromise and fixing it. (3) will be very difficult if the root cause is something like "the user clicks on every shiny thing they see". But if you're going to claim it's possible to do something truly meaningful about email forgery, then you must do all those steps first. Part D: email account compromises --------------------------------- And (C) isn't even all the compromised email accounts, because we also have to think about all the cases where only an email account (not an entire system) has been compromised. I don't know how many there are (nobody does), but I strongly suspect it's also an enormous (hundreds of millions or more) number. One way to approach an estimate of this is to observe the marketplaces where they're being sold, and in those places, there are lots people selling them in bulk: how many hundred thousand would you like? (Do recall that a few years ago, EVERY Yahoo email account was compromised. That was a neat 3 billion accounts right there. Every single Yahoo account was hacked - 3 billion in all https://money.cnn.com/2017/10/03/technology/business/yahoo-breach-3-billion-... And as anyone who's paying attention to the daily parade of security breaches and dataloss incidents is painfully aware: this isn't an isolated case. It's the norm.) This problem is exacerbated by the complete lack of support at some very large email operations: even when someone who's the real owner of one of these compromised email accounts detects it and make a good-faith effort to fix the problem...yeah, good luck with that. Their chances of recovering their own email account are pretty much zero thanks to awful procedures and non-existent customer service. Thus there's a high probabilty that any compromised email account is going to stay compromised until it no longer exists. Again, if you want to tell me that something serious is going to happen WRT email forgery: *you have to fix this too*. Let me know when that's done as well. Part E: email server compromises -------------------------------- And *then*, on top of (C) and (D), we have to consider the cases where email servers themselves have been compromised, and there's a steady parade of those. Hardly a day goes without another announcement of a massive security breach that involves most/all of some operation's infrastructure, Those announcements are only the tip of the iceberg, since pretty much everyone does their best to conceal such things until either the press reports on them and/or they're legally required to report them... and even *then* they'll sometimes persist in denial and minimization. I'm looking right at you, Oracle. Anyway, compromise of an email server means that it's not only possible to undetectably forge email from every account on it, it's also possible to create new ones and do the same with those. And of course the new owners of these servers have access to everything stored on or transiting them, which provides all kinds of resources that are useful in figuring out who to target, how best to target them, etc. And yet once again, if you want to tell me that something serious is going to happen WRT email forgery: *you have to fix this too*. First. Yeah, I'll wait for this too. Sure I will. Summary of (C) (D) (E): ------------------------ So whether (C) email accounts have been compromised via bot'd systems or (D) just the email accounts themselves have been taken over or (E) entire email servers have been comprommised, outbound messages from all of those will pass any anti-forgery check anybody feels like running. This problem is already well past "enormous" and will continue to get worse, as various ill-conceived bandaids (e.g. "let's get rid of passwords and use passkeys", a breathtakingly stupid and cruel idea) are haphazardly slapped on the problem because nobody wants to spend the massive resources it would take to solve it properly -- one system, one account, one server at a time. Nobody wants to do the hard boring work that might actually succeed in the real world *without* creating more problems than it claims to solve. Until then, we're living in a world where "one billion compromised email accounts" is very likely a serious underestimate. And there will be more tomorrow -- doubly so because attackers know everything I just said. They recognize that the ill-advised rollout of anti-forgery tech that doesn't actually solve the forgery problem has created a great opportunity for them...and they're all over it. Conclusions? Well, maybe. ;) ----------------------------- As I've said before, if you want to solve email forgery for a real-world value of "solve" at the scale of the Internet, then you have to solve the underlying security problems first. ALL OF THEM. And nobody's interested in doing that, because "doing nothing" and "pretending those problems don't exist" is not only far easier, it's far more profitable. Plus it's much more fun to invent technology and promote it and deploy it and declare "Mission Accomplished" and take credit for solving the problem than it is to do the hard work of actually solving the problem. So what's really transpired here, as a consequence of this headlong rush to "solve" email forgery, is that all these elaborate mechanisms have been deployed on top of an ecosystem that's broken all the way down. They're gold leaf on a rotting garbage pile. They don't work because they can't work. They've make-believe -- and unfortunately *users are being trained to believe in them*, which is going to lead to very bad real-world outcomes. Note well: none of this commentary has anything to do with the actual mechanics of SPF or DMARC or DKIM or anything else, which is why I haven't discussed their specifics. The specifics *do not matter*. If someone comes up with another one of these tomorrow and writes an RFC and releases code and tells us all about it: that's not going to work either -- for the same reasons. Yes, in some ideal best-case world, maybe, MAYBE, this elaborate anti-forgery tech might have a fighting chance. (Although: if we were living in that ideal best-case world, we probably wouldn't need it. Do recall that we did just fine for many years without any of this, modulo the occasional prankster or idiot.) But this is not the ideal best-case world. This is a world with massive problems that nobody wants to solve -- and as long as this writeup is, I've only listed some of them. (Yeah, it's worse. It's always worse.) ---rsk