another brick in the wall[ed garden]
Dear Sprint EVDO people, Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries. Queries from my lab: rs@click [14] % dig +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [15] % dig +tcp +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [16] % dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "bifrost" rs@click [17] % Queries from my laptop: Superfly:~ rs$ dig +short @192.148.252.10 version.bind. chaos txt "9.6.0-P1" Superfly:~ rs$ dig +tcp +short @192.148.252.10 version.bind. chaos txt ;; connection timed out; no servers could be reached Superfly:~ rs$ dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "ns1-kscymar06.spcsdns.net" Superfly:~ rs$ Guys, I send you money each month to deliver packets for me, not to invent new ways of being annoying (and breaking TSIG signed updates to dynamic DNS). Less is more. Please stop dinking with 10-minute-idle TCP sessions (which I complained about a year and a half ago) and knock it off with offering DNS service that I did not ask for. Sincerely, Your Disgruntled Customer, RS PS: No, I don't expect that this open letter will get you to fix the misbehavior, but if some Swedish guy comes along swinging a clue-bat at you guys I hope he whacks you a couple of times for me.
While you're at it, it would be nice if SPRINT also fixed the problems with ports TCP/25 and TCP/587. Another disgruntled SPRINT customer, Owen On May 14, 2009, at 10:48 AM, Robert E. Seastrom wrote:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
Queries from my lab:
rs@click [14] % dig +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [15] % dig +tcp +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [16] % dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "bifrost" rs@click [17] %
Queries from my laptop:
Superfly:~ rs$ dig +short @192.148.252.10 version.bind. chaos txt "9.6.0-P1" Superfly:~ rs$ dig +tcp +short @192.148.252.10 version.bind. chaos txt ;; connection timed out; no servers could be reached Superfly:~ rs$ dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "ns1-kscymar06.spcsdns.net" Superfly:~ rs$
Guys, I send you money each month to deliver packets for me, not to invent new ways of being annoying (and breaking TSIG signed updates to dynamic DNS). Less is more. Please stop dinking with 10-minute-idle TCP sessions (which I complained about a year and a half ago) and knock it off with offering DNS service that I did not ask for.
Sincerely,
Your Disgruntled Customer, RS
PS: No, I don't expect that this open letter will get you to fix the misbehavior, but if some Swedish guy comes along swinging a clue-bat at you guys I hope he whacks you a couple of times for me.
Can you be more specific? My TCP/465 and TCP/587 mail submission works great over Sprint. I'm not even trying to do submission on port 25 (in fact, my mail servers send rude messages if you try AUTH to a port 25 listener) so I can't speak to that. -r Owen DeLong <owen@delong.com> writes:
While you're at it, it would be nice if SPRINT also fixed the problems with ports TCP/25 and TCP/587.
Another disgruntled SPRINT customer,
Owen
On May 14, 2009, at 10:48 AM, Robert E. Seastrom wrote:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
Queries from my lab:
rs@click [14] % dig +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [15] % dig +tcp +short @192.148.252.10 version.bind. chaos txt "Just send your damn query already..." rs@click [16] % dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "bifrost" rs@click [17] %
Queries from my laptop:
Superfly:~ rs$ dig +short @192.148.252.10 version.bind. chaos txt "9.6.0-P1" Superfly:~ rs$ dig +tcp +short @192.148.252.10 version.bind. chaos txt ;; connection timed out; no servers could be reached Superfly:~ rs$ dig +tcp +short @192.148.252.10 hostname.bind. chaos txt "ns1-kscymar06.spcsdns.net" Superfly:~ rs$
Guys, I send you money each month to deliver packets for me, not to invent new ways of being annoying (and breaking TSIG signed updates to dynamic DNS). Less is more. Please stop dinking with 10-minute-idle TCP sessions (which I complained about a year and a half ago) and knock it off with offering DNS service that I did not ask for.
Sincerely,
Your Disgruntled Customer, RS
PS: No, I don't expect that this open letter will get you to fix the misbehavior, but if some Swedish guy comes along swinging a clue-bat at you guys I hope he whacks you a couple of times for me.
Owen DeLong wrote:
While you're at it, it would be nice if SPRINT also fixed the problems with ports TCP/25 and TCP/587.
Never tried 25, but I know 587 is fine through a tethered handset my (extremely non-technical) significant other uses daily. Shouldn't we all be using the submission port anyway? ;) ~Seth
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too. If you're aware of a mechanical way for them to tell the difference, we're all ears. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
I agree, running monitoring from my laptop at home at nights/weekends/vacations/holidays... I need to use most of those ports. My answer was VNP/tunnel everything. -----Original Message----- From: John Levine [mailto:johnl@iecc.com] Sent: Thursday, May 14, 2009 6:36 PM To: nanog@nanog.org Cc: rs@seastrom.com Subject: you're not interesting, was Re: another brick in the wall[ed garden]
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too. If you're aware of a mechanical way for them to tell the difference, we're all ears. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
I use SSH tunnels for all mail, but I have had no problems with DNS over Sprint EVD0 (the OP's issue). Regards Marshall On May 14, 2009, at 6:48 PM, Dave Larter wrote:
I agree, running monitoring from my laptop at home at nights/weekends/vacations/holidays... I need to use most of those ports. My answer was VNP/tunnel everything.
-----Original Message----- From: John Levine [mailto:johnl@iecc.com] Sent: Thursday, May 14, 2009 6:36 PM To: nanog@nanog.org Cc: rs@seastrom.com Subject: you're not interesting, was Re: another brick in the wall[ed garden]
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
If you're aware of a mechanical way for them to tell the difference, we're all ears.
Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
In message <20090514223605.88104.qmail@simone.iecc.com>, John Levine writes:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
And what's the next protocol that is going to be stomped on?
If you're aware of a mechanical way for them to tell the difference, we're all ears.
Well you can't answer a TSIG message without knowing the shared secret so you might as well just let it go through and avoid some percentage of support calls. Intercepting TSIG messages is guaranteed to generate a support call. Similarly intercepting "rd=0" is also guaranteed to generate a support call. You almost certainly have a interative resolver making the query which will not handle the "aa=0" responses. Similarly there is no sane reason to block DNS/TCP other than they can do it. Mark
Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies ", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
Disclaimer: I have a dog in this fight, since ThreatSTOP is dependent on DNS/TCP.
-----Original Message----- From: Mark Andrews [mailto:Mark_Andrews@isc.org] Sent: Thursday, May 14, 2009 4:59 PM To: John Levine Cc: nanog@nanog.org; rs@seastrom.com Subject: Re: you're not interesting,was Re: another brick in the wall[ed garden]
In message <20090514223605.88104.qmail@simone.iecc.com>, John Levine writes:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
And what's the next protocol that is going to be stomped on?
If you're aware of a mechanical way for them to tell the difference, we're all ears.
Well you can't answer a TSIG message without knowing the shared secret so you might as well just let it go through and avoid some percentage of support calls. Intercepting TSIG messages is guaranteed to generate a support call.
Similarly intercepting "rd=0" is also guaranteed to generate a support call. You almost certainly have a interative resolver making the query which will not handle the "aa=0" responses.
Similarly there is no sane reason to block DNS/TCP other than they can do it.
[TLB:] I can think of an argument they might make: that it is/could be used by bots as a fallback. However, redirecting DNS/UDP fits the model of "providing a better service for the average user"; blocking/redirecting TCP is more likely to break things a savvy user needs. Maybe someone with clue at Sprint can be persuaded that doing their own "OpenDNS" for UDP is probably a good thing for most uses, but doing it for TCP is a bad thing for those users who need TCP.
In message <70D072392E56884193E3D2DE09C097A91F3EBE@pascal.zaphodb.org>, "Tomas L. Byrnes" writes:
Disclaimer: I have a dog in this fight, since ThreatSTOP is dependent on DNS/TCP.
-----Original Message----- From: Mark Andrews [mailto:Mark_Andrews@isc.org] Sent: Thursday, May 14, 2009 4:59 PM To: John Levine Cc: nanog@nanog.org; rs@seastrom.com Subject: Re: you're not interesting,was Re: another brick in the wall[ed garden]
In message <20090514223605.88104.qmail@simone.iecc.com>, John Levine writes:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
And what's the next protocol that is going to be stomped on?
If you're aware of a mechanical way for them to tell the difference, we're all ears.
Well you can't answer a TSIG message without knowing the shared secret so you might as well just let it go through and avoid some percentage of support calls. Intercepting TSIG messages is guaranteed to generate a support call.
Similarly intercepting "rd=3D0" is also guaranteed to generate a support call. You almost certainly have a interative resolver making the query which will not handle the "aa=3D0" responses.
Similarly there is no sane reason to block DNS/TCP other than they can do it.
[TLB:] I can think of an argument they might make: that it is/could be used by bots as a fallback. However, redirecting DNS/UDP fits the model of "providing a better service for the average user"; blocking/redirecting TCP is more likely to break things a savvy user needs.
There is still no sane reason to block TCP. If they are intercepting DNS/UDP then they need to also intercept DNS/TCP as they will break all sites that cause "tc=1" to be set in the DNS/UDP reply.
Maybe someone with clue at Sprint can be persuaded that doing their own "OpenDNS" for UDP is probably a good thing for most uses, but doing it for TCP is a bad thing for those users who need TCP.
You can also add to the list of queries that you need to provide a clean path for any with DO=1, CD=1 or AD=1. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
On May 14, 2009, at 8:37 PM, Mark Andrews wrote:
[TLB:] I can think of an argument they might make: that it is/could be used by bots as a fallback. However, redirecting DNS/UDP fits the model of "providing a better service for the average user"; blocking/redirecting TCP is more likely to break things a savvy user needs.
There is still no sane reason to block TCP. If they are intercepting DNS/UDP then they need to also intercept DNS/TCP as they will break all sites that cause "tc=1" to be set in the DNS/UDP reply.
First, since when does it require a "sane" reason to do something? Second, and more importantly, John is right. Sprint is a for-profit business. If blocking UDP - or TCP or HTTP or whatever - makes them more money than not blocking it, they will do it. And rightly so. Of course, it is entirely possible management figured out blocking "DNS" was more profitable because the cost savings in lower call center volume more than offset the 4 people who dropped Sprint because of the block. So they told engineers to 'block DNS' and the engineers did that without knowing that blocking TCP port 53 was not more profitable, and perhaps was less profitable. Miscommunications abound between Engineering and Management. This should surprise few, and hopefully no one on NANOG. Assuming something like that happened, will a post to NANOG fix it? I don't know. Certainly has a non-zero chance. But trying to get Sprint, or any provider, to change because _you_ think what they are doing is not sane is, well, not sane. "Never appeal to a man's 'better nature,' he may not have one. Invoking his self-interest gives you more leverage." -- TTFN, patrick
On Sat, 16 May 2009, Patrick W. Gilmore wrote:
Assuming something like that happened, will a post to NANOG fix it? I don't know. Certainly has a non-zero chance. But trying to get Sprint, or any provider, to change because _you_ think what they are doing is not sane is, well, not sane.
In '02, I had a similar issue with Comcast, when they silently fired up transparent proxy servers. It became apparent when, while working on a remote web server, I was served up cached copies of the pages I was editing. My approach was two-pronged. First, I bitched loud and long on some security lists about the MITM attack. Not only was it abusive as it was, the potential for further abuse (tracking, ad insertion, theft of sensitive data and intellectual property...) was significant. Eventually, Ted Bridis of Associated Press picked it up and ran a story. The next day, the issue was on the front page of nearly every newspaper in the english speaking world, and then some, as well as network TV news. Comcast has a large customer base, particularly in the DC area, and a lot of very influential people (like federal judges) were not fond of having their research and recreational web surfing intercepted. The proxies went away within a few days, and several jurisdictions passed laws prohibiting this. I'd suspect Sprint is violating some of these laws. The other approach was; I sent exploit code addressed to one of my machines. Comcast's servers stole this code and choked on it. It's probably not illegal to send malicious code to a machine you own. If they stole it and choked on it, it's their problem. But with the legal system the way it is, you'll just have to use your imagination until the statute of limitations expires. Cheers, George
On May 17, 2009, at 4:34 AM, George Imburgia wrote:
On Sat, 16 May 2009, Patrick W. Gilmore wrote:
Assuming something like that happened, will a post to NANOG fix it? I don't know. Certainly has a non-zero chance. But trying to get Sprint, or any provider, to change because _you_ think what they are doing is not sane is, well, not sane.
In '02, I had a similar issue with Comcast, when they silently fired up transparent proxy servers. It became apparent when, while working on a remote web server, I was served up cached copies of the pages I was editing.
My approach was two-pronged. First, I bitched loud and long on some security lists about the MITM attack. Not only was it abusive as it was, the potential for further abuse (tracking, ad insertion, theft of sensitive data and intellectual property...) was significant. Eventually, Ted Bridis of Associated Press picked it up and ran a story. The next day, the issue was on the front page of nearly every newspaper in the english speaking world, and then some, as well as network TV news.
Comcast has a large customer base, particularly in the DC area, and a lot of very influential people (like federal judges) were not fond of having their research and recreational web surfing intercepted.
Then they were silly to think turning off the transparent proxies somehow allowed them not to be tracked. But then, most "influential people" are, at the very least, ignorant of technology.
The proxies went away within a few days, and several jurisdictions passed laws prohibiting this. I'd suspect Sprint is violating some of these laws.
You gave them a business reason (cost more to keep them then turn them off) to change their mind. Good for you. I doubt the same is true for Sprint modulo the laws you mention. And I'm wondering what laws these are, since intercepting port 43 is an extremely common practice. -- TTFN, patrick
On Thu, May 14, 2009 at 4:58 PM, Mark Andrews <Mark_Andrews@isc.org> wrote:
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
And what's the next protocol that is going to be stomped on?
I was going to say, "will the ISP also remove the DNS MITM the day that 99.9% of malware moves its command-and-control to the HTTP or other layer?". I figured why bother - but your point drives it home even further. dre
Subject: Re: you're not interesting, was Re: another brick in the wall[ed garden] Date: Fri, May 15, 2009 at 09:58:32AM +1000 Quoting Mark Andrews (Mark_Andrews@isc.org):
And what's the next protocol that is going to be stomped on?
Anything except http; at which point everything will move to http, and the firewalls are again useless. -- Måns Nilsson
And what's the next protocol that is going to be stomped on?
Anything except http; at which point everything will move to http, and the firewalls are again useless.
Um, if you think that http on consumer networks is transparent, I have some really bad news for you. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
Subject: Re: you're not interesting, was Re: another brick in the wall[ed garden] Date: Fri, May 15, 2009 at 10:10:26AM +0100 Quoting John R. Levine (johnl@iecc.com):
And what's the next protocol that is going to be stomped on?
Anything except http; at which point everything will move to http, and the firewalls are again useless.
Um, if you think that http on consumer networks is transparent, I have some really bad news for you.
If you change my sentence thusly: s/http/ what can be communicated over http/1;s/http/said channel/2; ..it all makes sense again, even "better" than before. -- Måns Nilsson
On May 14, 2009, at 10:07 PM, Mans Nilsson wrote:
Subject: Re: you're not interesting, was Re: another brick in the wall[ed garden] Date: Fri, May 15, 2009 at 09:58:32AM +1000 Quoting Mark Andrews (Mark_Andrews@isc.org):
And what's the next protocol that is going to be stomped on?
Anything except http; at which point everything will move to http, and the firewalls are again useless.
This is, indeed, already happening. In fact, I'm running an SMTP server with TLS on port 80 to get around SPRINT's existing braindamage. (or at least the braindamage they had at one point). Owen
-- Måns Nilsson
Anything traversing the edge. They are all revenue targets. Best, Martin On 5/14/09, Mark Andrews <Mark_Andrews@isc.org> wrote:
In message <20090514223605.88104.qmail@simone.iecc.com>, John Levine writes:
Dear Sprint EVDO people,
Your man-in-the-middle hijacking of UDP/53 DNS queries against nameservers that I choose to query from my laptop on Sprint EVDO is not appreciated. Even less appreciated is your complete blocking of TCP/53 DNS queries.
If I were an ISP, and I knew that approximately 99.9% of customer queries to random name servers was malware doing fake site phishing or misconfigured PCs that will work OK and avoid a support call if they answer the DNS query, with 0.1% being old weenies like us, I'd do what Sprint's doing, too.
And what's the next protocol that is going to be stomped on?
If you're aware of a mechanical way for them to tell the difference, we're all ears.
Well you can't answer a TSIG message without knowing the shared secret so you might as well just let it go through and avoid some percentage of support calls. Intercepting TSIG messages is guaranteed to generate a support call.
Similarly intercepting "rd=0" is also guaranteed to generate a support call. You almost certainly have a interative resolver making the query which will not handle the "aa=0" responses.
Similarly there is no sane reason to block DNS/TCP other than they can do it.
Mark
Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies ", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
-- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
participants (14)
-
Andre Gironda
-
Dave Larter
-
George Imburgia
-
John Levine
-
John R. Levine
-
Mans Nilsson
-
Mark Andrews
-
Marshall Eubanks
-
Martin Hannigan
-
Owen DeLong
-
Patrick W. Gilmore
-
Robert E. Seastrom
-
Seth Mattinen
-
Tomas L. Byrnes