May 2008 - Test - lists.nanog.org

[NANOG] Microsoft.com PMTUD black hole?
by Nathan Anderson/FSR 08 May '08

08 May '08

Hello, Has anyone else here seen problems with microsoft/msn/hotmail/live.com sites not performing PMTUD correctly? We have, for a while now, had people on our network complain of poor microsoft.com reachability, and discovered we can work around the issue by changing MSS on all TCP SYN as they go out of our network. I recently watched the whole conversation between msn.com and a host on our network (with the MSS rewrite disabled), and if I'm reading it right, we are following PMTUD protocol correctly by sending back ICMP type 3 code 4, but all Microsoft hosts seem to ignore this and continue to send packets back to our host with an MSS that is too large. I hope I'm wrong and that it is we who are doing something stupid, but after cruising Google for a while, I found a multitude of other complaints from people connected to other ISPs specifically about not being able to reach Microsoft web sites. It seems crazy that MS could have PMTUD broken for so long with nobody ever raising a complaint to them directly, though, which makes me wonder if there is another answer here that I'm missing. I sent the following message to a couple of addresses that I gleaned from ARIN WHOIS for the IP block in question and threw hostmaster in there just in case it went somewhere, but noc(a)microsoft.com appears to be defunct. I have yet to receive acknowledgment of receipt from the other address. Are there any microsoft.com admins that hang out here that can comment on this or get in touch with me, or is there perhaps someone on here with connections to the Microsoft NOC? (BTW, I stripped the referenced libpcap attachment off of this message to the list just so that I wouldn't accidentally incur the wrath of NANOG...if y'all want to see it, I'm happy to post it.) Thanks, -- Nathan Anderson First Step Internet, LLC nathana(a)fsr.com -------- Original Message -------- Subject: Microsoft/MSN/Live!/Hotmail behind blackhole router? Date: Thu, 01 May 2008 19:00:46 -0700 From: Nathan Anderson/FSR <nathana(a)fsr.com> To: hostmoaster(a)microsoft.com, noc(a)microsoft.com, iprrms(a)microsoft.com To microsoft.com NOC admins: I work for a regional ISP in the inland pacific northwest. May of our customers' connections have MTUs of less than 1500, and we get routine complaints from them that they have trouble reaching web sites that are under your administration. Usually we can fix the problem by "mangling" the TCP SYNs originating from our customers and headed to the world to reflect a lower value; however, we would rather not have to do that. The fact that we are REQUIRED to do this in order for your sites to be reachable by our customers strongly suggests that either the servers that respond to HTTP requests sent to www.microsoft/msn/hotmail/live.com are behind routers that are blocking ALL ICMP traffic sent their way -- even ICMP type 3 code 4 (packet too large, DF set), which is necessary in order for Path MTU Discovery to work -- or the servers themselves are not listening to the ICMP messages that we are sending their way when our routers are forced to drop a packet sent by you which is too large to be forwarded to a customer of ours. I set up a test connection "on the bench" so to speak, and had our router capture a copy of the conversation between our test client and www.msnbc.msn.com and forward that conversation encapsulated in TZSP to the same test client over a different interface. The capture clearly shows our test client establishing the TCP connection with MSNBC (SYN/SYN+ACK/ACK), and then goes on to show MSNBC send ethernet MTU-sized packets our way that an intermediate router of ours drops and responds with "packet too big, DF set." Despite this, MSNBC continues to retrasmit the original packet with the same payload and the same size back to us. We continue to respond "packet too big, DF set," but the MSNBC server never seems to get the message (literally). We see the same behavior with all sites across the board contained within the 207.46.0.0/16 space, regardless of actual hostname/FQDN. We also find this ironic considering that Microsoft published a Technet article a few years back on black hole routers and the problems they pose, found at http://technet.microsoft.com/en-us/library/bb878081.aspx (which we can't read/access unless we are mangling the MSS). We would appreciate it if Microsoft NOC admins would please look into the matter and take the appropriate corrective action: allowing ICMP type 3 code 4 messages through your routers/firewalls, and making sure that your servers respond to them appropriately as defined in RFC 1191. I have attached the capture we made of the conversation to this e-mail message in libpcap format for your analysis. The test client itself had a 1500 MTU to a desktop router, which in turn had an MTU of 1492 on its uplink to us. I am available to answer any additional clarifying questions you may have. Thank you for your time and attention to this matter. Regards, -- Nathan Anderson First Step Internet, LLC nathana(a)fsr.com

3 2

Re: [NANOG] Microsoft.com PMTUD black hole?
by Scott Weeks 08 May '08

08 May '08

---------- blaine(a)blaines.net wrote: --------------- First of all I would like to thank everyone for their support and concern. We certainly have a lot of things to "fix" at Microsoft. In fact, I can tell you that we have several brand new positions open (working on my team and for teams near mine) and could use more hands at the tiller. My apologies to the moderators for posting a help wanted but I figure since folks are expressing concerns with Microsoft networking they should have the opportunity to come over and help. --------------------------------------------------- Not support. Concern about M$ on soooo many levels. Your apologies should go to the other 10,000 folks that had to endure your improper email: http://www.nanog.org/listfaq.html#nocjobs -------------------------------------------- BOSD Network Engineer 231413 BOSD Senior Network Engineer 231414 -------------------------------------------- heh, you mean BSOD engineer? ;-) scott ----------------------------

1 0

[NANOG] PCH BGP Archive down?
by NetSecGuy 08 May '08

08 May '08

Anyone else not able to access the PCH BGP Archive? http://www.pch.net/resources/data/routing-tables/archive/ I'm getting a 403 Forbidden.

2 2

[NANOG] Looking for Flickr contacts
by gbromley＠intstar.com 08 May '08

08 May '08

I am having a bit of difficulty resolving a Flickr issue. Can someone point me to contact in Flickr? Cheers Gareth

1 0

[NANOG] Charter - Southern Oregon routing issues
by S. Ryan 07 May '08

07 May '08

I've called Charter and they deny they have an issue. I'm looking to speak to a Charter engineer who can specifically help with issues in the Southern Oregon service area. Multiple customers cannot check mail on 110 or use port 25 or 587. Looks like a loop: traceroute to 204.16.46.136 (204.16.46.136), 30 hops max, 40 byte packets 1 192.168.1.1 (192.168.1.1) 2.143 ms 1.168 ms 0.849 ms 2 10.218.0.1 (10.218.0.1) 8.523 ms 8.551 ms 14.098 ms 3 ge5-2.cr01.mdfd.or.charter.com (68.185.17.113) 31.033 ms 10.993 ms 9.398 ms 4 68-116-79-133.or.charter.com (68.116.79.133) 11.145 ms 18.148 ms 10.23 ms 5 68-116-79-134.or.charter.com (68.116.79.134) 19.288 ms 9.768 ms 14.054 ms 6 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 10.881 ms 12.468 ms 10.898 ms 7 68-116-79-134.or.charter.com (68.116.79.134) 16.426 ms 10.631 ms 11.4 ms 8 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 9.317 ms 10.047 ms 9.315 ms 9 68-116-79-134.or.charter.com (68.116.79.134) 11.117 ms 9.992 ms * 10 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 10.928 ms 20.391 ms 9.513 ms 11 68-116-79-134.or.charter.com (68.116.79.134) 11.349 ms 10.413 ms 10.222 ms 12 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 20.722 ms 9.746 ms 9.99 ms 13 68-116-79-134.or.charter.com (68.116.79.134) 10.682 ms 11.198 ms 11.253 ms 14 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 10.376 ms 21.943 ms 10.914 ms 15 68-116-79-134.or.charter.com (68.116.79.134) 11.039 ms 9.778 ms 17.995 ms 16 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 13.068 ms 10.622 ms 13.206 ms 17 68-116-79-134.or.charter.com (68.116.79.134) 11.783 ms 11.17 ms 9.351 ms 18 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 15.859 ms 13.637 ms 18.188 ms 19 68-116-79-134.or.charter.com (68.116.79.134) 9.532 ms 14.751 ms 16.703 ms 20 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 21.073 ms 11.547 ms 10.913 ms 21 68-116-79-134.or.charter.com (68.116.79.134) 18.878 ms 9.844 ms 17.044 ms 22 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 17.192 ms 12.964 ms 10.855 ms 23 68-116-79-134.or.charter.com (68.116.79.134) 9.931 ms 13.66 ms 17.299 ms 24 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 21.794 ms 11.108 ms 16.129 ms 25 68-116-79-134.or.charter.com (68.116.79.134) 17.288 ms 14.043 ms 12.522 ms 26 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 11.771 ms 14.531 ms 11.148 ms 27 68-116-79-134.or.charter.com (68.116.79.134) 11.697 ms 10.318 ms 9.069 ms 28 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 9.885 ms 15.104 ms 13.287 ms 29 68-116-79-134.or.charter.com (68.116.79.134) 21.404 ms 12.868 ms 10.294 ms 30 g1-0.er01.mdfd.or.charter.com (68.116.79.1) 10.807 ms 11.742 ms 12.197 ms Thank you for any assistance. I've even contacted Tier III at Charter and they're unwilling to help at all. I'm shocked they have a customer base still. Regards, Steve Ryan 541-842-8207

1 1

Re: [NANOG] Microsoft.com PMTUD black hole?
by Michael Sinatra 07 May '08

07 May '08

Kevin Oberman wrote: >> I agree with Iljitsch that it happens frequently, but I think I am >> justified in expecting more than that from Microsoft. Anything less >> would be unprofessional. > > And you would consider an organization that threatens someone who > complains publicly about its obvious incompetence "professional"? Absolutely not. That was actually the point of my statement, although I admit that it wasn't clear. > Many of Microsoft's people are highly professional, but the corporation, > as a whole, has been found to be large scale law breakers on two > continents and frequently incapable of even the most basic of technical > operations. I'm afraid that I don't see them as at all "professional". I > quit expecting any such behavior from them over a decade ago, probably > closer to two. And mentioning security and Microsoft is inviting bad > jokes and shudders. > > * Speaking only for myself * Agreed. michael

1 0

Re: [NANOG] Microsoft.com PMTUD black hole?
by Brandon Butterworth 07 May '08

07 May '08

> Has anyone else here seen problems with microsoft/msn/hotmail/live.com > sites not performing PMTUD correctly? I used to see it a lot when hosting on windows was popular and people realised they needed a firewall or decided to add a load balancer but broke PMTUD by leaving it enabled on the servers. I've not heard of it for some time so those people got a clue or moved to something else (or everyone worked around them) brandon

4 7

[NANOG] Google Contact
by Shane Thorson 07 May '08

07 May '08

All; I am having a bit of difficulty resolving a Google matter. I am hoping to get pointed in the right direction by someone from Google. Thanks ---------------------------------------------------------------------- NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

1 0

[NANOG] Bob Crooks/SaskTel/CA is out of the office.
by Bob Crooks 07 May '08

07 May '08

I will be out of the office starting 05/07/2008 and will not return until 05/12/2008.

1 0

Re: [NANOG] Microsoft.com PMTUD black hole?
by Iljitsch van Beijnum 07 May '08

07 May '08

On 6 mei 2008, at 23:48, Valdis.Kletnieks(a)vt.edu wrote: >> A more common approach is to rewrite the MSS option in all TCP SYNs >> with a smaller value so there won't be TCP segments large enough to >> trigger the problem. AFAIK, all boxes that do PPPoE do this. > And just the other day, you were saying: >> Very few people out there use an MTU significantly below 1500 >> bytes. A >> 1500-byte MTU will give you an _average_ packet size of ~1000 on >> long- >> lived TCP flows because there is one tiny ACK for every two full size >> data segments. Right. Why is that noteworthy? I have a lot more to say about MTU issues in this draft about negotating MTUs between two hosts/routers on a subnet so jumboframes can be deployed without manual configuration: http://www.ietf.org/internet-drafts/draft-van-beijnum-multi-mtu-02.txt > Apparently, there's a *reason* why RFC1122, section 3.3.3 says: > It is generally desirable to avoid local fragmentation and to > choose EMTU_S low enough to avoid fragmentation in any gateway > along the path. In the absence of actual knowledge of the > minimum MTU along the path, the IP layer SHOULD use > EMTU_S <= 576 whenever the destination address is not on a > connected network, and otherwise use the connected network's > MTU. Tell it to Microsoft and their ICMP-filtering friends...

1 0