Re: [NANOG] Microsoft.com PMTUD black hole?
`
Date: Tue, 06 May 2008 14:29:03 -0700 From: Nathan Anderson/FSR <nathana@fsr.com> Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
Now, although that makes sense, in order to avoid issues like the one we are facing with Microsoft, would it not make _more_ sense for the stack to look at the PMTU cache first, and then adjust its own MSS just for connections to that one host?
This _is_ Microsoft we're talking about, remember. 'sense' and 'Microsoft' are, at a =minimum= orthogonal to each other -- and may not even inhabit the same address-space. <wry grin> As for standards, it is official Microsoft policy to "embrace and extend", not to implement in a way compatible with the rest of the world. *sigh* I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends incrementally increasing-size packets, and uses the first one that -doesn't- get through as the size limit. <giggle>
Interestingly, Windows XP, Sp3, released today, describes changes in PMTUD behavior. Black Hole Router detection is now on by default: http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02 b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf
-----Original Message----- From: Robert Bonomi [mailto:bonomi@mail.r-bonomi.com] Sent: Tuesday, May 06, 2008 3:54 PM To: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
`
Date: Tue, 06 May 2008 14:29:03 -0700 From: Nathan Anderson/FSR <nathana@fsr.com> Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
Now, although that makes sense, in order to avoid issues like the one we are facing with Microsoft, would it not make _more_ sense for the stack to look at the PMTU cache first, and then adjust its own MSS just for connections to that one host?
This _is_ Microsoft we're talking about, remember. 'sense' and 'Microsoft' are, at a =minimum= orthogonal to each other -- and may not even inhabit the same address-space. <wry grin>
As for standards, it is official Microsoft policy to "embrace and extend", not to implement in a way compatible with the rest of the world. *sigh*
I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends incrementally increasing-size packets, and uses the first one that -doesn't- get through as the size limit. <giggle>
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
On May 6, 2008, at 8:51 PM, Tomas L. Byrnes wrote:
Interestingly, Windows XP, Sp3, released today, describes changes in PMTUD behavior.
Black Hole Router detection is now on by default:
http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02 b40c12982/Overview%20of%20Windows%20XP%20Service%20Pack%203.pdf
<http://download.microsoft.com/download/6/8/7/687484ed-8174-496d-8db9-f02b40c...
or http://tinyurl.com/323xb Regards
-----Original Message----- From: Robert Bonomi [mailto:bonomi@mail.r-bonomi.com] Sent: Tuesday, May 06, 2008 3:54 PM To: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
`
Date: Tue, 06 May 2008 14:29:03 -0700 From: Nathan Anderson/FSR <nathana@fsr.com> Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
Now, although that makes sense, in order to avoid issues like the one we are facing with Microsoft, would it not make _more_ sense for the stack to look at the PMTU cache first, and then adjust its own MSS just for connections to that one host?
This _is_ Microsoft we're talking about, remember. 'sense' and 'Microsoft' are, at a =minimum= orthogonal to each other -- and may not even inhabit the same address-space. <wry grin>
As for standards, it is official Microsoft policy to "embrace and extend", not to implement in a way compatible with the rest of the world. *sigh*
I -don't- believe the rumor that "PMTUD/Vista Ultimate" sends incrementally increasing-size packets, and uses the first one that -doesn't- get through as the size limit. <giggle>
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Tomas L. Byrnes wrote:
Interestingly, Windows XP, Sp3, released today, describes changes in PMTUD behavior.
Black Hole Router detection is now on by default:
As I pointed out in my post earlier today timestamped at 2:29PM, I was using an XP SP3 host to perform my tests with, and it made no difference. I also used BBR's DrTCP application to make sure that black hole router detection was, in fact, enabled on my XP box before commencing my packet captures. I cannot explain why it made no difference, but at the same time I don't know enough about how WinNT's black hole router detection works to begin speculating at this point. I do plan on looking into it, however. -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
All, A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen. Therefore, I would like to publicly apologize for my actions here. It was not my intention to "humiliate" Microsoft into compliance but rather to find a means of effective contact with them since none was to be found before today. However, I recognize that I did step over the line, especially with regards to one comment I made in an earlier post about "giving Microsoft too much credit." I apologize for this and retract this, and ask their forgiveness. As I promised, I will not be posting any more to this list regarding this issue unless it is to report the final verdict that I receive from my now-open ticket with Microsoft (thanks to this list, I found an effective contact), or to discuss the mechanics of PMTUD in general. Regards, -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
they try that intimidation every time a vulnerability or bug is revealed. laugh and post their overly-aggressive message on a public web site. randy
Nathan Anderson/FSR wrote:
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
Hang on a tick. Aren't you one of their customers...<looking through mail spool>...
As I pointed out in my post earlier today timestamped at 2:29PM, I was using an XP SP3 host to perform my tests with...
...why yes, you are. I can't think of any other supplier that would be so unprofessional and so lacking in business acumen as to say that their customer was UALIBI. Amazing. A fine case study of a person in customer contact undoing the work of millions of dollars in PR. Whatever you say about Steve Ballmer he's a great sales person at heart. He must despair at some of his staff. -- Glen Turner
On 07/05/2008, at 4:42 PM, Glen Turner wrote:
Amazing. A fine case study of a person in customer contact undoing the work of millions of dollars in PR.
I wouldn't worry too much about it, Glen. My observation is that the millions of dollars in PR isn't working very well either :-) - mark -- Mark Newton Email: newton@internode.com.au (W) Network Engineer Email: newton@atdot.dotat.org (H) Internode Systems Pty Ltd Desk: +61-8-82282999 "Network Man" - Anagram of "Mark Newton" Mobile: +61-416-202-223
Glen Turner wrote:
Amazing. A fine case study of a person in customer contact undoing the work of millions of dollars in PR. Whatever you say about Steve Ballmer he's a great sales person at heart. He must despair at some of his staff.
The rest of us however, despair at having to support their crap. Patrick Giagnocavo patrick@zill.net
On Tue, May 06, 2008 at 06:12:42PM -0700, Nathan Anderson/FSR wrote:
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
This is a typical Microsoft reaction: blame the messenger for their own incompetence, laziness, stupidity, and greed. I think you should post their assinine message so that it can receive the public ridicule it surely deserves. ---Rsk
Here is a brief update on the situation: I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security. Nevertheless, the person I have been in contact with is naturally not the final decision-maker on this issue and is going to continue to pass the issue on up the chain of command for me. So although this issue is not over and I do not have a final verdict from MS yet, I felt that, given that I don't know how much time to expect to pass between now and when that final verdict is rendered, it would be appropriate to let everybody here know what I have learned thus far. Hopefully public dissemination of this information factoid will prevent others in a position similar to mine from having to helplessly beat their heads into their keyboards. I, naturally, voiced my strong objection over this security policy, and attempted to make a reasoned argument with the contact I have over there. We will see what comes of this. Some have asked me to post copies of my private communication with my Microsoft contact here. I don't think it is appropriate for me to post copies of private communication without the other party's consent, so I will have to decline unless he first gives me said consent. Others have asked for valid contact information for the Microsoft NOC, since the ARIN records for their 207.46.0.0/16 do not appear to be up to date. I eventually found a working e-mail address from somebody off-list who pointed to the WHOIS lookup from TUCOWS for microsoft.comosoft.com (which I'm still not clear on what exactly this is...). The e-mail address that was gleaned from this lookup was msnhst@microsoft.com, which goes to the Microsoft Corporate Domains Team. They, in turn, forwarded my message on to msnalerts@microsoft.com, which generated a ticket # for me and is, as I understand it, the e-mail address I was looking for in the first place (leads to their network/system people). I hope this is helpful to others. Regards, -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
Nathan Anderson/FSR wrote:
Here is a brief update on the situation:
I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Although the need for your previous apology has already been questioned in this forum, the confirmation that they block not only certain ICMP types, but all ICMP, further vacates the need for any apology for criticizing this behavior in a pubic forum. It is disheartening for those of us who use and support MSFT's products to learn that their understanding of security lacks even the basic nuance to know not to block an entire--critical--portion of the Internet Protocol. Perhaps they should also block _all_ TCP and UDP as well, and then we can move on. I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional. *Speaking for myself only, of course!* michael
On 7 mei 2008, at 21:46, Michael Sinatra wrote:
MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Perhaps they should also block _all_ TCP and UDP as well, and then we can move on.
I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional.
Right. Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.) However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY! You can't have your cake and eat it too.
The remedy you have below is NOT the only one, and is, in fact, a non-sequitur in this case. PMTUD uses the DF (for Don't_Fragment) bit, and works by getting an ICMP Fragmentation needed response from the hop on the path where the packet is too large, not a fragmentation and forward, so the union of PMTUD packets and fragmented ones is 0. The network-level solution to ping of death is to BLOCK fragmented packets, and the way to ensure this doesn't self-deny-service is to perform PMTUD and Black-Hole Router discovery.
-----Original Message----- From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] Sent: Wednesday, May 07, 2008 1:35 PM To: Michael Sinatra Cc: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 21:46, Michael Sinatra wrote:
MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Perhaps they should also block _all_ TCP and UDP as well, and then we can move on.
I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional.
Right.
Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.)
However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY!
You can't have your cake and eat it too.
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Tomas L. Byrnes wrote:
The remedy you have below is NOT the only one, and is, in fact, a non-sequitur in this case.
How so? Iljitsch is suggesting that ICMP blockers originate packets without DF set if they are going to block the ICMP messages that PMTUD needs in order to work in the first place. That's what (I think) he means by "disabling path MTU discovery."
The network-level solution to ping of death is to BLOCK fragmented packets, and the way to ensure this doesn't self-deny-service is to perform PMTUD and Black-Hole Router discovery.
Which end are you talking about here, the servers or the client? If the servers, how do you expect them to do PMTUD if they _can't hear the ICMP messages_? Also, for some reason, as I pointed out before, XP black hole router discovery doesn't seem to be working for me for whatever reason. Does anybody have any clue why that might be the case? -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
On 7 mei 2008, at 23:08, Nathan Anderson/FSR wrote:
How so? Iljitsch is suggesting that ICMP blockers originate packets without DF set if they are going to block the ICMP messages that PMTUD needs in order to work in the first place. That's what (I think) he means by "disabling path MTU discovery."
Yes.
Also, for some reason, as I pointed out before, XP black hole router discovery doesn't seem to be working for me for whatever reason. Does anybody have any clue why that might be the case?
The problem is in the direction from M$ to you, so you can't fix that from your end. I wonder if they've installed SP3 on their servers...
Iljitsch van Beijnum wrote:
The problem is in the direction from M$ to you, so you can't fix that from your end. I wonder if they've installed SP3 on their servers...
Ah, you are right. I re-read the section on black-hole detection in http://technet.microsoft.com/en-us/library/bb878081.aspx more closely this time, and found that, yes, it only helps if the host trying to send the large packets has the feature enabled: "When PMTU black hole router detection is enabled, TCP tries to send segments with the DF flag set to 0 after several retransmissions of a segment are not acknowledged. If a segment with the DF flag set to 0 is acknowledged, the MSS is decreased and the DF flag is set to 1 in subsequent segments on the connection. Enabling PMTU black hole detection increases the maximum number of retransmissions that are performed for a given segment, and therefore has an effect on overall performance." I for some reason interpreted the advertisement of the black hole detection feature as being a help to clients impacted by the inability of the server to perform PMTUD. -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
I was responding to his post that blocking or disabling PMTUD was the way to avoid the ping of death, which is False, nothing more, nothing less. As far as who Iljitsch is, everyone misspeaks from time to time. Even those of us who have been at this for nearly 3 decades.
-----Original Message----- From: Nathan Anderson/FSR [mailto:nathana@fsr.com] Sent: Wednesday, May 07, 2008 2:08 PM To: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
Tomas L. Byrnes wrote:
The remedy you have below is NOT the only one, and is, in fact, a non-sequitur in this case.
How so? Iljitsch is suggesting that ICMP blockers originate packets without DF set if they are going to block the ICMP messages that PMTUD needs in order to work in the first place. That's what (I think) he means by "disabling path MTU discovery."
The network-level solution to ping of death is to BLOCK fragmented packets, and the way to ensure this doesn't self-deny-service is to perform PMTUD and Black-Hole Router discovery.
Which end are you talking about here, the servers or the client? If the servers, how do you expect them to do PMTUD if they _can't hear the ICMP messages_?
Also, for some reason, as I pointed out before, XP black hole router discovery doesn't seem to be working for me for whatever reason. Does anybody have any clue why that might be the case?
-- Nathan Anderson First Step Internet, LLC nathana@fsr.com
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
On 7 mei 2008, at 23:20, Tomas L. Byrnes wrote:
I was responding to his post that blocking or disabling PMTUD was the way to avoid the ping of death, which is False, nothing more, nothing less.
I never said that disabling PMTUD will get rid of the ping of death, what I said was that if your system is susceptible to a ping of death you may be tempted to filter ICMP but if you do that then you need to disable PMTUD because PMTUD + ICMP filtering = breakage.
As far as who Iljitsch is, everyone misspeaks from time to time. Even those of us who have been at this for nearly 3 decades.
After making the jump to academia I often feel a bit long in the tooth between all these students. But considering that (apparently) some people have been posting flames on NANOG for 30 years makes me feel young in comparison. :-)
Sorry if I misunderstood what you were saying. I thought you said that they couldn't have their cake and eat it too, as in protect against Ping of death, AND do PMTUD. As for those flames, well, that was a long time ago, in a valley not too far from here ;-). Can we have the 'net before the endless September back?
-----Original Message----- From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] Sent: Wednesday, May 07, 2008 2:40 PM To: Tomas L. Byrnes Cc: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 23:20, Tomas L. Byrnes wrote:
I was responding to his post that blocking or disabling PMTUD was the way to avoid the ping of death, which is False, nothing more, nothing less.
I never said that disabling PMTUD will get rid of the ping of death, what I said was that if your system is susceptible to a ping of death you may be tempted to filter ICMP but if you do that then you need to disable PMTUD because PMTUD + ICMP filtering = breakage.
As far as who Iljitsch is, everyone misspeaks from time to time. Even those of us who have been at this for nearly 3 decades.
After making the jump to academia I often feel a bit long in the tooth between all these students. But considering that (apparently) some people have been posting flames on NANOG for 30 years makes me feel young in comparison. :-)
Tomas L. Byrnes wrote:
As far as who Iljitsch is, everyone misspeaks from time to time. Even those of us who have been at this for nearly 3 decades.
I was simply LOLing at the fact that you found it necessary to give him a link to the NetHeaven article is all. ;-) -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
Some Edumacation on the topic is here: http://www.netheaven.com/pmtu.html
-----Original Message----- From: Iljitsch van Beijnum [mailto:iljitsch@muada.com] Sent: Wednesday, May 07, 2008 1:35 PM To: Michael Sinatra Cc: nanog@merit.edu Subject: Re: [NANOG] Microsoft.com PMTUD black hole?
On 7 mei 2008, at 21:46, Michael Sinatra wrote:
MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Perhaps they should also block _all_ TCP and UDP as well, and then we can move on.
I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional.
Right.
Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.)
However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY!
You can't have your cake and eat it too.
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
Tomas L. Byrnes wrote:
Some Edumacation on the topic is here:
You do know who it is that you are responding to, right? :) http://www.oreillynet.com/pub/au/970 -- Nathan Anderson First Step Internet, LLC nathana@fsr.com
Iljitsch van Beijnum <iljitsch@muada.com> writes:
Now Microsoft is also the company that built the OS that could be crashed by a maliciously crafted fragmented IP packet, so maybe there's something to this security policy. (One hopes that this bug and others like it are now fixed.)
Although the fact that Microsoft block all icmp makes me wonder which unfixed icmp related security holes they know about... I am not saying that there are any such holes in current Windows versions, but I will certainly not use a Windows server in an environment where I could receive icmp after learning that Microsoft themselves don't trust Windows' icmp handling. After all, Microsoft must have a reason to block all icmp. Or?
However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY!
You can't have your cake and eat it too.
But maybe the death of icmp is worth some sort of ceremony? Cake or not. Bjørn
Bjørn Mork wrote:
Iljitsch van Beijnum <iljitsch@muada.com> writes:
<snip>
After all, Microsoft must have a reason to block all icmp. Or?
However, in that case the only workable course of action would be TO DISABLE PATH MTU DISCOVERY!
You can't have your cake and eat it too.
But maybe the death of icmp is worth some sort of ceremony? Cake or not.
Oddly enough there is a draft on the subject of icmp filtering recomendations is making the rounds. http://tools.ietf.org/wg/opsec/draft-gont-opsec-icmp-filtering-00.txt The opsec working group (opsec@ietf.org) and the authors would appreciate feedback from operators on the subject. thanks joelja
Bjørn
_______________________________________________ NANOG mailing list NANOG@nanog.org http://mailman.nanog.org/mailman/listinfo/nanog
On 8 mei 2008, at 9:53, Joel Jaeggli wrote:
Oddly enough there is a draft on the subject of icmp filtering recomendations is making the rounds.
http://tools.ietf.org/wg/opsec/draft-gont-opsec-icmp-filtering-00.txt
The opsec working group (opsec@ietf.org) and the authors would appreciate feedback from operators on the subject.
Speaking as someone who isn't interested in reading an explanation of what happens when the message is filtered for every ICMP message known to man, I find this a completely useless document: I can't find the recommendations. Either they're there but impossible to find by looking at the table of contents or searching for "recommend", or they're not there in which case the title is EXTREMELY misleading. Also: 2.1.1.5.4. Operational/interoperability impact if blocked Filtering this error message breaks the Path-MTU Discovery mechansim described in [RFC1191]. This is completely insufficient because it doesn't mention that 99% of all TCP traffic on today's internet uses PMTUD and filtering these messages leads to broken connectivity towards destinations that have an MTU lower than the source (lower than 1500 in practice). Please spell check and five levels of numbering is considered bad style.
A few comments on your comments below. RM=for(1) {manage_risk(identify_risk(product[i++]) && (identify_threat[product[i++]))} Donald.Smith@qwest.com giac
-----Original Message----- From: opsec-bounces@ietf.org [mailto:opsec-bounces@ietf.org] On Behalf Of Iljitsch van Beijnum Sent: Thursday, May 08, 2008 3:24 AM To: Joel Jaeggli Cc: guillermo@gont.com.ar; opsec@ietf.org; NANOG list Subject: Re: [OPSEC] [NANOG] Microsoft.com PMTUD black hole?
On 8 mei 2008, at 9:53, Joel Jaeggli wrote:
Oddly enough there is a draft on the subject of icmp filtering recomendations is making the rounds.
http://tools.ietf.org/wg/opsec/draft-gont-opsec-icmp-filtering-00.txt
The opsec working group (opsec@ietf.org) and the authors would appreciate feedback from operators on the subject.
Speaking as someone who isn't interested in reading an explanation of what happens when the message is filtered for every ICMP message known to man, I find this a completely useless document: I can't find the recommendations. Either they're there but impossible to find by looking at the table of contents or searching for "recommend", or they're not there in which case the title is EXTREMELY misleading.
I believe a table of what to filter where was recommended. I hope that table includes filtering and ratelimiting from, through, and to. However blindly accepting recommendations without understanding the possibly ramifications such filtering can have on your network is not wise.
Also:
2.1.1.5.4. Operational/interoperability impact if blocked Filtering this error message breaks the Path-MTU Discovery mechansim described in [RFC1191].
This is completely insufficient because it doesn't mention that 99% of all TCP traffic on today's internet uses PMTUD and filtering these messages leads to broken connectivity towards destinations that have an MTU lower than the source (lower than 1500 in practice).
I suspect your statistics. I don't believe the number is anywhere near 99% but haven't seen a study that would support any actual % numbers of traffic that relies on PMTUD. If your aware of such a study/research I would be interested in reviewing the results. Again filtering THROUGH a device is probably not advisable filtering TO your device might be advisable.
Please spell check and five levels of numbering is considered bad style. _______________________________________________ OPSEC mailing list OPSEC@ietf.org https://www.ietf.org/mailman/listinfo/opsec
This communication is the property of Qwest and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
On Wed, 7 May 2008, Michael Sinatra wrote:
Nathan Anderson/FSR wrote:
Here is a brief update on the situation:
I have been in contact with someone at Microsoft's service operations center, who has confirmed for me that MS does in fact block _all_ ICMP at the edge of their network, that they are aware that this will in fact break PMTUD, and that they have no current plans to change this practice which they have implemented in the interest of security.
Although the need for your previous apology has already been questioned in this forum, the confirmation that they block not only certain ICMP types, but all ICMP, further vacates the need for any apology for criticizing this behavior in a pubic forum. It is disheartening for those of us who use and support MSFT's products to learn that their understanding of security lacks even the basic nuance to know not to block an entire--critical--portion of the Internet Protocol. Perhaps they should also block _all_ TCP and UDP as well, and then we can move on.
I agree with Iljitsch that it happens frequently, but I think I am justified in expecting more than that from Microsoft. Anything less would be unprofessional.
I wonder if MS knows about: ICMP Packet Filtering v1.2 from 2003: http://www.cymru.com/Documents/icmp-messages.html Only been around 5 years or so. Hopefully MS people reading this email will take note, read the entire page and implement what everyone else has been doing for a number of years. -Hank
Nathan Anderson/FSR wrote:
Nevertheless, the person I have been in contact with is naturally not the final decision-maker on this issue and is going to continue to pass the issue on up the chain of command for me. So although this issue is not over and I do not have a final verdict from MS yet, I felt that, given that I don't know how much time to expect to pass between now and when that final verdict is rendered, it would be appropriate to let everybody here know what I have learned thus far. Hopefully public dissemination of this information factoid will prevent others in a position similar to mine from having to helplessly beat their heads into their keyboards.
Let's also not ignore the generally overworked IT administrator at any small or medium sized enterprise. He/she may not be (as many folks I've run into are) of the mistaken impression that ICMP *is* bad and leaves you vulnerable to all sorts of things like SMURF. There are even tools out there that "test" your vulnerability by "pinging" you and do other investigations. I know of a tool that a major financial institution uses when certifying your networks security -- that scrapes the version number from your ESTMP banner to decide whether you comply or not (and other banners). (Rather than actually testing for a specific vulnerability). Simply blocking all of these packets from their test host gives you a high passing score; possibly a perfect one. [Irony and humor aside...] Many non-SP IT folks think they understand TCP, grudgingly accept UDP for DNS from external sources and think everything else is bollocks. Many *might* have a fit if they saw Microsoft accepting ICMPs because that seems inconsistent with their knowledge of turn-the-knob network security. To their view, their Linksys/Netgear/whathaveyou COTS firewalls block everything too. I don't think I'm exaggerating here. Just a thought, not saying its a good one or whose fault it is... Deepak Jain AiNET
On 7-May-2008, at 17:07:06, Deepak Jain wrote:
Many non-SP IT folks think they understand TCP, grudgingly accept UDP for DNS from external sources and think everything else is bollocks. Many *might* have a fit if they saw Microsoft accepting ICMPs because that seems inconsistent with their knowledge of turn- the-knob network security. To their view, their Linksys/Netgear/ whathaveyou COTS firewalls block everything too.
I don't think I'm exaggerating here.
No, you are not. I have seen the same from "firewall engineers" at large companies, people who, supposedly, have done "network security" for years. Even after showing them numerous Web sites detailing current best practices, especially Rob Thomas's fine site, these folks would not change their practices. Some days it is hard to not give in to the "I give up" feelings.
On Wed, 7 May 2008, Deepak Jain wrote:
I know of a tool that a major financial institution uses when certifying your networks security -- that scrapes the version number from your ESTMP banner to decide whether you comply or not (and other banners). (Rather than actually testing for a specific vulnerability). Simply blocking all of these packets from their test host gives you a high passing score; possibly a perfect one. [Irony and humor aside...]
Cisco PIX/ASA firewalls in SMTP fuxup mode are so incredibly broken. Possibly the worst SMTP implementation ever. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ FISHER GERMAN BIGHT: VARIABLE 3, BUT EASTERLY 4 OR 5 IN SOUTH GERMAN BIGHT. SLIGHT. FOG PATCHES. MODERATE OR GOOD, OCCASIONALLY VERY POOR.
First of all I would like to thank everyone for their support and concern. We certainly have a lot of things to "fix" at Microsoft. In fact, I can tell you that we have several brand new positions open (working on my team and for teams near mine) and could use more hands at the tiller. My apologies to the moderators for posting a help wanted but I figure since folks are expressing concerns with Microsoft networking they should have the opportunity to come over and help. We have an INCREDIBLE amount of interesting work ahead of us. I can't really speak to it directly but I can tell you that this is a really good place to be if you think big. We have a very interesting playbook for the next few years. Keep in mind that this is one of the few spots where thinking big can actually result in action. The current positions... Principal Network Engineer 204091 Senior Network Engineer 185014 SR PM 220793 IT/Ops PM 2 220797 Network Engineer 3 226032 Group Manager, Core Engineering 227621 Senior Network Engineer 229347 BOSD Network Engineer 231413 BOSD Senior Network Engineer 231414 If you think you have what it takes go check out these positions at http://www.microsoft.com/careers and apply (the numbers to the right are the job codes). Microsoft is an incredible place to work if you truly enjoy what you do. To be honest I don't read as much NANOG as I did a number of years ago. I had a couple friends point this thread out to me. I hope you are all doing well. Regards, Blaine
On Wed, 07 May 2008 12:24:57 -0700 Nathan Anderson/FSR <nathana@fsr.com> wrote:
Here is a brief update on the situation:
<snip>
Team. They, in turn, forwarded my message on to msnalerts@microsoft.com, which generated a ticket # for me and is, as I understand it, the e-mail address I was looking for in the first place (leads to their network/system people).
Doesn't look like it unfortunately. I've just tried to use this email address to advise them of some routing issues we're having with new APNIC IP ranges (114/8 specifically), and I've just got: -- This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: msnalerts@microsoft.com SMTP error from remote mail server after RCPT TO:<msnalerts@microsoft.com>: host mailb.microsoft.com [131.107.115.215]: 550 5.1.1 User unknown -- Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Hi, On Sat, 17 May 2008 10:20:06 +0930 Mark Smith <nanog@85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org> wrote:
On Wed, 07 May 2008 12:24:57 -0700 Nathan Anderson/FSR <nathana@fsr.com> wrote:
<snip>
-- This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:
msnalerts@microsoft.com SMTP error from remote mail server after RCPT TO:<msnalerts@microsoft.com>: host mailb.microsoft.com [131.107.115.215]: 550 5.1.1 User unknown --
Somebody from Microsoft kindly contacted me off list, the email address is msnalert@microsoft.com - note the removed "s". Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Thus spake "Nathan Anderson/FSR" <nathana@fsr.com>
A member of Microsoft's GNS network escalations team saw my postings on NANOG about this issue and took offense at my use of this forum to raise this issue with them, and criticized me as being unprofessional and lacking in business acumen.
First, it's "unprofessional and lacking in business acumen" for someone to criticize their customers to their face. As one manager taught me, "The customer may not always be right, but they're never wrong." Second, it's their own damn fault for not maintaining their contact information properly in public databases. If the only option they leave you is to post to NANOG, because they don't respond to (or even accept) direct requests to the listed contacts, then that's what you have to do. Many companies are guilty of the latter, and we all get the benefit of seeing the state of their customer service for reference when making future buying decisions. Very few are arrogant enough to do the former, though. S Stephen Sprunk "God does not play dice." --Albert Einstein CCIE #3723 "God is an inveterate gambler, and He throws the K5SSS dice at every possible opportunity." --Stephen Hawking
participants (21)
-
Bjørn Mork -
Blaine Christian -
Deepak Jain -
Glen Turner -
Hank Nussbacher -
Iljitsch van Beijnum -
Joel Jaeggli -
Mark Newton -
Mark Smith -
Marshall Eubanks -
Michael Sinatra -
Nathan Anderson/FSR -
Patrick Giagnocavo -
Randy Bush -
Rich Kulawiec -
Robert Bonomi -
Smith, Donald -
SML -
Stephen Sprunk -
Tomas L. Byrnes -
Tony Finch