RE: Winstar says there is no TCP/BGP vulnerability
Please forgive me if I'm naive and/or ask a stupid question, but is there any reason (besides your platform not supporting it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router all my v4 BGP sessions are MD5ed (v6 not there yet). Michel. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Joe Rhett Sent: Tuesday, April 20, 2004 8:07 PM To: Rodney Joffe Cc: NANOG Subject: Re: Winstar says there is no TCP/BGP vulnerability I've left your entire message below so that one can see I've removed nothing. Winstar has made NONE of the statements you are interpreting from their response. They have simply stated that they don't support it at this moment in time. I'll grant you that they could have answered "when" or "why" or "what else". But they certainly didn't say anything you are suggesting that they have said. <joke>Should we ever meet, I'll remember to never turn down a beer. You might think I'm pro-prohibition or something...</joke> On Tue, Apr 20, 2004 at 01:44:44PM -0700, Rodney Joffe wrote:
Perhaps we are all making too much of this...
It appears that Winstar feels that there is no need for MD5 authentication of peering sessions. One of our customers has just had the following response from Winstar following a request to implement
MD5
on their OC3 connection to Winstar. My first suggestion is to locate another upstream provider (they have 3 already).
However, perhaps someone from Winstar would care to help us all understand what the alternative solution is to securing the session via MD5? I would *love* an alternative to the 5 days of work we've just gone through.
-----Original Message----- From: Justin Crawford - NMCW Engineer [mailto:jcrawford@winstar.net] Sent: Tuesday, April 20, 2004 11:13 AM To: xxxxxx Subject: Re: *****SPAM***** MD5 implimentation on BGP
xxxxx,
Winstar does not currently run MD5 authentication with our peers.
Thanks
Justin
Thank you for your time and business
Justin Crawford Winstar NMCW Ph: 206-xxx.xxxx
Has anyone else run in to this with Winstar?
-- Rodney Joffe CenterGate Research Group, LLC. http://www.centergate.com "Technology so advanced, even we don't understand it!"(SM)
-- Joe Rhett Chief Geek JRhett@Isite.Net Isite Services, Inc.
On Apr 20, 2004, at 11:29 PM, Michel Py wrote:
Please forgive me if I'm naive and/or ask a stupid question, but is there any reason (besides your platform not supporting it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router all my v4 BGP sessions are MD5ed (v6 not there yet).
There is serious operational overhead in maintaining sync'ed passwords between separate organizations. IOW: Eventually someone will screw up and lose the password. When they do, the session goes down, and probably for far longer than if some miscreant tries to RST it via the "vulnerability". Actual data: Over the past three plus years an organization with on the order of a dozen MD5-ized BGP sessions has has multiple down sessions due to, for instance, a peer doing standard (for them) password rotation and forgetting to inform the organization. Each time incurred a minimum of several hours downtime, once stretching into several days as the peer could not figure out what was wrong and get the right person with the password to give it to the organization. Over the past three plus years with over 1000 non-MD5-ized BGP sessions, the same organization experienced exactly *ZERO* seconds of downtime identified as due to RST-style attacks. And certainly no prolonged outages due to it. Add to that the additional CPU overhead some people have reported, making it easier to packet the router to its knees, and MD5 looks like a cure worse than the disease. All that said, it is your router, your peers, your decision. I would never dream of telling anyone who wanted MD5 to not do it. I just don't understand people who want to do it. Especially when they could be doing things like filtering at the leaf nodes and forcing their vendors to support the TTL hack. But that's me. -- TTFN, patrick
Hi, NANOGers. ] Actual data: Over the past three plus years an organization with on the ] order of a dozen MD5-ized BGP sessions has has multiple down sessions ] due to, for instance, a peer doing standard (for them) password ] rotation and forgetting to inform the organization. Yep, that's a problem - a PROCESS problem. The definition of insanity is repeating the same behavior over and over and expecting a different result. ;) Saying that we've not seen any RST attacks may be correct, but it's not a predictor of future activity. No one can do more than model what may come next. Prior to February 2001 no one had seen massive DDoS either. Anyone still think DDoS isn't a problem? We manage well over 150 peering sessions with MD5 passwords in place. This includes bogon peering, route-server peering, and production traffic peering. This has grown over the past three years. The total number of MD5-related outages: zero. In other words, your mileage may vary. :) Test any feature. Think about how to manage that feature, both in the deployment stage and in steady-state. I don't advocate the use of any feature, be it MD5, MPLS, et al. without careful consideration of the support ramifications of it. Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
On Apr 21, 2004, at 12:11 AM, Rob Thomas wrote:
] Actual data: Over the past three plus years an organization with on the ] order of a dozen MD5-ized BGP sessions has has multiple down sessions ] due to, for instance, a peer doing standard (for them) password ] rotation and forgetting to inform the organization.
Yep, that's a problem - a PROCESS problem. The definition of insanity is repeating the same behavior over and over and expecting a different result. ;)
So you think continuing to use MD5 and not see down sessions due to lost passwords would be insane? Also, PROCESSES are part of running a network. As any large network provider, they will tell you the process of keeping the network up costs more than the fiber the network runs on. In fact, operational support and complexity is one of the main reasons large networks give for not peering with other networks.
Saying that we've not seen any RST attacks may be correct, but it's not a predictor of future activity. No one can do more than model what may come next. Prior to February 2001 no one had seen massive DDoS either. Anyone still think DDoS isn't a problem?
Of course not. But I also see a reason to do DDoS attacks. Rob, you are the biggest proponent of the underground economy I have ever seen. Tell me why a miscreant would waste his time - at *least* 30 minutes (lowest report I've seen, and possibly hours or days) - trying to reset a BGP session when with less effort they could take down an entire router? I can come up with reasons, but they are pretty few and far between, especially compared with reasons to take out a whole router. Can you tell me why these miscreants - people who are lazy just like us - would not rather packet the router off the 'Net than do this slow, possibly useless exercise?
We manage well over 150 peering sessions with MD5 passwords in place. This includes bogon peering, route-server peering, and production traffic peering. This has grown over the past three years. The total number of MD5-related outages: zero.
In other words, your mileage may vary. :)
Awesome news. And your milage *will* vary, since you cannot control separate organizations. -- TTFN, patrick
RT> Date: Tue, 20 Apr 2004 23:11:28 -0500 (CDT) RT> From: Rob Thomas RT> We manage well over 150 peering sessions with MD5 passwords RT> in place. This includes bogon peering, route-server peering, CYMRU bogon (et al.) route servers are an example of where MD5 or IPSec definitely is a good idea. However, most peer/peer and carrier/downstream BGP sessions aren't multihop spanning a network or three. Of course, if ingress SAV were universal... Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses : blacklist@brics.com -or- alfra@intc.net -or- curbjmp@intc.net Sending mail to spambait addresses is a great way to get blocked.
May be, it is reasonable to have a simple MD-5 key - I mean, without a rotation, use e-mail to exchange it instead of the phone, do not generate but use simple password, and so on. If this key is never changed, then risk to lost a session is very low, and I do not see _any_ reason to keep it on rotation plan (hacker must know too much and can damage too little, in this case). Even such keys as '415' or 'monday' will prevent TCP attacks in alll cases - if single attack require 5 - 30 minutes for the one hit, then no any way exists to use dictionary 'guess' for password cracking. Now, we can see a _histeria_ around this problem; but yes, when it will coll down (1 - 2 weeks), it will be a time to make a reasonable improvements. ----- Original Message ----- From: "Patrick W.Gilmore" <patrick@ianai.net> To: <nanog@merit.edu> Cc: "Patrick W.Gilmore" <patrick@ianai.net> Sent: Tuesday, April 20, 2004 8:49 PM Subject: Re: Winstar says there is no TCP/BGP vulnerability
On Apr 20, 2004, at 11:29 PM, Michel Py wrote:
Please forgive me if I'm naive and/or ask a stupid question, but is there any reason (besides your platform not supporting it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router all my v4 BGP sessions are MD5ed (v6 not there yet).
There is serious operational overhead in maintaining sync'ed passwords between separate organizations. IOW: Eventually someone will screw up and lose the password. When they do, the session goes down, and probably for far longer than if some miscreant tries to RST it via the "vulnerability".
Actual data: Over the past three plus years an organization with on the order of a dozen MD5-ized BGP sessions has has multiple down sessions due to, for instance, a peer doing standard (for them) password rotation and forgetting to inform the organization. Each time incurred a minimum of several hours downtime, once stretching into several days as the peer could not figure out what was wrong and get the right person with the password to give it to the organization.
Over the past three plus years with over 1000 non-MD5-ized BGP sessions, the same organization experienced exactly *ZERO* seconds of downtime identified as due to RST-style attacks. And certainly no prolonged outages due to it.
Add to that the additional CPU overhead some people have reported, making it easier to packet the router to its knees, and MD5 looks like a cure worse than the disease.
All that said, it is your router, your peers, your decision. I would never dream of telling anyone who wanted MD5 to not do it. I just don't understand people who want to do it. Especially when they could be doing things like filtering at the leaf nodes and forcing their vendors to support the TTL hack.
But that's me.
-- TTFN, patrick
On Tue, 20 Apr 2004, Michel Py wrote:
Please forgive me if I'm naive and/or ask a stupid question, but is there any reason (besides your platform not supporting it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router all my v4 BGP sessions are MD5ed (v6 not there yet).
Michel.
there is the issue of changing the keys during operations without impacting the network, eh? Having to bounce every bgp session in your network can be pretty darned painful... if you change the key(s) of course. If you don't you might as well not have keys, since adding the 3 lines of C code required to Paul Watsons' program making it do the hashing certainly won't be a big deal, eh?
"Christopher L. Morrow" <christopher.morrow@mci.com> writes:
there is the issue of changing the keys during operations without impacting the network, eh? Having to bounce every bgp session in your network can be pretty darned painful... if you change the key(s) of course. If you don't you might as well not have keys, since adding the 3 lines of C code required to Paul Watsons' program making it do the hashing certainly won't be a big deal, eh?
I've added keys without bouncing the sessions... doesn't seem to cause any difficulties at all. You just add the password clause on both ends within the window for a BGP keepalive timeout. Worst case, this line: Milwaukee#sho ip bgp neigh 203.176.61.22 | inc md5 Flags: passive open, nagle, gen tcbs, md5 Milwaukee# is lying, and the md5 won't actually come up until some nogoodnik or bad fortune causes the session to bounce. 12.0S. ---Rob
On Wed, 21 Apr 2004, Robert E. Seastrom wrote:
"Christopher L. Morrow" <christopher.morrow@mci.com> writes:
there is the issue of changing the keys during operations without impacting the network, eh? Having to bounce every bgp session in your network can be pretty darned painful... if you change the key(s) of course. If you don't you might as well not have keys, since adding the 3 lines of C code required to Paul Watsons' program making it do the hashing certainly won't be a big deal, eh?
I've added keys without bouncing the sessions... doesn't seem to cause any difficulties at all. You just add the password clause on both ends within the window for a BGP keepalive timeout. Worst case, this line:
yup, this is the expected behaviour at a certain level of code... I don't recall that level but I'm sure a cisco person could give us the rundown :) Basically, as I understand the explanation given to me, there is no defined manner to deal with: 1) changing keys 2) adding keys 3) removing keys in the RFC for this (2385). So, atleast Cisco, implemented key change in a sane manner, if you change the key packets immediately start heading out with the new key used as the hash key. The far side starts logging 'bad key' messages but the session doesn't reset and updates keep attempting to be sent, until you either change the key, or the session timeouts hit. For adding keys, I have experienced the following: Apr 21 17:01:45.278: %BGP-5-ADJCHANGE: neighbor <ip> Down - Password change on 12.0S and 12.1(19)(non-s) code trains... on passwd change: Apr 21 17:03:36.117 GMT: %BGP-5-ADJCHANGE: neighbor <ip> Down Password change So, this is obviously suboptimal. The good news is code releases up the chain seem to have this fixed, getting there is a chore, but will make MD5 more operationally managable in the long term, and thus more widely deployed, eh? (hopefully)
That isn't the point of my post. Whether or not you think X is a good idea, having someone technical say "we don't support X currently" does not mean a host of other things like "we think X is a bad idea" or any other nonsense like that. On Tue, Apr 20, 2004 at 08:29:34PM -0700, Michel Py wrote:
Please forgive me if I'm naive and/or ask a stupid question, but is there any reason (besides your platform not supporting it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router all my v4 BGP sessions are MD5ed (v6 not there yet).
Michel.
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Joe Rhett Sent: Tuesday, April 20, 2004 8:07 PM To: Rodney Joffe Cc: NANOG Subject: Re: Winstar says there is no TCP/BGP vulnerability
I've left your entire message below so that one can see I've removed nothing. Winstar has made NONE of the statements you are interpreting from their response. They have simply stated that they don't support it at this moment in time. I'll grant you that they could have answered "when" or "why" or "what else". But they certainly didn't say anything you are suggesting that they have said.
<joke>Should we ever meet, I'll remember to never turn down a beer. You might think I'm pro-prohibition or something...</joke>
On Tue, Apr 20, 2004 at 01:44:44PM -0700, Rodney Joffe wrote:
Perhaps we are all making too much of this...
It appears that Winstar feels that there is no need for MD5 authentication of peering sessions. One of our customers has just had the following response from Winstar following a request to implement
MD5
on their OC3 connection to Winstar. My first suggestion is to locate another upstream provider (they have 3 already).
However, perhaps someone from Winstar would care to help us all understand what the alternative solution is to securing the session via MD5? I would *love* an alternative to the 5 days of work we've just gone through.
-----Original Message----- From: Justin Crawford - NMCW Engineer [mailto:jcrawford@winstar.net] Sent: Tuesday, April 20, 2004 11:13 AM To: xxxxxx Subject: Re: *****SPAM***** MD5 implimentation on BGP
xxxxx,
Winstar does not currently run MD5 authentication with our peers.
Thanks
Justin
Thank you for your time and business
Justin Crawford Winstar NMCW Ph: 206-xxx.xxxx
Has anyone else run in to this with Winstar?
-- Rodney Joffe CenterGate Research Group, LLC. http://www.centergate.com "Technology so advanced, even we don't understand it!"(SM)
-- Joe Rhett Chief Geek JRhett@Isite.Net Isite Services, Inc.
-- Joe Rhett Chief Geek JRhett@Isite.Net Isite Services, Inc.
participants (8)
-
Alexei Roudnev
-
Christopher L. Morrow
-
E.B. Dreger
-
Joe Rhett
-
Michel Py
-
Patrick W.Gilmore
-
Rob Thomas
-
Robert E. Seastrom