We have a potential customer that is asking for us to enable MD5 authentication on a TCP connection between two BGP peers? Is this still common practice today? Any potential problems or gotchas to keep in mind? Thanks! -- Brian Stengel KINBER Director of Operations bstengel@kinber.org 412.254.3481 Skype: brian_stengel KINBER - Keystone Initiative for Network Based Education and Research - www.kinber.org PennREN - Pennsylvania's Research and Education Network
On 1/27/12 11:26 AM, Brian Stengel wrote:
We have a potential customer that is asking for us to enable MD5 authentication on a TCP connection between two BGP peers? Is this still common practice today? Any potential problems or gotchas to keep in mind?
Sprint requires it to enable remote triggered blackhole. ~Seth
On Fri, Jan 27, 2012 at 2:51 PM, Seth Mattinen <sethm@rollernet.us> wrote:
On 1/27/12 11:26 AM, Brian Stengel wrote:
We have a potential customer that is asking for us to enable MD5 authentication on a TCP connection between two BGP peers? Is this still common practice today? Any potential problems or gotchas to keep in mind?
Sprint requires it to enable remote triggered blackhole.
lots of folks still use it yes. is it helpful? maybe? maybe not? is this peering over a shared media (like a 10base-T hub). You might point out that you'll be enabling this, then promptly writing the 'secret' on a large whiteboard in your noc... because chances are the config won't include it in rancid and ... you don't have a place to store these securely that's not prone also to outages :( also, customers wander through your NOC, so...
On Fri, 27 Jan 2012, Christopher Morrow wrote:
lots of folks still use it yes. is it helpful? maybe? maybe not? is this peering over a shared media (like a 10base-T hub).
You might point out that you'll be enabling this, then promptly writing the 'secret' on a large whiteboard in your noc... because chances are the config won't include it in rancid and ... you don't have a place to store these securely that's not prone also to outages :(
also, customers wander through your NOC, so...
All that may be true, but still, the random hacker in Romania who wants in on their BGP session won't know the secret...probably. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Fri, Jan 27, 2012 at 3:32 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Fri, 27 Jan 2012, Christopher Morrow wrote:
lots of folks still use it yes. is it helpful? maybe? maybe not? is this peering over a shared media (like a 10base-T hub).
You might point out that you'll be enabling this, then promptly writing the 'secret' on a large whiteboard in your noc... because chances are the config won't include it in rancid and ... you don't have a place to store these securely that's not prone also to outages :(
also, customers wander through your NOC, so...
All that may be true, but still, the random hacker in Romania who wants in on their BGP session won't know the secret...probably.
1) that person doesn't exist 2) they need a LOT more info about what's going on anyway 3) I bet they will get a copy of the config from at least: a) vendor data sources b) ebay purchases of gear c) pwning a noc-worker and getting things done from there. There are far better ways to skin this cat.
MD5 on BGP sessions is the canonical example of a cure worse than the disease. There has been /infinitely/ more downtime caused by MD5 than the mythical attack it protects again. (This is true because anything times zero is still zero.) It is far easier to take a router out than try to calculate the number of RSTs per second you can get through to the RE without your guesses being dropped / throttled, then waiting hours or days to watch a BGP session flap. Amazingly awesome attack, because as everyone knows BGP sessions never flap on their own, so a random session flapping every day or six will totally freak out the provider in question. And all that ignores the fact every router vendor fixed the ephemeral port selection & window size issues half a decade ago, so those "days" it takes to reset a single BGP session are actually more like months or years. Remember, miscreants are lazy, impatient, and frequently clueless. Who would want to reset a BGP that will come back up in 30-90 seconds when you can packet an entire router off the 'Net easier, more quickly, and for longer a period? Unfortunately, Network Engineers are lazy, impatient, and frequently clueless as well. They read something from 1906 that says "$FOO IS GOOD!!1!1!" and force every peer to subscribe to their own ideal without understanding the underlying technology or rationale. Your network, your decision. On my network, we do not do MD5. We do more traffic than anyone and have to be in the top 10 of total eBGP peering sessions on the planet. Guess how many times we've seen anyone even attempt this attack? If you guessed more than zero, guess again. I am fully well aware saying this in a public place means someone, probably many someones, will try it now just to prove me wrong. I still don't care. What does that tell you? STOP USING MD5 ON BGP. -- TTFN, patrick
On Fri, Jan 27, 2012 at 3:52 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:
MD5 on BGP sessions is the canonical example of a cure worse than the disease. There has been /infinitely/ more downtime caused by MD5 than the mythical attack it protects again. (This is true because anything times zero is still zero.)
I don't disagree with patrick here... but 'infinitely more', is hard to measure :) "Most likely there have been far more lengthy outages due to lost/changed/incorrect key material than were caused by the problem this is meant to solve for." -chris
It is
On 27-01-12 21:52, Patrick W. Gilmore wrote:
Who would want to reset a BGP that will come back up in 30-90 seconds when you can packet an entire router off the 'Net easier, more quickly, and for longer a period?
+1 Actually, when you have lot of MD5 BGP session coming up at the same time (a connection to internet exchanges went up), you have longer convergence time because of higher cpu load. MD5 offers no security advantages and in some cases it causes more downtime by slowing down convergence. -- Grzegorz Janoszka
On Jan 27, 2012, at 3:52 PM, Patrick W. Gilmore wrote:
Your network, your decision. On my network, we do not do MD5. We do more traffic than anyone and have to be in the top 10 of total eBGP peering sessions on the planet. Guess how many times we've seen anyone even attempt this attack? If you guessed more than zero, guess again.
I am fully well aware saying this in a public place means someone, probably many someones, will try it now just to prove me wrong. I still don't care. What does that tell you?
STOP USING MD5 ON BGP.
I would generally say: If you are on a p2p link or control the network, then yeah, you don't need md5. If you are at a shared medium (e.g.: IX) I do recommend it there, as it will help mitigate cases where someone can hijack your session by putting your IP/ASN whatnot on the router. The threat (Attack) never became real and we've now had enough time that even the slowest carriers are running fixed code. - Jared
2012/1/27 Jared Mauch <jared@puck.nether.net>:
On Jan 27, 2012, at 3:52 PM, Patrick W. Gilmore wrote:
Your network, your decision. On my network, we do not do MD5. We do more traffic than anyone and have to be in the top 10 of total eBGP peering sessions on the planet. Guess how many times we've seen anyone even attempt this attack? If you guessed more than zero, guess again.
I am fully well aware saying this in a public place means someone, probably many someones, will try it now just to prove me wrong. I still don't care. What does that tell you?
STOP USING MD5 ON BGP.
I would generally say: If you are on a p2p link or control the network, then yeah, you don't need md5. If you are at a shared medium (e.g.: IX) I do recommend it there, as it will help mitigate cases where someone can hijack your session by putting your IP/ASN whatnot on the router.
The threat (Attack) never became real and we've now had enough time that even the slowest carriers are running fixed code.
- Jared
I kind of agree that there isn't much of a vector here, but I don't agree that MD5 hurts if done correctly. Is it really that hard to find a semi-secure place to store passwords for an entire company? There's also the question of engineering standards. Is it an aging practice? Probably... Is it worth spending time to update it and train everyone not to use it? Probably not. I'll be happy when the world realizes that it's ok to let gig-e auto-negotiate. I've never really seen MD5 cause issues.
On Fri, Jan 27, 2012 at 6:35 PM, Keegan Holley <keegan.holley@sungard.com> wrote:
realizes that it's ok to let gig-e auto-negotiate. I've never really seen MD5 cause issues.
I have run into plenty of problems caused by MD5-related bugs. 6500/7600 can still figure the MSS incorrectly when using it. It used to be possible for that particular box to send over-sized frames out Ethernet ports with MD5 enabled, which of course were likely to be dropped by the neighboring router or switching equipment (perhaps even carrier Ethernet equipment.) Obviously that can be a chore to troubleshoot. Sometimes we choose to use it. Sometimes we don't. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
2012/1/27 Jeff Wheeler <jsw@inconcepts.biz>:
On Fri, Jan 27, 2012 at 6:35 PM, Keegan Holley <keegan.holley@sungard.com> wrote:
realizes that it's ok to let gig-e auto-negotiate. I've never really seen MD5 cause issues.
I have run into plenty of problems caused by MD5-related bugs.
6500/7600 can still figure the MSS incorrectly when using it. It used to be possible for that particular box to send over-sized frames out Ethernet ports with MD5 enabled, which of course were likely to be dropped by the neighboring router or switching equipment (perhaps even carrier Ethernet equipment.) Obviously that can be a chore to troubleshoot.
Sometimes we choose to use it. Sometimes we don't.
--
Bugs are a different argument though. If you could call something harmful because a single vendor codes it wrong there would be far fewer windows users in the world. (I know it's friday, but please no one change the subject to OS's)
I am in the camp of no MD5 in general and more specifically IX. It is a real pain to manage MD5 and no network in my experience has ever implemented a sustainable solution. There is no BCP that folks follow so generally its a verbal agreement that someone in either party will maintain the record. This works until that operator leaves the job and the MD5 is in their email box which is no longer accessible. I would say this is pretty common, I have inherited quite a few networks where I had to deal with this problem. Also most common places where people store MD5's are not in secure locations. I would argue that even though you connect via shared medium in the case of an IX you can still use TTL. Zaid On 1/27/12 3:20 PM, "Jared Mauch" <jared@puck.nether.net> wrote:
On Jan 27, 2012, at 3:52 PM, Patrick W. Gilmore wrote:
Your network, your decision. On my network, we do not do MD5. We do more traffic than anyone and have to be in the top 10 of total eBGP peering sessions on the planet. Guess how many times we've seen anyone even attempt this attack? If you guessed more than zero, guess again.
I am fully well aware saying this in a public place means someone, probably many someones, will try it now just to prove me wrong. I still don't care. What does that tell you?
STOP USING MD5 ON BGP.
I would generally say: If you are on a p2p link or control the network, then yeah, you don't need md5. If you are at a shared medium (e.g.: IX) I do recommend it there, as it will help mitigate cases where someone can hijack your session by putting your IP/ASN whatnot on the router.
The threat (Attack) never became real and we've now had enough time that even the slowest carriers are running fixed code.
- Jared
On Jan 27, 2012, at 6:20 PM, Jared Mauch wrote:
On Jan 27, 2012, at 3:52 PM, Patrick W. Gilmore wrote:
Your network, your decision. On my network, we do not do MD5. We do more traffic than anyone and have to be in the top 10 of total eBGP peering sessions on the planet. Guess how many times we've seen anyone even attempt this attack? If you guessed more than zero, guess again.
I am fully well aware saying this in a public place means someone, probably many someones, will try it now just to prove me wrong. I still don't care. What does that tell you?
STOP USING MD5 ON BGP.
I would generally say: If you are on a p2p link or control the network, then yeah, you don't need md5. If you are at a shared medium (e.g.: IX) I do recommend it there, as it will help mitigate cases where someone can hijack your session by putting your IP/ASN whatnot on the router.
As much as this scares me, I am going to disagree with Jared. If another member on the IX fabric wants to do something bad, then spoofing your address and causing BGP sessions to flap is the least of your worries. I've personally configured thousand of sessions at dozens of IXes for well over a decade. I have yet to see a single case where MD5 would have been useful. OTOH, it has caused quite a bit of downtime. There is no perfect solution, everything has issues. Past performance is no guarantee of future profits. All you can do is try your level-headed best to keep the packets flowing as quickly, reliably, and cheaply as possible. MD5 is a detriment to _all three_ of those goals. -- TTFN, patrick
On Fri, 27 Jan 2012 15:52:41 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Unfortunately, Network Engineers are lazy, impatient, and frequently clueless as well.
While the quantity of peering sessions I've had is far less than yours, once upon a time when I had tried to get MD5 on dozens of peering sessions I learned quite a bit about those engineers and those networks. I got to find out who couldn't do password management, who never heard of MD5 and who had been listening to Patrick. :-) All good input that inform what else I might want to do to protect myself from those networks or who I wouldn't mind having a business relationship with. John
I suppose so but BFD certainly has alot more moving parts then adding MDF checksums to an existing control packet. I'm not saying everyone should turn it on or off for that matter. I just don't see what the big deal is. Most of the shops I've seen have it on because of some long forgotten engineering standard. 2012/1/30 John Kristoff <jtk@cymru.com>:
On Fri, 27 Jan 2012 15:52:41 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Unfortunately, Network Engineers are lazy, impatient, and frequently clueless as well.
While the quantity of peering sessions I've had is far less than yours, once upon a time when I had tried to get MD5 on dozens of peering sessions I learned quite a bit about those engineers and those networks. I got to find out who couldn't do password management, who never heard of MD5 and who had been listening to Patrick. :-) All good input that inform what else I might want to do to protect myself from those networks or who I wouldn't mind having a business relationship with.
John
My thoughts are that you should filter traffic routed directly to your BGP speaking devices, traffic routing through a edge device and to an edge device are treated differently. BGP session protection using a MD5 password by itself is not securing the control plane, but it is a component of an overall secure edge posture. For example, md5 protection, plus edge filtering polices, plus ttl security, plus ........., make for a more secure edge. Also, It does not matter how many attempts compromising a BGP session occurs, it only takes one, so why not nail it down. Mike On Tue, Jan 31, 2012 at 12:39 AM, Keegan Holley <keegan.holley@sungard.com>wrote:
I suppose so but BFD certainly has alot more moving parts then adding MDF checksums to an existing control packet. I'm not saying everyone should turn it on or off for that matter. I just don't see what the big deal is. Most of the shops I've seen have it on because of some long forgotten engineering standard.
2012/1/30 John Kristoff <jtk@cymru.com>:
On Fri, 27 Jan 2012 15:52:41 -0500 "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Unfortunately, Network Engineers are lazy, impatient, and frequently clueless as well.
While the quantity of peering sessions I've had is far less than yours, once upon a time when I had tried to get MD5 on dozens of peering sessions I learned quite a bit about those engineers and those networks. I got to find out who couldn't do password management, who never heard of MD5 and who had been listening to Patrick. :-) All good input that inform what else I might want to do to protect myself from those networks or who I wouldn't mind having a business relationship with.
John
From: harbor235 <harbor235@gmail.com>
Also, It does not matter how many attempts compromising a BGP session occurs, it only takes one, so why not nail it down.
Because downtime is a security issue too, and MD5 is more likely to contribute to downtime (either via lost password, crypto load on CPU, or other) than the problem it purports to fix. The goal of a network engineer is to move packets from A -> B. The goal of a security engineer is to keep that from happening. A business needs to weigh the cost and benefit of any given approach, and MD5 BGP auth does not come out well in the of situations. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com
On 31/01/2012 16:40, David Barak wrote:
Because downtime is a security issue too, and MD5 is more likely to contribute to downtime (either via lost password, crypto load on CPU, or other) than the problem it purports to fix. The goal of a network engineer is to move packets from A -> B. The goal of a security engineer is to keep that from happening. A business needs to weigh the cost and benefit of any given approach, and MD5 BGP auth does not come out well in the of situations.
cpu load is negligible and is done in hardware on several platforms. Lost passwords can occur but if you have properly stored configuration backups, they shouldn't be a major problem. Also, they can be trivially decrypted from C/J configuration files.
From my point of view, MD5 passwords serve two purposes:
1. they prevent intentional session hijacking at IXPs when IP addresses get re-used and new IP address assignees suddenly notice that some people haven't torn down their old BGP sessions to the previous users of the address 2. they can be used to convince security auditors that the network is secure and that they can now sod off and stop harassing me, kthxbai Other people may have other reasons for liking / not liking them. Nick
Sounds like we want a well thought out plan in place in case there is a screw up with an org's lack of planning and management capabilities.......... Mike On Tue, Jan 31, 2012 at 12:56 PM, Nick Hilliard <nick@foobar.org> wrote:
On 31/01/2012 16:40, David Barak wrote:
Because downtime is a security issue too, and MD5 is more likely to contribute to downtime (either via lost password, crypto load on CPU, or other) than the problem it purports to fix. The goal of a network engineer is to move packets from A -> B. The goal of a security engineer is to keep that from happening. A business needs to weigh the cost and benefit of any given approach, and MD5 BGP auth does not come out well in the of situations.
cpu load is negligible and is done in hardware on several platforms. Lost passwords can occur but if you have properly stored configuration backups, they shouldn't be a major problem. Also, they can be trivially decrypted from C/J configuration files.
From my point of view, MD5 passwords serve two purposes:
1. they prevent intentional session hijacking at IXPs when IP addresses get re-used and new IP address assignees suddenly notice that some people haven't torn down their old BGP sessions to the previous users of the address
2. they can be used to convince security auditors that the network is secure and that they can now sod off and stop harassing me, kthxbai
Other people may have other reasons for liking / not liking them.
Nick
On 1/31/12, Nick Hilliard <nick@foobar.org> wrote:
On 31/01/2012 16:40, David Barak wrote:
Because downtime is a security issue too, and MD5 is more likely to contribute to downtime (either via lost password, crypto load on CPU, or other) than the problem it purports to fix. The goal of a network engineer is to move packets from A -> B. The goal of a security engineer is to keep that from happening. A business needs to weigh the cost and benefit of any given approach, and MD5 BGP auth does not come out well in the of situations.
cpu load is negligible and is done in hardware on several platforms. Lost passwords can occur but if you have properly stored configuration backups, they shouldn't be a major problem. Also, they can be trivially decrypted from C/J configuration files.
From my point of view, MD5 passwords serve two purposes: .. snip ..
2. they can be used to convince security auditors that the network is secure and that they can now sod off and stop harassing me, kthxbai
+1 It isn't worth the time or effort trying to get an exception to their 'best practice'. Lee
On 1/27/12 12:35 , Christopher Morrow wrote:
On Fri, Jan 27, 2012 at 3:32 PM, Jon Lewis <jlewis@lewis.org> wrote:
On Fri, 27 Jan 2012, Christopher Morrow wrote:
lots of folks still use it yes. is it helpful? maybe? maybe not? is this peering over a shared media (like a 10base-T hub).
You might point out that you'll be enabling this, then promptly writing the 'secret' on a large whiteboard in your noc... because chances are the config won't include it in rancid and ... you don't have a place to store these securely that's not prone also to outages :(
also, customers wander through your NOC, so...
All that may be true, but still, the random hacker in Romania who wants in on their BGP session won't know the secret...probably.
1) that person doesn't exist 2) they need a LOT more info about what's going on anyway 3) I bet they will get a copy of the config from at least: a) vendor data sources b) ebay purchases of gear c) pwning a noc-worker and getting things done from there.
There are far better ways to skin this cat.
I don't think md5 is that great, but I absolutely wouldn't use a clear text password if I'm going to use anything at all. I don't think shared seceret management is dramatically harder than any other form of of configuration management, modula rekeying requires coordination with a third party and is therefore hard. joel
All that may be true, but still, the random hacker in Romania who wants in on their BGP session won't know the secret...probably.
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net |
One thing I will do at shared peering switches is to also configure static ARP or IPv6 neighbor entries in the router for my peers. This protects against some new arrival on the switch accidentally configuring one of my peer's IP addresses on their gear and blowing up my session. That does cause some problems when a peer does maintenance that changes their MAC address, but I notice it fairly quickly.
participants (17)
-
Brian Stengel
-
Christopher Morrow
-
David Barak
-
George Bonser
-
Grzegorz Janoszka
-
harbor235
-
Jared Mauch
-
Jeff Wheeler
-
Joel jaeggli
-
John Kristoff
-
Jon Lewis
-
Keegan Holley
-
Lee
-
Nick Hilliard
-
Patrick W. Gilmore
-
Seth Mattinen
-
Zaid Ali