- NANOG - lists.nanog.org

RE: Winstar says there is no TCP/BGP vulnerability
by Peering 21 Apr '04

21 Apr '04

> If you have a fully redundant internal BGP, and are running all > 12.2S/12.3/12.2T, then you can rather safely do the internal BGP > passwords without a customer notice, expecting no session drop but > knowing if one did you'd have routes via a second BGP reflector > anyway. Just an FYI, when we changed sessions that had full routing tables on them at 2am CST with barely any traffic, it causes malloc failures on the Cisco 7513. So it's something I would reserve for a maintenance window if you have one.

1 0

RE: Winstar says there is no TCP/BGP vulnerability
by Michel Py 21 Apr '04

21 Apr '04

> David Luyer wrote: > 98 of the first 100 did not reset. Today, > I did another 12 and only one failed. Thanks for the feedback. > If you have a fully redundant internal BGP, and are running > all 12.2S/12.3/12.2T, then you can rather safely do the > internal BGP passwords without a customer notice, expecting > no session drop but knowing if one did you'd have routes via > a second BGP reflector anyway. Ack. >>> Christopher L. Morrow wrote: >>> use a route-map to add/remove metric or localpref? or any >>> other settable thing on your side? or prepend or .... >> Michel Py wrote: >> Based on what criteria? Both the peer and the transit >> announce the same prefix with the same AS-PATH length. I >> agree that in many cases, favoring the route coming from >> the transit provider would work, > Iljitsch van Beijnum wrote: > Huh? You don't pay for peering traffic by the megabit, so > the idea is to always prefer routes from peers. Indeed, but we were talking about what to do with routes coming from the peer that are not supposed to. Legit routes announced by the peer will naturally be preferred, either because the prefix is longer than the one received from transit, or because the AS-PATH is shorter as the prefix is connected directly to the peer. Michel.

1 0

RE: Winstar says there is no TCP/BGP vulnerability
by Michel Py 21 Apr '04

21 Apr '04

Christopher / David, > Christopher L. Morrow wrote: > if you mean resetting sessions to change keys I believe > it's more code dependent than anythingn else. That was my point: I thought that changing keys without resetting the session was tied to a specific version of the "S" train that I have never seen it on anything lower than a 7200. Anyway, given David Luyer's post, it appears that unless you are willing to accept the risk of an unplanned session reset, you'd better have a planned outage for it. > David Luyer wrote: > Have done around 100 of these in the past 24 hours. It's > not related to platform AFAIK - we've successfully done > the changes on a lowly 2651 and 3620 without outages, but > a 7200 with older IOS did have an outage. Given the context, I assume that you have added MD5 to sessions that did not have it previously, I am correct? Then, do you mean by "without outages" that the session was not reset by the password add/change? If I may ask, how many out of 100 did not reset? > Christopher L. Morrow wrote: > For pure: "Don't blow me up with prefixes" just limit the > maximum-prefix to some # over your expected peer's list. Please allow me to try to make my point again: you store the expected peer maximum-prefix somewhere in your management system. I do understand the added complexity, but in the big scheme of things would it be _that_ more difficult to store a comma-delimited string or something that contains the prefixes that could be announced by that peer instead of the maximum-prefix? Yes, it generates more work to update the database, but OTOH it provides the LIII engineer with a lot more to troubleshoot issues. Is it simply not worth the work at your scale? >> - There are cases (such as the peer being a tier-2 customer of >> UUNET and me being a tier-3 customer of UUNET via a different >> tier-2) when the routes seen coming from the peer will have the >> same length AS-PATH than the ones coming from my transit, some >> other BGP tie-breaking criteria favoring the peer over the >> transit, leading to disaster. > use a route-map to add/remove metric or localpref? or any > other settable thing on your side? or prepend or .... Based on what criteria? Both the peer and the transit announce the same prefix with the same AS-PATH length. I agree that in many cases, favoring the route coming from the transit provider would work, but what guarantees it? What we are trying to define is the idiot-proof setup for peers; what if the misconfiguration is with the transit? >> - In theory, I could add a "route-map blah deny 1" that matches >> everything, then manipulate the subsequent seqs at will, then >> remove the "route-map blah deny 1"; in this situation though, >> I do not see a clear advantage over clearing the session. >> What am I missing? > you could tftp in your config change, that doesn't cause the > problems... then just wait for next update time. I don't see much of a difference. AFAIK when you tftp a config into the running-config, it is appended to the existing config same as if you pasted the commands into conf t. What happens when the next update time happens in the middle of tftp merging the old route-map with the new one? Michel.

5 6

RE: Winstar says there is no TCP/BGP vulnerability
by Peering 21 Apr '04

21 Apr '04

We do prefix-filter all our peers, both customer and transit. We also use as-path filters. It does seem to help us avoid insertion of invalid routes and other issues (especially since some people we peer with don't do the same on their side). As far as stability and process problems, we're too busy working on the instability of the Ciscos we're on now to notice, particularly the problem with BGP scanner taking up all the CPU every 60 seconds. We're preparing to move from an ATM core on Alcatel ATM switches with a Cisco edge to an IP-MPLS core on Juniper M-20s with M-20s (and a few Ciscos in smaller cities) on the edge. Hopefully that will improve our stability. We're pretty excited about the Junipers (the network geeks like me here are drooling). Diane Turley Network Engineer Xspedius Communications Co. 636-625-7178 -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W.Gilmore Sent: Wednesday, April 21, 2004 10:12 AM To: nanog(a)merit.edu Cc: Patrick W.Gilmore Subject: Re: Winstar says there is no TCP/BGP vulnerability On Apr 21, 2004, at 10:38 AM, Jared Mauch wrote: > On Wed, Apr 21, 2004 at 10:19:10AM -0400, Patrick W.Gilmore wrote: >> >>> Yes, it generates more work to update the database, >>> but OTOH it provides the LIII engineer with a lot more to >>> troubleshoot >>> issues. Is it simply not worth the work at your scale? >> >> Exactly. >> >> And you do not have to be at 701's scale for this to not work. > > We've not had these issues and have been using > bgp passwords/md5 for years. We do have a fancy configuration > managment system in place, whereby people put things into the database > first before they configure the router. Sorry, in this particular post, we were (or at least I was) talking about having prefix filters for all your peers. I know I've talked a lot about MD5 lately, just thought it would be a nice change of subject. :) If you do prefix filter all your peers, that is impressive. Do you get out of sync a lot? Does it help keep the network more stable? Or do process problems make it worse than just max-prefixes on a peer? -- TTFN, patrick

1 0

RE: Xspedius / E.Spire as wellRe: Winstar says there is no TCP/BGP vulnerability (fwd)
by Peering 21 Apr '04

21 Apr '04

All, Xspedius (formerly e.spire/ACSI) is in the process of converting BGP connections to MD5. We have already done so with both of our transit providers and we're working on contacting customers that we do BGP with. If anyone is a customer of Xspedius and wants to convert to MD5, please email me or open a ticket with the NOC and request that it be referred to Dwayne Chin in Tier II DMC, who is working with me on this project. Otherwise, we will be contacting you directly. Regards, Diane Turley Network Engineer Xspedius Communications Co. peering(a)xspedius.com ---------- Forwarded message ---------- Date: Tue, 20 Apr 2004 15:30:30 -0600 From: " John Brown (CV)" <jmbrown(a)chagresventures.com> To: Rodney Joffe <rjoffe(a)centergate.com> Cc: NANOG <nanog(a)merit.edu> Subject: Xspedius / E.Spire as wellRe: Winstar says there is no TCP/BGP vulnerability Seems Xspedius aka E.SPire aka ACSI doesn't feel that MD5 is important on their BGP sessions either. Based on the ticket we filed last week, Managment does not feel its warranted to make these changes. On the other hand, SPRINT was willing and able to take MD5 session info right away. WAY TO GO SPRINT. On Tue, Apr 20, 2004 at 01:44:44PM -0700, Rodney Joffe wrote: > > Perhaps we are all making too much of this... > > It appears that Winstar feels that there is no need for MD5 > authentication of peering sessions. One of our customers has just had > the following response from Winstar following a request to implement > MD5 on their OC3 connection to Winstar. My first suggestion is to > locate another upstream provider (they have 3 already). > > However, perhaps someone from Winstar would care to help us all > understand what the alternative solution is to securing the session > via MD5? I would *love* an alternative to the 5 days of work we've > just gone through. > > > -----Original Message----- > > From: Justin Crawford - NMCW Engineer [mailto:jcrawford@winstar.net] > > Sent: Tuesday, April 20, 2004 11:13 AM > > To: xxxxxx > > Subject: Re: *****SPAM***** MD5 implimentation on BGP > > > > xxxxx, > > > > Winstar does not currently run MD5 authentication with our peers. > > > > Thanks > > > > Justin > > > > Thank you for your time and business > > > > Justin Crawford > > Winstar NMCW > > Ph: 206-xxx.xxxx > > Has anyone else run in to this with Winstar? > > -- > Rodney Joffe > CenterGate Research Group, LLC. > http://www.centergate.com > "Technology so advanced, even we don't understand it!"(SM)

1 0

RE: Winstar says there is no TCP/BGP vulnerability
by Michel Py 21 Apr '04

21 Apr '04

> Patrick W.Gilmore wrote: > And when that process involves customers calling to ask > why they can't get to XXX web site (no pun intended - > I'm sure no one would filter a pr0n site :), it is much > more than "a bitch", it is a CLM/CEM. But you're missing something fundamental here: for non-tier-1s, the very purpose of peering is to offload your transit. If you filter the routes you accept from the peer, you don't shoot yourself in the foot, all you do is to send traffic to that prefix to transit, which is what you would have done in the first place if you were not peering. In other words, if your ingress route-map to filter prefixes received from the peer is incomplete, your only loss is that you will not reap the benefits of peering for the omitted prefix(es). I never said that prefix-filtering peering should shoot for perfection; if a few prefixes slip through the cracks and they're worth a modemload of traffic, who cares? Michel.

1 0

BGP session reset in one packet [where a looking glass or route server is available]
by David Luyer 21 Apr '04

21 Apr '04

It's not the general case, however... Some looking glass CGIs (in some cases, into production routers) permit "sh ip bgp nei <x>" -- try typing "sum" and then "nei x.x.x.x" into the "show ip bgp" box on a looking glass CGI, or using the command on a route server with CLI access. This gives you: Local host: [...], Local port: 179 Foreign host: [...], Foreign port: 29626 [...] iss: 770717974 snduna: 770746699 sndnxt: 770746699 sndwnd: 15472 irs: 431124262 rcvnxt: 440258849 rcvwnd: 15433 delrcvwnd: 951 [...] A traceroute will give you the information you need to estimate the TTL, and from there it's a single packet to reset the BGP session. If the looking glass is into a dedicated router, this achieves little more than causing the other routers to have to continually re-send their tables to the looking-glass; but if it's into a production router, well, it's a little more significant. And that _is_ sufficient to do a BGP session reset in one packet as the router is happily handing out sequence numbers, source and destination IP, source and destination port. Summary: if you don't want to do TCP MD5 auth on your BGP sessions, at least restrict "show tcp" and "show ip bgp nei" in all looking glasses and make sure your peers are not permitting them either in their directly attached routers. David.

1 0

RE: Winstar says there is no TCP/BGP vulnerability
by Michel Py 21 Apr '04

21 Apr '04

Christopher / Patrick, > Christopher L. Morrow wrote: > I wasn't clear and for that I'm sorry. Except in the later > code trains, or until the recent past (1 year or so) changing > the BGP MD5 auth bits required the session to be reset. Then I'm the one sorry because I never got it to work (I have not tried hard, I have to say); I always considered the session reset to be annoyance that was part of life. Dumb question: on what platforms is this working? If my memory is correct nothing below the 7200; I have seen numerous cases of peering with platforms such as 3600. > no, removing a route-map or adding a new one is painful, > changing it is just normal, no bounce required. Maybe I have something to learn here. - The reason I have a route-map in for a peer's BGP session is that I want to want to shield myself against a misconfigured peer that announces the entire world to me. - There are cases (such as the peer being a tier-2 customer of UUNET and me being a tier-3 customer of UUNET via a different tier-2) when the routes seen coming from the peer will have the same length AS-PATH than the ones coming from my transit, some other BGP tie-breaking criteria favoring the peer over the transit, leading to disaster. - I have equally been burned by filtering using a regexp that accepts only routes that appear to originate from the peer. Looks good on paper but when the peer configures aggregates of their own that leak in their BGP, becomes messy. Also been burned with peers configuring next-hop-self. - I think I have a reasonably good understanding of route-maps; however I have not found so far a mechanism that could guarantee that, when adding/removing seqs from it, there is no transient condition that causes the peer wrongly announcing the entire world to you to get all his crud into my own table. - In theory, I could add a "route-map blah deny 1" that matches everything, then manipulate the subsequent seqs at will, then remove the "route-map blah deny 1"; in this situation though, I do not see a clear advantage over clearing the session. What am I missing? > wrong program, I was referring to his reset_tcp.c program, Oh, I see. I thought that you meant that saving the password somewhere was useless, as any side could recover it by decrypting it from the config. My mistake. > Remember that once you put the passwd in the config it's exposed > and the next NOC engineer that leaves will have a copy emailed to > his home email account before he stomps out :( You're preaching the choir on this. That being said, the point was how MD5 could diffuse a DOS attack based on the vulnerability mentioned earlier; the correlation between a BGP TCP RST DOS attack (especially against a big networks such as yours) and a NOC engineer leaving is not quite 1:1. As a matter of fact, if was a NOC engineer planning a DOS attack against the company I work for, I would stay in just to have the inside picture. > Patrick W.Gilmore wrote: > No medium or large network filters their peers on prefix. And > very few small networks do either. Prefix count, maybe, but > not individual prefixes. There are far too many changes on far > too many peers per day to keep up. Exactly the same argument as using MD5 or not. When you build this as part of the operation, yes it means time _and_ money but overall not that much more and the bigger the network the lesser the cost-per-router. I do concede that people such as Rob and myself are sometimes living on another planet, as there we operate a few networks for fun, not for money. Nevertheless, time for pet projects is limited is as well as day-time job ones, and if we lobby for something such as edge filtering or bogon route-servers, it's because it can be worked out. We just have the luxury to try without having to justify the time wasted on it. > Not to mention far, far, far too many chances to screw up > and get the dreaded phone call in the middle of the night. Irrelevant: 1) Look at your watch. 2) That's what rookies are for. A NOC rookie screwing up _one_ BGP session does not mean my phone rings. I'm sure that everyone understands that, when I was a rookie, I _never_ screwed up a session in the middle of the night when nobody cares :-) Michel.

3 2

RE: why use IPv6, was: Lazy network operators
by Michel Py 21 Apr '04

21 Apr '04

[consolidated some posts] > Alex Bligh wrote: > As an IPv6 skeptic I would note that some protocols NAT > extremely badly (SIP for instance), and the bodges to fix > it are costly. So if IPv6 means I can avoid NAT, that can > actually save $$$. Likely the market will find some other way, which is not to use a protocol that has problems in 80% of environments and to use one that works smoothly everywhere; have a look at Skype... Trouble crossing NAT has always been an excuse for people that design antiquated protocols. To some extent NAT is a benefit here as it will help to get rid of these. NAT is a reality; designing a protocol that does not cross it will only doom said protocol, not remove NAT. > Petri Helenius wrote: > We need one (or more) of the p2p vendors to support it. And why are they not doing it? More work, zero gain. Today, a p2p app has to cross NAT nicely and has to work over IPv4 nicely. Why bother with IPv6? It won't bring more users in. From the user's side: why bother with IPv6 since it works fine with v4? (if it was not working fine they would not use it in the first place). > Then IPv6 traffic will explode in three months to ~10-15% > of all internet traffic In your dreams. How much does threedegrees traffic account for? 0.0001%? 0.001%? Compare to Kazaa. > Patrick W.Gilmore wrote: > Dunno what your problem is, I have no problem getting as much > address space as I need as long as I can justify it. Perhaps > you need to speak to your provider? Agree. Actually, the situation is even worse than this: I have numerous customers that stockpile IPv4 addresses that they don't need just because they can have them (just in case). A typical 400-user organization with NAT needs only a dozen or two IPv4 addresses; however, I see more and more requesting 2 class Cs from their provider because they can justify the number. And there are number of bigger enterprises that multihome for the month they request their portable address space in order to get it, and then drop BGP and the second provider. > Iljitsch van Beijnum wrote: > [IPv6] Renumbering is much easier. What a joke. Have a look at this: http://www.ietf.org/internet-drafts/draft-ietf-v6ops-renumbering-procedu re-00.txt Then, ever tried to renumber a Windows 2000 domain controller? And please, save me the "Microsoft is crud" thing. 95% percent of the networks I renumbered had more than one. 75% of the renumbering hassle is orthogonal to the protocol being renumbered. > So currently, multi6 is looking at approaches that allow transport > protocols to jump addresses in the middle of a session. Which will be developed just the same for IPv4. > Paul Jakma wrote: > [snip darth vader] > Iljitsch van Beijnum wrote: > Michel, you forgot to include the audio: > http://www.bgpexpert.com/darkside.mp3 Cut/paste casualty! I requested the file from you 2 days ago for this very purpose! Paul, I'm surprised you missed the "dark side" thing. > Iljitsch van Beijnum wrote: > Michel is no longer in the IPv6 business, Wrong. I'm currently in the anti-IPv6 business. The dark side. > Paul Jakma wrote: > (how's your MHAP doing? I dumped it. > How's Iljitsch's geo-assigned addressing proposal? Right behind MHAP in oblivion land. At this very time, I think Iljitsch is wondering how to deal with Darth Py and Darth Jakma... > Well, let's be honest, name one good reason why you'd want > IPv6 (given you have 4)? And, to be more on-topic, name one > good reason why a network operator would want it? Especially > given that, apart from the traditional bleeding edges > academic networks), no customers are asking for it. You're preaching the choir. > But there _will_ be NAT, that is the very premise of this > discussion, as offered by Paul Vixie. And Tim Chown, and me, and plenty of others. > So that one doesnt count, unless you knock down the premise: > There will be site-local and NAT with v6 because of the > multihoming problem. I used to think that way, but no longer. When we started ipv6mh, there was still a chance that providing a reasonable multihoming solution would get IPv6 out the mud hole. Trouble is that there were developments in other sectors of IPv6 that I was not able to foreseen have changed the situation to a point where IPv6 multihoming is no more that a bug on the windshield of IETF backroom politics, to re-use Vixie's words. For everyone, here's the bottom line: - Today, what to do with IPv6 is simple: nothing. Whether you are an end-user/small business, large enterprise or provider everyone is in the same situation: is costs money to upgrade, causes trouble, not the only thing we have to do anyway, there is no demand and therefore no ROI. It is urgent to wait. IPv6 is in a very similar situation ISDN was some time ago: I Still Don't Need. - - - - - Tomorrow, IPv4 will get the small upgrades that are needed. Michel.

3 3

RE: Winstar says there is no TCP/BGP vulnerability
by Michel Py 21 Apr '04

21 Apr '04

Patrick / Christopher, >> Michel Py wrote: >> Please forgive me if I'm naive and/or ask a stupid question, >> but is there any reason (besides your platform not supporting >> it) _not_ to MD5 your BGP sessions? Geez, on my _home_ router >> all my v4 BGP sessions are MD5ed (v6 not there yet). > Patrick W.Gilmore wrote: > There is serious operational overhead in maintaining sync'ed > passwords between separate organizations. IOW: Eventually > someone will screw up and lose the password. > [large snip] Thanks for the insight. I have an even dumber question: Context: set aside the MD5, the way I configure BGP sessions/traffic from/to peers is as follows: a) A generic (configured in the peer-group) route-map to filter the routes I announce to the peer to be only my blocks. b) A specific-to-the-peer route-map to filter the routes I receive from the peer to the peer's blocks, as agreed in the beer drinking meeting ^H^H^H^H BLPA. This route map is not entirely specific, as I also put in stuff such as deny RFC1918 routes ;-) c) A generic access-list filtering ingress traffic from the peer to me to allow only traffic which DA is mine. (cracks me up if the peer sets a default to me :-) d) A generic access-list filtering egress traffic from me to the peer to allow only traffic which SA is mine. Now, the dumb question: Given: 1) The context above especially item b 2) Christopher Morrow's comments below Explain me what having or not having the MD5 password changes. Either you're small and/or stupid and do it manually, or you have an automated system that does it for you. > Christopher L. Morrow wrote: > there is the issue of changing the keys during operations > without impacting the network, eh? Having to bounce every > bgp session in your network can be pretty darned painful... > if you change the key(s) of course. See above: Changing the route-map is equally painful. > If you don't you might as well not have keys, since adding > the 3 lines of C code required to Paul Watsons' program > making it do the hashing certainly won't be a big deal, eh? I'm weak with C. Besides adding "neighbor x.x.x.x password 7 " below "enable-password 7 " for each peer (which requires recompiling, how annoying) would you care sharing the 3 said lines for the code below :-) Michel. #include <stdio.h> #include <ctype.h> char xlat[] = { 0x64, 0x73, 0x66, 0x64, 0x3b, 0x6b, 0x66, 0x6f, 0x41, 0x2c, 0x2e, 0x69, 0x79, 0x65, 0x77, 0x72, 0x6b, 0x6c, 0x64, 0x4a, 0x4b, 0x44 }; char pw_str1[] = "password 7 "; char pw_str2[] = "enable-password 7 "; char *pname; cdecrypt(enc_pw, dec_pw) char *enc_pw; char *dec_pw; { unsigned int seed, i, val = 0; if(strlen(enc_pw) & 1) return(-1); seed = (enc_pw[0] - '0') * 10 + enc_pw[1] - '0'; if (seed > 15 || !isdigit(enc_pw[0]) || !isdigit(enc_pw[1])) return(-1); for (i = 2 ; i <= strlen(enc_pw); i++) { if(i !=2 && !(i & 1)) { dec_pw[i / 2 - 2] = val ^ xlat[seed++]; val = 0; } val *= 16; if(isdigit(enc_pw[i] = toupper(enc_pw[i]))) { val += enc_pw[i] - '0'; continue; } if(enc_pw[i] >= 'A' && enc_pw[i] <= 'F') { val += enc_pw[i] - 'A' + 10; continue; } if(strlen(enc_pw) != i) return(-1); } dec_pw[++i / 2] = 0; return(0); } usage() { fprintf(stdout, "Usage: %s -p <encrypted password>\n", pname); fprintf(stdout, " %s <router config file> <output file>\n", pname); return(0); } main(argc,argv) int argc; char **argv; { FILE *in = stdin, *out = stdout; char line[257]; char passwd[65]; unsigned int i, pw_pos; pname = argv[0]; if(argc > 1) { if(argc > 3) { usage(); exit(1); } if(argv[1][0] == '-') { switch(argv[1][1]) { case 'h': usage(); break; case 'p': if(cdecrypt(argv[2], passwd)) { fprintf(stderr, "Error.\n"); exit(1); } fprintf(stdout, "password: %s\n", passwd); break; default: fprintf(stderr, "%s: unknow option.", pname); } return(0); } if((in = fopen(argv[1], "rt")) == NULL) exit(1); if(argc > 2) if((out = fopen(argv[2], "wt")) == NULL) exit(1); } while(1) { for(i = 0; i < 256; i++) { if((line[i] = fgetc(in)) == EOF) { if(i) break; fclose(in); fclose(out); return(0); } if(line[i] == '\r') i--; if(line[i] == '\n') break; } pw_pos = 0; line[i] = 0; if(!strncmp(line, pw_str1, strlen(pw_str1))) pw_pos = strlen(pw_str1); if(!strncmp(line, pw_str2, strlen(pw_str2))) pw_pos = strlen(pw_str2); if(!pw_pos) { fprintf(stdout, "%s\n", line); continue; } if(cdecrypt(&line[pw_pos], passwd)) { fprintf(stderr, "Error.\n"); exit(1); } else { if(pw_pos == strlen(pw_str1)) fprintf(out, "%s", pw_str1); else fprintf(out, "%s", pw_str2); fprintf(out, "%s\n", passwd); } } }

3 2