Hello, There is a routing loop while accesing my network 194.9.82.0/24 from some networks on the Internet. | This is a test done from lg.above.net looking glass. 1 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 2 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 0 msec 3 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 8 msec 8 msec 0 msec 4 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 5 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 6 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 4 msec 7 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 64 msec 0 msec 4 msec 8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 4 msec 0 msec 11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 4 msec 0 msec 4 msec| How do i aproach to fix this issue Regards Felix -- Best Regards, Felix Bako Network Engineer Africa Online, Kenya Tel: +254 (20) 27 92 000 Fax: +254 (20) 27 100 10 Email: fbako@africaonline.co.ke Aim:felixbako * Africa Online Disclaimer and Confidentiality Note * This e-mail, its attachments and any rights attaching hereto are, unless the context clearly indicates otherwise, the property of Africa Online Holdings (Kenya) Limited and / or its subsidiaries ("the Group"). It is confidential and intended for the addressee only. Should you not be the addressee and have received this e-mail by mistake, kindly notify the sender, delete this e-mail immediately and do not disclose or use the same in any manner whatsoever. Views and opinions expressed in this e-mail are those of the sender unless clearly stated as those of the Group. The Group accepts no liability whatsoever for any loss or damages, however incurred, resulting from the use of this e-mail or its attachments. The Group does not warrant the integrity of this e-mail, nor that it is free of errors, viruses, interception or interference. For more information about Africa Online, please visit our website at http://www.africaonline.com
If it continues for any length of time then contact above.net. To find their contact information, check their registrar. e.g. whois above.net gets you Technical Contact: AboveNet Communications, Inc. dns@ABOVE.NET AboveNet Communications, Inc. 50 W SAN FERNANDO ST STE 1010 SAN JOSE, CA 95113-2414 US 408-367-6673 fax: 408-367-6688 If that does not help, then you can solicit for better contact information (from NANOG.) I am betting above.net knows about this and is already working on it. Good luck! --Patrick Darden -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Felix Bako Sent: Friday, March 14, 2008 3:34 PM To: nanog@merit.edu Subject: Routing Loop Hello, There is a routing loop while accesing my network 194.9.82.0/24 from some networks on the Internet. | This is a test done from lg.above.net looking glass. 1 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 2 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 0 msec 3 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 8 msec 8 msec 0 msec 4 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 5 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 6 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 4 msec 7 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 64 msec 0 msec 4 msec 8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 4 msec 0 msec 11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 4 msec 0 msec 4 msec| How do i aproach to fix this issue Regards Felix -- Best Regards, Felix Bako Network Engineer Africa Online, Kenya Tel: +254 (20) 27 92 000 Fax: +254 (20) 27 100 10 Email: fbako@africaonline.co.ke Aim:felixbako * Africa Online Disclaimer and Confidentiality Note * This e-mail, its attachments and any rights attaching hereto are, unless the context clearly indicates otherwise, the property of Africa Online Holdings (Kenya) Limited and / or its subsidiaries ("the Group"). It is confidential and intended for the addressee only. Should you not be the addressee and have received this e-mail by mistake, kindly notify the sender, delete this e-mail immediately and do not disclose or use the same in any manner whatsoever. Views and opinions expressed in this e-mail are those of the sender unless clearly stated as those of the Group. The Group accepts no liability whatsoever for any loss or damages, however incurred, resulting from the use of this e-mail or its attachments. The Group does not warrant the integrity of this e-mail, nor that it is free of errors, viruses, interception or interference. For more information about Africa Online, please visit our website at http://www.africaonline.com
[more accurate subject line] On Mar 14, 2008, at 1:33 PM, Felix Bako wrote:
Hello, There is a routing loop while accesing my network 194.9.82.0/24 from some networks on the Internet.
| This is a test done from lg.above.net looking glass.
1 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 2 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 0 msec 3 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 8 msec 8 msec 0 msec 4 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 5 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 6 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 0 msec 4 msec 7 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 64 msec 0 msec 4 msec 8 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) [MPLS: Label 80 Exp 0] 0 msec 4 msec 0 msec 9 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 4 msec 0 msec 0 msec 10 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) [MPLS: Label 78 Exp 0] 0 msec 4 msec 0 msec 11 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 4 msec 0 msec 4 msec|
According to RIPE BGP play data looks to me like AS 6461 (Abovenet) began announcing 194.9.82.0/24 about 10 hours ago, pulling traffic away from AS 39615 and triggering your reachability problems (Note times are UTC): # 1/361 2008-03-15 03:05:27 Path Change from 29636 6461 2914 8513 25228 36915 rrc01 195.66.224.132 to 29636 2914 6461 # 2/361 2008-03-15 03:05:27 Route Announcement 20485 2914 6461 rrc01 195.66.224.212 .... About 17 minutes later AS 6461 they withdrew the route announcement: # 41/361 2008-03-15 03:22:56 Route Withdrawal ( 4777 2497 2914 6461 ) rrc06 202.249.2.20 .... And another 12 minutes or so later they began announcing it again: # 42/361 2008-03-15 03:35:26 Path Change from 29636 6461 2914 8513 25228 36915 rrc01 195.66.224.132 to 29636 2914 6461 ... Seemed to be a bunch more instability with this prefix around 5:53: # 66/361 2008-03-15 05:53:40 Route Announcement 25462 6461 rrc07 194.68.123.157 ... And then some withdraws around 7:43: # 183/361 2008-03-15 07:43:48 Path Change from 8468 6453 6461 rrc01 195.66.224.151 to 8468 3491 25228 25228 25228 25228 25228 36915 ... With considerable oscillation for around 40 minutes between the legit path via AS 36915 and the path via AS 6461. And the latest was this transition from AS 6461 back to the 36915 path about 2 hours ago, but only by a few ASNs, I suspect because those ASNs explicitly modified policy (either preference or filtering) to de_prefer the AS 6461 path. This is illustrated pretty nicely with BGP play: # 335/361 2008-03-15 14:59:43 Route Withdrawal ( 1916 3549 6461 ) rrc15 200.219.130.4 # 361/361 2008-03-15 15:00:27 Path Change from 13645 3356 6461 rrc11 198.32.160.150 to 13645 3491 25228 25228 25228 25228 25228 36915 BGP Play applet here: http://www.ris.ripe.net/bgplay/applet.html? Although most folks are definitely still preferring the AS 6461 path. An interesting bit is that the current announcement on routeviews directly from AS 6461 has Community 6461:5999 attached: ... 6461 64.125.0.137 from 64.125.0.137 (64.125.0.137) Origin IGP, metric 0, localpref 100, valid, external, best Community: 6461:5999 ... According to this, that community is used for "internal prefixes": http://onesc.net/communities/as6461/ "6461:5999 internal prefix" A "sh ip bgp community 6461:5999" currently yields 130 prefixes with Origin AS of 6461 and that community. Nothing more specific than a /24, although many many adjacent prefixes that would presumably be aggregated normally are announced as well. The closest adjacent prefix to 194.9.82/24 they're announcing is 194.9.40/24, which is one of their prefixes: *> 194.9.40.0 64.125.0.137 0 0 6461 i *> 194.9.82.0 64.125.0.137 0 0 6461 i Unfortunately, the AS6461 forwarding loops still exists, and most ASNs still appear to be preferring their path over yours per BGP AS path route selection rules: --- danny@pork% date Sat Mar 15 11:55:27 MDT 2008 ... 14 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 188.278 ms 172.714 ms 174.984 ms 15 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 176.234 ms 174.013 ms 174.109 ms 16 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 173.230 ms 172.892 ms 174.765 ms 17 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 174.721 ms 175.256 ms 174.738 ms 18 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 174.437 ms 220.815 ms 180.961 ms 19 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 177.564 ms 181.966 ms 174.771 ms 20 ten-gige-2-2.mpr2.ams2.nl.above.net (64.125.26.70) 176.028 ms 174.269 ms 174.365 ms 21 ten-gige-2-2.mpr1.ams2.nl.above.net (64.125.26.69) 175.626 ms 175.381 ms 175.831 ms 22 ge-1-2-0.mpr1.ams1.nl.above.net (64.125.26.74) 174.046 ms 174.841 ms 174.388 ms 23 ten-gige-1-1.mpr1.ams2.nl.above.net (64.125.26.73) 174.861 ms 174.857 ms 175.475 ms ... My recommendation, stay on the phone with Abovenet (via your upstream, and their upstream if necessary) until you see a withdraw for the route on routeviews from AS 6461: telnet route-views.routeviews.org sh ip bgp 194.9.82.0/24 -danny
A bit more analysis of this at the moment, and a few recommendations and related pointers is available here: http://tinyurl.com/2nqg2a -danny
Unlike the Youtube outage where PTA had issued a directive asking all ISPs to block Youtube - What is the reason most often cited for such mishaps? The reason i ask this is because the ISPs that "inadvertently" hijack someone elses IP space, need to explicitly configure *something* to do this. So, what really are they trying to do there? Thanks, Glen On Sun, Mar 16, 2008 at 1:36 AM, Danny McPherson <danny@tcb.net> wrote:
A bit more analysis of this at the moment, and a few recommendations and related pointers is available here:
-danny
On Sat, Mar 15, 2008 at 9:09 PM, Glen Kent <glen.kent@gmail.com> wrote:
Unlike the Youtube outage where PTA had issued a directive asking all ISPs to block Youtube - What is the reason most often cited for such mishaps? The reason i ask this is because the ISPs that "inadvertently" hijack someone elses IP space, need to explicitly configure *something* to do this. So, what really are they trying to do there?
I've seen two popular reasons for doing it accidentally - Fat fingers when configuring IP addresses by hand - Using old routing protocols such as IGRP or RIP and autosummarizing routes, usually done by a customer of an ISP that doesn't bother filtering carefully. This doesn't give you a /24 address by accident, but it lets you take two /24 subnets of a Class B or Class A and turn them into an advertisement for the whole network. A popular reason from longer ago was enterprises that used arbitrary addresses for their internal networks, which was safe because they'd never be connected to the real internet. RFC1918 has made that problem mostly go away, but as recently as 1995 I had a customer who was a bank that was using University of Toronto IP addresses internally. We were working on their databases, not their networks, so while we strongly recommended they renumber some time soon, it wasn't happening during our project. -- ---- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.
A popular reason from longer ago was enterprises that used arbitrary addresses for their internal networks, which was safe because they'd never be connected to the real internet. RFC1918 has made that problem mostly go away, but as recently as 1995 I had a customer who was a bank that was using University of Toronto IP addresses internally. We were working on their databases, not their networks, so while we strongly recommended they renumber some time soon, it wasn't happening during our project.
italian isps are notorious for using us military and other non-announced networks for infrastructure. i get a bit of a giggle out of it now. but boy was i shocked when i first did a traceroute from some public network in bologna years back. randy
On Sat, Mar 15, 2008, Danny McPherson wrote:
A bit more analysis of this at the moment, and a few recommendations and related pointers is available here:
Its a good writeup. :) It almost sounds like Felix should talk to some friendly SP's and organise /25 origination + tunneling into various places into the US. Or is this concept reminiscent of my exposure to this world circa 1999? :) Adrian
On Sat, Mar 15, 2008 at 11:57:50AM -0600, Danny McPherson wrote:
An interesting bit is that the current announcement on routeviews directly from AS 6461 has Community 6461:5999 attached: ... 6461 64.125.0.137 from 64.125.0.137 (64.125.0.137) Origin IGP, metric 0, localpref 100, valid, external, best Community: 6461:5999 ...
According to this, that community is used for "internal prefixes":
http://onesc.net/communities/as6461/
"6461:5999 internal prefix"
A "sh ip bgp community 6461:5999" currently yields 130 prefixes with Origin AS of 6461 and that community.
Hi Danny, Unless things have changed since I left in '05, 6461:5999 is the outbound community set on internally-originated prefixes. You would expect to see it on prefixes "owned" by AS6461 (such as 216.200/16) as well as address space announced on behalf of customers (i.e., prefixes "belonging" to customers who have no ASN and/or no desire to run BGP). Prefixes learned from another customer would have :5998 and those learned from a peer would have :5997, IIRC. These outbound translations are/were only performed on customer BGP sessions, which makes sense in this case since the session to route-views is/was configured like any other customer session. All it really tells you is that for whatever reason, that prefix was "manually" injected into BGP, most likely as a redist'ed static. Anyway, it's possible that this was intended due to an AUP issue but it's unlikely that they'd intentionally propagate the /24 in that case. At least when I was there, AboveNet had a separate system for injecting routes into BGP (for TE, abuse, etc) that automatically set no-export on those routes. In addition to making the process a lot less error-prone it helped contain any mistakes due to the automatic no-export. The only time you added a static route was when you WANTED to announce it. Beyond that, I have no idea why 6461 would have originated this route. My guess would be that someone who didn't understand the implications of their action added it as a static route for whatever reason, but that's nothing more than a guess. Seems like I've heard Randy voice an opinion on the local/global thing once before. :-) --Jeff
participants (8)
-
Adrian Chadd -
Bill Stewart -
Danny McPherson -
Darden, Patrick S. -
Felix Bako -
Glen Kent -
Jeff Aitken -
Randy Bush