We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there. Anyone else seeing evidence of this or being affected? [1] http://cyclops.cs.ucla.edu/ -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
On 11/5/09 16:30, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
I'm seeing alerts for AS13214 advertising our prefixes from cyclops also. However a quick look at a few looking glasses and route servers doesnt seem to show any rogue advertisments, and we havent see any drop in traffic as yet. Vince
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Seeing the same issues with AS13214 and no corresponding drop in traffic, route views doesn't show any rogue adverts for out prefixes either. -James On May 11, 2009, at 9:01 AM, Vincent Hoffman wrote:
On 11/5/09 16:30, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
I'm seeing alerts for AS13214 advertising our prefixes from cyclops also. However a quick look at a few looking glasses and route servers doesnt seem to show any rogue advertisments, and we havent see any drop in traffic as yet.
Vince
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
"Ravioli is the square root of pasta." - Max K., Age 11
It looks like Cyclops is seeing these from AS 48285, but I see no indication they are being advertised to any production upstream provider. Our /16 is being alerted in Cyclops, but I can not find any advert on any looking glass.
From Cyclops: BGP protocol Time (UTC) W/A/B Peer IP Peer ASN Prefix AS_PATH Origin NEXT_HOP LOCAL_PREF MED Community Atomic Agg Aggregator BGP4MP 1242044196 A 194.71.0.1 48285 128.227.0.0/16 48285 13214 INCOMPLETE 194.71.0.1 0 0 NAG
Robert D. Scott Robert@ufl.edu Senior Network Engineer 352-273-0113 Phone CNS - Network Services 352-392-2061 CNS Phone Tree University of Florida 352-392-9440 FAX Florida Lambda Rail 352-294-3571 FLR NOC Gainesville, FL 32611 321-663-0421 Cell -----Original Message----- From: James Kelty [mailto:jkelty@pandora.com] Sent: Monday, May 11, 2009 12:20 PM To: nanog@nanog.org Subject: Re: Anomalies with AS13214 ? Seeing the same issues with AS13214 and no corresponding drop in traffic, route views doesn't show any rogue adverts for out prefixes either. -James On May 11, 2009, at 9:01 AM, Vincent Hoffman wrote:
On 11/5/09 16:30, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
I'm seeing alerts for AS13214 advertising our prefixes from cyclops also. However a quick look at a few looking glasses and route servers doesnt seem to show any rogue advertisments, and we havent see any drop in traffic as yet.
Vince
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
"Ravioli is the square root of pasta." - Max K., Age 11
Robert D. Scott wrote:
It looks like Cyclops is seeing these from AS 48285, but I see no indication they are being advertised to any production upstream provider. Our /16 is being alerted in Cyclops, but I can not find any advert on any looking glass.
That's what I'm seeing as well. It's possible that 13214 is broken but not causing an issue except to their customers. Or 48285 is broken or just giving bad data to Cyclops. Cyclops has hundreds of monitors and this is the only one showing the issue. I suspect that if there's a real problem it isn't affecting anyone other than 48285 and maybe 13214. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Same here. Cyclops reporting an origin change but we are seeing no change in traffic levels. Still investigating at the moment... 2009/5/11 Jay Hennigan <jay@west.net>
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
On Mon, 11 May 2009, Russell Heilling wrote:
Same here. Cyclops reporting an origin change but we are seeing no change in traffic levels. Still investigating at the moment...
Somewhere, something is confused. I'm seeing cyclops report some of my prefixes with origins of 6364 (correct), 13214 6364 (no), and 6364 13214 (not right either). I'm also not seeing any unusual reduction of input traffic. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Randy doing testing again? Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Yeah, interesting contact name on this: person: Fredrik Neij address: DCPNetworks address: Box 161 address: SE-11479 Stockholm address: Sweden mnt-by: MNT-DCP phone: +46 707 323819 nic-hdl: FN2233-RIPE source: RIPE # Filtered Christopher Morrow wrote:
On Mon, May 11, 2009 at 12:41 PM, David Freedman <david.freedman@uk.clara.net> wrote:
Randy doing testing again?
13214 != 3130
On 12/05/2009, at 4:47 AM, David Freedman wrote:
Yeah, interesting contact name on this:
person: Fredrik Neij address: DCPNetworks address: Box 161 address: SE-11479 Stockholm address: Sweden mnt-by: MNT-DCP phone: +46 707 323819 nic-hdl: FN2233-RIPE source: RIPE # Filtered
Dispatch someone from IETF, that is on in Stockholm right now. Actually, Paul Jakma might be there, dispatch him if it really is a Quagga bug. -- Nathan Ward
Subject: Re: Anomalies with AS13214 ? Date: Wed, Jul 29, 2009 at 12:27:56AM +1200 Quoting Nathan Ward (nanog@daork.net):
On 12/05/2009, at 4:47 AM, David Freedman wrote:
Yeah, interesting contact name on this:
person: Fredrik Neij address: DCPNetworks address: Box 161 address: SE-11479 Stockholm address: Sweden mnt-by: MNT-DCP phone: +46 707 323819 nic-hdl: FN2233-RIPE source: RIPE # Filtered
(yes, it is him.)
Dispatch someone from IETF, that is on in Stockholm right now.
Won't help. Neij is 12 time zones away. But he is aware of the problem. -- Måns Nilsson
anyone but me find it "unusual" that we accept behaviours by some that we would find unacceptable by others... its stuff like that which provides my strongest motivation for things like SIDR... --bill On Mon, May 11, 2009 at 05:41:36PM +0100, David Freedman wrote:
Randy doing testing again?
Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
On Mon, 11 May 2009, bmanning@vacation.karoshi.com wrote: I certainly do. This time it is a config error, next time it will be researcher X doing some testing for a NANOG paper, and the time after that it will be some RBN test to see if anyone cares anymore to look deeply into what they are trying to pull off. Our level of sensitivity will eventually be nullified and we will all be the worse for it. -Hank
anyone but me find it "unusual" that we accept behaviours by some that we would find unacceptable by others...
its stuff like that which provides my strongest motivation for things like SIDR...
--bill
On Mon, May 11, 2009 at 05:41:36PM +0100, David Freedman wrote:
Randy doing testing again?
Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
.-- My secret spy satellite informs me that at Mon, 11 May 2009, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
It seems it was picked up by route-views4. Non of the RIS peers seem to have seen this. Looking at the raw bgp data from route-views4: AS13214 leaked a full table (~266294 prefixes) with 13214 as OriginAS to AS48285 which is a routeviews4 peer. Routeviews4 saw these announcements as: ASpath 48285 13214. It seems to have happend twice: ~ 11:03:45 GMT to 12:16:31 GMT (here AS48285 start announcing a valid path to routeviews again) then a few seconds later again: ~ 12:16:36 GMT to 12:18:14 GMT After that AS48285 announced ânormalâ ASpath to routeviews again. So looks like it wasnât a global hijack, it was only seen by one routeview peer. This is a very similar event as the one we saw on November 11 2008: http://bgpmon.net/blog/?p=80 This again shows that itâs hard to determine if an event is a ârealâ hijack or not. Some will say itâs irrelevant some want to be notified in all cases. Based on received feedback regarding the November 11 event, BGPmon.net implemented peer thresholds (http://bgpmon.net/blog/?p=88). Cheers, Andree
On Mon, May 11, 2009 at 2:29 PM, Andree Toonk <andree+nanog@toonk.nl> wrote:
.-- My secret spy satellite informs me that at Mon, 11 May 2009, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
It seems it was picked up by route-views4. Non of the RIS peers seem to have seen this.
Looking at the raw bgp data from route-views4: AS13214 leaked a full table (~266294 prefixes) with 13214 as OriginAS to AS48285 which is a routeviews4 peer. Routeviews4 saw these announcements as: ASpath 48285 13214.
Since 48285 == robtex, is it possible TPB was just setting up a monitoring/route-feed session to robtex and either missed their outbound policy or sent them the wrong form of outbound policy (full routes not customer only routes)?? -chris
Hello, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
I have also seen this today for our prefixes where Cyclops reported the as path "48285 13214". After sending an e-mail to both ASN I got the following answer from AS48285: "Our transit 13214 had interesting router problems affecting bgp origins for the entire bgp table. The next-hop and thus routing was still working fine though. Since we collect bgp data from several transits and announce it to multiple route servers and for our own publicly available bgp-tools, it looked worse than it was, but as far as we can tell it was actually never propagated by them to the Internet except to downstreams, where traffic still worked, although via an unusually short path." Regards, Christian Seitz Network Operations
Hi all, First, thanks for using Cyclops, and thanks for all the Cyclops users that drop me a message about this. It seems some router in AS13214 decided to originate all the prefixes and send them to AS48285 in the Caymans, all the ASPATHs are 48285 13214. The first announcement was on 2009-05-11 11:03:11 UTC and last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn afterwards) As indicated in the Cyclops alerts, only a single monitor(AS48285) in route-views4 detected this leak. I checked on other neighbors of AS13214 and they seem fine, so it seems it was only a single router issue. This incident shows the advantage of having a wide set of peers for detection, it seems Cyclops was the only tool to detect this incident. Given the amount of banks and financial institutions in the Caymans, i would otherwise have raised a red flag, but it seems this case was an unintentional misconfig by AS13214. Would appreciate any further comment on the tool, and happy cyclopying! --Ricardo the Cyclops guy http://cyclops.cs.ucla.edu On May 11, 2009, at 8:30 AM, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
2009/5/11 Ricardo Oliveira <rveloso@cs.ucla.edu>:
Hi all,
First, thanks for using Cyclops, and thanks for all the Cyclops users that drop me a message about this.
It seems some router in AS13214 decided to originate all the prefixes and send them to AS48285 in the Caymans, all the ASPATHs are 48285 13214. The first announcement was on 2009-05-11 11:03:11 UTC and last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn afterwards)
It looks like AS13214 are misbehaving again... We have just started receiving cyclops alerts indicating that AS13214 is announcing our prefixes again: Alert ID: 4927389 Alert type: origin change Monitored ASN,prefix: 78.154.96.0/19 Offending attribute: 78.154.96.0/19-13214 Date: 2009-07-28 08:30:56 UTC Duration: 00:00:01 (hh:mm:ss) No. monitors: 1 (http://cyclops.cs.ucla.edu/view_monitors.html?aid=4927389) Announced prefix: 78.154.96.0/19 Announced ASPATH: 48285 13214 BGP message: http://cyclops.cs.ucla.edu/show_myalert.html?aid=4927389 I guess ROBTEX didn't implement ingress filters after the last episode...
As indicated in the Cyclops alerts, only a single monitor(AS48285) in route-views4 detected this leak. I checked on other neighbors of AS13214 and they seem fine, so it seems it was only a single router issue.
This incident shows the advantage of having a wide set of peers for detection, it seems Cyclops was the only tool to detect this incident. Given the amount of banks and financial institutions in the Caymans, i would otherwise have raised a red flag, but it seems this case was an unintentional misconfig by AS13214.
Would appreciate any further comment on the tool, and happy cyclopying!
--Ricardo the Cyclops guy http://cyclops.cs.ucla.edu
On May 11, 2009, at 8:30 AM, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
-- Russell Heilling http://perlmonkey.blogspot.com "The amazing ability of the bee to adapt herself often helps the beekeeper to overcome the results of his ignorance." - Brother Adam
On Tue, 28 Jul 2009, Russell Heilling wrote:
It looks like AS13214 are misbehaving again... We have just started receiving cyclops alerts indicating that AS13214 is announcing our prefixes again:
There is talk about this being a new Quagga bug redist:ing kernel routes into BGP. I'm yelling at them for not having outgoing route filters to handle the possibility after what happened last time. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, Jul 28, 2009 at 11:50:02AM +0100, Russell Heilling <chewtoy@s8n.net> wrote a message of 75 lines which said:
No. monitors: 1
That's why it's good to use BGP alarm systems with a peer threshold. I recommend BGPmon <http://bgpmon.net/> (today, I run it with a peer thershold of 1 because the problem is rare enough but I can raise it if necessary). AFAIK, Cyclops does not have this functionality.
On Tue, Jul 28, 2009 at 11:50:02AM +0100, Russell Heilling <chewtoy@s8n.net> wrote a message of 75 lines which said:
I guess ROBTEX didn't implement ingress filters after the last episode...
It *seems* (I do not know them in detail) that Robtex <http://www.robtex.com/>, AS 48285, is dedicated to measurements, not to IP transit. If so, it makes sense for them to accept everything. If I'm right, it means Cyclops was wrong to have a monitor in an AS which is not a "real" operator.
On Tue, Jul 28, 2009 at 11:50:02AM +0100, Russell Heilling <chewtoy@s8n.net> wrote a message of 75 lines which said:
I guess ROBTEX didn't implement ingress filters after the last episode...
I simply asked them and they told me that DCP (AS 13214) is simply their transit provider so they cannot put a max-prefixes or list the prefixes announced in an ACL.
Russell Heilling wrote:
2009/5/11 Ricardo Oliveira <rveloso@cs.ucla.edu>:
Hi all,
First, thanks for using Cyclops, and thanks for all the Cyclops users that drop me a message about this.
It seems some router in AS13214 decided to originate all the prefixes and send them to AS48285 in the Caymans, all the ASPATHs are 48285 13214. The first announcement was on 2009-05-11 11:03:11 UTC and last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn afterwards)
It looks like AS13214 are misbehaving again... We have just started receiving cyclops alerts indicating that AS13214 is announcing our prefixes again:
We are seeing the same thing for two of our prefixes: Offending attribute: 66.251.224.0/19-13214 Offending attribute: 66.146.192.0/19-48285 Pretty annoying --steve
Seeing the same thing here. Had alerts from Cyclops roll in for all 7 of our prefixes at: 2009-07-28 08:30:26, lasted 35 mins or so: Alert ID: 4910940 Alert type: origin change Monitored ASN,prefix: 174.137.112.0/20 Offending attribute: 174.137.112.0/20-13214 Date: 2009-07-28 08:30:26 UTC Duration: 00:00:01 (hh:mm:ss) --kyle On Tue, Jul 28, 2009 at 7:53 AM, sjk<sjk@sleepycatz.com> wrote:
Russell Heilling wrote:
2009/5/11 Ricardo Oliveira <rveloso@cs.ucla.edu>:
Hi all,
First, thanks for using Cyclops, and thanks for all the Cyclops users that drop me a message about this.
It seems some router in AS13214 decided to originate all the prefixes and send them to AS48285 in the Caymans, all the ASPATHs are 48285 13214. The first announcement was on 2009-05-11 11:03:11 UTC and last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn afterwards)
It looks like AS13214 are misbehaving again... We have just started receiving cyclops alerts indicating that AS13214 is announcing our prefixes again:
We are seeing the same thing for two of our prefixes:
Offending attribute: 66.251.224.0/19-13214
Offending attribute: 66.146.192.0/19-48285
Pretty annoying
--steve
Isn't this the second time that AS13214 seemed to have made a "unintentional misconfig"? On Mon, May 11, 2009 at 3:05 PM, Ricardo Oliveira <rveloso@cs.ucla.edu>wrote:
Hi all,
First, thanks for using Cyclops, and thanks for all the Cyclops users that drop me a message about this.
It seems some router in AS13214 decided to originate all the prefixes and send them to AS48285 in the Caymans, all the ASPATHs are 48285 13214. The first announcement was on 2009-05-11 11:03:11 UTC and last on 2009-05-11 12:16:32 UTC, there were 266,289 prefixes leaked (they were withdrawn afterwards)
As indicated in the Cyclops alerts, only a single monitor(AS48285) in route-views4 detected this leak. I checked on other neighbors of AS13214 and they seem fine, so it seems it was only a single router issue.
This incident shows the advantage of having a wide set of peers for detection, it seems Cyclops was the only tool to detect this incident. Given the amount of banks and financial institutions in the Caymans, i would otherwise have raised a red flag, but it seems this case was an unintentional misconfig by AS13214.
Would appreciate any further comment on the tool, and happy cyclopying!
--Ricardo the Cyclops guy http://cyclops.cs.ucla.edu
On May 11, 2009, at 8:30 AM, Jay Hennigan wrote:
We're getting cyclops[1] alerts that AS13214 is advertising itself as
origin for all of our prefixes. Their anomaly report shows thousands of prefixes originating there.
Anyone else seeing evidence of this or being affected?
[1] http://cyclops.cs.ucla.edu/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
-- --sharlon
On Tue, Jul 28, 2009 at 09:45:28AM -0400, Sharlon R. Carty <me@sharloncarty.net> wrote a message of 57 lines which said:
Isn't this the second time that AS13214 seemed to have made a "unintentional misconfig"?
participants (20)
-
Andree Toonk -
bmanning＠vacation.karoshi.com -
Christian Seitz -
Christopher Morrow -
David Freedman -
Hank Nussbacher -
James Kelty -
Jay Hennigan -
Jon Lewis -
Kyle McLerren -
Mans Nilsson -
Mikael Abrahamsson -
Nathan Ward -
Ricardo Oliveira -
Robert D. Scott -
Russell Heilling -
Sharlon R. Carty -
sjk -
Stephane Bortzmeyer -
Vincent Hoffman