Qwest has confirmed a DOS attach against two of their Juniper routers in the NY POP. I believe they had a UDP attack last week also (maybe on Saturday). This time the DOS was a TCP attack on the 100Mb management interface on the Juniper, leaving the box unable to pass packets, hence BGP stays up and a full routing table but you cannot get anywhere. Patrick We saw the same issue on Saturday morning too (eastern time). BGP sessions to Qwest stayed up, but no data was moving across Qwest links. Kind of ugly. Emails to Qwest seeking information on Saturday went unanswered. Drew Linsalata The Gotham Bus Company Internet Server and Carrier Neutral Co-Location http://www.gothambus.com
From: Eric Gauthier <eric@roxanne.org> Date: Wed, 8 May 2002 10:43:57 -0400 To: nanog@merit.edu Subject: Qwest outage in NY
Heya...
Just found out that Qwest had an outage in one of their NY pops (jfk?) that disrupted access out of the Boston area for about 20 minutes. The master ticket (I believe) is 566548. Does anyone know what happened?
Thanks,
Eric :)
On Wed, May 08, 2002 at 12:32:00PM -0400, Patrick McEvilly wrote:
Qwest has confirmed a DOS attach against two of their Juniper routers in the NY POP. I believe they had a UDP attack last week also (maybe on Saturday). This time the DOS was a TCP attack on the 100Mb management interface on the Juniper, leaving the box unable to pass packets, hence BGP stays up and a full routing table but you cannot get anywhere.
Ok I'll bite... What crackpipe are you smoking from? If the link from the RE to the PFE (the fxp1) became saturated, or enough packets hit the RE to blow away the processor, BGP (and the CLI, and everything else) would certainly fall over. Much like with any other router using distributed forwarding, if the management processor dies, the traffic will continue to forward until the routing protocols timed out and the rest of the network stopped sending it traffic. The attack would then stop hitting the box in question, it would come back up, and the cycle would repeat. This assumes that there are actual routing protocols, in the case where it's statically routed the box just stays down. :) But Juniper is more resilient to this form of attack than most, and you have the ability to filter packets going to the RE on any IP rev. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Let me clarify, our directly connected Qwest router was not under DOS attack so BGP stayed up and we had a full routing table. The router that got hosed was 3 router hops into their backbone and it was definitely hosed good. :-) -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Richard A Steenbergen Sent: Wednesday, May 08, 2002 1:21 PM To: Patrick McEvilly Cc: nanog@merit.edu Subject: Re: Qwest outage In NY On Wed, May 08, 2002 at 12:32:00PM -0400, Patrick McEvilly wrote:
Qwest has confirmed a DOS attach against two of their Juniper routers in the NY POP. I believe they had a UDP attack last week also (maybe on Saturday). This time the DOS was a TCP attack on the 100Mb management interface on the Juniper, leaving the box unable to pass packets, hence BGP stays up and a full routing table but you cannot get anywhere.
Ok I'll bite... What crackpipe are you smoking from? If the link from the RE to the PFE (the fxp1) became saturated, or enough packets hit the RE to blow away the processor, BGP (and the CLI, and everything else) would certainly fall over. Much like with any other router using distributed forwarding, if the management processor dies, the traffic will continue to forward until the routing protocols timed out and the rest of the network stopped sending it traffic. The attack would then stop hitting the box in question, it would come back up, and the cycle would repeat. This assumes that there are actual routing protocols, in the case where it's statically routed the box just stays down. :) But Juniper is more resilient to this form of attack than most, and you have the ability to filter packets going to the RE on any IP rev. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
On Wed, 8 May 2002, Patrick McEvilly wrote:
Let me clarify, our directly connected Qwest router was not under DOS attack so BGP stayed up and we had a full routing table. The router that got hosed was 3 router hops into their backbone and it was definitely hosed good. :-)
Sounds like another point for the whole argument for separate management networks and separation of the device control plane from the route processor/packet forwarding plane. Nice to know some of the big guys don't quite do it either ;-) jms
Qwest has confirmed a DOS attach against two of their Juniper routers in the NY POP. I believe they had a UDP attack last week also (maybe on Saturday). This time the DOS was a TCP attack on the 100Mb management interface on the Juniper, leaving the box unable to pass packets, hence BGP stays up and a full routing table but you cannot get anywhere.
The story I just got from Qwest (from a NOCie who was reading from their ticket, so take this with a grain of salt) made it sound like that were flooded with bogus routes from some BGP peer. I tend to believe what you wrote above though. I mean, getting a bunch of bogus routes via a BGP peer doesn't seem like the kind of thing where you'd call the vendor onsite (several Qwest NOC'ies stated that Juniper was onsite) whereas a large-scale DOS might... Anyways, that's the scoop that I've got /me returns to lurking Eric :)
Qwest has confirmed a DOS attach against two of their Juniper routers in
NY POP. I believe they had a UDP attack last week also (maybe on Saturday). This time the DOS was a TCP attack on the 100Mb management interface on
We have been working an issue with Qwest for the past two months where they simple black hole all our traffic for no known reason. We had an escalation procedure to get directly to the Ops Eng group when this event started this morning as we are still trying to find out what causes it in the past. Today's event had the very same symptoms as before but one router hop further into the network from the past 2 times it happened. Below is what the Ops Eng guy told us happened (very reluctantly). -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Eric Gauthier Sent: Wednesday, May 08, 2002 4:11 PM To: nanog@merit.edu Subject: Re: Qwest outage In NY the the
Juniper, leaving the box unable to pass packets, hence BGP stays up and a full routing table but you cannot get anywhere.
The story I just got from Qwest (from a NOCie who was reading from their ticket, so take this with a grain of salt) made it sound like that were flooded with bogus routes from some BGP peer. I tend to believe what you wrote above though. I mean, getting a bunch of bogus routes via a BGP peer doesn't seem like the kind of thing where you'd call the vendor onsite (several Qwest NOC'ies stated that Juniper was onsite) whereas a large-scale DOS might... Anyways, that's the scoop that I've got /me returns to lurking Eric :)
participants (4)
-
Eric Gauthier -
Patrick McEvilly -
Richard A Steenbergen -
Streiner, Justin