At 03:54 PM 8/20/2007, Dan Armstrong wrote:

We're going crazy up here, I'm trying to nail down where exactly the problem is - We don't use Cogent anywhere, but we're having terrible problems with Bell and many sites in Europe...

Bell uses Cogent in a large way. The second traceroute was from an IP in their AS (577) out. I am prepending out Cogent, but Bell does everything it can not to use Teleglobe so I am having problems influencing their routes to come back that way. They also have a very odd path out of Chicago. This is from a site in Toronto (source IP in AS577) back to me peering with Cogent's router in Toronto... Toronto, Chicago, Kansas, Texas, Washington, Boston, Albany, Toronto. Thats quite the milk run. Usually its Toronto, Chicago, Toronto. % traceroute -f 6 -q1 199.212.134.3 traceroute to 199.212.134.3 (199.212.134.3), 64 hops max, 40 byte packets 6 64.230.229.5 (64.230.229.5) 4.846 ms 7 64.230.242.97 (64.230.242.97) 4.030 ms 8 64.230.147.14 (64.230.147.14) 14.213 ms 9 206.108.103.142 (206.108.103.142) 14.216 ms 10 p13-0.core01.ord01.atlas.cogentco.com (154.54.11.29) 101.348 ms 11 te3-1.mpd01.ord01.atlas.cogentco.com (154.54.1.206) 102.507 ms 12 t2-4.mpd01.mci01.atlas.cogentco.com (154.54.2.233) 108.993 ms 13 t4-2.mpd01.iah01.atlas.cogentco.com (154.54.5.221) 108.496 ms 14 t2-2.mpd01.dca01.atlas.cogentco.com (154.54.2.145) 110.221 ms 15 t8-2.mpd01.bos01.atlas.cogentco.com (154.54.1.105) 129.529 ms 16 g2-0-0.core01.bos01.atlas.cogentco.com (154.54.2.213) 108.507 ms 17 p13-0.core01.alb02.atlas.cogentco.com (154.54.7.41) 116.526 ms 18 p14-0.core01.yyz01.atlas.cogentco.com (66.28.4.218) 118.463 ms 19 v3491.mpd01.yyz01.atlas.cogentco.com (154.54.5.78) 217.912 ms 20 v3492.mpd01.yyz02.atlas.cogentco.com (154.54.5.82) 225.491 ms 21 sentex.demarc.cogentco.com (38.104.158.78) 217.134 ms 22 i3-vl-814 (67.43.129.242) 65.576 ms 23 shell1 (199.212.134.3) 66.221 ms ---Mike

Mike Tancsa wrote:

...

Does anyone have any details about the Cogent outage that started this morning (9am GMT-400) and is still continuing ? If its a fibre cut between Montville (NJ?) and Cleveland OH (http://status.cogentco.com/) why is it so bad in Chicago and Albany locations ? Is there really that little excess capacity ?

My connection out of Toronto is pretty bad via Albany

3 g8-22.mpd01.yyz02.atlas.cogentco.com (38.104.158.77) 7.470 ms 6.754 ms 6.481 ms 4 v3493.mpd01.yyz01.atlas.cogentco.com (154.54.5.85) 6.981 ms 6.730 ms 6.984 ms 5 g2-0-0-3490.core01.yyz01.atlas.cogentco.com (154.54.5.73) 6.482 ms 7.175 ms 5.974 ms 6 p4-0.core01.alb02.atlas.cogentco.com (66.28.4.217) 105.954 ms 112.055 ms 111.426 ms 7 p6-0.core01.bos01.atlas.cogentco.com (154.54.7.42) 115.413 ms 117.090 ms 113.816 ms

and Bell's through Chicago is even worse

6 64.230.229.5 (64.230.229.5) 12.572 ms 36.983 ms 200.187 ms 7 64.230.242.97 (64.230.242.97) 4.685 ms 5.439 ms 3.645 ms 8 64.230.147.14 (64.230.147.14) 14.351 ms 15.344 ms 14.387 ms 9 206.108.103.142 (206.108.103.142) 14.374 ms 14.280 ms 14.255 ms 10 p13-0.core01.ord01.atlas.cogentco.com (154.54.11.29) 156.616 ms * 142.150 ms 11 te3-1.mpd01.ord01.atlas.cogentco.com (154.54.1.206) 135.199 ms 138.900 ms * 12 t2-4.mpd01.mci01.atlas.cogentco.com (154.54.2.233) 152.292 ms 149.956 ms 148.095 ms 13 t4-2.mpd01.iah01.atlas.cogentco.com (154.54.5.221) 149.047 ms 150.556 ms 151.232 ms

---Mike

-------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike

On Mon, 20 Aug 2007, Mike Tancsa wrote:

Bell uses Cogent in a large way. The second traceroute was from an IP in their AS (577) out. I am prepending out Cogent, but Bell does everything it can not to use Teleglobe so I am having problems influencing their routes to come back that way. They also have a very odd path out of Chicago. This is

Bear in mind that doing "everything they can not to use Teleglobe" probably involves local preference. Local preference comes before AS path length in the BGP selection order, so nothing you can do with prepending is going to help. You'll need to either keep them from seeing the undesirable path at all (drop the announcement, ask your upstreams to limit its propagation, etc.) or convince Bell not to use it. Depending on the setup, you may be able to limit route propagation with communities, or it may require some phone calls. -Steve

Date: Mon, 20 Aug 2007 23:25:12 +0100 From: "Rod Beck" <Rod.Beck@hiberniaatlantic.com> Sender: owner-nanog@merit.edu

As opposed to 'unintentionally sabotaged'? I think there is some redundancy there ...

Sorry for the cheap shot, it was just too tempting.

Roderick S. Beck Director of EMEA Sales Hibernia Atlantic 1, Passage du Chantier, 75012 Paris http://www.hiberniaatlantic.com Wireless: 1-212-444-8829. Landline: 33-1-4346-3209 AOL Messenger: GlobalBandwidth rod.beck@hiberniaatlantic.com rodbeck@erols.com ``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.

If someone sabotages a rail to stop a train and the derailment takes out the fiber that is buried in the right-of-way, is that unintentional sabotage? At least of the fiber? Just asking...;-} -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

Date: Mon, 20 Aug 2007 19:17:21 -0400 From: Deepak Jain <deepak@ai.net>

...

If someone sabotages a rail to stop a train and the derailment takes out the fiber that is buried in the right-of-way, is that unintentional sabotage? At least of the fiber?

That's not sabotage at all.

As it relates to the fiber, its not "deliberately and maliciously destroying [fiber]"... unless the goal was to cause a derailment which hurt the train operator both from the operation of the train and subsequent revenues they might lose from fiber operations along the same right-of-way -- then *that* would be sabotage.

In the case you outlined above, barring other motivations, the fiber would be "collateral damage".

That said, Cogent's page says nothing about sabotage (status.cogentco.com) and I can't find the reference on teliasonera's page.... Link please?

Too many people missed the emoticon. Clearly the fiber damage in the case I gave was collateral damage. It would have been sabotage on the rail line and the derailed train. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

Eric Spaeth wrote:

This appears to be affecting Telia as well. Here was their last update:

"Concerning the cable break near Cleveland we have been informed that the cables have been intentionally sabotaged. The provider informed that they need to change the whole damaged fibre part and that is 3600 feet. Fibre has been ordered and ETA is 1900 UTC. Once the fibre arrives they need to blow it into the 3600 feet long duct before the splicing can start."

Sounds like a DHS/FBI investigation will be starting soon. Eesh.. if we start having to secure 500,000 route miles of fiber routes against sabotage, um... well, I guess I'll have to become a fiber installation contractor. :) DJ

Eric Spaeth wrote:

This appears to be affecting Telia as well. Here was their last update:

"Concerning the cable break near Cleveland we have been informed that the cables have been intentionally sabotaged. The provider informed that they need to change the whole damaged fibre part and that is 3600 feet. Fibre has been ordered and ETA is 1900 UTC. Once the fibre arrives they need to blow it into the 3600 feet long duct before the splicing can start."

-Eric

Um, anyone have any information on how, why and who might be involved with this sabotage? I'm not suggesting an increase in the threat level, but it might be nice to know if the "Free Silicon" movement has decided to make this a day of protest or something.... Anyone spots anything, please post! -- Jeff Shultz