On Sat, Oct 08, 2005 at 12:45:32AM -0400, Jon Lewis wrote:

On Fri, 7 Oct 2005, David Hubbard wrote:

...

...

I don't remember seeing this public notice from Level(3) posted.... Wouldn't that be "without notice from Level(3)"?

They notified Cogent, not the public. Cogent chose to

I think it's also interesting, that AFAIK, Level3 didn't give their own customers any advance notice. We're a customer. I saw nothing about this until it hit nanog. We're multi homed, so the impact on us was unnoticed.

I don't know how you missed it, but as far as I can tell every sales rep (3) has was mobilized to call every customer or potential customer they have ever spoken to during the month of Augusst, informing them about the receipt of a depeering notice from Level 3 and offering affected customers zero-commit Cogent ports. Among the people I know who took that offer, Cogent actually worked quickly (quicker than normal I would say) to get service up before the event. There was also a post to NANOG about the depeering in early September. While Cogent may not have taken out a full page ad in the New York Times for it, they certainly made every effort to inform those who needed to know, and they were never ambiguous about the fact that they were fully expecting to be segmented from (3). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

DISCLAIMER: From one of the clueless During this entire debaucle, I never saw any mention of: 1) Cogent sending "transit" traffic to Level3, which leads me to believe that all the traffic from Cogent through the peering points was actually *destined* for Level3 customers. Does the routing support this idea? Is it safe to assume the opposite, also... that only traffic destined for Cogent customers came through the Level3 peering points? And that Level3 had one and only one path to Cogent (no one else providing transit for them to Cogent AS'es?) 2) Level3 making any contingency for their own customers to reach Cogent networks (any announcements to their own customers) 3) Possible traffic issues. Was Cogent guilty of not transporting the Level3-bound packets within the Cogent network to the closest point-of-entry peer to the host in the Level3 network, therefore "costing" Level3 transit of their own packets? In other words is it also a traffic engineering issue? Are some of the business issues solvable by proper engineering and filtering (or statistics-jockeying)? ----- Original Message ----- From: "Jon Lewis" <jlewis@lewis.org> To: <nanog@merit.edu> Sent: Friday, October 07, 2005 9:45 PM Subject: RE: Level 3's side of the story

On Fri, 7 Oct 2005, David Hubbard wrote:

...

...

I don't remember seeing this public notice from Level(3) posted.... Wouldn't that be "without notice from Level(3)"?

They notified Cogent, not the public. Cogent chose to

I think it's also interesting, that AFAIK, Level3 didn't give their own customers any advance notice. We're a customer. I saw nothing about this until it hit nanog. We're multi homed, so the impact on us was unnoticed.

Suppose you're a single homed L3 or Cogent customer doing regular business with a single homed Cogent or L3 customer. If your provider gave you several weeks notice, and if you realized the coming problem, you might take some steps to work around the issue, depending on how important your internet communications are. Do the typical peering NDAs forbid giving customers this sort of notice? Is it better to surprise them with a multi-day outage and then give them 30 days notice that it's going to happen again??

...

...

Splendid, that gives the world sufficient time to accept Cogent's offer of 1 year free service.

This is not the first time Cogent has used their customers as pawns in peering disputes, I don't know if I'd jump on the bandwagon so quickly (spoken as a customer of both companies).

If you're multihomed and using Cogent as a cheap bandwidth whore, does it matter if their cheap bandwidth gives you 155k routes instead of 168k routes? After all, if its cheap and off-loads enough traffic from your more expensive 168k route circuits, isn't it doing what you bought it for?

Also, is 30 days really enough time for anyone to get a free connection to Cogent? I mean if you're in a building they're already in, and its just a cross connect, sure that can be done quickly...but at least around here, getting any sort of high bandwidth circuit (>T1) can take months. IIRC, the UNE DS3 connecting our office to the rest of our network was several months late.

---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

-- No virus found in this incoming message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.11.13/124 - Release Date: 10/7/2005

-- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.11.13/124 - Release Date: 10/7/2005

Given that at least part of JC Dill's comments were directly lifted from an e-mail I sent him, I feel compelled to put them side by side. JC's comments: In a message written on Sat, Oct 08, 2005 at 07:24:06AM -0700, JC Dill wrote:

Consider a simple hypothetical closest-exited network setup (hot potato routing) between 2 peers:

ISP Eyeballs: Router-E1----2,000 Mile Link----Router-E2----Customer | | | | peering peering | | | | ISP Content: Server--Router-C1---2,000 Mile Link-----Router-C2

When the customer on ISP E (Eyeballs) requests content (web page, music file, etc.) from the server on ISP C (Content), packets travel like this:

Customer->Router-E2->Router-C2->Router-C1->Server

When the server returns traffic to the customer, traffic goes like this:

Server->Router-C1->Router-E1->Router-E2->Customer

The problem is the customer->server direction would typically be a 500 byte request and 64 byte ACK packets, where as the server->customer data includes many 1500 byte data packets. So, ISP Eyeballs may carry 2Mbs of data over its 2,000 mile link, where as ISP Content will only carry 128Kbs over its 2,000 mile link.

Even though both companies met in the middle, ISP Content shifted some of its costs to ISP Eyeballs.

Back when most ISPs had the same types of traffic (even mixes of content and eyeballs), they had even ratios which equalized this effect, as it was happening the same amount in both directions. But as some ISPs started specializing in one type of content or the other, uneven flows were produced. Some bean-counter felt that these uneven flows meant that the network that was sending more traffic should now pay for transit, even though this traffic was traffic that their own customers were requesting and paying them to transmit!

There are ways to deal with it though, like cold potato routing.

My message to JC: In a message written on Thu, 6 Oct 2005 15:07:05 -0400, Leo Bicknell wrote:

That's not how it works. Consider a simple, closest exited network setup:

ISP A: Router1-------2,000 Mile Link-----Router2----Customer | | | | peering peering | | | | ISP B: Server----RouterA-------2,000 Mile Link-----RouterB

When the "customer" requests a web page from the "server", packets travel like this:

Customer->Router2->RouterB->RouterA->Server

When the server returns traffic to the customer, traffic goes like this:

Server->RouterA->Router1->Router2->Customer

The problem is the customer->server direction is a 500 byte request, and 64 byte ACK packets, where as the server->customer data includes lots of 1500 byte data packets. So, ISP A may carry 2Mbps of data over it's 2,000 mile link, where as ISP B will only carry 128k over it's 2.000 mile link.

Even though both companies met in the middle, ISP B shifted it's costs to ISP A.

The theory is that having an even ratio equalizes this effect, as it's happening the same amount in both directions. There are other ways to deal with it though, like cold potato routing and other tricks.

In practice it becomes a much more complicated issue, but in many cases due to how routing works, geographies involved, and others routing policies (eg, customers not advertising their routes to all providers) there are very real, very expensive inequalities. Large ISP's work together to equalize them to the extent possible.

It's not so much that I mind quoting without attribution, it's that I mind interleaving original bits with new bits as that can lead to confusion later least anyone put A and B together. On the issue at hand, surface it to say costs are a complicated issue. This is a rather simplified view of the world, and does not take into consideration many of the things that are going on. For instance, many content providers buy cheap Cogent bandwidth and dump traffic on Cogent, but DO NOT advertise ANY prefixes to Cogent, or at least, not the ones generating traffic. So in my example if we assume "customer" is Level 3, and "Server" is someone buying from Cogent, the path from customer to server may not transit Cogent's network at all. Level 3 may send that to say, AT&T, who's also a service provider for the person with the server. There's also a lot of other considerations that come down to various people's choices. How many people bought ATM cards and had them be the only ATM cards in their entire network because you needed them to connect to AADS, Mae-East, and UUNet? That's extra cost because of those other entities technology choices. People with OSR's at peering points love GigE interconnects, people with GSR's generally don't like them. People with GSR's may like OC-3 interconnects, those with M160's probably hate them. If all layer 1, layer 2, and layer 3 technology had the same cost/megabit(/mile) then it would all be the same, but the fact is it doesn't, and so based on a providers other business assets they will see very different costs. In some cases both sides are "right". Believe me I've been in many discussions that went like "I can only do GigE right now cuz it's all I can afford", well "I can only do OC-12 right now because it's all I can afford". Level 3 could be pulling out of particular peering points, which happen to be where Cogent is located, and Cogent is not where they are going to be in the future. Cogent could have been using localpref to artificially raise or lower level 3 traffic. Level 3 may have wanted to upgrade full circuits that were dropping packets every day and Cogent refused, causing them to terminate the agreement for failing to work with them on the problem. I could come up with a hundred other reasons this happened. For better or for worse it is all under NDA I'm sure, and frankly if it were our NDA I'm not sure what Cogent has said so far would be acceptable. I find it rather sad that "engineers" would be trying to solve a problem when they don't in fact know the true cause of the problem. All we've seen in a single symptom, down peering. We have no idea what has, or has not been said between them for the last few months. What the graphs look like, what the netflow statistics show. What the costs are to both parties, and how much they make on the topic. It's a wonder the internet works as well as it does, not because of Level 3 and Cogent partitioning, but because of the lack of clue on the part of the people who are (supposedly) running it. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org

On Sat, Oct 08, 2005 at 07:24:06AM -0700, JC Dill wrote:

Cogent was a "tier 1" until prior de-peering incidents left them unable to reach other networks. They solved this by buying filtered transit thru Verio to reach the networks they couldn't reach via peering.

For the record, Cogent was never a Tier 1. They have never had Sprint peering (unless you count the 30 seconds between acquisition of a company that did have it, and the depeering notice, years ago). Cogent's history of depeering debacles, at least as best as I can remember them, is: ATDN (AS1668) depeers Cogent, December 18 2002. http://www.cctec.com/maillists/nanog/historical/0212/msg00366.html http://www.cctec.com/maillists/nanog/historical/0212/msg00412.html ATDN is in the process of shutting off legacy transit and peering on its path to tier-1-dom, and disconnects Cogent due to ratio (also technically Cogent is still on a trial peering session). At this time, Cogent is a full transit customer of AboveNet (AS6461), ATDN is still a full transit customer of Level 3, and Cogent is a peer of Level 3. Following the depeering, Cogent shifts 100% of the traffic to their (3) peers, which become severely congested nearly 24/7 for several weeks. Despite being able to send some traffic to AboveNet transit, they decide to leave traffic congested to (3) to see if ATDN will repeer (not knowing that AOL customers don't know what peering is and thus won't be nearly as vocal as Cogent's customers). Traffic stays congested until Cogent's peering capacity with (3) is upgraded. ATDN later switches their routing with (3) from transit to customer-only routes (removing the last of their transit paths), at which point Cogent shifts traffic to newly acquired Verio transit to reach them. Teleglobe (AS6453) depeers Cogent, some time in Feb 2005? Don't ask me why but I can't find a NANOG thread discussing this. Teleglobe depeers Cogent due to various ratio and market pressure issues. Of note is that Cogent has recently entered the Canadian market where Teleglobe has a strong presence, and has started giving away free or nearly free transit to large inbound networks. Teleglobe is a Sprint customer, and Cogent reaches Sprint through Verio. Teleglobe is caught completely off-guard when Cogent refuses to accept the route via Sprint transit, and blocks traffic between the networks. This continues for several days, until eventually routes are leaked/added from Teleglobe to SAVVIS (AS3561), who Cogent peers with. This continues for a few days more until Teleglobe finally agrees to repeer Cogent. France Telecom (OpenTransit/AS5511) depeers Cogent, April 14 2005 http://www.merit.edu/mail.archives/nanog/2005-04/msg00484.html FT depeers Cogent due to, well, a variety of issues and general unhappiness surrounding Cogent's entrance into their markets through the purchase of Lambdanet. FT is a Sprint customer, Cogent is already receiving Sprint routes via Verio but intentionally blocks these routes so that they have no path to FT. The rumored resolution to the dispute is that a FT customer sues Cogent in France, and a French judge either does or is about to fine the hell out of Cogent unless connectivity is restored. At this point Cogent caves, and begins accepting the routes via Sprint (via Verio). Of course I am certain there are a lot more depeerings (both from and to Cogent) that did not make the news, but these are the big notable events that dramatically impacted connectivity. For anyone keeping score, the last two times Cogent was depeered, it responded by intentionally blocking connectivity to the network in question, despite the fact that both of those networks were Sprint customers and thus perfectly reachable under the Sprint transit Cogent gets from Verio. While no one has come forward to say if the Cogent/Verio agreement is structured for full transit or only Sprint/ATDN routes, Cogent has certainly set a precedent for intentionally disrupting connectivity in response to depeering, as a scare tactic to keep other networks from depeering them.

L3 was hoping to force Cogent to increase that transit to include the traffic destined for L3's customers, thus raising Cogent's transport costs at no additional (transport) cost to L3.

As I've already pointed out, L3 depeering Cogent is in fact a major revenue loss for L3. Not only will they not make any money off of Cogent (since we both know Cogent will NEVER give them money for direct transit), but Cogent will heavily depref them and shift many many gigabits of traffic away from L3 and onto their competitors, traffic that L3 was previously billing their customers for. They'll also lose customers during the unreachability, and even if Cogent buckles and buys transit they'll lose some outbound traffic from their multihomed customers due to a longer as-path length to reach Cogent and many of Cogent's routes (11k of them remember). Let me be perfectly clear here, under absolutely no line of logic will L3 see an increase in revenue from this, period. If you think they will, you don't understand how the Internet works. What L3 will see from this is a REDUCTION IN BILLABLE TRAFFIC AND BACKBONE UTILIZATION.

...

3) Possible traffic issues. Was Cogent guilty of not transporting the Level3-bound packets within the Cogent network to the closest point-of-entry peer to the host in the Level3 network, therefore "costing" Level3 transit of their own packets?

Possible, in fact probable. Most ISPs hand off traffic to peers under a "hot potato" policy, they hand it off at the closest point where they connect. If the traffic is equal in both directions then this works. If the traffic is not equal, then this lowers the cost of the network that has high outbound traffic, as the other network bears the brunt of the total cost for transporting the combined traffic between their respective customers.

Do you know why people hot potato traffic? Because MEDs suck. In addition to the obvious aggregation issues (for example how do you assign a MED value to 4.0.0.0/8, it is used around the world), they usually end up producing sub-optimal routing. IGP cost is a view of what it costs YOU to get the packet off your network. MED values set to the opposing network's IGP cost is a view of what it costs THEN to get the packet off their network. Neither is a complete view of reality, and the MED view just happens to be worse. Consider a simple scenerio, You operate a major network, you peer with someone who operates a major network, you both intelligently aggregate your prefixes and work with your customers to make certain that everything in BGP maps to a specific geographic region, and you both interconnect with each other in the usual "maxium reasonable extend possible" locations (New York, Ashburn, Chicago, Dallas, San Jose, Los Angeles, Seattle, Atlanta, Miami) across the US. Now lets say you have a customer who is in Chicago, and they're sending data to a customer in, oh lets say Denver. In hot-potato routing, you get the packet off your network in Chicago, and then the other network uses its more complete and detailed understanding of where this packet is going within its own network to know that Chicago->Denver is a straight shot. In a cold potato situation however, you are only looking at the other network's IGP cost, not your own. Denver is pretty much dead center in the middle of San Jose, Chicago, and Dallas, and which one is "closer" is really up for grabs. On the vast majority of networks, Dallas is actually closest by IGP cost, with San Jose a close second, and Chicago a close third. If you're cold potato'ing to try and improve routing, even under the most ideal conditions possible (which given the current financial state of the carriers involved RARELY happens these days), you're going to end up hauling packets to some out of the way place like SJC or DFW, and then the other network is going to end up hauling packets back to Denver. You both lose the "saving money by hauling traffic less" game, and your customers lose in suboptimal routing. The heart of the problem is that you need to consider your cost + their cost to have meds be effective, even if you solved all the implementation issues that you will never practically solve. Unfortunately since two networks have no way to coordinate metrics on the same "scale" (my 43ms may be 4300 igp cost, your 43ms may be 43, and joe bob's 43ms cost may be 9182), you have no reliable way to "add" the two costs. Now, networks who are looking for equity in the ratios have a choice. They can either: * Spend thousands of man hours deaggregating (and then listen to you complain about poluting the routing table with prefixes) * Spend millions of dollars deploying more gear into more interconnection locations in areas of network presense but not peering presence (Denver, St Louis, Kansas City, New Orleans, Tampa, Phoenix, etc etc etc), all in areas without well defined peering locations where they are likely to end up in buildings across the block but which cost thousands of dollars to connect, or * Establish these as smaller interconnections across telco circuits, again spending thousands of dollars a month more in circuits, hundreds of thousands of dollars in ports, tens of thousands of man hours managing capacity at five dozen new interconnections around the world, all while reducing to almost zero the ability for a major interconnection to fail over to another major interconnection during a maintenance, fiber cut, network event, etc. * Break their customers routing in the process of doing all this. -OR- * Depeer said network, expect that they will buy transit from Verio or any of the other dozens of networks who provide this service, and that whoever ends up interconnecting with them to deliver the traffic will have equitable traffic. Now, which one do you think they're going to pick?

There are ways to deal with it though, like cold potato routing.

Spoken like someone who has never dealt with the reality of running a large network, or dealt with customers wondering why you are routing their traffic across the country and back again. Anyone who values the quality of their connectivity will stick to arm-chair engineering and not actually building a network this way. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Sat, Oct 08, 2005 at 04:37:29PM -0400, Patrick W. Gilmore wrote:

You mention at least twice (here and in the FT depeering paragraph) that Cogent "accepts" the routes.

It is entirely possible, and in fact likely IMHO, that the prefixes were never offered by Verio to Cogent. Cogent pays Verio for partial transit, why would Verio give Cogent more ASes than they paid for? If Verio doesn't announce the prefixes, how can Cogent filter them?

Of course, I do not have enable on Cogent or Verio border routers, so I cannot say for certain. Would anyone who _knows_ care to enlighten us?

Because FT and Teleglobe are both full transit customers of Sprint, with full global routes in, and full propagation out (this is verifiable via many looking glasses). You aren't seriously going to claim that Cogent has a contract with Verio which says "We will give you partial transit aka only Sprint routes, but not Sprint routes to certain Sprint customers like FT and Teleglobe", and that Cogent is throwing up its hands and saying "sorry our contract doesn't give us routes to you, Verio won't let us change it, what are we going to do?" are you? But while we're on the subject, how do you think Sprint feels about Verio selling Cogent "Sprint routes only"? I of course am not privy to exact wording of the peering contract between Verio and Sprint (and if I was I sure as hell wouldn't be talking about it on NANOG), but on many peering agreements there is usually a clause that contains words like "shall not give or sell nexthop to others". At the very least, it is down-right unfriendly. Verio peers with FT and Teleglobe directly already, which means that in order for them to send Cogent those routes via Sprint (which Cogent now clearly still uses, 174 2914 1239 5511), they must have Cogent directly connected on the same routers as their Sprint interconnections, and have a virtual RIB set up, into which they import only the Sprint routes. That does raise some interesting questions about how the contract is written. But I agree, we're just speculating, and I'm certain no one is going to give us an answer publicly. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Sat, 8 Oct 2005, Richard A Steenbergen wrote:

the last two times Cogent was depeered, it responded by intentionally blocking connectivity to the network in question, despite the fact that both of those networks were Sprint customers and thus perfectly reachable under the Sprint transit Cogent gets from Verio. While no one has come forward to say if the Cogent/Verio agreement is structured for full transit or only Sprint/ATDN routes, Cogent has certainly set a precedent for intentionally disrupting connectivity in response to depeering, as a scare tactic to keep other networks from depeering them.

i dont see it like that.. and you reapply your view in your later email to me. cogent and level3 were peers. level3 want to change that, the only solution level3 will consider is for cogent to purchase transit with another provider (sprint/verio) or pay them direct. whether cogent's contract with verio could provide it transit to level3 for the same price is irrelevant. the fact is cogent currently does not use verio for this and they do not want to add a number of Gbps to their transit service theres nothing special about level3 being tier-1 and cogent being tier-2 with verio transit. the status of these networks is not of issue, both sides have a right to decide whether to connect via settlement free peering or not. of course for level3 to transit to cogent would be inconceivable to them, but thats ego / economics / marketing, not a principle of networking that either network could use transit to reach the other is an engineering point, that neither wants to pay to do so is a business point. and this is a business problem. Steve

On Sat, Oct 08, 2005 at 04:54:52AM -0700, Eric Louie wrote:

DISCLAIMER: From one of the clueless

During this entire debaucle, I never saw any mention of:

1) Cogent sending "transit" traffic to Level3, which leads me to believe that all the traffic from Cogent through the peering points was actually *destined* for Level3 customers. Does the routing support this idea? Is it safe to assume the opposite, also... that only traffic destined for Cogent customers came through the Level3 peering points? And that Level3 had one and only one path to Cogent (no one else providing transit for them to Cogent AS'es?)

Neither Level3 nor Cogent utilize a middleman called "transit" to reach each other. Sure, Cogent has Verio as its transit, but Verio is used as a partial transit to only reach some spot nets (e.g. Sprint, France Telecom, ATDN, etc). You may want to better understand the difference between a transit relationship and a peering relationship. Here is exerpt from wbn's Equinix paper: 8<--- Transit is the business relationship whereby one ISP provides (usually sells) access to all destinations in its routing table. --->8 (Note: Also several ISPs offer "regional transit" or "limited/partial transit", by providing regional routes instead of full routing table access. Cogent's Verio transit service is partial transit.). 8<--- Peering is the business relationship whereby ISPs reciprocally provide access to each others' customers (no access to whole internet is provided). --->8

2) Level3 making any contingency for their own customers to reach Cogent networks (any announcements to their own customers)

This is unheard of unfortunately to date. But as mentioned, Cogent has made numerous contacts through sales channel to accomodate customer needs before the disconnection.

3) Possible traffic issues. Was Cogent guilty of not transporting the Level3-bound packets within the Cogent network to the closest point-of-entry peer to the host in the Level3 network, therefore "costing" Level3 transit of their own packets? In other words is it also a traffic engineering issue?

No, that was BBN vs. Exodus depeering superbowl back in 90's, which was resolved through cold potato routing. Level 3 claims Cogent is sending far more traffic than Level3 to Cogent. Thus, Level3's viewpoint is that Cogent relies on them more than they rely on Cogent. Thus, it no longer makes sense in their view point to maintain a free interconnection as there is no similar balance of traffic ratio. Cogent claims Level3 depeered because of competingly low market prices for transit bandwidth sold by Cogent. Cogent further claims that it is Level3 who requested Cogent to send more traffic. So... which side is really true, no one knows at this point, but these are what both sides are stating.

Are some of the business issues solvable by proper engineering and filtering (or statistics-jockeying)?

Yes and no. Cogent could technically deprefer Level3 peering partially or totally to reduce the egress traffic to Level3 peering. Since many of Level3 customers are multihomed, deprefering Level3 peering to certain extent would cause significant chunk of bits to reroute to Cogent's other peers who are providing additional transit to said customers. But you have to realize though that often times, significant peering disputes arise from business and political related issues, not technical. There are lot of technical peering disputes going on everyday, and majority of those can be resolved quietly without even impacting users other than a "scheduled maintenance" event. The current depeering dispute is really more or less "desperation depeering" as Richard pointed out. Thus it is not necessarily initiated by technical disputes that can be resolved using traffic engineering, but trying to see who blinks and pays up. Some may call it "extortion", some may call it "the depeeree deserved it" or some may call it "both sides burning bridges." and other views, etc.. James -- James Jun Infrastructure and Technology Services TowardEX Technologies Office +1-617-459-4051 x179 | Mobile +1-978-394-2867 james@towardex.com | www.towardex.com

On Oct 8, 2005, at 11:42 AM, John Curran wrote:

Cold potato routing alone is insufficient in many cases, and some form of settlement becomes necessary.

http://www.cctec.com/maillists/nanog/historical/9808/msg00517.html

This does not in any way explain why cold-potato routing is insufficient. But I seriously doubt a consensus will be found in this forum, or between John & me. :) Let's just say that there _are_ technical means to make "traffic imbalances" either (close to) fair, or reverse the financial burden. And almost every hoster I know (and certainly some of the ones involved in the original BBN / Exodus / Above.Net / etc. fracas) were perfectly willing to take on more of the cost so the eyeball network actually has a financial incentive to peer. -- TTFN, patrick

DISCLAIMER: From one of the clueless i'm not picking on you as theres a huge amount of misinformation being posted on

On Sat, 8 Oct 2005, Eric Louie wrote: this thread but perhaps a private email to someone with questions might be better. having said that, perhaps this will serve to educate..

During this entire debaucle, I never saw any mention of:

1) Cogent sending "transit" traffic to Level3, which leads me to believe that all the traffic from Cogent through the peering points was actually *destined* for Level3 customers. Does the routing support this idea? Is it safe to assume the opposite, also... that only traffic destined for Cogent customers came through the Level3 peering points? And that Level3 had one and only one path to Cogent (no one else providing transit for them to Cogent AS'es?)

peering is all about exchanging bgp on your customers with the peer, it excludes sending routes from another peer which is usually called transit

2) Level3 making any contingency for their own customers to reach Cogent networks (any announcements to their own customers)

understand the inability to reach cogent was the desired result for level3, had a contingency been put in place level3 would have been heading in the opposite direction to which they are moving (they are moving to force cogent to buy transit, not moving to pay for their connectivity to cogent nor to keep the current settlement free arrangement)

3) Possible traffic issues. Was Cogent guilty of not transporting the Level3-bound packets within the Cogent network to the closest point-of-entry peer to the host in the Level3 network, therefore "costing" Level3 transit of their own packets? In other words is it also a traffic engineering issue?

cogent has not got a transit provider giving them level3 routes (as far as we understand) and they have not gone and setup any such transit arrangement whilst waiting for the depeering. it is not allowed for you to send traffic to another network via a peering. so eg if you peer with me we will send you only our customer routes, if you forcibly get cogents routes and route them over us without our permission you are stealing bandwidth and we will either depeer, sue, or both. to obtain this 'permission' would mean us acting as a transit for you and we'd probably want some money to do that. (consider i do not exchange money with my peers, if one peer uses me to reach another peer nobody is paying me for the use of my network).

Are some of the business issues solvable by proper engineering and filtering (or statistics-jockeying)?

short answer: no, this is a political and business problem not one of engineering. longer answer: level3 may be claiming that either cogent has insufficient traffic, that it has too much outbound or that it isnt in enough geographic locations. cogent could invest money to comply with level3s peering requirement. but this ultimately results in cogent spending money either to meet a new peering requirement or paying level3 direct to maintain a settlement peering. Steve

----- Original Message ----- From: "Jon Lewis" <jlewis@lewis.org> To: <nanog@merit.edu> Sent: Friday, October 07, 2005 9:45 PM Subject: RE: Level 3's side of the story

...

On Fri, 7 Oct 2005, David Hubbard wrote:

...

...

I don't remember seeing this public notice from Level(3) posted.... Wouldn't that be "without notice from Level(3)"?

They notified Cogent, not the public. Cogent chose to

I think it's also interesting, that AFAIK, Level3 didn't give their own customers any advance notice. We're a customer. I saw nothing about this until it hit nanog. We're multi homed, so the impact on us was unnoticed.

Suppose you're a single homed L3 or Cogent customer doing regular business with a single homed Cogent or L3 customer. If your provider gave you several weeks notice, and if you realized the coming problem, you might take some steps to work around the issue, depending on how important your internet communications are. Do the typical peering NDAs forbid giving customers this sort of notice? Is it better to surprise them with a multi-day outage and then give them 30 days notice that it's going to happen again??

...

...

Splendid, that gives the world sufficient time to accept Cogent's offer of 1 year free service.

This is not the first time Cogent has used their customers as pawns in peering disputes, I don't know if I'd jump on the bandwagon so quickly (spoken as a customer of both companies).

If you're multihomed and using Cogent as a cheap bandwidth whore, does it matter if their cheap bandwidth gives you 155k routes instead of 168k routes? After all, if its cheap and off-loads enough traffic from your more expensive 168k route circuits, isn't it doing what you bought it for?

Also, is 30 days really enough time for anyone to get a free connection to Cogent? I mean if you're in a building they're already in, and its just a cross connect, sure that can be done quickly...but at least around here, getting any sort of high bandwidth circuit (>T1) can take months. IIRC, the UNE DS3 connecting our office to the rest of our network was several months late.

---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

-- No virus found in this incoming message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.11.13/124 - Release Date: 10/7/2005

On Sat, 8 Oct 2005, Jon Lewis wrote:

If you're multihomed and using Cogent as a cheap bandwidth whore, does it matter if their cheap bandwidth gives you 155k routes instead of 168k routes? After all, if its cheap and off-loads enough traffic from your more expensive 168k route circuits, isn't it doing what you bought it for?

I wonder if Cogent customers can get their money back if more than 0.1% of their packets go to L3 single-homed networks ;) http://www.cogentco.com/htdocs/internet.php?current=10 Product Features: FULL SLA: Our SLA guarantees 99.99% network availability, 99.9% packet delivery, less than 50 milliseconds roundtrip latency and proactive outage notification within 15 minutes. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan