A couple weeks later than expected, but as of Oct 5 02:51AM EDT it looks like 3356 and 174 are no longer reachable. lg.level3.net: Show Level 3 (Washington, DC) BGP routes for 38.9.51.20 No matching routes found for 38.9.51.20. www.cogentco.com looking glass: Tracing the route to www.Level3.com (209.245.19.42) 1 f29.ba01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 4 msec 0 msec 2 * * * 3 * * * I guess the earlier reports of (3)'s lack of testicular fortitude may have been exagerated after all. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, 5 Oct 2005, Richard A Steenbergen wrote:
It's sure causing a few headaches here. (from level3 looking glass) Show Level 3 (London, England) BGP routes for 38.9.51.20 No matching routes found for 38.9.51.20 As of 16:22 BST Level3 still seems to have no routes for cogent's space. thats about 5 hours now. Vince
I opened a billing/support ticket with Cogent. I'm not planning on paying my bill or continuing the contract if they cannot provide full BGP tables and full Internet transport (barring outages). Luckily I have 2 other providers so I can still reach Level 3. Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money. -Matt On Oct 5, 2005, at 11:29 AM, Vince Hoffman wrote:
-- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
On Wed Oct 05, 2005 at 11:50:52AM -0400, Matthew Crocker wrote:
We tried the same line with Level3 - and were told "Tough, we're not paying service credit. The transit still works, just its coverage is slightly different".
Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money.
Indeed, that's the natural next step. Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director | * Domain & Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: info@bogons.net *
On 5-Oct-2005, at 11:57, Simon Lockhart wrote:
Cogent seem to be suggesting an alternate solution, which involves them giving away free transit to l(3) customers. This might be useful for some here who are feeling pain. [I realise that there are probably at least nine sides to this story and that the text here represents just one of them.] http://status.cogentco.com/
This is what I just got from Cogent support. I'm still waiting on the billing dispute ticket. I've already told our payables department to not pay any Cogent invoices, this should get fun. Hell, I wish Verio never sold me to Cogent in the first place, it is all their fault :/ <quote> Hello, As of 5:30 am EDT, October 5th, Level(3) terminated peering with Cogent without cause (as permitted under its peering agreement with Cogent) even though both Cogent and Level(3) remained in full compliance with the previously existing interconnection agreement. Cogent has left the peering circuits open in the hope that Level(3) will change its mind and allow traffic to be exchanged between our networks. We are extending a special offering to single homed Level 3 customers. Cogent will offer any Level 3 customer, who is single homed to the Level 3 network on the date of this notice, one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3. Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe. For status updates and further information on the special offering -- please see our status page at http://status.cogentco.com </quote> -Matt On Oct 5, 2005, at 11:57 AM, Simon Lockhart wrote:
-- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
On 10/5/05, Matthew Crocker <matthew@crocker.com> wrote:
Or, more likely, pointless... I have to wonder though, if your network access is this mission critical, why are you single homed?
Hell, I wish Verio never sold me to Cogent in the first place, it is all their fault :/
Wow, if I had a penny for everytime I heard "It's all Verio's fault" I'd have left there a very rich man and would now be cruising on some yacht in the bahamas avoiding hurricanes and figuring out where to import the next case of Scotch. Thanks for the trip down memory lane.... -doug
Since Cogent was depeered couple times before and each time they fairly quickly setup transit connection (without statement like the one below), I seem to be getting an impression that they are a lot more angry this time for whatever reason and it might not be resolved quite as easily. Am I wrong to think that this is not quite the game of chicken this time and neither one will provide connection to the other unless a lot and lot of outside pressure is put on them to resolve it? On Wed, 5 Oct 2005, Matthew Crocker wrote:
On Oct 5, 2005, at 3:46 PM, william(at)elan.net wrote:
Or perhaps it would just cost them so much more to fix this time they decided not to? I don't know, but it seems like a possibility. Plus, define 'quickly'. FT was many days, nearly a week.
That was true every time. -- TTFN, patrick
At 11:50 AM 05/10/2005, Matthew Crocker wrote:
Perhaps they took the 3 in level3 to mean 3 routes ? ;-) cogent-gate# show ip bgp regex 174 .* 3356$ Network Next Hop Metric LocPrf Weight Path * 63.135.164.0/24 38.112.9.13 15001 100 110 174 701 18905 14745 1239 3356 i * 194.32.125.0 38.112.9.13 18000 100 110 174 1273 9121 3549 3356 i * 194.32.127.0 38.112.9.13 18000 100 110 174 1273 9121 3549 3356 i Rather than swamping their desk with the same ticket, can you report back to NANOG what they tell you ? ---Mike
On Oct 5, 2005, at 11:50 AM, Matthew Crocker wrote:
I'm curious where in your contract you think Cogent guaranteed you connectivity to Level 3? Most transit contracts only guarantee packet delivery to the edge of their own networks. I'm pretty sure Cogent is doing that. (Hell, they have lots of spare capacity now. :) Of course, I would claim that the word "transit" has certain implications, but IANAL. (I'm not even an ISP. :) So perhaps someone could enlighten me on how one would go about asking for credits for this ... disconnectivity. Also, I've already heard from customers of L3 single-homed providers that L3 will _NOT_ be issuing credits. So I guess the question goes for L3 contracts as well.
Maybe I can buy the new 'Cogent - it is almost the Internet' service for less money.
Maybe. Would you pay L3 for "almost the Internet" as well? There is nothing wrong with "partial transit". If we could get partial transit at 50% off full transit pricing, we would absolutely consider it - depending on things like which "part" of the Internet is served. And why aren't people asking for partial transit pricing from providers who do things like filter smaller prefixes (because they are too cheap or too dumb to run a real backbone) or entire countries (say, for spam) or other things? -- TTFN, patrick
My original contract was with NTT/Verio which Cogent purchased last year when Verio nuked their Boston POP. I'm having the contract dug out of the archives to look at what it says. IMHO I pay Cogent for Transit to the whole Internet, If I wanted partial transit or local peering I would order/contract and pay for that. Cogent is not currently providing me full transit service. I really don't care who pulled the plug, it is Cogents job to fix it for me as I am their customer.
Most also have a clause to cover the inter-AS links, making sure that they are not overloaded.
Yes, if the price were right. -- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
On Wed, 5 Oct 2005, Matthew Crocker wrote:
"Isn't BGP supposed to work around this sort of thing?" This comes down to a little more than just "depeering" -- at least in the BGP sense. There's active route filtering going on as well if connectivity is dead; after all, I can bet the house that at least one of Cogent's network edge peers has connectivity to Level3, and vice versa.
So perhaps the question you should be asking is: Why didn't routes for these networks fall over to the other upstream peers which *are* capable of moving the packets? Surely MCI, AT&T, Sprint, and others would carry the packets to the right place. I can see the paths right here....
What nature of clause? I consider deliberately filtering prefixes or origin ASs to be a violation of common backbone BGP use. Too bad there aren't Equal Access laws for tier1s. <slyly evil grin> -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
They did, and I'm not down. I see Level 3 via Sprint and GNAPs/CENT just fine. I didn't lose any connectivity to Level 3 at all. Bits moving down different pipes, not a big deal to me technically. The fact remains that Cogent is not providing the service I'm paying them for and they need to get it fixed. If that means picking up transit from another Tier 1 to get to Level 3 or making amends with Level 3 to get the existing peering working again. It doesn't matter to me, I just don't like paying for stuff I'm not getting. In the grand scheme of things I'm paying A LOT for my Cogent bandwidth (it started off as Verio remember).
Ewww, I'll put up with these occasional pissing matches and build around them to avoid any government regulations. -Matt -- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
On Wed, 5 Oct 2005, Matthew Crocker wrote:
No, I mean: Why didn't *your upstream's* routes fall back to *their* other peers, who should be perfectly capable of transiting those packets? The thinly veiled implication there is that "full mesh" is not a long term effective way to run the backbone level transit, because dropping one peer without an alternate path means that we get broken transit. Yum. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
On Oct 5, 2005, at 2:04 PM, Todd Vierling wrote:
You are very, very confused on how the Internet works. Which network do you run, sir? I find it useful to know which networks have engineers who understand not only "conf t" but also what the commands they type actually mean. -- TTFN, patrick
On Wed, 5 Oct 2005, Patrick W. Gilmore wrote:
Obviously my attempt at being the cynic didn't play out to some folks. My mention of BGP being capable of "working around" the lost connectivity was an attempt to illustrate one of the Stupid Factors of these peering agreements, as in the portion of my followup that you quoted:
This was my real meaning. These backbone leve "free" peering agreements that are still common form the [all too conveniently forgotten] SPOF of the tier-1 space. The price push that this depeering causes may be an artifact of market pressure. Personally, I see both parties as having their fingers on the wrong buttons. But when taking the role of the cynic, I couldn't care less about what the market implications are -- I care that end users are f00ked over in the aftermath. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
At 01:49 PM 10/5/2005, Matthew Crocker wrote:
Unless you had an outage of both your Sprint and GNAPS circuits, at which point it'd be a REALLY big deal. I've got two upstreams, with Cogent being one of them. I'm getting L(3) through my other, but if the other were to drop, I would not have routes to L(3). Just how many links should an edge provider need to buy to ensure uninterrupted service in the event of a circuit failure? Two used to be enough if reasonably chosen.
I have the same concern. I'm buying transit from Cogent. I'm sure the Cogent marketing materials did sell this on the basis of it covering traffic anywhere on the Internet, whether the fine print of the contract did or not.
So the edge providers either suffer, or spend a lot on lawyers to add penalty clauses to contracts for when their upstreams get in pissing matches with other backbones. Not fun either way.
On 10/5/05, Matthew Crocker <matthew@crocker.com> wrote:
So, where's the problem, exactly?
fact remains that Cogent is not providing the service I'm paying them for and they need to get it fixed.
Really? As you already pointed out, your packets are reaching their destination. So, they don't "need" to get anything "fixed." What utter nonsense... *shakes head and walks away* -doug
On Oct 5, 2005, at 2:47 PM, Douglas Dever wrote:
Um, I only have 2 routes to Level 3 when I should have 3 routes and I'm paying for 3 routes...
Ok, I *pay* Cogent for 'Direct Internet Access' which is IP Transit service. I *cannot* get to part of the internet via Cogent right now. I also *pay* Sprint and GNAPS for 'Direct Internet Access' and I can get to all parts of the internet via their networks. I *used* to be triple redundant to *all* of the Internet but now I only have *two* connections to Level 3. My packets are reaching their destination because I'm smart enough to be multi-homed, that doesn't remove the responsibility of Cogent to do what I *pay them to do*. Cogent is *not* providing complete Internet access, I really don't care who's fault it is.
What utter nonsense...
*shakes head and walks away*
Is it really that hard to understand? As a paying Cogent customer I expect to be able to get to the Internet through them. Isn't that the business they are in?
-doug
-- Matthew S. Crocker Vice President Crocker Communications, Inc. Internet Division PO BOX 710 Greenfield, MA 01302-0710 http://www.crocker.com
Not to add fuel to the fire, but many IP contracts with my upstreams have a clause, which is very similar across vendors: "VENDOR cannot guarantee the peering sessions between our ourselves and other companies and/or networks. There is no guarantee of end to end connectivity between you as a CUSTOMER and other non-VENDOR controlled networks." While it actually has meaning now, I am not sure you'd get a vendor to delete that from an agreement. On Wed, 5 Oct 2005, Matthew Crocker wrote:
-- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben Net Access Corporation, 800-NET-ME-36, http://www.nac.net
Matthew Crocker wrote:
Right now *neither* Cogent nor Level3 are providing complete Direct Internet Access. This is a self-solving problem - why would anyone buy internet access (or renew existing contracts as they expire) from either of these networks when neither of them connect to the complete internet?[1] Either one of them caves in and buys access to the other (or to another network that will transit to the other), or they repeer, or they BOTH lose sales. Your weapon is your sales rep. Use your sales rep to make your position clear to your network provider. If you are presently a customer of either network you need to contact them now. Contact your account manager and demand service credit. Contact your sales rep and notify them that you will not be renewing your present service with them or buying any service from in the future unless you get your service credit AND this problem is immediately fixed. For extra measure, copy the message to your other providers. This puts them on notice that if they ever try such a stunt they will end up with the same problem. Then start your search for a replacement provider. If every Cogent and Level3 customer did this today, this problem would be solved by the end of the week, guaranteed. Service credit demands will get their attention but if every current customer says "we are going to cancel this service and buy from someone else who actually connects to the complete internet and not just a subset of the internet" then the complaints flooding in from the SALES department will really get their attention. Neither of them can afford to have their sales stream dry up while they play chicken. jc [1] Then there's the publicity problem. I can't find any news articles online about this depeering. Write or call your local newspaper (or favorite tech magazine) and explain why this is news and you want them to cover it. A flurry of articles about how some businesses can no longer access the "complete internet" due to a connection and routing spat between 2 large backbone internet providers will be juicy news for the business section and add to each network's sales problems.
I tend to think this is oversimplification. The big picture risk, cogent will be judged now by their actions, lest they run the risk of being de-peered by others. A few low yield, short term customers crying about rebates, could in comparison be quite insignificant.
jc
-- James
James Spenceley wrote:
You say that as if the only move to be made is on Cogent's side. What about L3? If every L3 customer complained to L3, demanded service credits, claimed the contract was in default, and swore to never buy from L3 again, maybe L3 would budge instead. My point is that if BOTH of them look to lose business over this it will push them back to the negotiating table to find a way to resolve it to their mutual satisfaction because if they do NOT resolve it they both ARE going to lose sales over it. Depeering never makes sense to me. Customers of both companies are expecting their vendor to connect them to the customers of the other company. These customers are each paying their respective vendor for this service. Why should one vendor pay the other for this traffic that is mutually beneficial to them while the other pays nothing? Why does the amount of bandwidth or the direction it travels make any difference? The customers are PAYING for the bandwidth. If each vendor pays their own costs to a peering point then they should be passing that cost onto their respective customers as part of the *customer's* bandwidth bill. RAS posted:
So Level3 is pissing off 7.5% of their own customers by failing to transmit their traffic to Cogent, and return traffic back to the customer. How does L3 justify this behavior to those customers? Somehow I don't think the customer is going to be too happy with an explanation that goes like this: "Even though you pay us for internet access, we want this other network to pay us TOO so that we get paid double for transmitting your traffic to and from that other network rather than doing a mutual exchange like we do with other big networks." Yeah, right, that's going to go over real well. If I were single-homed at L3 my answer would be "If my money isn't good enough to get you to provide the internet access I pay for, then I'll take my money elsewhere to another company who isn't intentionally breaking my internet access as they try to double-dip and get paid twice for the same the traffic." If anyone had prefix lists from before and after this depeering, it would be a great sales opportunity for other networks to contact those poor stranded customers and offer them REAL internet access. (Just don't email them, that would be spam! Let your fingers do the dialin'.) jc
On Wed, Oct 05, 2005 at 04:51:12PM -0700, JC Dill wrote:
Point being, they don't both feel that the traffic is mutually beneficial. Peering relationships end when either one of the parties decided they don't want to swap bits for free any more, at which point notice is given, and the two parties make arrangements to deliver their traffic via alternate means.
At the risk of wasting everyone's time by explaining how and why peering and transit works on a mailing list which is supposed to be loaded with people who are experts on the subject (HAH), I think you're missing quite a few concepts that are central to the debate. First of all, do you have any idea how much it costs to operate a sizable network? Empty routers alone cost hundreds of thousands of dollars, then each OC192 or 10GE card costs hundreds of thousands of dollars more. You need routers in pairs for reliability, interconnected, and you need an aggregation layer to efficiently terminate customers. You need to interconnect individual nodes across a diverse backbone, if you're operating a fiber network you need to pay for LH DWDM equipment which costs hundreds of thousands of dollars for every card used along the path, which must be repeated many times, equaling hundreds of cards. You then need to operate metro fiber networks to deliver the bits to the specific buildings where the customer is located, which means more fiber costs, WDM costs, possibly telco circuit costs. We're already in the hundreds of millions of dollars on the cheap side, billions on the expensive side, of capex, plus millions in operational expenses. And we haven't even covered colocation, powered, techs, crossconnects, peering interfaces, traffic management, network engineers, corporate overhead, sales staff, etc. And Cogent sells transit for $10/Mbps. How much traffic do you think is OUT there to pay for this? Yes they lose money hand over fist doing it (though not nearly as high a burn rate in absolute dollars as some other networks *cough*), and it can only be sustained by investors pumping money into the system over and over again, under the delusion that somehow they'll come out of it making money, but it does happen. Despite the delicious irony that Level 3 was the original disruptive pricing carrier many years ago, Cogent is the disruptive pricing carrier now. They actually run a pretty lean and efficient shop, which is how they're able to keep the losses as low as they do, but Cogent makes what little money it does make on being able to deliver massive amounts of bandwidth to carriers all over the Internet for as cheap as possible. Much of its traffic is hosting based and outbound only, which means that they closest-exit ("hot potato") bits onto other networks, and leave those networks responsible for an extremely disproportionate amount of the longhaul, costing millions of dollars alone on an ongoing basis. All the while, Cogent undercuts the market of every other carrier who isn't as efficient as they are, leading to massive losses, bankruptcy filing after bankruptcy filing, out of court reorganizations and purchases for foreign companies, etc. In case you missed it, this industry is in a state of utter financial disaster on almost all fronts right now. Billion dollar companies are surviving on scraps and what they can sell from previously purchased infrastructure at any price, with no ability to handle the need for upgrades when the existing infrastructure runs out. Networks everywhere are running out of money, pissed off, and realizing that the access to their customers which they provide for free to Cogent is what is enabling Cogent to operate and offer the pricing that it does. They don't feel that it is of mutual benefit any more, and they want it to stop. Level 3 also delivers much more "expensive" traffic, is a real Tier 1, has almost 6x the customer base of Cogent by prefix count (though probably roughly the same or even less overall traffic). It is not unreasonable to expect them to want to capture as much settlement as possible from networks putting traffic into theirs, infact this is a critical requirement in order for them to compete. As with any Tier 1 network, a huge amount of the traffic stays on-net. Of course reality is far more complex than even that little generic description above. Peering is highly political, affecting sales, potential customers, and industry goodwill positively and negatively whether you do or don't peer, and whether you do so with the same policy ("fairly") for all networks. If you look at Cogent's previous routing, it is blatantly clear that they were never going for ratio balance with Level 3. It is entirely possible that Level 3 was asking them for traffic with which they could bill their customers, rather than delivering that traffic via the competition. Of course the only thing you're going to see on this list about why L3 felt the need to depeer Cogent in this particular instance is pure speculation, but there ARE very real and very steep costs associated with peering and peering policies at all levels. These costs do need to be taken into account, and L3 certainly needs all the money it can get its hand on (which is why it has depeered many more networks than just Cogent recently). If anything, I'm personally surprised they didn't do this sooner. Cogent is one of the few networks out there with the balls to put their customer base on the line in this way to fight such a depeering (possibly because their customer base consists so largely of multihomed and/or extremely cheap networks who are willing to endure this and other issues to save a buck). They certainly don't want to look weak on this issue, or other networks with whom they may have less than perfect peering relationships will certainly move on taking similar steps. What happens here could set the tone for many more future depeerings, or lack there of, and in some ways the survival of both companies and the course of the IP transit industry at large. Both networks are highly motivated and have perfectly reasonable positions which make sense to them, hence they are at an impass which can only be resolved through suffering. To force either one to peer by law or regulation would be even more disruptive to the industry as a whole, which may even lead to a complete collapse of the existing peering system as we know it. After all, if anyone can become or retain an "undeserved" peer by the BGP equivalent of saying "you can't make me route packets to you any other way", we could see this kind of thing happening a lot more often, trading short term customer satisfaction for long term economical gain. If forced to keep "undeserved" peers in the long term, peering may vanish completely, to be replaced by some other means of settlement (at least without government funded networks at the cost of billions). Some of the few insights on this list on the subject of depeering have been provided by folks like Vijay Gill and Dan Golding, if you want to get a better feel for why these large networks are motivated to depeer others I highly recommend reading their posts before jumping to any conclusions that the way you see peering as a small network operator has ANY relationship to the way a large network operator has to see it. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wednesday 05 October 2005 15:52, JC Dill wrote:
I would think in NANOG that one would know the simple fact that 'The Complete Internet' is complete and utter fiction, and does not exist. What does exist is a complex, dynamic, even stochastic set of relationships between autonomous networks, who can pick and choose their relationships at whim. We conveniently label this collection of relationships as 'The Internet' and erroneously treat it as an individual, which it is not. Sometimes these individual networks are even antagonistic to each other; boo-hoo. Cry me a river; they've done what they've done and most SLA's for transit don't cover traffic outside the transit network. Take it up with your upstream, who will probably simply say it's not their problem (and it's not, unless your upstream is Cogent or Level3). You cannot reach what doesn't exist, and the Compleat Internet does not exist. All philosophy aside, it does bother me that a simple single depeering can cause such an uproar in a network supposedly immune to nuclear war (even though the Internet was not designed from the start to survive nuclear war; Paul Baran's packet-switching work aside; reference 'Where Wizards Stay Up Late' which quotes Taylor and others on the origins of the ARPAnet portion of the old Internet). I shudder to think of what would happen if there were to be a real problem (I mean, really, one link (out of many thousands) is down and the Sky Is Falling!). What happened to resiliency? 'Hold her steady, steady, Mr. Sulu.' -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
In a message written on Thu, Oct 06, 2005 at 06:36:00PM -0400, Lamar Owen wrote:
I've seen a lot of comments about the "disruption" caused by this depeering event, and what would happen if $bad_thing happened. I point you back a few weeks to when the hurricane hit. You need look no further to see people offering up their assistance to those in need. Look back further to 9-11, and people offering networking help to those who's infrastructure was damaged. I have no doubt that if the Level 3 / Cogent issue had been caused by a pre-emptive nuclear strike and the nation was called to arms that virtually every ISP that connects to both would be offering them free transit to get them reconnected. Indeed, I could log into my routers now and fix the Cogent / Level 3 problem with about 3 minutes of typing. It would cost my company thousands of dollars to do so, so I'm not going to do it. As I said before, right now this is a business problem. By the same token, if we were just attacked and Level 3 and Cogent were both, together, asking for help I'd log in and have them working as fast as I could type. I bet others would as well. Level 3 and Cogent are able to fix their own problems in this case, either by making up, or by entering into a business relationship with a third party to fix it. This is also a problem that they, themselves created. That's the difference here. I've got a new set of rules to add to this thread: If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment. You really don't understand what's going on here, and it's not, I repeat, not a technical problem. There is nothing wrong with the technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
I agree. Though many of the people who meet the second criteria don't even have enable anymore. :) That said, the business relationships that have evolved have certainly overwhelmed -- or rather use a very specific definition of connectivity/reachability, etc. When two transit free networks peer, its often in many locations over many different political regions (time zones, geographies, pick your term). Deliberately and voluntarily taking that down does not change the stability of the underlying architecture. Certainly anyone who runs any network of any size knows very well that the Internet does not survive conscience, deliberate breakage well at all -- nor was it architected to. Put a little knowledge into a border router that is a part of the "Internet" and watch the chaos you can create. Further, the "survivability" we talk about also requires that the end nodes, clients, networks, ISPs design for fault-tolerance. This would imply no single-connections. Like all de-peerings, this creates the most hardship for those Enterprise customers (and smaller customers) that either don't have the know how to know they need multiple providers and portable space or the smaller customers that can't afford it [business model or actual finance]. Those of us who are customers of both networks or customers of neither network wouldn't even notice. I think Cogent's offer of providing free transit to all single homed Level3 customers is particularly clever and being underpublicized. I wouldn't be surprised if Cogent is in more buildings than Level3 with a high degree of overlap with the entire Level3 lit network. That could be a very nasty "competitor" to force into your customers awareness by your own action (or inaction) -- especially if your customer is single homed to you and realizes now that isn't enough of the "Internet" for them. Deepak
* deepak@ai.net (Deepak Jain) [Fri 07 Oct 2005, 02:29 CEST]:
I guess a significant part of the single-homed networks behind Level(3) would be in PA space owned by them, and thus will find the initial step towards multihoming very hard to take (renumbering into PI or their own PA space). -- Niels. -- "Calling religion a drug is an insult to drugs everywhere. Religion is more like the placebo of the masses." -- MeFi user boaz
Its absolutely a high bar. It is no higher than changing providers which I would probably advocate to anyone who asked my opinion who was single homed. However the "to-whom" question looms larger and larger. The list of transit-free providers that have not forcibly depeered another network is growing short indeed. If Cogent were looking for an opportunity as a solution provider, they could provide PBR route-maps and the following suggestions: If you are an enterprise customer, many services like your main DNS servers, web server, etc. could gain IPs in both spaces and you could set up a proxy that can tell one space from another and reach both spaces. This would solve many of the access provider and webhosting provider problems out there. The ones that would stay broken are specialized applications and websites that are highly sensitive to proxies, etc. But as as a community, I think NANOGers would agree... something that smells like connectivity is still better than none. Deepak
On Oct 6, 2005, at 8:32 PM, Niels Bakker wrote:
Renumber why? If they have a /24, all they need is an AS & a BGP capable router. If they don't have a /24 or larger, then they will either need to renumber, or NAT, or some other fun magic. But the upper bound on the difficult of such exercises is exactly equal to changing providers. -- TTFN, patrick
Patrick W. Gilmore wrote:
Yes, and now that we know enough publically available details, it's time as a community to assist in moving all NANOG ISPs from Level3 to Cogent without renumbering.
It's unlikely that they have an AS when not already multihomed. If everybody with a /24 got an AS, we'd run out quickly. Bad policy. However, we should assist everybody without an AS and at least /24 to move to Cogent without renumbering. That means the blocks should be reassigned. That requires registry assistance. To avoid routing table explosion, we probably need to identify adjacent blocks and encourage them to move to Cogent, too. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
William Allen Simpson wrote:
It seems to me that the recent discussion between Leo Bicknell and Lamar Owen is relevant. A network partition affecting hundreds of thousands (millions according to the Boston Globe article) is closely akin to a natural disaster. In this case, an _unnatural_ disaster. I remember presenting a paper at IETF over a decade ago about assigning IP addresses to exchanges instead of carriers. That would allow rapid failover, and promote competition. Therefore, I propose that we identify those ISPs moving from Level3 to Cogent that are also connected to an *IX. The IP block should be reassigned by the registry to the IX. We should encourage everybody else moving that requires a circuit change to connect to an IX where Cogent is present. That should help routing tables immensely! -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
wsimpson@greendragon.com (William Allen Simpson) writes:
and i remember hearing you present this.
IX's are not, alas, the center of the internet, no matter what anybody's marketing might say. the vast majority of traffic exchange is by private interconnect, which might be an IX crossconnect, but might just as likely be some kind of ISO-L1 or ISO-L2 link through a metro or telco or whatever. there is no justification for making an IX into an address allocator or address aggregator, unless it's an ISO-L3 IX with regulatory powers, which isn't the norm in most places around the world, and isn't likely to become such, even if it were uncontroversial, which it also isn't. -- Paul Vixie
On Fri, 7 Oct 2005, William Allen Simpson wrote: > I remember presenting a paper at IETF over a decade ago about assigning > IP addresses to exchanges instead of carriers. Yes, that's been debunked many times over at this point. Still, it occurs to someone new, periodically. The problem is that the ISPs which matter are at more than one IX, so it results in massively fragmented routing, completely defeating aggregation. The cure is worse than the disease. -Bill
Bill Woodcock wrote:
Actually, your memory is incomplete. Wasn't it Deering who did the analysis that it would actually _decrease_ the _global_ table to mere hundreds, but increase the internal routing tables of large carriers? And didn't the analysis show that the maximum routing table of those large carriers was still in the few thousand range? Today's routing table reports show 167203 and 171237 respectively. Certainly we're much worse off now! And given the perspective of another decade of history, I'd argue that without the incentive to join IXs, we still have the problem that most "ISPs that matter" aren't connected to _any_ IX. (Noted as well by Paul Vixie in an earlier message.) -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
Randy Bush wrote:
Hmmm, I'll look for my old files (1993 was quite awhile ago in machine years) to see whether I still have that stuff. If I can find it, we should probably correspond privately about any errors in the analysis -- and how an analysis might be done today.
OK, on this I'll bite, as it's operational. On 07 Oct 2005, Paul Vixie wrote: # IX's are not, alas, the center of the internet, no matter what anybody's # marketing might say. the vast majority of traffic exchange is by private # interconnect, which might be an IX crossconnect, but might just as likely # be some kind of ISO-L1 or ISO-L2 link through a metro or telco or whatever. Are you agreeing with Bill Woodcock, who wrote "ISPs which matter are at more than one IX"? Perhaps you are quibbling that the "ISPs that matter" are not the "vast majority of traffic"? Or do you have some other insight on how to do massive moves quickly, without renumbering and without damaging the routing tables? -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
i am saying that paul did not say that most isps that matter are not connected to any ix, which can be clearly [not] seen in the text of his you quoted. what he also did not say, but you put words into his keyboard to contradict was that ... most isps that matter are actually at ixen, though they do not exchange large amounts of traffic there. ixen do not much like the gigabits of traffic exchanged between the major isps, and the isps don't particularly wish to expose that traffic to ixen. note that this is a somewhat us-centric view, but does extend somewhat. but what does this have to do with a collection of newcomers, amnesiacs, and pontificants [re]learning the facts of life about peering, depeering chicken, etc? no answer required. randy
Randy Bush wrote:
Unless you have data that is not published, there is no factual support for your initial statement. In the context of the earlier discussion, "ISPs that matter" would be those that have an AS, and are multi-homed, and have PI space. As far as I can tell from the data, _most_ are _not_ at ixen. And as I wrote, I argue that this is because of the lack of incentive. The incentive that was proposed many years ago -- when we already saw this coming and were small enough to do something about it -- was assignment of IP addresses based on IXen. That also had the distinct advantage that ARIN and others would be small one person operations, instead of policy-laden multi-million dollar boondoggles. The funding would more appropriately go to Ixen, which have an actual operational function.
The facts of life are that because of short-sighted oligopolists that don't follow good engineering practices, we're going to become heavily regulated. And we're not going to like it! But that's another thread. This one is devoted to rapidly moving singly-homed Level(3) customers without renumbering and without destroying the routing table. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
On Oct 8, 2005, at 8:40 AM, William Allen Simpson wrote:
Let's cut this thread short right here. Since you and Randy clearly have different definitions of "ISPs that matter", can we not spend 100 posts on what would be two people arguing different sides of different points? The sad part is you knew (or at least should have known, if you honestly believe your opinion os worth wasting over 10,000 people's time) what Randy meant. -- TTFN, patrick
Patrick W. Gilmore wrote:
Since I started the thread subject, and the phraseology was given by Bill Woodcock, I'm pretty sure _I_ know what _I_ was talking about. And suspected that Randy talking about something else or being otherwise pedantic (as has been his forte from time to time), so explicitly asked: # Perhaps you are quibbling ... This thread is about how to do massive moves quickly, without renumbering and without damaging the routing tables. Or do you think that (according to media reports) millions of singly- homed users will just renumber in a few days or weeks? My experience hasn't been that sanguine. Your other posts seemed to be more realistic. Let's keep the eye on the operational ball. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
On Oct 8, 2005, at 1:58 PM, William Allen Simpson wrote:
I was not questioning that you knew what you meant.
So you admit you knew that Randy and you different definitions, making the thread an argument over semantics rather than something operationally relevant.
This thread is about how to do massive moves quickly, without renumbering and without damaging the routing tables.
The thread might have been. Your post was not.
Your other posts seemed to be more realistic. Let's keep the eye on the operational ball.
Wish I could say the same. Talking about how to move that many people that quickly isn't really operational, since is not going to happen - even if it were technically possible. But at least it's close. Arguing over semantics is not. -- TTFN, patrick
um, no. or maybe yes. that's a different issue altogether than what i said.
Perhaps you are quibbling that the "ISPs that matter" are not the "vast majority of traffic"?
i don't know or care what isp's matter. i do know what i said, from personal experience as a co-founder and later president of PAIX, is what i meant.
Or do you have some other insight on how to do massive moves quickly, without renumbering and without damaging the routing tables?
the last idea i heard in that regard was IPv6's A6/DNAME dns architecture, which i strongly supported, and which would have given IPv6 a qualitative rather than quantitative advantage over IPv4. other than that, tli's comments on the thread where he finally claimed that IPv6 was "too little, too soon", whereas what we needed wasn't more bits but smarter routing, were the last intelligent words spaketh on this topic. i guess that means, no, i havn't got quick renumbering in my pocket. but i do know that the IX's don't have it either. let's talk about this again every ten years until one of us dies, OK? -- Paul Vixie
Paul Vixie wrote:
um, no. or maybe yes. that's a different issue altogether than what i said.
Paul, that message wasn't directed to you, it was to Randy.
i don't know or care what isp's matter. i do know what i said, from personal experience as a co-founder and later president of PAIX, is what i meant.
Again, to Randy. You and Woody have far more experience in this than anybody else in the field. Actually, I was somewhat surprised that after all these years and all the IX announcements that so few are actually using them. But not entirely surprised. In my small rural Mississippi experience, even though we were the first ISP there (or maybe because we were the first), we couldn't get a single incoming competitor to agree to interconnect even with a little mutual frame relay to exchange locally. Even though we were the only ISP for Ol' Miss students, when Ol' Miss itself connected via some "free" BellSouth program that kept us from competing, Ol' Miss was prohibited in the contract from interconnecting with us. Meaning all the packets went via Dallas to Atlanta to Jackson just to cross the street. The delays were a constant support irritant.
As to the former, the _original_ IPv6 was supposed to be a minimal change, rather than a wholesale redesign. The redesigners were IPv7 and IPv8 (ISO CLNP cum TUBA). As to the latter, speaking as a member of the original IPv6 design team, that is the same sentiment we shared. 64 bit addresses, smarter routing, only what was needed. The original name was "Simpler" IP.
Well, as it was 10 years from IPv4 to IPv6, it's been 12+ now, so maybe it's time to design the successor to IPv6.... ;-) -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
William Allen Simpson wrote:
If a single-homed network moves from L3 to Cogent, how would they benefit? Would they not still be cut off from a significant percentage of the Internet? Is it reasonable to think that numerous /24's from L3's IP space could be reassigned elsewhere without causing significant trouble for L3 and others? Even if it could work, what would be the justification for taking L3's property?
For educational purposes, could someone elaborate on how this would work? If you're a Level3 customer with Level3 PA space (assumed, since you're already assumed to be single-homed, and therefore very unlikely to need PI or BGP) and move to a Cogent circuit with Cogent PA space, then you'd be able to once again reach Cogent's view of the 'net, but then lose Level3's view of the 'net. If, on the other hand, you move to a Cogent circuit, but keep your Level3 PA space, wouldn't that at least require Cogent to announce all of these "recircuited" customers' Level3 blocks? This could stop working if Level3 filters those announcements, again resulting in non-reachability for existing Level3 downstreams? Or, on the other hand, is Cogent's offer not exclusive of maintaining the customer's existing Level3 circuit as well, in which case the customer will probably incur more pain with juggling two circuits while not speaking BGP in the first place? Or, is there another hand? Thanks. -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
news flash!!! "if you use a provider's address space, you're locked in!" if you don't qualify for your own address space, see http://www.google.com/search?client=safari&rls=en-us&q=vixie+multihoming+without+bgp&ie=UTF-8&oe=UTF-8 if you decide to use a provider's address space, then you can either pay the higher cost of locked-in transit pricing, or you can pay the higher cost of having to renumber whenever you want to break the lock. in a pay-me-now-or-pay-me-later scenario, you have to pick "now" vs. "later". (it's a pity that the internet, for all its power, cannot alter that rule.) -- Paul Vixie
It should be noted that if one opts for 'later', you can do quick and dirty games with NAT. Do not renumber, change providers and put a NAT between yourself and your provider. This will continue to work until such time as your original PA space is reassigned and then you will not be able to reach the new assignee. This allows for quick moves, but creates the mortgage of an eventual renumbering. Folks who take this approach are likely to renumber into RFC 1918 space. Before you break out the blowtorches, I'm *not* claiming that this a good way of doing things. It's a hack. It's expedient. ;-) Regards, Tony
In both of those cases, how much core infrastructure was damaged? (I've read the threads on this list (with interest, particularly some of the comments about many core paths from Atlanta to Houston not tranisting NOLA), and have read the archives of several lists on the earlier event; when the backbone providers want to do so, they can cooperate very nicely. When they don't want to do so, it gets ugly). If a tornado took out a major peering point, that would be different.
Yes, you would be correct. Which offers an interesting thought: why would it be important for you then but not now? If the issue impacts your customers, then why not spend the 3 minutes reconfiguring your router(s)? (obviously, if it doesn't impact your customers, then ignore that).
In other words, this problem is a problem simply because people can't be bothered to fix the problem because it's just a customer service issue, and not 'helping out fellow backbone providers?' Shades of the old backbone cabal here. (yes, a healthy dose of cynicism there)
As I said before, right now this is a business problem.
Absolutely. But who's business? Hint: the two parties involved aren't the only ones with a business stake in this issue. Of course, I'm not telling you something you don't already know.
I've got a new set of rules to add to this thread:
If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment.
Again, shades of the old backbone cabal. How do you know that I have neither? I have both, in fact, even though the negotiations for the SFI didn't pan out due to regulated carrier issues (after all, one must arrange transit to a peering point, and, while I have the Cisco 12000's sitting here, they are here and not there).
You really don't understand what's going on here, and it's not, I repeat, not a technical problem.
Of course. I fully understand that. There is nothing wrong with the
technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies.
There is something very wrong with the whole business if two players' business models and business decisions can make this much of an uproar. When other businesses won't help their customers see the illusion of it being fixed, something is wrong. That's why you multihome in the first place. And if you have customers, and you are single-homed, your business plan stinks anyway. That goes for content consumer as well as content producer customers. But you are very right; it is not a technical problem and never was. Why? Because technical problems don't typically take this long to fix, and, as you said, if enough people cared enough this could easily be a nonissue. Aside: love the domain name, Leo. -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
In a message written on Fri, Oct 07, 2005 at 10:40:50AM -0400, Lamar Owen wrote:
I venture any other ISP of importance either has direct connectivity to Cogent and Level 3, and/or buys transit from someone who does. All but the smallest most trivial ISP's are multi-homed. Those that are have seen no result from this, by and large. I can all my customers can get to both.
No, it doesn't affect anyone else's customers. Period. "Fixing it" in this case would be offering Charity to Level 3 and Cogent, and offering your competitors Charity, particularly for their own mistake is not high on most business plans. There's a very large difference between offering charity to a competitor, and keeping the industry going in the face of disaster. To suggest the two are related at all is just absurd. If someone wants to cut their network off from someone else for a business reason they will be able to do that whatever the design of the network may be. Level 3 and Cogent are both actively causing this outage. It's not some grand design failure. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On 10/5/05 3:02 PM, "Matthew Crocker" <matthew@crocker.com> wrote:
Break your contract for non-performance and call it a day. Cogent and L(3) deciding to depeer is an operational issue. Crocker Communications getting shafted on a transit contract? Not so much. - Dan
On Oct 5, 2005, at 2:47 PM, Douglas Dever wrote:
On 10/5/05, Matthew Crocker <matthew@crocker.com> wrote:
I think you are confused. Without making value judgments or saying what L3 / Cogent _should_ do, I think Matthew is saying that he paid Cogent for connectivity to the internet. So if his GNAPS circuit dies, he does not want to be cut off from L3 end users. Right now, he has no such guarantee. Exactly which part of this do you think is nonsense? Or do you think redundant circuits only need to be partially redundant? And, making value judgments, I can say (for me personally) that if I buy transit, I also expect to get to the _whole_ Internet, at least to a few decimal places. Neither L3 nor Cogent are providing that today. -- TTFN, patrick
On 10/5/05, Patrick W. Gilmore <patrick@ianai.net> wrote:
Having re-read this thread, you're right, I was confused. I mis-read what Matthew was telling me. I interpreted this as his routes from Cogent to L3 were now moving through Sprint as opposed to being directly connected - as opposed to him having multiple carriers. Obviously, that isn't the case. My bad. Sorry, Matthew. -doug
The part that defines "internet" implicitly as including any computer he wishes to reach regardless of its actual connectivity or policy. IMO, if a site or provider is not making a genuine effort to exchange traffic with anyone else willing to make a similar effort, it's not part of the internet. Neither Cogent nor Level 3 can force someone who does not wish to accept their traffic into doing so. All they can do is make a reasonable effort to exchange traffic with anyone else who will make such an effort. Level 3 cut of Cogent's connectivity. Until and unless they give some reason that makes sense, they are no longer making the effort and are not part of the internet. The fact that Cogent could make a spectacular effort and get connectivity is not relevant. Cogent could run a 100Mpbs line from their neaest POP to the machine in my garage that isn't connected to anything else and you could reach it. That doesn't mean I get to say my machine is an internet host you can't reach. My views may be colored though. I've heard Cogent's side of the story and nobody seems to know what Level3's side is. DS
On Wed, 05 Oct 2005 19:27:24 PDT, David Schwartz said:
If I had a garden, things would grow *so* wonderfully next year if I spread this stuff on it. So are you saying that if *your* AS was peered at a dozen places, and you dropped *one* because it wasn't cost-effective, that you wouldn't be part of the Internet, even though you still had 11 peers going full blast? By the same logic, Cogent isn't part of the Internet *EITHER*, because they're not bending over backwards to buy transit to get the L3 routes accessible again. For that matter, AS1312 isn't part of the Internet either, because we're only connected at 2 major points at the moment, and we're not making much of an effort to get connectivity to places that for one reason or another don't see a routing announcement for us, or we don't see their announcement. And I'm sure that with 180K routes, there's gotta be at least a dozen that we can't actually talk to... But oddly enough, I *seem* to be on the Internet. What's wrong with this picture?
Is being a tier-1 now a good or bad sales argument when selling internet access ? -- Tony Sarendal - dualcyclone@gmail.com IP/Unix -= The scorpion replied, "I couldn't help it, it's my nature" =-
On Thu, 6 Oct 2005, tony sarendal wrote:
Is being a tier-1 now a good or bad sales argument when selling internet access ?
Personally I think it's good strategy to multihome with one "tier-1" and one not so "tier-1". The ones further down the foodchain are more likely to be "peering whores" and therefore provide better connectivty to others like them. It's more likely someone skimps on connections they pay per meg for than peering links, therefore it's in my expereience more likely to be uncongested on peering links than transit links. So my answer to your question is "it depends". Using Tier-1:s as your only uplinks means everybody else is paying per meg to send traffic to you, is that what you want? -- Mikael Abrahamsson email: swmike@swm.pp.se
On Thu, Oct 06, 2005 at 01:33:38PM +0200, Mikael Abrahamsson wrote:
Sometimes yes, sometimes. no. With a transit provider I can call a sales rep and have a new circuit installed in 30 days or I start getting SLA credits. If said provider is doing something wrong, I can vote with my wallet and take my business to someone else who will do better. With a peer, even a friendly one, you are at the mercy of the cashflow, capacity, goodwill, and traffic engineering clue of another network that is essentially out of your control. Some folks are really good at peering, and some folks are really really bad at peering. Ask anyone who does enough peering and they will have a list of network about whom they will say "if we didn't send them X amount of traffic, we would shut their non-responsive prefix-leaking non-upgrading frequent-outage asses off in an instant". Just because a network is big and important doesn't mean that they are taking proper steps to manage the traffic and ensure reliable peering, or even that there is anyone manning the helm at all. And then there is AT&T... But that is an issue for another day. :) In my experience, since there is "no revenue" associated with the peering port on the other side, even very big networks who depend on reliable peering for their business manage to sit on necessary upgrades to peers for months or even years longer than they would if it was a customer port. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On 10/6/05 6:43 AM, "tony sarendal" <dualcyclone@gmail.com> wrote:
Is being a tier-1 now a good or bad sales argument when selling internet access ?
Its a great sales argument. That's why everyone claims to be one. It just sounds SO good. And its not like the Peering Police are going to enforce it. What does it mean in real life? Nothing. Nada. An organization's SFI status is a particularly poor criteria for choosing a transit provider. There are so many better factors to use - support, packet loss, price, latency, availability, provisioning speed - you name it, its a better criteria than SFI status. - Dan
>> Is being a tier-1 now a good or bad sales argument when >> selling internet access ? > Its a great sales argument. That's why everyone claims to be > one. It just sounds SO good. And its not like the Peering > Police are going to enforce it. What does it mean in real > life? Nothing. Nada. An organization's SFI status is a > particularly poor criteria for choosing a transit > provider. There are so many better factors to use - support, > packet loss, price, latency, availability, provisioning speed > - you name it, its a better criteria than SFI status. packet loss and latency to *where*? before replying, consider that most of a leaf's traffic is either to/from another leaf of a tier-1 to which they're (possibly indirectly) downstream, or to/from the tree of a tier-1 which peers with the tier-1 to which they're attached. if tier-n, where n > 1, is buying transit from tier-1s, which they have to do, then the price game seems to be pretty determined unless one likes to run at a loss or is cross- subsidizing from some other product line. all your bases are belong to us. :-) randy
On 10/6/05 10:30 AM, "Randy Bush" <randy@psg.com> wrote: >>> Is being a tier-1 now a good or bad sales argument when >>> selling internet access ? >> Its a great sales argument. That's why everyone claims to be >> one. It just sounds SO good. And its not like the Peering >> Police are going to enforce it. What does it mean in real >> life? Nothing. Nada. An organization's SFI status is a >> particularly poor criteria for choosing a transit >> provider. There are so many better factors to use - support, >> packet loss, price, latency, availability, provisioning speed >> - you name it, its a better criteria than SFI status. > > packet loss and latency to *where*? before replying, consider > that most of a leaf's traffic is either to/from another leaf of > a tier-1 to which they're (possibly indirectly) downstream, or > to/from the tree of a tier-1 which peers with the tier-1 to > which they're attached. > Consider this: A Tier 1 (SFI network) with congested peering links vs a non-SFI network with wide open transit pipes. I know I'd pick the latter. Latency when all inter-network links are uncongested is going to be pretty low in any case. > if tier-n, where n > 1, is buying transit from tier-1s, which > they have to do, then the price game seems to be pretty > determined unless one likes to run at a loss or is cross- > subsidizing from some other product line. > > all your bases are belong to us. :-) > > randy > Dan
On Thu, 6 Oct 2005, tony sarendal wrote:
Is being a tier-1 now a good or bad sales argument when selling internet access ?
its the same as it always was, its a marketing positive. but thats because the market is dumb. if you wish to make your purchasing decision on 'tier-1' status thats up to you, but i'll be looking at performance, price, strategy, service level and what type of supplier i want for a company like mine. cogent is cheap and you get what you pay for. level3 is mid-price, but they really dont care much about their customers (or thats what i found). perhaps you want better customer service or to deal with a smaller company to gain their attention and respect. choose your supplier based on your own criteria, not someone elses or on who has the most marketing points. Steve
On 06/10/05, Stephen J. Wilcox <steve@telecomplete.co.uk> wrote:
I didn't mean for this to sound so much like a question, but I belive I posted before my first cup of coffee. This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. Now, is being a tier-1 now a good or bad sales argument when selling internet access ? /Tony going for more coffee
On Thu, 6 Oct 2005, tony sarendal wrote:
how would purchasing from a tier-2 be any different (and by historical definition cogent is a tier-2), i've seen networks intentionally block routes from competitors for various reasons, and ultimately this is about your level of connectivity to the parts of the Internet anyone can have connectivity issues either by choice or by accident, you have to decide whether your chosen supplier is giving you a service level you are happy with for the price, what the risks are and what failure modes they are likely to present. Steve
On Oct 6, 2005, at 10:19 AM, tony sarendal wrote:
s/every time it happens/every time it happens to YOUR upstream People on Sprint, AT&T, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity.
Now, is being a tier-1 now a good or bad sales argument when selling internet access ?
It's still a good argument, because Marketing != Reality. :) -- TTFN, patrick
On 06/10/05, Patrick W. Gilmore <patrick@ianai.net> wrote:
Patrick, it happens to every PA customer who buys his service from one of the Tier-1 providers active in the de-peering. If a PA customer buys his service from a non-tier1 this will most likely not happen, unless that provider has bought transit in a very unwise way. The entire point is that it's not always good to be too close to tier-1 space. PS. sorry about the double-post Patrick.
On 10/6/05 10:37 AM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly)
Dan
On Fri, 7 Oct 2005, Daniel Golding wrote:
Or "single-home" to a tier-2 -- or tier-1.5, or whatever you want to call it in marketing newspeak -- that provides multihoming of their own networks, and get a netblock from their space. Often, that can be more cost effective (even these depeering situations notwithstanding) than single-homing to a tier-1. -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
On Oct 7, 2005, at 11:54 AM, Paul Vixie wrote:
Paul, I think that's unnecessarily one dimensional. The needs of business to be connected in a reliable fashion are above and beyond being for or against CIDR. Rather, they are the requirements for the routing architecture that the Internet has yet to fulfill. Single homing is bad simply from a reliability standpoint, and the only true technological impediment to everyone multi-homing is cost and the routing architecture. Consider the ability of the average consumer to make use of WiFi to provide mutual backup connectivity to his neighbors with alternate last mile providers. As the cost goes to zero, everyone will want to multi-home. Regards, Tony
well, sure, my answer is only valid if pigs do not have rocket boosters. if you're going to talk about how fast pigs could fly if they had rocket boosters then i'm very interested but i consider it a change of subject.
yea, verily, that is so. but "do not single home" is not practical advice as of the date on this particular milk carton. unless you can do it without bgp. see http://www.google.com/search?client=safari&rls=en-us&q=vixie+multihoming+without+bgp&ie=UTF-8&oe=UTF-8 if you're wondering how long we've been fiddling around with THAT tune here. -- Paul Vixie
On Fri, 7 Oct 2005, Daniel Golding wrote:
disagree. i know networks who multihome to avoid this kind of problem but introduce new problems with greater risk because they are unable to run bgp properly (be it from inadequate hardware, bad config, bad administration) my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management Steve
On Sat, 08 Oct 2005 20:41:55 BST, "Stephen J. Wilcox" said:
I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Anybody want to start a pool on how many new AS numbers will get issued as a result of this tiff, and what percent will commit a BGP whoopsie that impacts more than just themselves within the first 6 months? On the other hand, I see a business opportunity to sell new customers insurance against self-inflicted gunshot wounds to the feet here. Some providers might even consider selling a managed service at a slight loss, just for self-defense.. :)
any ISP likely to be involved in a peering dispute is a reliability risk, and whether it's because others keep de-peering them or because they keep de-peering others, doesn't matter. i liked the advice heard here the other day-- if you have to single home, do it through a tier-2 or tier-1.5 ISP without transit-free aspirations. they'll remain connected to the riskier ISP's no matter what the riskier ISP's are doing to each other this week. -- Paul Vixie
Connection restored between feuding Net providers http://news.yahoo.com/s/nm/backbone_dc;_ylt=ArskJPD_l3TpJ01SroWSOdQjtBAF;_yl...
its only temporary, level3 have given a temporary stay until 9th nov On Sun, 9 Oct 2005, Henry Linneweh wrote:
Connection restored between feuding Net providers
http://news.yahoo.com/s/nm/backbone_dc;_ylt=ArskJPD_l3TpJ01SroWSOdQjtBAF;_yl...
On Sat, 8 Oct 2005, Valdis.Kletnieks@vt.edu wrote:
Sure, but consider is it worse to have a very small number of complaining customers who cant get to a bit of the web for 2 or 3 days, or a complete outage to the Internet for a few hours because of a problem you cant fix. I see the latter occurring quite frequently, in particular I see support queries about loss of connectivity to large parts of the Internet which on inspection was caused by dampening because the ISP was flapping. I'm just saying, you fix one problem and create a whole bunch of new ones and it depends on the customer as to which results in the optimum situation. Steve
Yes, indeed, I think it makes sense to multihome my humble enduser pc. Right now all I can get is aDSL and it does not matter what provider because they all use DTAG.DE infrastructure. Maybe cable will be choce. It is not as fast as aDSL at least not here and it will take another two or three years until they deploy it. If it does not get shot on site again by the regulation office or the cartell office again. So I will end up having a cable-modem speaking ethernet/PPPoE and an aDSL-modem speaking ethernet/ tcp/ip and DHCP. My ip adresses probably will be 84.167.xxx.xxx for aDSL and 24.xxx.xxx.xxx for the cable. I can talk to no-ip.com, they will allow a second ip for host_look("84.167.252.166","echnaton.serveftp.com","1420295334"). host_name("84.167.252.166","p54A7FCA6.dip.t-dialin.net"). Its entry will look a bit like this one: host_look("81.88.34.51","Kunden2.KONTENT.de","1364730419"). host_name("81.88.34.51","kunden2-1.kontent.de"). host_look("81.88.34.52","Kunden2.KONTENT.de","1364730420"). host_name("81.88.34.52","kunden2-2.kontent.de"). So I will end up with 3 names and 2 ip addresses for my humble host. Do I need BGP now or OSPF or can I rely on RIP. Do I need an AS number? How do I get it? Imagine not a fool like me is asking this but some 32K end users of DTAG.DE connected to a DSLAM at Franfurt/Main in germany. I guess the number of end users disconnected be Cogent and Level 3 is not much smaller. Asbestos parapluis opened. Shoot now! Peter and Karin Dambier :) Valdis.Kletnieks@vt.edu wrote:
-- Peter and Karin Dambier Public-Root Graeffstrasse 14 D-64646 Heppenheim +49-6252-671788 (Telekom) +49-179-108-3978 (O2 Genion) +49-6252-750308 (VoIP: sipgate.de) +1-360-448-1275 (VoIP: freeworldialup.com) mail: peter@peter-dambier.de http://iason.site.voila.fr http://www.kokoom.com/iason
Look into multi6 - which basically proposes new network layer above ip but below tcp and that new layer would provide common end-point for system with multiple ip addresses. A closer possibility right now is dns "multi-homing" based on incoming request ip, i.e. dns server would answer with one provider ip address if they are coming from cogent routed ip space and for another from l3 routed ip space. This requires integration of bgp routing data with dns which has only been done by private implementations (I'm sure you all know who I mean) so far, but it would be a worthy project to do a open-source implementation of this technique if fragmentation of the internet continues to happen or becomes permanent. On Sun, 9 Oct 2005, Peter Dambier wrote:
-- William Leibzon Elan Networks william@elan.net
On Sun, 9 Oct 2005, william(at)elan.net wrote:
Look into multi6 - which basically proposes new network layer above ip
multi6 is dead, long live shim6... attend and discuss in Vancouver. (also, I'm fairly sure it's not going to help if you only have a single provider)
On Wed, 05 Oct 2005 19:27:24 PDT, David Schwartz said:
Being part of the Internet is not about communicating with 11 people and not the twelfth. It's about communicating with *anyone* (quite literally) that's willing to make a sufficient effort to communicate with you using the standards and practices that have evolved.
Bending over backwards was never required. As I said in the part you cut off when you replied, nobody has to run a line all the way to the server in my basement that isn't connectected to anything at all. What they do have to do is make a reasonable effort to communicate with anyone who is willing to make a similar effort. When you contract for Internet access, you are contracting to reach everyone who wants to reach you. This "want" is not a mental thing, it's an action of making the effort to connect to people.
If they make an effort to talk to you, and you do not make a similar effort to talk to them, then you're not part of the Internet. The Internet is the network that has resulted from this philosophy. It is this philosophy that makes it the Internet.
But oddly enough, I *seem* to be on the Internet. What's wrong with this picture?
What's wrong is that you are misrepresenting yourself and your connection philosophy. You are, through your providers and peers making that effort. Buying Internet access from someone who purports to provide it is one way of trying to connect with anyone who tries to connect with you. DS
There is another point here. For anyone signing contracts where the buyer has significant bargaining power with the seller, you can specifically stipulate that connectivity to the seller's network is not-good-enough to save them from paying an SLA event or indeed breaching the contract. (What is good enough is left as an exercise to the reader). Sellers may wish to push that risk onto the buyer's, but if history is any judge, buyer's are remiss in accepting that liability and risk without a significant financial incentive. (such as a huge discount over prevailing rates). Deepak
On Wed, Oct 05, 2005 at 01:41:50PM -0400, Todd Vierling wrote:
No there really isn't.
Cogent does not buy transit from MCI, AT&T, or Sprint. Level 3 does not buy transit from MCI, AT&T, or Sprint. You can only be a "tier 1" and maintain global reachability if you peer with every other tier 1. Level 3 is obviously the real thing, and Cogent is "close enough" (at least in their own minds :P) that they won't buy real transit, only spot routes for the few things that they are missing (ATDN and Sprint basically). There is no route "filtering" going on, only the lack of full propagation due to transit purchasing decisions, or in this case the lack thereof. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, Oct 05, 2005 at 02:08:01PM -0400, Richard A Steenbergen wrote:
Exactly. And this is why Cogent's statement to the public (and their customers) is an outright lie. Level 3 isn't "denying Level 3's customers access to Cogent's customers and denying Cogent's customers access to Level 3 customers.". It's just that they deny Cogent settlement-free direct peering anymore. Cogent can get the L3 and L3 customer routes elsewhere if they want. But Cogent doesn't. It's Cogents decision to break connectivity, not L3's. If I would be a Cogent customer, I would have a _very_ warm word with my sales rep why they are trying to bs me with those kind of statements and think that I actually am dumb enough to believe that. Regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On 10/5/05, Daniel Roesen <dr@cluenet.de> wrote:
Oh man, I have to jump in here for a moment. Not that I agree with what happened, but to refute your claim that Cogent can get L3 elsewhere, it goes both ways. L3 can also get Cogent connectivity elsewhere. This is a big game of chicken, it will be interesting to see who backs down first.
Well, as I somewhat said above, there will always be three sides to every story. Side 1, Side 2 and the truth. Each side has a case, it's up to the lawyers now to sort it all out. charles
On Wed, Oct 05, 2005 at 03:44:10PM -0400, Charles Gucker wrote:
Of course it gets both ways. The point is that Level 3 is a real tier 1, and Cogent not. Cogent _tries_ to become tier 1, but doesn't achieve it (actually, seems to make steps back, see OpenTransit and now Level 3). Ras has written an IMHO very good explanation of the situation and I don't have to add anything to it really.
Yup. Seeing that OpenTransit won, and Level 3 beeing a fatter chicken, I'd be somewhat surprised of L3 would lose it. But as I don't know the US market, I might be wrong. *shrug*. Either way, L3 _is_ a tier 1, and Cogent isn't. That gives L3 a certain advantage they try to play out right now. Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On 05 Oct 2005, at 13:44 , Charles Gucker wrote:
Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? I mean, Cogent can pay someone to route to L3 or L3 can fix what they did on their side, they have no need to go anywhere but their own routers, right? -- Lewis Butler, Owner Covisp.net 240 S Broadway #203, 80209 mobile: 303.564.2512 fx: 303.282.1515 AIM/ichat: covisp xdi: http://public.xdi.org/=lewisbutler
On Fri, Oct 07, 2005 at 02:53:02AM -0600, Lewis Butler wrote:
Simply put, yes. Longer answer, Level(3) would have to "kiss and make up" with Cogent before the sessions would be coordinated to be turned up. There would certainly have to be a renewed level of communication between these two networks to come up with this result.
Well, there are three options here. -> Both networks kiss and make up, ending up turning up the pre-existing peering session, or possibly additional peering sessions. -> Cogent obtains transit from another provider to Level(3). -> Level(3) obtains transit from another provider to Cogent. Business decisions do not always make sense, stubbornness can very easily get in the way of a proper decison[1]. charles [1] As outlined in this thread, one person's proper decision may not be another person's.
On Oct 7, 2005, at 12:56 PM, Charles Gucker wrote:
I seriously doubt L3 would have to do anything but revert to the last known good configuration. If Cogent has done anything to those BGP configurations, they're not only being silly, but they are being disingenuous, since they said they left the configs in place. Of course, that only gets the bits flowing, it doesn't solve the underlying issue. -- TTFN, patrick
cgucker@onesc.net (Charles Gucker) writes:
Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right?
that's what this press release says: http://www.cogentco.com/htdocs/press.php?func=detail&person_id=62 disclaimer-- my employer has friendly relations with both Level(3) and Cogent. -- Paul Vixie
On 07 Oct 2005 19:00:46 +0000, Paul Vixie <vixie@vix.com> wrote:
First off, that's not my quote. ;-) Second, it would appear routes are once again beng exchanged between Level(3) and Cogent. BGP routing table entry for 209.244.0.0/14, version 103309841 Paths: (1 available, best #1, table Default-IP-Routing-Table) Not advertised to any peer 174 3356, (aggregated by 3356 4.68.0.12) 66.28.1.1 from 66.28.1.1 (66.28.1.1) Origin IGP, metric 1000, localpref 100, valid, external, atomic-aggregate, best Community: 174:21000 16631:1000
From an outside view, it seems like Level(3) caved in to customer demand, but what the true outcome is, nobody will know [publically].
charles
Yeah, we just noticed the same.. BGP routing table entry for 38.0.0.0/8, version 23735501 Paths: (3 available, best #1, table Default-IP-Routing-Table) Flag: 0x220 Advertised to peer-groups: core Advertised to non peer-group peers: 64.39.2.107 212.100.225.49 3356 174, (received & used) 195.50.112.205 from 195.50.112.205 (4.68.0.240) Origin IGP, metric 0, localpref 100, valid, external, best Community: 3356:3 3356:86 3356:575 3356:666 3356:2010 Charles Gucker wrote:
-- ------------------------------------------------------ Tom Sands Chief Network Engineer Rackspace Managed Hosting (210)447-4065 ------------------------------------------------------
On Oct 5, 2005, at 3:11 PM, Daniel Roesen wrote:
I'm wondering why you decided it's Cogent's responsibility to pay someone for transit to Level3 and not vice versa? I'm not saying you're wrong (or right)... but I really don't think it's as clear cut as "cogent must...". Or if it is, it's not obvious to me :)
What is Level3 telling customers? Is it more accurate?
On Oct 5, 2005, at 3:11 PM, Daniel Roesen wrote:
I think you and I have a different definition of "deny" and "decision". Cogent was connected to L3. Level 3 TOOK ACTIVE STEPS to sever that relationship. Cogent, this moment, has their routers, ports, and configurations ready, willing, and able to accept and send packets to and from L3. Please explain to me why you think Cogent is the bad actor here? By your logic, Level 3 is denying customers access to Cogent because they are perfectly capable of buying transit from Verio. All that said, it is entirely possible L3 was justified in their actions because Cogent was abusing the peering relationship. For instance, Cogent may have been forcing L3 to carry long haul traffic for Cogent instead of buying their own fiber / routers / whatever. If so, L3 probably feels their decision to terminate the peering relationship is on sound moral, ethical, and financial ground. However, L3 still typed in the config lines which caused this bifurcation. Whether that makes it their "fault" is open to debate. But it certainly is not Cogent's fault just 'cause Cogent can buy transit - so can L3. -- TTFN, patrick
On Wed, Oct 05, 2005 at 03:51:34PM -0400, Patrick W. Gilmore wrote:
At this moment I stand ready willing and able to accept free interconnection from L3. If I then ask my transit providers to intentionally block announcement of L3 routes so that they are unreachable, is it L3's fault that they don't give me free peering? L3 told them it was coming on August 15th, they had around 50 days of notice, and they intentionally decided to remain unreachable in order to fight the depeering. Not that it may not have been a perfectly valid play on their part to sacrifice current customer happiness for long term interconnection capacity and financial viability, but don't make one side out to be "evil" and the other "good". It takes two to tango, and what we have here is two participants who are both very willing to make certain the other side in unreachable while pointing fingers at the other party for their half of the mess. A more honest position would be to man up and say "yes we broke half of the connectivity, they broke the other half, and we're going to stay like this until someone gives". But then again when has honesty ever been a part of marketing? -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Oct 5, 2005, at 5:01 PM, Richard A Steenbergen wrote:
There is a difference between not doing something and doing something. L3 does not currently peer with you. Not peering with you tomorrow does not take action on their part. Shutting down links does. Does this make L3 bad? Of course not. But neither does it make Cogent bad. (Which you know since you read my whole post, right?) However, it does make L3 the instigator of the disconnectivity.
Honestly, RAS, you are spouting more marketing than I am. L3 BROKE THE CONNECTIVITY. Not half of it, all of it. Cogent may or may not have done things which precipitated this action. But L3 took the steps. If you want to stay away from marketing, don't muddy the waters with things like "we broke half". As an L3 customer, I am upset that I cannot reach Cogent. As a Cogent customer I am upset I cannot reach L3. But that's not "blame" in the peering sense, that's just me upset over paying money for services not rendered. As an "objective NANOG poster", I do not know who is at fault. Not even sure I care. If there even is one. Am I "at fault" for not wanting to talk to you? Aren't I allowed to decide to whom I speak and whom I avoid? On balance, this is bad for the Internet. No matter who is at fault, the Internet is less useful than it was. Even if Cogent buys transit to get to L3, it will still be less useful than it was. Connectivity will be less robust, which hurts all of us. Sad day for the Internet. But we'll get over it. -- TTFN, patrick
On Wed, Oct 05, 2005 at 03:51:34PM -0400, Patrick W. Gilmore wrote:
I think you and I have a different definition of "deny" and "decision".
I agree that my usage of words was highly suboptimal to express what I wanted to express. See my other answer.
Yes, but Cogent actively risked that that this happens, and L3 did only took active steps to sever that DIRECT relationship, but does (AFAIK) nothing to prevent connectivity _at_all_ (which Cogent IMHO does claim). They just make it more costly for Cogent. Cogent doesn't want to pay the price, so no connectivity. OF COURSE L3 could start to buy transit... but as a real tier 1 they are prolly in the position that they won't need to. A real tier 1 depeering another tier 1 would be a completely different story though. :-)
Please explain to me why you think Cogent is the bad actor here?
I wouldn't say "bad actor". The current situation is just the result of L3 playing out their tier 1 card, and Cogent not being a tier 1 but not wanting to buy more transit. Given that Cogent was not yet on the same "eye level" (no pun intended) with Level 3, I as a hypothetical Cogent customer would blame Cogent to not having made provisions for that case. Again, I said that from the perspective of a Cogent customer knowing "the hierarchy" out there. Of course, there are shades of grey between black and white.
By your logic, Level 3 is denying customers access to Cogent because they are perfectly capable of buying transit from Verio.
L3 is tier 1, Cogent is tier 2. L3 tries to make the gap larger. Cogent doesn't want to get the L3 routes via their transit Verio.
If so, L3 probably feels their decision to terminate the peering relationship is on sound moral, ethical, and financial ground.
I'm not sure wether "moral" and "ethics" are significant factors in such peering battles (anymore)... especially with such offer like Cogent's to L3 customers. Anyway, I knew it was a mistake to post right after I sent it off. I hate peering politics (having worked for a former tier 1 and being losely involved into peering stuff there made me a burned kid), and I should firmly stay out of (doing and discussing) it - especially in a language which isn't my native one. :-( Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On Wed, Oct 05, 2005 at 11:11:05PM +0200, Daniel Roesen wrote:
I think the obvious answer from Cogent, though perhaps phrased a little bit nicer, is: for the price you pay for transit, sit down, shut up, and wait for us to resolve the issue in a way that lets us keep selling you cheap transit. On Cogent's side is the fact that the vast majority (at least by traffic volume) of its customers understand that you don't get a free lunch in this world. You don't pay the price that Cogent charges for transit AND get your butt wiped during every little issue, there just isn't enough money to go around. The majority of Cogent customers are multihomed, and rely on Cogent as a "it works pretty well most of the time" solution. If a Cogent customer wants to continue paying the price that they do for transit, they will sit down and be quiet, let (3) customers (who are more high-dollar and high-touch almost across the board) do the complaining, and try to have the situation resolved in favor of (3) giving in so Cogent can continue to dump bits to them for cheap. The confusion comes when folks expect to have their cake and eat it too. You don't pay Cogent enough money for them to buy transit to L3 when L3 wants to depeer them. That is why smart people who use Cogent multihome. Anyone who doesn't understand this is not understanding the simple economic realities of the product they are buying. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Oct 5, 2005, at 5:11 PM, Daniel Roesen wrote:
This is silly. "I don't buy transit, but you buy partial transit, so if I shut down the interconnection links (SFI or otherwise), you should buy more transit to get to me." HUH? B does not in any way follow A. This is no different than MCI de-peering Sprint. Both sides knows what will happen long before they do it.
I'm very confused by what you said there. WTF has "the hierarchy" got to do with my bits getting to you? Better question: As a "hypothetical Level 3 customer", would you blame Cogent or L3? "Tier One" is marketing. Even the technical idea of "buys no transit" is BS. It might be well defined, but that doesn't make the term _useful_. Cogent and L3 had _no_ interconnectivity besides the direct peering relationship. L3 knew it, Cogent knew it. L3 made a decision to sever that direct relationship, and bifurcation ensued. This was not only not a surprised, it was expected. Whether Cogent is a "tier one" or not is irrelevant to the decision, and the resulting effects. When you can't reach the web / mail / etc. server you need, does it matter if your network is big or small, tier one or tier five? Not to me, I'm just interested in getting packets from point A to point B. From your posts, it sounds like you are OK with buying partial transit at full price - as long as you buy it from a "Tier One" provider. -- TTFN, patrick
So if you're a Level(3) customer and you ask their network to exchange packets with an IP address reachable only through Cogent, and they can't do so, how does that differ from the reverse situation? If you pay a network to move bits for you, they should be making the best effort to move those bits. Level(3) is now making less of an effort. So is Cogent. Customers of both networks lose out, and have a right to be irritated. I wonder what the traffic levels and ratios involved are. I'm sure that's a big part of the untold story. Kevin
----- Original Message ----- From: "Daniel Roesen" <dr@cluenet.de> To: <nanog@nanog.org> Sent: Wednesday, October 05, 2005 2:11 PM Subject: Re: Cogent/Level 3 depeering
Some would argue about Cogent being a tier 1 carrier. I honestly don't know anymore. All I do know, is that when I was still provisioning circuits within PSINet years ago, PSINet was on the verge of being a tier 1 as they had bilateral peering with the majority of the other tier 1 carriers at the time. Now, when Cogent took over the PSINet fiber backbone, I've no idea if they kept those peering points hot or not. If they did, then they should have plenty of pathing to L3 even with the direct peer being down. As you also said, I would think that if that traffic isn't getting through from Cogent's net to L3, there's an issue with Cogent's routing of that traffic unless L3 has placed a direct acl prohibiting any Cogent IP space from entering their network. That's a big if though simply because of the amount of traffic that will get just blown away by doing that. -- Micheal Patterson Senior Communications Systems Engineer 405-917-0600 Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
On 05/10/2005, at 8:41 PM, Todd Vierling wrote:
"Isn't BGP supposed to work around this sort of thing?"
Ok, I'll state the obvious first .... BGP is a routing protocol, the economics of its implementation bears no resemblance to implied or otherwise connectivity.
That would assume that cogent is paying someone to transit their routes to L3. Which I can deduce is not the case.
I'm not familiar with the concept of a 'common backbone BGP use policy". The best analogy I can think of is .... "A commodity appears at first sight an extremely obvious, trivial thing. But its analysis brings out that it is a very strange thing, abounding in metaphysical subtleties and theological niceties." -- Karl Marx.
-- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
-- James
Todd Vierling wrote:
</lurk> Maybe not, the depeering L3 is involved in is sort of like blackmail, we can all thank the indicted ex-CEO of WorldCom, Bernie Ebbers, for the modern peering "There can only be one" rule set. Big guys double dip, and little guys are paying half the big guys double dip... great deal if you can con someone into accepting it, or are big enough to -force- them into accepting it. Case in point. L3 wants CoGent to kneel, and kiss the ring, nothing more, nothing less. "They must smell blood in the water".
Well, we know who -your- *transit* providers are.... * cough *
Some providers, a legacy of course, are "transit free", and rely on direct routes.. Soon, there won't be many of these left... and it will be a non-issue. "There can only be *one* !" - WorldCom chant, Circa 1995.
Anyone who provides -peering-, instead of transit, actively filters routes, as SOP.
Too bad there aren't Equal Access laws for tier1s. <slyly evil grin>
Like I said, light a fire, and lets burn Bernie at the stake! "I saw him fly up into the sky with the Devil himself !" * :-P (* no GOP affiliated ex-CEO's were harmed, or -actually- threatened, in the making of this post. Like FOX news, this post is classified as "Entertainment" and may or may not accurately portray actual facts.. ;-) <lurk>
On Wed, Oct 05, 2005 at 01:22:37PM -0500, Richard Irving wrote:
Peering and depeering is very much like playing poker, sometimes you smell blood, sometimes you've got a good hand, and sometimes you are just bluffing your ass off and hoping for the best. Experienced people can tell the difference. The reality is that the particular kind of depeering which Level 3 is engaged in at the moment (on a wide variety of fronts besides just Cogent or even XO) is more like "desperation depeering" (brandish the sword and see who is willing to pay you money to leave them alone), and historically is an indication of serious financial problems in the very near future. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Richard Irving wrote:
Because you were there at the time Ebbers was going around? Do you have any idea of how this works? I am going to go ahead and say no.
Time to quote Geoff Huston one more time. "A true peer relationship is based on the supposition that either party can terminate the interconnection relationship and that the other party does not consider such an action a competitively hostile act. If one party has a high reliance on the interconnection arrangement and the other does not, then the most stable business outcome is that this reliance is expressed in terms of a service contract with the other party, and a provider/client relationship is established" There is no big guy double dipping conspiracy theory here, it's really very simply laid out and explained in the above paragraph. Really a work of art that should be printed out and nailed to the forehead of the Peering Coordinators everywhere. So, part of this entire debacle is terminology. We need to kill the use of the heavily overloaded word Peer and replace it with what it tries to mean - Settlement Free Interconnect or SFI. Most of the so-called Peering Coordinators need to realize that what they really are doing is better described by SFI Secretary instead.
Lets apply Geoffs Razor to the above. I think L3 thinks that the value they get from connecting with Cogent is not justifying their cost, so they are pulling the plug. "Smelling blood in the water" indeed. Welcome to the inet-access of the new millennium.
WorldCom didn't know what IP SFI was in 95. Perhaps you mean UUNET/MFS?
Like I said, light a fire, and lets burn Bernie at the stake!
"I saw him fly up into the sky with the Devil himself !" *
:-P
<lurk>
Best to stay in lurk mode for a while longer. /vijay
Richard Irving wrote:
Brzzzt! lost both points. My prior email was rirving@onecall.net, Charter Nanog member. 8-) [Querying whois.arin.net] [whois.arin.net] Name: Irving, Richard B Handle: RI69-ARIN Company: One Call Internet Address: One Call Internet Inc. Address: 701 Congressional Blvd. City: Carmel StateProv: IN PostalCode: 46032 Country: US Comment: Old Internet Fossil ;) RegDate: 1995-11-29 Updated: 2002-09-05 Phone: +1-317-805-3742 (Office) Email: rirving@onecall.net "In the beginning it wasn't all sharks in suits, I swear!"
vijay gill wrote:
Yada-Yada. *DO* try to be less vitriolic, TIA.. "Those of you who know think you know it all, irritate those of us who -really- do. " - C'ya! :-P as-set: AS-ONECALL descr: ONECALL transit AS's, and VNAP pathways members: AS-INDNET, AS-INDYNET, AS11820, AS87, AS5072, AS1767, AS5689, AS6402, AS7206, AS7900, AS8169, AS10680, AS11069, AS11550, AS11114, AS22311, AS11780, AS10694, AS12277, AS13394, AS12074, AS10403, AS10718, AS6571, AS6164, AS11126, AS27443, AS11106, AS21997, AS-IEI, AS-IONENET, AS-21903, AS8011, AS26212
On Wed, Oct 05, 2005 at 04:01:51PM -0500, Richard Irving wrote:
Brzzzt! lost both points.
My prior email was rirving@onecall.net, Charter Nanog member.
What is this "Charter Nanog member" and how does one get to be it? Do you get a cool t-shirt with it? -dorian
Or if you change "1995" above to "1997," which was when UUNET 1st announced the end of "free of charge" interconnections, then WorldCom would be correct, as they had just recently purchased UUNET. At least that's the date I have in the following paper I wrote a while back: http://www.2sparrows.org/Sean/rit/final%20thesis.pdf It's fairly outdated now but may be a good read for those that haven't been around that long and seem to be a bit confused on peering vs. transit and other such things. At least the 1st 1/2 of it before it ventures off into telco realms. /Sean
At 03:37 PM 10/5/2005, Justin M. Streiner wrote:
And any ISPs who buy wholesale dialup services from either Level(3) or Cogent. Both of them offer such services, and many ISPs use these wholesale outfits rather than building their own infrastructure in every city. The dialup case results in a very large number of users of a large number of ISPs being single-homed to one or the other of these outfits. Keep that in mind too when you next sign a contract for wholesale dialup service.
Matthew Crocker wrote:
Undereducated rant to follow... While I realize that the "nuke survivable" thing is probably an old wives tale, it seems ridiculous that "the Internet" can't adjust by routing any packets that used to go directly from Cogent to Level 3 though some 3rd (and) 4th (and) 5th set of providers that are connected in some fashion to both... Level 3 and Cogent can't be operating in a vacuums - if we can get to Kevin Bacon in 6 degrees, Level 3 and Cogent should be able to get to each other in under 30 hops through other providers. And why isn't this apparently happening automatically? Pardon the density of my brain matter here, but I thought that was what BGP was all about? I welcome any education the group wishes to drop on me in this matter. -- Jeff Shultz
On Wed, Oct 05, 2005 at 10:43:51AM -0700, Jeff Shultz wrote:
This isn't 6bone. People don't give transit to everyone w/o being asked for or otherwise being purchased for :) James -- James Jun Infrastructure and Technology Services TowardEX Technologies Office +1-617-459-4051 x179 | Mobile +1-978-394-2867 james@towardex.com | www.towardex.com
At 01:43 PM 05/10/2005, Jeff Shultz wrote:
And why isn't this apparently happening automatically? Pardon the density of my brain matter here, but I thought that was what BGP was all about?
The assumption you are making is that Cogent has a full view from someone of all prefixes outside AS174 and that someone is providing a full view of AS174 to their downstreams/peers etc. My guess is this is not the case. ---Mike
On 5-Oct-2005, at 13:43, Jeff Shultz wrote:
For most ISPs, normal practice is to advertise your own routes and those of your customers to your peers, and to your transit providers. To your customers, you advertise everything. If someone decides to stop peering with you, you reach them through one or more of your transit providers. A relatively small number of providers are transit-free -- that is, they rely solely on customer and peering connections to reach the entire Internet. When a transit-free ISP loses a peer, there is no transit path to fall back on. While it's undoubtedly true that there are third parties who interconnect and peer with both cogent and layer(3), the fact that those are peering connections and not customer/transit connections means that the third parties are unlikely to advertise cogent routes to layer(3), and vice versa. This is a money issue. ISPs don't generally give away transit for free. Joe
On Oct 5, 2005, at 1:43 PM, Jeff Shultz wrote:
If nobody filtered BGP at all (in or out), you would have the state you are expecting. However, you would have both a capacity problem, and an economic failure, as you may well end up with cogent trying to send all (much) of it's level3 destined traffic through a customer's connection with much smaller pipes... or overloading it's connectivity to one of its other peers. The economic failure comes because now you're expecting a third party to transit packets between cogent and level3 without being paid for it (and some of those connections are metered).
John Payne wrote:
Okay. I always figured that the difference between peering and transit was that you paid for one and not the other. I had no idea that when you bought transit from someone, you weren't automatically buying transit to _all_ of that providers other connections. Interesting. Balkanization of the Internet anyone? As one other commenter hinted at, it does sound like a recipe for encouraging multi-homing, even at the lowest levels. How many ASN's can the system handle currently? -- Jeff Shultz
On 5-Oct-2005, at 15:22, Jeff Shultz wrote:
It's a 16-bit number; 0 isn't used, and the IANA (per RFC 1930) reserves 64512 through 65535 for private use. So from a resource allocation perspective, the answer is 64511. There is a proposal to introduce 32-bit AS numbers. See, for example: http://www.ietf.org/internet-drafts/draft-ietf-idr-as4bytes-11.txt http://www.ietf.org/internet-drafts/draft-huston-idr-as4bytes- survey-00.txt We have talks scheduled for Los Angeles about AS number exhaustion. Joe
On Wed Oct 05, 2005 at 10:43:51AM -0700, Jeff Shultz wrote:
Yes, it could have - I'm led to believe that one of the parties does purchase transit. However, moving all that traffic over transit rather than peering would cost them a significant amount of money - and as they're running their transit service at extremely low cost, they probably would find it hard to fund the use of transit to reach the other party. Simon -- Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration * Director | * Domain & Web Hosting * Internet Consultancy * Bogons Ltd | * http://www.bogons.net/ * Email: info@bogons.net *
Simon Lockhart wrote:
Okay, here is how I see this war... which seems to be the proper term for it. 1. Level 3 is probably annoyed at Cogent for doing the extremely low cost transit thing, thus putting price pressures on other providers - including them. So they declared war. 2. Level 3's assault method is to drop peering with Cogent, in hopes this will force Cogent to purchase transit to them in some fashion (does Level 3 have an inflated idea of their own worth?), also forcing them to raise prices and hopefully (for Level 3) returning some stability to the market. 3. Cogent's counter-attack is to instead offer free transit to all single homed Level 3 customers instead, effectively stealing them (and their revenue) from Level 3... and lowering the value of Level 3 service some amount as well. 4. Next move, if they choose to make one, is Level 3's. Fun. I think I'll stay in the trenches. -- Jeff Shultz
----- Original Message ----- From: "Jeff Shultz" <jeffshultz@wvi.com> To: "Simon Lockhart" <simon@slimey.org> Cc: "NANOG list" <nanog@nanog.org> Sent: Wednesday, October 05, 2005 2:35 PM Subject: Re: Cogent/Level 3 depeering
Could be that a bilateral peer contract isn't being fulfilled and L3 got tired of taking the full load of the traffic. PSInet killed the peer with C&W for that very reason, regardless of what was told to the general public about it years ago. C&W simply wouldn't provision their peering OC3 so PSINet killed theirs. Without know all sides of this one, and having access to the router configs at each side, no one will be able to really say who's breaking routing or who's got an active acl up and who doesn't. Traffic flow is apparently still broken otherwise, with these two peering as they do with over tier 1's, bgp should have settled the problem as intended. My guess is that either one or even both sides may still have active static routes in place breaking bgp routing. -- Micheal Patterson Senior Communications Systems Engineer 405-917-0600 Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
On Wednesday 05 October 2005 03:12 pm, Micheal Patterson wrote:
Cogent support just told me that they cannot reroute any of the traffic currently being dropped due to active filters on L3 equipment dropping all Cogent IP blocks.......don't know how true, but that's what they are saying.... Chris
Is this wrong? Two sides: a) cogent is directly responsible for the accelerated pace of transit pricing errosion, by almost an order of magnitude when they started. b) perhaps someone else would have done it?
I think I'd bet that if L3 depeered Cogent, the last place cogent would go to buy transit to L3 would be L3.
This is a free enterprise machine we live in. I laud Cogent for this action. It shows chutzpah (see: cajones).
4. Next move, if they choose to make one, is Level 3's.
I agree with the Honorable Mr. Steenbergen. I will be watching http://finance.yahoo.com/q/sec?s=LVLT
Fun. I think I'll stay in the trenches.
It will be fun, until the point at which this happens, and the depeered sues the depeeree. It will then become further fun when a unwise, uneducated judge in a court of equity will enter a status-quo injunction, forcing the two parties to peer. Tis what we need: court enforced peering. I can't imagine how this could happen, however. (please be sure you detected the sarcasm in the last statement). -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben Net Access Corporation, 800-NET-ME-36, http://www.nac.net
Alex Rubenstein wrote:
I'm not making value judgements on anything that has happened - both sides think that either tactically or strategically what they are doing is for the best. But when I said "purchase transit to them in some fashion" that allowed buying it from a third party as well - as long as it reaches L3 eventually. -- Jeff Shultz Network Technician Willamette Valley Internet Customer Service: 9am-5pm Weekdays Stayton: 503-767-1984 Salem: 503-399-1984 info@wvi.com Tech Support: 24/7/365 Stayton: 503-769-3331 Salem: 503-390-7000 tech@wvi.com
On Wed, Oct 05, 2005 at 10:43:51AM -0700, Jeff Shultz wrote:
Internet connectivity is only as good as the people who are willing to buy it. If you wanted to connect to the Internet, you would pay someone money to deliver the packets to/from you to the complete Internet, yes? They do this by either connecting with every other network out there, or by in turn paying someone else to deliver the data that you paid them to deliver. This is called transit. Now, sometimes when two networks of roughly equal size and value to each other have customer bases that need to talk to each other, they will set up circuits between the two and not charge each other for the traffic passed over it, for the SOLE purposes of exchanging traffic with each others' customer base. This is called peering. If you carry this trend all the way out to the maximum extent possible, you end up with a network that is so big that it doesn't have to pay anyone else to "deliver the bits for it", it interconnects with EVERYONE else that it would send bits to via peering, and everything else is a customer. This is called a "tier 1", of which there are only a handful (not counting marketing-land, where everyone claims to be a tier 1). So, what you have here is a battle of wills between two very large networks. One is a legitimate "tier 1" (and one of the biggest IP networks in the world), the other is "really really close", only a couple of networks away from being a tier 1. The network who is "really really close" is still buying transit to reach a few destinations, but they want to be a tier 1. This means that the transit they are buying is not "full transit" in the way that you would normally think of it, instead they are buying "selected routes" to the few remaining networks they don't peer with. This is a kind of "tier 1 by technicality", not having "earned" it through true "settlement free peering" the way that a true tier 1 has, but by intentionally paying your transit provider to "emulate" peering with the remaining networks who they don't peer with directly. Now, when said "big" peer comes along and says "we don't want to swap traffic for free with you any more", the smaller network doesn't want to let them go. Besides the obvious fact that they don't want to have to start paying money for traffic that was previously free, they don't want to look "weak" by caving in and buying transit, incase other networks who previously peered with them decide that they can depeer and force said network to pay THEM money for transit too. So, the smaller network intentionally chooses to remain unreachable and not buy transit, under the hopes that the customers of the larger network will complain enough that they are forced to "repeer". So, the bottom line is that the two networks "could" be reachable to each other if they wanted to, but they are intentionally choosing not to do so. Level 3 "could" repeer Cogent (which Cogent wants but Level 3 doesn't), and Cogent "could" buy transit (which Level 3 wants but Cogent doesn't), but it is currently a matter of waiting to see which side will blink first under the pain of pissed off customers who can't reach the full Internet. Whichever one blinks first loses. Cogent has successfully used this tactic in the past (Teleglobe), and unsuccessfully tried it as well (OpenTransit). But that said, the Internet is working the way that it is intended. I believe folks have reported that Level 3 saw a loss of around 1200 prefixes from Cogent, and Cogent saw a loss of around 4300 prefixes from Level 3. Out of a customer base of 11k and 57k respectively, this is relatively small (11% of Cogent's customer base and 7.5% of Level 3's customer base), since only single homed customers are affected. Unfortunately you can't make two networks who don't want to directly connect with each other or pay someone else to connect to the other network talk to each other if they don't want to. Usually these things iron themselves out within a few days, but these are certainly two of the largest and most pigheaded networks to go up against each other, so it could be interesting. Whining about it as a customer is one way to try and convince one side or the other to cave sooner, but you can pretty much be guaranteed that someone will end it before some judicial, regulatory, or law making body steps and makes them. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Wed, 5 Oct 2005, Jeff Shultz wrote:
If agreements are in place with those other providers to carry the traffic, then sure. Remember that when backbones peer with each other, they typically (and as normally dictated by peering policies on both sides) only announce their own routes and the routes of their downstream customers and agree not to announce a default route to each other. They do not announce a full routing table to each other. Upshot: When provider X de-peers provider Y, single-homed customers of either provider will likely have problems reaching single-homed sites of the other. Some of it comes down to the mob-rule mentality. The hope (though not often publicized :-) ) is that the de-peerER will force the de-peerEE into buying transit from them to get the de-peerEE's customers to stop calling in saying "I can't get to site BLAH - FIX THIS!" The de-peerEE (or their customers) may opt to try their case in the court of public opinion and try to get the de-peerER to reverse their stance and stop being 'the bad guy' :-) Irresistable force, immovable object. I now return you to your regularly scheduled programming. jms
Justin M. Streiner wrote:
That assumes that both are transit-free networks. We've already heard that Cogent buys some form of fill-in transit from Verio, perhaps for a subset of {AOL/ATDN, Sprint, others}. Assuming such, why aren't routes appearing in L3's routers with paths matching _2914_174_? L3 might be filtering them out (automatically or manually; perhaps filters haven't auto-generated yet), or Cogent might be requesting that they be filtered (Verio's community structure allows this quite easily). Conversely, why aren't routes appearing in Cogent's routers with paths matching _2914_3356_? If Cogent is buying transit from Verio, they should be receiving a full table. Anyone have any inside connections to expose the truth(s) here? pt
Just got off the phone with Cogent - no real resolution in sight. They say they have escalated this to their CEO (about damned time!), but do not expect resolution today anyway. They have issued a press release which can be found at http://telephonyonline.com/home/news/cogent_level_3_100505/ for whatever that's worth. Chris
Finally, some press taking notice: http://www.techworld.com/opsys/news/index.cfm?NewsID=4531 -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
William Allen Simpson wrote:
Finally, some press taking notice: http://www.techworld.com/opsys/news/index.cfm?NewsID=4531
More at: <http://news.com.com/Network+feud+leads+to+Net+blackout/2100-1038_3-5889592.html> <http://www.broadbandreports.com/shownews/68174> <http://www.hostingtech.com/?m=show&id=964> <http://searchnetworking.techtarget.com/newsItem/0,289139,sid7_gci1132045,00.html> and of course the obligatory slashdot thread: <http://ask.slashdot.org/askslashdot/05/10/05/2247207.shtml?tid=95&tid=187&tid=4> jc
On 10/5/05 1:43 PM, "Jeff Shultz" <jeffshultz@wvi.com> wrote:
Undereducated rant to follow...
Don't ever rant when uneducated. Its like driving angry. (/groundhog day)
They can. Cogent has transit and is preventing traffic from traversing its transit connection to reach Level(3). Level(3) does not have transit - they are in a condition of settlement free interconnection (SFI). The ball is in Cogent's court. This is not the first time or the second that they have chosen to partition. I'm not judging them on this, its a strategic call. However, considering the have backed down the last couple times, Level(3) would be smart to call their bluff.
This is about more than routing protocols. Its about how the Internet (large I) works. Peers, transit, etc. Not usually found in your favorite BGP book. Think of the SFI networks in the center. They may not be the largest networks in terms of traffic, but they have complete mesh SFI relationships with each other. Then, there are numerous other networks with some degree of SFI - think of them as a donut surrounding the SFI core. They are interconnected with a subset of the SFI networks and to each other in order to "short circuit" those networks who won't perform settlement free interconnection with them. Cogent has been attempting to establish SFI relationships with many of Level(3)'s customers since Level(3) threatened them with depeering. This is so that a) the partition is less painful and/or b) they'll have to buy less transit. We will now return this thread to the normal stream of "why is Cogent broken", "Level(3) is a bunch of meanies", and "my traceroutes feel FUNNY". ;) - Daniel Golding
On Oct 5, 2005, at 10:25 PM, jmalcolm@uraeus.com wrote:
I think Dan is trying to say that since Cogent buys transit, it is Cogent's responsibility to ensure they have connectivity to L3, not L3's responsibility to get to Cogent. Of course, he is dead wrong, but that's the way I read what he wrote. -- TTFN, patrick
ok, vijay popping up is not totally surprising, but twice? dorian was a bit of a surprise. but you, joe? coming out of the woodwork? the lack of clue in this thread must be *really* painful. 96 messages in the thread since 11:30 gmt, and maybe one screen worth of hard content, vjay's quote of geoff being half a screen. or, to be terse, it's called PEERing. it would be cool if folk carefully read the whole thread and picked out the clues before hitting the keyboard. randy
At 10:46 PM 10/5/2005, you wrote:
It's pretty evident that this has been a clue-free thread... even more telling are the "L3 should do this" and the "Cogent should do this" messages. In the end, there's nothing in this thread that's gonna make one whit of difference... those guys will either work it out or they won't (and Cogent will buy transit), and life will continue. Personally, I see a lot of panties in great big wads over this, yet not one of the owners of said undergarments can actually *do* anything about this situation. I'm not a customer of either of these companies, so I really don't have a lot of operational interest in this whole magilla... therefore, let the flames begin.
Peter Kranz Founder/CEO - Unwired Ltd -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dave Stewart Sent: Wednesday, October 05, 2005 8:22 PM To: nanog@nanog.org Subject: Re: Cogent/Level 3 depeering At 10:46 PM 10/5/2005, you wrote:
It's pretty evident that this has been a clue-free thread... even more telling are the "L3 should do this" and the "Cogent should do this" messages. In the end, there's nothing in this thread that's gonna make one whit of difference... those guys will either work it out or they won't (and Cogent will buy transit), and life will continue. Personally, I see a lot of panties in great big wads over this, yet not one of the owners of said undergarments can actually *do* anything about this situation. I'm not a customer of either of these companies, so I really don't have a lot of operational interest in this whole magilla... therefore, let the flames begin.
On Oct 5, 2005, at 4:13 PM, Daniel Golding wrote:
Cogent does purchase transit from Verio to Sprint, AOL, and other locations (but not to Level 3). Perhaps Dan would like to explain why that is relevant to the discussion at hand? Or why that puts the "ball" in Cogent's court? And no, L3's "SFI" status does not mean it's Cogent's fault. It is strange that people have to be reminded no network has the "right" to use any other network's resources without permission. Most people realize this in one direction. For instance, the "tier ones" love to point out Cogent has no "right" to peer with Level 3. Absolutely correct. What some people seem to forget is that Level 3 has no right to force Cogent to buy transit to get to Level 3. If Level 3 doesn't mind not being able to pass packets to Cogent, that's fine. If they do mind, they need to figure out a way to solve the problem - with Cogent. The inverse is true as well. As RAS said, it takes two to tango. This problem will be solved "soon" (in human time - days, weeks at most). One of the networks may go out of business, but that "solves" the problem because there would no longer be locations on the Internet someone couldn't reach. I suspect it will be solved by less drastic means. -- TTFN, patrick P.S. Does anyone else get that Baby Bell feeling whenever someone talks about being a "Tier One"?
--- "Patrick W. Gilmore" <patrick@ianai.net> wrote:
This is where you lost me: if there is no obligation for an SFI between them, then each player absolutely can force the other to buy transit to reach them. The way it plays out is this: whichever player's customers are more upset about the inability to reach the other will force that player to blink and either buy transit or make some other arrangement. The term "peering" is useful to describe SFI, because there is an implied equivalence between the players: i.e. it would hurt them both equally to partition. As was said by someone earlier, if it is more valuable to one party than the other, the business relationship is skewed, and ripe for a conversion to a settlement-based interconnection.
heh. I'm certain we're about to see the Nth iteration of the "who's a Tier One Provider" discussion, and I'll repeat: there are two contexts for "tier one" - marketing and routing. In marketing, everyone with a big, national network is a tier-one. In routing, definitions differ, and whatever definition is used, it's a smaller set than the marketing bunch... David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com ______________________________________________________ Yahoo! for Good Donate to the Hurricane Katrina relief effort. http://store.yahoo.com/redcross-donate3/
On 10/6/05 1:41 AM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:
Since you demanded it - Cogent buys transit. Regardless of what their route filters are currently set to, or what communities or arrangements they have with Verio, its transit. They purchase bandwidth to access other networks. Although I have not seen their transit contract, its not a stretch to say that they can use these connections to reach L3. I realize they may claim otherwise, but I have personal experience with them lying about their transit arrangements. And no, not some call center rep or NOC guy, either. Try a Cogent executive.
And no, L3's "SFI" status does not mean it's Cogent's fault.
There is no fault here. This is a business arrangement for all concerned. Cogent can make a configuration change to use their transit to reach Level(3). Level(3) has depeered them. I don't think anyone is "right" or "wrong". Generally, when one plays the peering game and loses, one eats it. Cogent however, is putting up a fight first. I don't blame them - its what I would do. However, they must face the music with their customers.
Sure. Cogent is free to offer a partial routing table and take their chances with their customers.
Usually these situations resolve in 2 - 10 days. At least, that's been the pattern. My prediction is that Cogent will fold, because they have in the past. Of course, I can't speak to Level(3)'s intestinal fortitude. - Dan
On Oct 6, 2005, at 9:11 AM, Daniel Golding wrote:
I think you are confused. If Cogent pays Verio to receive (for instance) only 1239 prefixes, and to propagate 174 prefixes only to 1239, then Cogent cannot "make a configuration change" to fix things. It would require a contractual change. But even if they could, why does this put the onus only on Cogent? Cogent has just as much right to not spend money to reach L3 as L3 has to not spend money to reach Cogent. Perhaps we are miscommunicating. I am not saying Cogent should not buy transit to reach L3. It is a business decision, not a technical argument. I am saying your idea of "Cogent buys transit, therefore the ball is in Cogent's court" is Just Plain Wrong. The "ball" is in _both_ of their "courts".
If you think the inverse of the above is also true, we agree. However, you posts have absolutely at least implied (and I would argue outright claim) that L3 should not be expected to do anything because they are in the "SFI club", and Cogent should do something because they "buy transit". Perhaps we do agree more than I thought. Did I misunderstand your comments about SFI and balls and courts and stuff? Do you think this situation is bilateral, or does one side have more responsibility to ensure interconnectivity than the other? -- TTFN, patrick
On Wed, Oct 05, 2005 at 11:50:52AM -0400, Matthew Crocker wrote:
Every depeering, someone pops up to talk about how much of a stink they are going to make and how many service credits they are going to ask for because they aren't getting a full table, even though they are multihomed and aren't relying on those routes to maintain reachability, because "transit means you are supposed to provide a view of the entire Internet". Not that I don't sympathize, but let me save you some time and energy by assuring you right now that Cogent doesn't give a flying crap about your whining on this issue, and anyone in "customer service" who says they do is being paid to nod and smile until 5pm. You will never see a dime in credits, so if you're actually going after the money and not just complaining for the sake of complaining or to try and encourage them to cave, save your energy for an issue you can actually win. :) Understand that both sides believe they are in the right in their own heads, or it would never have come this far in the first place. You can threaten them with cancellation of future business (or an illegal breach of contract and best of luck in collections I guess), but there is no way you're going to see a dime or make them stop under the terms of your contract. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
In the end, both providers lose, as customers buy real Internet transit from someone else. OTOH, the industry as a whole probably gains. I have a client who's massively overprovisioned, multihomed with multiple Ts each to 3 or 4 providers now after being bitten a couple years ago when singlehomed to C&W and they depeered PSI. Funny that those PSI customers are getting screwed again now. On Wed, 5 Oct 2005, Christopher Woodfield wrote:
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Wed, 2005-10-05 at 06:01:15 -0400, Richard A Steenbergen proclaimed...
I guess the earlier reports of (3)'s lack of testicular fortitude may have been exagerated after all. :)
Luckily, many of us have ipv6 tunnels that managed to help us get around this. See, ipv6 has a purpose, afterall! :-)
So this is all well and good while some measure of V6 is tunneled, but one should be wondering what these games of chicken mean to V6 when it is native. Given that most organizations won't meet the qualifications to be multi-homed, stunts like this will have a greater impact then this one is having today. Doesn't exactly leave a warm fuzzy that the current direction for IPv6 services is sane.... David On Oct 5, 2005, at 12:08 PM, eric-list-nanog@catastrophe.net wrote:
On Wed, Oct 05, 2005 at 02:17:09PM -0700, David Sinn wrote:
Indeed. Unfortunately (or actually, may this is rather fortunate?) there is practically no money value yet in IPv6, so we may be at least a year (or more) away from seeing the first major v6 depeering dispute. But nevertheless, given the imperfect state of multihoming for edge sites in IPv6, such depeering war will be significantly more detrimental to customers who cannot justify for a /32 or a "special infrastructure" /48 prefix allocation from the RIRs. Let see how multihoming proposals (e.g shim6, relaxed RIR allocation policy requests, etc et al) turn out in the next few months. IPv6 operators should probably want to pay close attention to multihoming proposals and any commercial developments in v6 world in the next year or two perhaps. If multihoming solutions don't really turn out well and v6 is appearing to become more ubiquitous, it may be a plausible idea to start opening up your route-filters to accept /48 prefix-lengths before the first depeering happens :) James -- James Jun Infrastructure and Technology Services TowardEX Technologies Office +1-617-459-4051 x179 | Mobile +1-978-394-2867 james@towardex.com | www.towardex.com
On Thu, 6 Oct 2005 bmanning@vacation.karoshi.com wrote:
Which is probably a GOOD thing, right? -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
As Randy pointed out, this conversation has been fairly clue free. Working as a peering coordinator for a large ISP I can tell you that most of the posts in this thread have been so wrong it makes me laugh. ISP's are businesses, and let me tell you that peering is no exception. People seem to like to think that "settlement free interconnections" are "free", but nothing could be further from the truth. You have to buy routers and the cards that go in them, provision transport services to the peering location, and then on to the peer. Provide enough backhaul in your network from where your customers are located to where the peers are located. You have to pay lawyers to review contracts, engineers to configure and troubleshoot sessions, and managers to negotiate the whole deal. The budget for "settlement free interconnections" at a major ISP can run into the millions of dollars. Two major ISP's may have 8xOC-48 between them. That's probably $200,000 in router costs alone. Yes, sometimes you can get a $200 cross connect, but sometimes you also have to have the $6k/month circuit, for each one, creating a $42,000/month cost. That's a half million dollars a year. When you look at the requirements, geographic diversity, volume, ratio, number of routes what is really happening is the companies are trying to make sure there is some balance of costs. It doesn't have to be a 50/50 split...everyone uses their own assets to reduce their costs, but there has to be some equality. Personally, I'm not a fan of the technical requirements to make them equal, but rather like to negotiate equality, but everyone has their own approach. Back to the issue at hand. What I can tell from this situation is that Level 3 thought they were paying a lot of costs for very little return on investment. The idea that Level 3 would take this action because Cogent was selling cheaper seems a bit far fetched to me. Level 3 knows this is going to hurt their customers as well. Indeed, I'll bet this went all the way to the Level 3 CEO for approval first, because they knew their was going to be pain. This isn't some router cowboy going nuts in the middle of the night, this is a business backed into a corner. Why? We'll never know the real story. Maybe L3 is paying for third party circuits because cogent doesn't touch their network and it's costing them a boatload. Maybe L3 wanted to move to GigE peering for cheap high density ports, and Cogent wouldn't budge from using OC-3's because their routers don't have great GigE density. Maybe traffic between the two has dropped to 20 megs, and so L3 doesn't think maintaining ports is a justified cost. Maybe the ratio is 20:1, and that was finally enough for L3 to feel they were carrying too much of the transport cost. Most likely it's a combination of all of these issues. Bottom line is some engineer had to dress up and go over to the land of suits and explain to them that yes, Level 3 was going to totally screw their own customers, but it was also going to save $X, where $X is probably fairly large, and so they really had no choice. Least I seem like I'm on Level 3's side, it may well be that they have high costs due to their own stupidity. Perhaps they cut a deal with Equinix for $5,000/month cross connects. Perhaps they pay list price for their routers. Perhaps they are about to go down the tubes. As for those who want to re-architect the Internet to "fix" this problem, please go away. It's not a technical problem. It's a business problem. Two companies, each responsible for their own bottom line couldn't find an economically feesable way to interconnect. Any attempt to "force" the interconnection (either via regulation, transit through third parties, etc) will RAISE prices for all involved. The key here is that the peering was not economically viable for some reason. It will be interesting to see how this is resolved in the end. As time passes, there will be increased pressure on both companies to fix this problem. Single homed customers are going to think twice about connecting to either one. My own observations? This appears to be happening to Cogent a lot lately. This makes me wonder if part of the reason they have been able to offer lower costs is by finding ways to take advantage of peers and transfer costs to them which is causing the peers to notice and take action. I also find Cogent's practice of offering Level 3 customers free service unseemly. They are partially responsible for the outage, and are trying to use that fact to lure customers. That makes me wonder if they've written off Level 3 entirely already...after all if you're planning on working out a deal with someone you don't rub salt in their wounds as a first step. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
participants (64)
-
Albert Meyer
-
Alex Rubenstein
-
andrew matthews
-
Bill Woodcock
-
bmanning@vacation.karoshi.com
-
Charles Gucker
-
Chris Stone
-
Christopher L. Morrow
-
Christopher Woodfield
-
Daniel Golding
-
Daniel Roesen
-
Daniel Senie
-
Dave Stewart
-
David Barak
-
David Schwartz
-
David Sinn
-
Deepak Jain
-
Dorian Kim
-
Douglas Dever
-
eric-list-nanog@catastrophe.net
-
Henry Linneweh
-
Henry Yen
-
James
-
James
-
James Spenceley
-
JC Dill
-
Jeff Shultz
-
jmalcolm@uraeus.com
-
Joe Abley
-
John Curran
-
John Payne
-
Jon Lewis
-
Justin M. Streiner
-
Lamar Owen
-
Leo Bicknell
-
Lewis Butler
-
Matthew Crocker
-
Micheal Patterson
-
Mikael Abrahamsson
-
Mike Tancsa
-
Niels Bakker
-
Patrick W. Gilmore
-
Paul Vixie
-
Pekka Savola
-
Pete Templin
-
Peter Dambier
-
Peter Kranz
-
Randy Bush
-
Richard A Steenbergen
-
Richard Irving
-
Robert E.Seastrom
-
Sean Butler
-
sigma@smx.pair.com
-
Simon Lockhart
-
Stephen J. Wilcox
-
Todd Vierling
-
Tom Sands
-
Tony Li
-
tony sarendal
-
Valdis.Kletnieks@vt.edu
-
vijay gill
-
Vince Hoffman
-
William Allen Simpson
-
william(at)elan.net