I often question why\how people build networks the way they do. There's some industry hard-on with having a few ginormous routers instead of many smaller ones. I've learned that when building Internet Exchanges, the number of networks that don't have BGP edge routers in major markets where they have a presence is quite a bit larger than one would expect. I heard a podcast once (I forget if it was Packet Pushers or Network Collective) postulating that the reason why everything runs back to a few big ass routers is that someone decided to spend a crap-load of money on big ass routers for bragging rights, so now they have to run everything they can through them to A) "prove" their purchase wasn't foolish and B) because they now can't afford to buy anything else. There seems to be a bit of overstatement with respect to how large
There's no reason why Tampa doesn't have a direct L3 adjacency to Miami, Atlanta, Houston, and Charlotte over diverse infrastructure to all four. Obviously there's room to add\drop from that list, but it gets the point across.
On 5/17/18 6:24 AM, Mike Hammett wrote: these are... alcatel/nokia 7750 (L3's newer PE platform) is large but not outlandish and they've been deployed for a couple years. it's relatively similar in capacity or a to to the devices that many of us interconnect with them using. Most of their customers probably though not always need less fib then they need on a PE router. There is a longer time-scale overhang from the choice to design of MPLS core networks 15-20 years ago where PE routers have more to do fib wise then do cores (which may well be larger and simpler, since most of what they do is label switching), that drives the selection of what hardware goes in the edge in ways than an IP only carrier might make different choices (e.g. this big fib/queue/buffer router might have been a large l3 switch). the number of paths available into and out a market seems somewhat orthogonal to the number of routers.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
----- Original Message -----
From: "David Hubbard" <dhubbard@dino.hostasaurus.com> To: nanog@nanog.org Sent: Wednesday, May 16, 2018 11:59:42 AM Subject: Curiosity about AS3356 L3/CenturyLink network resiliency (in general)
I’m curious if anyone who’s used 3356 for transit has found shortcomings in how their peering and redundancy is configured, or what a normal expectation to have is. The Tampa Bay market has been completely down for 3356 IP services twice so far this year, each for what I’d consider an unacceptable period of time (many hours). I’m learning that the entire market is served by just two fiber routes, through cities hundreds of miles away in either direction. So, basically two fiber cuts, potentially 1000+ miles apart, takes the entire region down. The most recent occurrence was a week or so ago when a Miami-area cut and an Orange, Texas cut (1287 driving miles apart) took IP services down for hours. It did not take point to point circuits to out of market locations down, so that suggests they even have the ability to be more redundant and simply choose not to.
I feel like it’s not unreasonable to expect more redundancy, or a much smaller attack surface given a disgruntled lineman who knows the routes could take an entire region down with a planned cut four states apart. Maybe other regions are better designed? Or are my expectations unreasonable? I carry three peers in that market, so it hasn’t been outage-causing, but I use 3356 in other markets too, and have plans for more, but it makes me wonder if I just haven't had the pleasure of similar outages elsewhere yet and I should factor that expectation into the design. It creates a problem for me in one location where I can only get them and Cogent, since Cogent can't be relied on for IPv6 service, which I need.
Thanks