Has your management received enough shocks to system to make talking
about provider communications worthwhile yet? Or are they still in a
state of denial? Or have the shocks not been big enough to be noticed?
The last couple of months have seen many of the same types of outages
over and over again, yet provider communications during these outages
seems minimal, or even non-existent. Having an outage lists is a nice
thing for your sales people to say, but if no information is given
during an outage, what was the point of setting up the list?
Hint: in the absence of pro-active notifications, more and more people
are going to start their own active monitoring, i.e. your most critical
machines are going to get pinged to death by more and more people checking
if those critical machines are up. I wonder what Netscape's current ping
rate is?
Acts of God and other natural disasters seem to get modest reporting.
But anything else seems to be treated like a tree falling in the forest,
if I don't say anything maybe no one will notice it fell down.
That works great, sometimes. But it also means we have no way to
collect reliable data on how well the "Internet" is doing. Which means
every once in a while an acorn falls out of a tree, and hits chicken
little reporter on the head, and we end up with stories on TV how the
sky is falling. Then we get called into our management's office to
explain what we plan on doing about the sky falling.
Overall, most of us, I hope, understand the sky is not falling. But
as an industry we have no data to back that up. If one provider goes
off the air for eight hours, they may have shot a big hole in their
99.9% reliability. But what does that mean for the Internet as a
whole? How much customer traffic was affected? Do we even know
how much customer traffic exists on the Internet as a whole? The
so-called denominator problem. In other words when one provider
carrying 10% or 90% of the Internet traffic has problems, does it
affect the total amount of Internet traffic, or is a substantial
portion of the traffic re-routed around the damaged provider?
I've actually been a bit surprised just how little my total traffic
volume has been effected even when major providers or exchange points
go off the air for periods of time. Maybe we've achieved TYMNET/TELENET
nirvana, and any useful site on the net is connected to all of the
IRCs :-).
Q: What's the worst thing that can happen to a backbone?
A: Go down, and no one notice.
Fiber cuts, operator errors, software bugs, etc are normal network
events. For the Internet, the unusual thing is supposed to be when a
single event has a disruptive affect on more than just that one system.
With only modest pre-planning, and good communications between providers,
the Internet continues flapping on its merry way through most of these
events. The myth of the nuclear bomb and the Internet is somewhat true,
the net will keep working, but routing is going to be sub-optimal at the
node which gets nuked.
The problem is as an industry we don't seem to have a good grasp of
exactly what is the root cause of most the problems. Sure, you can
say fiber cuts are the number one cause of Internet outages, but how
do you know? Maybe its power failures, or configuration errors, or
any number of other things. And my biggest question, why do single
events seem to have such a widespread effect on the Internet? Is it
a design error, or an assembly error?
Second hint: if you only have two name servers, and they are both on the
same subnet, your life is going to suck when that subnet gets nuked.
Spring planting and backhoe season is just around the corner, do you
have a second fiber strand?
Ok, what's this got to do with provider communications?
Q: Find the value of the Internet traffic denominator, so then statements
like 40% of the Internet traffic make sense.
A: Get all Internet providers to give KC? the inbytes/outbytes of their
network cloud for anonymization and consolidation.
Q: What are the top three causes of Internet outages? Nanog did the
top three problems, but not the causes.
A: Show me your trouble tickets.... Ok, how to normalize the data?
Q: Do you want active pinging or provider pro-active notification?
A: Software is Internet law. Develop an Internet (not single-point)
status board for providers that don't cooperate. A software
message system for providers that do cooperate, i.e. spin
control. Decentralize, and provider control.
Q: How good are the alternate paths in the Internet?
A: Unfortunately we don't get to find out until the disaster.
Q: Who reports Internet outages first? CNN or the provider?
A: CNN seems to still be reporting them first.
--
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
Affiliation given for identification not representation
