On 31-aug-2005, at 10:04, Mark Foster wrote:

I do see a risk in the move toward IP systems at the edge. At the core is a different story to at least some degree. Twas also pointed out that British Telecom are heading down the same track as Telecom NZ, and their rollout should be completed earlier. I trust therefore that it has all been thought out in terms of robustness and the like.

There are two types of VoIP: voice over a private, tightly controlled IP network, and voice over the public internet. Now obviously the latter is a risky proposition, as it imports all the limitations of the internet into the voice service. Apart from the fact that many parts of the internet aren't all that robust (but some are), this is a problem because voice and IP react differently to congestion collapse, which invariably happens to some degree in big emergencies. With IP, delays and packet loss build up, slowing everything down, but allowing many protocols to continue to work at a reduced rate. With PSTN, initiating calls starts failing more and more, but when you get through, you generally get to talk because you get a reserved piece of the scarce bandwidth. With VoIP, packet loss and delay eventually make the service useless. So VoIP fails harder than either traditional IP apps and PSTN. However, voice over a private network isn't entirely trouble-free, even though the private network can be designed such that congestion is a less fatal problem. And it does have the advantage that it allows IP routing protocols to route ongoing calls around failed parts of the network. On the other hand, in a circuit switched network you can do all kinds of interesting stuff (such as restarting all your control software) without breaking your sessions. We're only now seeing this in IP, and I think it's not really possible to reach the same levels with IP routing even in the long run. And then there is all this SIP stuff, which I'm (thankfully) only superficially familiar with, but never seemed particular robust to me. And voice over any kind of packet infrastructure introduces significant additional delays. I think in 10 years or so we'll realize that TDM isn't so bad after all.

Eesh... I grabbed a copy of this thing. In a cursory over-read... I am afraid if people (people defined by lim(clue) -> 0) start implementing datacenters by this guide. This would be a BRILLIANT document as the reading material for a college-level course. However, I'd be concerned if a CxO reads this and assumes they are great if the document has no conflicts with their implementation and they think they are in good shae. Before I comment publicly on the issues I think I have with it, I want to verify that the points I raise aren't covered in some sort of disclaimer about being "out of scope" etc. Essentially 90% of the conversations folks have on nanog about datacenter designs are outside of what this advocates building (in a very cursory overread). DJ Chris Gilbert wrote:

[snip] The Telecommunications Industry Association (TIA) -- the people who brought you the CAT standards for unshielded twisted pair cabling -- recently undertook a vast challenge to publish a definitive document encompassing best practices and design considerations for every single aspect of the modern data center.

The standard, entitled Telecommunications Infrastructure Standard for Data Centers, TIA-942, weighs in at 148 pages, and covers everything from site selection to rack mounting methods. [/snip]

Link: http://searchdatacenter.techtarget.com/originalContent/0,289142,sid80_gci112...

Also: http://www.tiaonline.org/media/press_releases/index.cfm?parelease=05-46

I seem to remember some folks asking questions about such a thing here in the past... so I hope this isn't a duplicate of an old thread.

In any case, has anyone here looked over the documents and/or have any comments on them?

It seems to me (however I have not yet read it) that something such as this could be quite useful to IT students and others who don't have the field experience.

-- Regards Chris Gilbert

We have already been asked about where our datacenters fit in with the TIA942 spec in several RFPs! It does cover some good topics, but it also leaves out the design and structure of many things which are far more likely to cause an outage than the copper and fiber physical plants.

Yeah... and it introduces/codifies the concept of "tiers" of datacenters... Yet, its possible to be have "tier 4" access to telecommunications while being a "tier 1" datacenter to operate those telecommunications, or vice versa. What bothers me as significantly as this tier stuff is that redundancies, procedures, staffing, testing, policies are only mentioned, but not actually discussed (such as the why's, or how to test for the condition). They refer to specific technologies... like "RAID" as an application for a "tier 4" facility. They mention colocation and internet data centers, but don't discuss or even address how your facilities survivability is not fundamentally affected by non-carrier grade equipment being installed by customers -- yet, not surprisingly, the "tier 4" definition specifically talks about all the equipment installed in the datacenter. There is lots of hand waving... like "beware the EPO". And yet, it doesn't discuss how facilities like Exodus's NJ facility that had all the power outages or Equinix/Ashburn and Equinix/Chicago which presumably meet at least, the Tier-3 specifications by design... still fail when they are implemented poorly. That 99.99% and above availability have more to do with maintenance and procedures than the equipment you installed initially. Its more of a document I'd expect to spend a ridiculous some of money to have a consultant produce, not someone who should know better. Great college guide book to discuss "issues" though. Deepak Jain AiNET

On Thu, Sep 01, 2005 at 10:54:26PM +1200, Simon Lyall wrote:

also "The Practice of System and Network Administration" by Limoncelli and Hogan has a few pointers as well but on a smaller scale and it feels a little old at times.

Tom tells me he's prepping a seconfd edition for late 96; anyone who has comments on the first edition should codify them and ship them to him now. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Designer Baylink RFC 2100 Ashworth & Associates The Things I Think '87 e24 St Petersburg FL USA http://baylink.pitas.com +1 727 647 1274 "NPR has a lot in common with Nascar... we both turn to the left." - Peter Sagal, on Wait Wait, Don't Tell Me!

With VoIP, packet loss and delay eventually make the service useless. So VoIP fails harder than either traditional IP apps and PSTN.

That is only in theory. In practice, during times of impending congestion collapse, IP network operators reconfigure the network to cope. For instance when DDoS is detected, people set up ACLs and trigger black hole routes. I think that it is possible for network operators to define an analogous action plan to stave off congestion collapse in an emergency situation. I'm not sure exactly what that action plan would look like, but I'm sure other list members will have plenty of good ideas. If you'll recall, just a few days ago people were talking about how they informally identified IP connectivity to emergency response sites so that those sites could be given priority in restoring service. We just need to sit down and talk these things over with our local emergency response organizations and learn where network operators can become part of the solution.

On the other hand, in a circuit switched network you can do all kinds of interesting stuff (such as restarting all your control software) without breaking your sessions. We're only now seeing this in IP, and I think it's not really possible to reach the same levels with IP routing even in the long run.

MPLS may have the edge here because you can have backup paths and fast reroute to keep traffic flowing if you have an orderly plan for rebooting routers.

And voice over any kind of packet infrastructure introduces significant additional delays.

Experience with the Inter-NOC phone system http://www.pch.net/inoc-dba/ seems to suggest otherwise. Some kinds of packet infrastructure only introduce insignificant delays. It would be interesting to know if any of the academics among us have studied the behavior of a SIP-based VoIP network during various types of failure and congestion scenarios. I suspect that problems will be mostly found under certain specific sets of conditions and if we know what those conditions are and how they impact voice services, then we can plan actions to mitigate the problems. One thing that IP network operators can do is throw bandwidth at a problem by "shedding load", i.e. killing traffic that is deemed non-essential. This would free bandwidth for traffic that is deemed "important". This has nothing to do with QoS per se becaus it can be implemented in many ways up to and including unplugging sites that generate non-essential traffic. All indications are that the next few decades will see an increased number of emergency situation like the tsunami, terror attacks in major cities, hurricanes, earthquakes. We have gotten very good at running the network through normal times, maybe we should now focus on how to keep it running through times of extreme stress. --Michael Dillon

Iljitsch van Beijnum wrote:

There are two types of VoIP: voice over a private, tightly controlled IP network, and voice over the public internet. Now obviously the latter is a risky proposition, as it imports all the limitations of the internet into the voice service.

I'm not so sure; someone cuts an ISDN-30 into our building and the sky falls down. Someone cuts some fibre carrying IP and life (and communications) carry on .. Perhaps you've made a fair and good comment on the marurity of most off-the-shelf voip products or implementations. But the key, in my mind, is that VoIP across the internet, when done well, imports all of the opportunities of internet routing into voice service. -a

Valdis.Kletnieks@vt.edu wrote:

It's clearly possible to find telco engineers with 5/10/15 years experience in running PSTN (might even find somebody with 40-50 years? :). It's possible to find network engineers with lots of BGP experience. Where do you find a senior engineer with 5+ years experience in enterprise-scale VoIP deployment?

Deployable enterprise VoIP products existed in 1998. So it would be somebody who was there doing it back then? Goes 5+ with a margin. Pete

On 31-aug-2005, at 21:19, Andy Davidson wrote:

...

There are two types of VoIP: voice over a private, tightly controlled IP network, and voice over the public internet. Now obviously the latter is a risky proposition, as it imports all the limitations of the internet into the voice service.

I'm not so sure; someone cuts an ISDN-30 into our building and the sky falls down.

Yes, single homing sucks.

Someone cuts some fibre carrying IP and life (and communications) carry on ..

You can get your ISDN 30 over redundant fibers too, that's not the problem.

Perhaps you've made a fair and good comment on the marurity of most off-the-shelf voip products or implementations. But the key, in my mind, is that VoIP across the internet, when done well, imports all of the opportunities of internet routing into voice service.

You say that as if it's a good thing. :-) I think in the long run, it makes sense to have end-to-end IP calls over the internet. However, this is not going to be as reliable as the PSTN for many years to come, because there are is no inter-AS QoS deployment, routing protocols take their sweet time (180 seconds BGP timeout anyone?) and the internet is becoming fairly non-transparent because of all the goo people keep pouring into the machinery in the name of security and the like. However, using the public internet as a local loop is bad. Here in the Netherlands, the incumbent telco isn't allowed to lower its prices, but everyone (including the incumbent telco) can sell voice minutes to PSTN destinations over an IP "local" loop for any price they want. So basically they're forced to kill off the local leg of the PSTN to be able to compete on medium/long distance. This is not good. Not so long ago, when there was a failure in the long distance infrastructure, you could still make local calls. With the current "intelligent" networks that's not always the case anymore, but if the emergency number stuff is done properly, you can still call 911/112 when the long distance stuff is down. With inet local loop that will no longer be the case in most cities. But then, people don't really care about this, as cell is in the exact same boat and huge numbers of people rely on just their cell phone and no longer have a fixed line (in Europe at least).

Michael.Dillon@btradianz.com wrote:

...

But then, people don't really care about this, as cell is in the exact same boat and huge numbers of people rely on just their cell phone and no longer have a fixed line (in Europe at least).

I have read accounts that suggest that cellphone subscribers from New Orleans only have one way service. In other words, if you left New Orleans with your cellphone then you can make outgoing calls but no-one can call you. I don't know how widespread this is, but knowing that there has to be an SS7 switch in New Orleans directing those incoming calls to your new location, I can imaging that loss of such a switch would create problems.

It is sometimes the case in disasters that people from inside can call out but that people from outside can't call in because the circuits into the disaster area become overloaded. This would hold true especially in the case where many people in the disaster area have no access to working phones, so those with working phones can easily get a free outbound circuit - meanwhile frantic friends and family clog up the incoming circuits trying to reach phones that are out of service or people who simply aren't near the phone and who can't answer but those calls still tie up circuits each time they are attempted. I've had several reports that cell phone users who can't make *or* receive calls are successfully sending *and* receiving SMS. It could be that the problem is one of not enough cell channels and working phone circuits for all the phone calls people want to make, but that the SMS channel is not overloaded and thus SMS traffic can zip on thru (when the cell has power and can reach a working cell tower). jc

SMS is all IP these days. Look at mBlox for more data. Others like them. The real key to SMS (and IP) that makes VoIP and PSTN so different is SMS does not need to be real time, and it does not have to be in order (i.e. packet 2 does not have to come after packet 3 etc.....). It delivers whatever data it can when it can.... Thus extremely low bandwidth required (1 bit per second would take a long time to get the message through, but it would make it) -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Michael.Dillon@btradianz.com Sent: Friday, September 02, 2005 5:57 AM To: nanog@merit.edu Subject: Re: Replacing PSTN with VoIP wise? Was Re: Phone networks struggle in Hurricane Katrina's wake

I've had several reports that cell phone users who can't make *or* receive calls are successfully sending *and* receiving SMS. It could be

that the problem is one of not enough cell channels and working phone circuits for all the phone calls people want to make, but that the SMS channel is not overloaded and thus SMS traffic can zip on thru (when the

cell has power and can reach a working cell tower).

This was my personal experience during the July 7th terrorist attacks in London. I couldn't make or receive voice calls but SMS did get through both incoming and outgoing. However the delivery of SMS messages was sometimes delayed by as much as an hour. SMS takes far less network bandwidth than voice calls. Originally it was implemented as part of the control network of GSM (rather like SS7) but I believe that most carriers now simply use IP networks to carry their SMS traffic. By now it has become clear that the response to the New Orleans disaster has been completely screwed up because of lack of reliable communications in and out of the city. There are tons of food, water, medical supplies and personnel hung up on edge of the city because no-one seems to know what is needed, where it is needed, how to get it there, etc. --Michael Dillon

Michael.Dillon@btradianz.com wrote:

A similar problem would be created if a web server relied on DNS that was only hosted on servers in New Orleans.

Do you (or somebody) know of recent numbers of what percentage of domains have all their DNS servers in; a) same subnet b) same AS c) same geographical "zone" (faultline, floodplain, coastline, etc.) Obviously if everything served by the DNS is co-located in the same failed facility, it does not matter that much if the names resolve to IP addresses which are unreachable anyway. Pete