Eric Germann writes:
Well our friends at Internic apparently forgot how to update their servers
*.root-servers.net from here is showing a serial of 1997072500 with no updates made Friday night showing (11:30am EDT July 27).
While is is easy to blame the InterNIC and NSI for all the evils in the world, the InterNIC of five years ago (or even one year ago) is not the InterNIC that exists today. Last week I went to Network Solutions to meet with Dave Holtzman (SVP of Engineering) on an unrelated matter, and while I was there I got a chance to observe first hand the metamorphosis that has transpired over the past year. Here are just a few points that everyone should be aware of: * NSI completely replaced senior level management about nine months ago. Most of those folks were responsible for the arrogant, unresponsive attitude that NSI displayed for so many years. NSI replaced those folks with the most qualified people they could find. For example, Gabe Battista, NSI's CEO, was President of Cables and Wireless before coming to NSI last year. Dave Holtzman was one of IBM's most-talented senior engineers, and also has a substantial business background. There are dozens of other examples throughout the organization. * Since the transistion in the management structure, the call support area has gone from three people to 50 people, and the engineering support desk has gone from two to over 30 (and they are trying to hire more people right now). Hours have been extended from 5 PM EDT to 9 PM EDT, and plans are underway to make InterNIC support a 24x7 operation very soon. * The operations area is much like you would expect any large industrial organization to have -- a controlled environment computer room with raised floors, fire and water detection alarms, racks of equipment and systems, hot spare systems, several large Cisco routers including a spare Cisco 7500, three T3 links to separate backbone carriers, a UPS system with 200 lead batteries capable of running the entire facility for more than 24 hours, and a large deisel generator with enough fuel capacity to run for two weeks. The entire facility is secured with access cards, limiting people to those areas that they work in. [BTW, the entire InterNIC moved from its old location to its new location one month ago without any outage -- this is a true test of the InterNICs resources and expertise to be able to move an operation that large without impacting the rest of us.] * Effort is underway to define and document formal procedures for every activity that goes on in the InterNIC. Systems are being implemented everywhere to make the InterNIC as fault tolerant as possible. Based on what I saw, problems like the partial DNS update two weeks ago will be a thing of the past very shortly. I had the opportunity to talk with Richard Walsh, the Director of Operations, about the DNS problem and I am convinced it will never happen again -- there are too many automated systems and check procedures in place to allow a repeat of the previous partial update problem. * There is a complete focus on customer service in the InterNIC now. With a 1600+ percent increase in customer supprt personnel in just the past few months (which continues to grow as they are able to find and train people), the folks at NSI now talk about quality of service, response time, escalation procedures, and other customer support concepts that were anathma to the organization just one year ago. [Case in point -- before Dave Holtzman started posting on NANOG a few weeks ago, I can't remember anytime when an SVP at the InterNIC cared enough to post anything to the NANOG list.] Before I started US Net four years ago, I spent 16 years managing software engineering projects for KPMG Peat Marwick and American Management Systems (before that I taught CS at UofMD) -- I know what it takes to build, operate, and manage an industrial-strength operations center, and Network Solutions has one. I know what the response to this is -- "These folks are the InterNIC, they SHOULD be doing all these things!". And I agree 100%. The difference is they ARE doing these things now. This is a direct result of the new management at the InterNIC. Also, a lot of folks have made derogatory comments about NSI going public. Frankly, I think this is a good idea -- it will give them the financial depth to be able to support the InterNIC at a level that will benefit all of us. Based on what I saw, and knowing the skills of Battista and Holtzman, NSI is sure to be one of the best stock deals of the '90s. [FWIW, I have no connection to the InterNIC what so ever -- like the rest of you, I rely on them for their domain registration services only.] So if your idea of the InterNIC is two engineers, two computers, and a half-dozen support personnel with an attitude, forget it -- it is not that way anymore. Dave Holtzman wants to make the InterNIC a world-class support organization that puts reliability and customer support first. From what I have seen, he is well on his way ... Dave Stoddard US Net Incorporated 301-572-5926 dgs@us.net
On Mon, Jul 28, 1997 at 12:19:34PM -0400, David Stoddard wrote:
While is is easy to blame the InterNIC and NSI for all the evils in the world, the InterNIC of five years ago (or even one year ago) is not the InterNIC that exists today. Last week I went to Network Solutions to meet with Dave Holtzman (SVP of Engineering) on an unrelated matter, and while I was there I got a chance to observe first hand the metamorphosis that has transpired over the past year. Here are just a few points that everyone should be aware of:
And David proceeds to tell us why the InterNIC is so great. Ok, David. Fine They are equipped technically. As you note, of course, they _have_ to be, they're the InterNIC. But answer this: why is it that _you_ have to be the one to tell us this? The major complaint I hear about the InterNIC isn't that they're arrogant, it's that their a black hole. Or worse, a roach motel: complaints check in, but they don't check out. That's exactly the problem I have with them. They need a PR person, with an extensive background in the culture of the net, and the power to get things done... and they need one _now_. Cheers, -- jr 'silent period or not' a -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "People propose, science studies, technology Tampa Bay, Florida conforms." -- Dr. Don Norman +1 813 790 7592
On Mon, 28 Jul 1997, Jay R. Ashworth wrote:
The major complaint I hear about the InterNIC isn't that they're arrogant, it's that their a black hole. Or worse, a roach motel: complaints check in, but they don't check out.
That's exactly the problem I have with them. They need a PR person, with an extensive background in the culture of the net, and the power to get things done... and they need one _now_.
Exactly. If NSI doesn't have someone who head up pr doing the work of informing the net community of what problems it's having, then rumors start happening. Rumors spread out, and get picked up by the media. Since the media is obviously a part of this list (we sure get quoted a lot), these stories of NSI outages start making the news. The price of NSI stock drops, and NSI's stockholders are not happy. Life is not good for NSI. So, doesn't it make sense to just get someone to do the PR job? Joe Shaw - jshaw@insync.net NetAdmin - Insync Internet Services
At 2:04 PM -0500 7/28/97, Joe Shaw wrote:
On Mon, 28 Jul 1997, Jay R. Ashworth wrote:
The major complaint I hear about the InterNIC isn't that they're arrogant, it's that their a black hole. Or worse, a roach motel: complaints check in, but they don't check out.
That's exactly the problem I have with them. They need a PR person, with an extensive background in the culture of the net, and the power to get things done... and they need one _now_.
Exactly. If NSI doesn't have someone who head up pr doing the work of informing the net community of what problems it's having, then rumors start happening. Rumors spread out, and get picked up by the media. Since the media is obviously a part of this list (we sure get quoted a lot), these stories of NSI outages start making the news. The price of NSI stock drops, and NSI's stockholders are not happy. Life is not good for NSI. So, doesn't it make sense to just get someone to do the PR job?
Because of the pending IPO, the InterNIC is in a 90 day quiet period. Even if they have hired a PR person/group/agency, that entity couldn't really do their job until after the IPO. I believe that this is an SEC regulation. ____________________________________________ Wayne D. Correia tel:415-826-6000 http://www.domain.net fax:415-826-6100
On Mon, Jul 28, 1997 at 02:04:56PM -0500, Joe Shaw wrote:
On Mon, 28 Jul 1997, Jay R. Ashworth wrote:
The major complaint I hear about the InterNIC isn't that they're arrogant, it's that their a black hole. Or worse, a roach motel: complaints check in, but they don't check out.
That's exactly the problem I have with them. They need a PR person, with an extensive background in the culture of the net, and the power to get things done... and they need one _now_.
Exactly. If NSI doesn't have someone who head up pr doing the work of informing the net community of what problems it's having, then rumors start happening. Rumors spread out, and get picked up by the media. Since the media is obviously a part of this list (we sure get quoted a lot), these stories of NSI outages start making the news. The price of NSI stock drops, and NSI's stockholders are not happy. Life is not good for NSI. So, doesn't it make sense to just get someone to do the PR job?
Sorry, Joe. Ms. Baker at DEC tells me we're off topic. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "People propose, science studies, technology Tampa Bay, Florida conforms." -- Dr. Don Norman +1 813 790 7592
At 12:19 07-28-97 -0400, David Stoddard <dgs@us.net> wrote:
[Case in point -- before Dave Holtzman started posting on NANOG a few weeks ago, I can't remember anytime when an SVP at the InterNIC cared enough to post anything to the NANOG list.]
Well, it's nice that he posted something, but why did he have to wait until 16 hours after the problem occurred [1]? Someone else noticed the problem and sent a message to NANOG in UNDER 3 MINUTES [2]. Holtzman also made no mention in this notice about how a sysadmin overrode the "quality assurance" mechanisms or that they have made any effort to prevent future occurances of the same problem(s). Too little, too late. Stephen "From: David Holtzman <dholtz@internic.net> "Subject: NSI bulletin 097-004 | Root Server Problems "To: nanog@merit.edu "Date: Thu, 17 Jul 1997 22:52:18 +0500 (GMT) " "On Wednesday night, July 16, during the computer-generation of the "Internet top-level domain zone files, an Ingres database failure resulted "in corrupt .COM and .NET zone files. Despite alarms raised by Network "Solutions' quality assurance schemes, at approximately 2:30 a.m. (Eastern "Time), a system administrator released the zone file without regenerating the "file and verifying its integrity. Network Solutions corrected the "problem and reissued the zone file by 6:30 a.m. (Eastern Time). " "Thank you. "David H. Holtzman "Sr VP Engineering, Network Solutions "dholtz@internic.net "Date: Thu, 17 Jul 1997 00:33:58 -0700 (PDT) "From: Taner Halicioglu <taner@isi.net> "To: NANOG <nanog@merit.edu> "Subject: root nameservers broken again? " "bleh... is it me, or are the root nameservers hosed again? -- Unsolicited commercial/propaganda email subject to legal action. Under US Code Title 47, Sec.227(a)(2)(B), Sec.227(b)(1)(C), and Sec.227(b)(3)(C), a State may impose a fine of not less than $500 per message. Read the full text of Title 47 Sec 227 at http://www.law.cornell.edu/uscode/47/227.html
On Jul 28, Stephen Sprunk <sprunk@csi.net> wrote:
Well, it's nice that he posted something, but why did he have to wait until 16 hours after the problem occurred [1]? Someone else noticed the problem and sent a message to NANOG in UNDER 3 MINUTES [2]. Holtzman also made no mention in this notice about how a sysadmin overrode the "quality assurance" mechanisms or that they have made any effort to prevent future occurances of the same problem(s).
To join this with another thread, I think it'd be a Good Thing[TM] if the InterNIC were to start up an outage list...it sure would be popular. My only request (should any 'NIC people be reading this and taking me seriously, not that I think I deserve to be taken seriously too often) is that it be free of advertisements (even for NSI) or new payment schemes or /anything/ that isn't an outage that affects the provider community.
Too little, too late.
But, better than most of us have grown accustomed to expect. I really don't think anybody here is trying to say that the InterNIC is now perfect, only that they've improved. ********************************************************* J.D. Falk voice: +1-415-482-2840 Supervisor, Network Operations fax: +1-415-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." *********************************************************
participants (6)
-
David Stoddard
-
J.D. Falk
-
Jay R. Ashworth
-
Joe Shaw
-
Stephen Sprunk
-
Wayne D. Correia