Though I can name off several 'credentialed' network engineering gurus and the 'bibles' of network engineering, a recent discussion about the source of network operations 'best practices' left me speechless, and curious. Who is/are the network operations equivalents of people like Peter Drucker and Jack Welch--people who are looked at not only has role models for operations success, but as luminaries in the industry for having established and educated the masses about best practices? What would be the network operations equivalents to revered business tomes like "The Practice of Management", "Seven Habits of Highly Successful People", "The G.E. Way" (and a variety of others that populate the shelves of your friendly local executive)? Most network-oriented training seems to focus on the technology, not on operations (and those subtle but ever so critical differences between knowing how something is /supposed/ to work and how it /really/ works, and all of the effort it takes to create a smooth-running operations engine). What are the network operations equivalents to business programs such as Six Sigma? What about something similar to the various leading institutions of business management, institutions that study of how networks are operated (and used), and develop training and methodologies for better operations practices? Pete.
inter-domain routing is a little short in the tooth to have luminaries the way that "20th century capitalism" has luminaries. only time will tell if our luminaries are are better than theirs. joelja On Thu, 6 Dec 2001, Pete Kruckenberg wrote:
Though I can name off several 'credentialed' network engineering gurus and the 'bibles' of network engineering, a recent discussion about the source of network operations 'best practices' left me speechless, and curious.
Who is/are the network operations equivalents of people like Peter Drucker and Jack Welch--people who are looked at not only has role models for operations success, but as luminaries in the industry for having established and educated the masses about best practices?
What would be the network operations equivalents to revered business tomes like "The Practice of Management", "Seven Habits of Highly Successful People", "The G.E. Way" (and a variety of others that populate the shelves of your friendly local executive)?
Most network-oriented training seems to focus on the technology, not on operations (and those subtle but ever so critical differences between knowing how something is /supposed/ to work and how it /really/ works, and all of the effort it takes to create a smooth-running operations engine).
What are the network operations equivalents to business programs such as Six Sigma? What about something similar to the various leading institutions of business management, institutions that study of how networks are operated (and used), and develop training and methodologies for better operations practices?
Pete.
-- -------------------------------------------------------------------------- Joel Jaeggli Academic User Services joelja@darkwing.uoregon.edu -- PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -- The accumulation of all powers, legislative, executive, and judiciary, in the same hands, whether of one, a few, or many, and whether hereditary, selfappointed, or elective, may justly be pronounced the very definition of tyranny. - James Madison, Federalist Papers 47 - Feb 1, 1788
On Thu, 06 Dec 2001 20:40:36 MST, Pete Kruckenberg <pete@kruckenberg.com> said:
What would be the network operations equivalents to revered business tomes like "The Practice of Management", "Seven Habits of Highly Successful People", "The G.E. Way" (and a variety of others that populate the shelves of your friendly local executive)?
We still don't know enough about how to actually run a network to have luminaries. The occasional bright candle, maybe. Right now, most of the suits are still trying to figure out why the non-suits think the suits "Just Dont Get It" (the PHB syndrome). Unfortunately, the Adams brothers (Scott and Douglas) seem to have quite a bit of relevance (or more importantly - a large percentage of the people working in *your* trenches *think* they have relevance). And both authors share the trait that they tend to be much more an entertaining read than anything labeled a 'tome' ;) /Valdis P.S. Yes, I know about the incorrect genealogical term 2 paragraphs back. ;)
Well, I'd say the network should be able to run itself. Like, just plug the wire in, and it works. I do not see any _technical_ reason why it shouldn't be like that. Now, back to the regular Yet Another Useless Feature Sold To Us By OFRV discussions :) --vadim On Fri, 7 Dec 2001 Valdis.Kletnieks@vt.edu wrote:
On Thu, 06 Dec 2001 20:40:36 MST, Pete Kruckenberg <pete@kruckenberg.com> said:
What would be the network operations equivalents to revered business tomes like "The Practice of Management", "Seven Habits of Highly Successful People", "The G.E. Way" (and a variety of others that populate the shelves of your friendly local executive)?
We still don't know enough about how to actually run a network to have luminaries. The occasional bright candle, maybe. Right now, most of the suits are still trying to figure out why the non-suits think the suits "Just Dont Get It" (the PHB syndrome).
Unfortunately, the Adams brothers (Scott and Douglas) seem to have quite a bit of relevance (or more importantly - a large percentage of the people working in *your* trenches *think* they have relevance).
And both authors share the trait that they tend to be much more an entertaining read than anything labeled a 'tome' ;)
/Valdis
P.S. Yes, I know about the incorrect genealogical term 2 paragraphs back. ;)
business management is in more need of luminaries as the business failure rate is orders of magnitude higher than the internet packet drop rate. randy
Randy Bush wrote: > business management is in more need of luminaries as the business failure > rate is orders of magnitude higher than the internet packet drop rate. Yes, that's interesting. I hadn't looked at it that way. I guess you could say that Internet routing is basically successful, since something like 99.X% of packets are delivered to their destination, whereas business is basically a bust, since 90% of business fail in the first 18 months (or something like that). That's obviously an apples-to-oranges comparison, since the packets have a short natural lifespan within which to succeed, whereas the "natural lifespan" of a business is open-ended. But back to the point, why we don't need luminaries in this business: We're an engineering culture, and we celibrate the oral tradition of didactic tales of failure, rather than the cult of personality. The handing down of tales of failure, which is what we do for entertainment and social reinforcement, is what allows us as engineers to build upon the successes of previous generations and avoid replicating their failures. Celebrating personality is irrelevant to our social construct; it serves no function. The luminaries of our culture, if they exist, are the engineers who made the famous mistakes which are indelibly commited to our collective memory, from which the greatest number of other engineers have learned a lesson without replicating the learning experience first-hand. -Bill
On Fri, Dec 07, 2001 at 09:01:22AM -0800, Bill Woodcock wrote:
The luminaries of our culture, if they exist, are the engineers who made the famous mistakes which are indelibly commited to our collective memory, from which the greatest number of other engineers have learned a lesson without replicating the learning experience first-hand.
-Bill
Nomination: Jeff "Ampersand" Rizzo. For teaching us all just *why* it is that you don't want to redistribute BGP into your IGP. Or so goes the tale relayed to me. While I worked at the relevant company at the relevant time, I didn't work with the network there until a year or two later. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://users.lightbearer.com/lucifer/
Jeff "Ampersand" Rizzo. For teaching us all just *why* it is that you don't want to redistribute BGP into your IGP. Or so goes the tale relayed to me. While I worked at the relevant company at the relevant time, I didn't work with the network there until a year or two later.
I thought it was the op of 7007 that did that? Alex
On Mon, Dec 10, 2001 at 10:21:55AM -0500, alex@yuriev.com wrote:
Jeff "Ampersand" Rizzo. For teaching us all just *why* it is that you don't want to redistribute BGP into your IGP. Or so goes the tale relayed to me. While I worked at the relevant company at the relevant time, I didn't work with the network there until a year or two later.
I thought it was the op of 7007 that did that?
I don't have a timeline to know which happened first; 2551 was down, or at least the majority of it was, for something on the order of 48-72 hours. The rendition I heard assigned the moniker because one of the major news outlets said that the mistake which triggered it was "like a misplaced ampersand". It was certainly the one on the wall beside his legacy Chevy's sombrero. -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://users.lightbearer.com/lucifer/
On Mon, Dec 10, 2001 at 10:18:48AM -0700, Joel Baker wrote:
I don't have a timeline to know which happened first; 2551 was down, or at least the majority of it was, for something on the order of 48-72 hours. The rendition I heard assigned the moniker because one of the major news outlets said that the mistake which triggered it was "like a misplaced ampersand". It was certainly the one on the wall beside his legacy Chevy's sombrero.
My oh my, how the versions differ. As was recounted to me, the outage was about 19 hours, and was due to the semantics of Cisco config mode. Something like: router ospf 1234 redistribute bgp subnets route-map blah (everything fine, now let's turn it off...) no redistribute bgp subnets route-map blah So tell me, does this turn off the redistribution, or just remove the route-map... :) And this is certainly worth remembering. Had I not known of this, I could likely have made the same mistake at some point. -c
On Mon, Dec 10, 2001 at 11:15:25AM -0800, Clayton Fiske wrote:
On Mon, Dec 10, 2001 at 10:18:48AM -0700, Joel Baker wrote:
I don't have a timeline to know which happened first; 2551 was down, or at least the majority of it was, for something on the order of 48-72 hours. The rendition I heard assigned the moniker because one of the major news outlets said that the mistake which triggered it was "like a misplaced ampersand". It was certainly the one on the wall beside his legacy Chevy's sombrero.
My oh my, how the versions differ.
As was recounted to me, the outage was about 19 hours, and was due to the semantics of Cisco config mode. Something like:
router ospf 1234 redistribute bgp subnets route-map blah
(everything fine, now let's turn it off...)
no redistribute bgp subnets route-map blah
So tell me, does this turn off the redistribution, or just remove the route-map... :)
And this is certainly worth remembering. Had I not known of this, I could likely have made the same mistake at some point.
Interesting. I wonder if we're discussing two separate events. The one I knew of supposedly involved a regex with a misplaced .*, which seems like a fairly fundamentally different lesson. Both valuble, though... (and thus is demonstrated one of the hazards of a purely oral tradition... human memories are notorious for showing all of the properties of most complex neural networks, including partial matching triggering more than one pattern...) -- *************************************************************************** Joel Baker System Administrator - lightbearer.com lucifer@lightbearer.com http://users.lightbearer.com/lucifer/
Interesting. I wonder if we're discussing two separate events. The one I knew of supposedly involved a regex with a misplaced .*, which seems like a fairly fundamentally different lesson. Both valuble, though...
I was downstream from 2551 at the time of their, um, big event. A botched regex that led to external BGP routes being redistributed into OSPF is what sources that I considered reliable explained to me as the cause. Reports on the length of the outage varied, but I think official reports were from the time of the initial failure until their core was stable, because my network was down well over 24 hours, and I seem to recall the "official" times as shorter. As for the previous question in this thread, 12.1 that I'm currently running treats "no red bgp subn route-m <name>" the same as "no red bgp" (for any value of <name>, including values that don't match what's already there). I have no reason to believe that the 11.x that Netcom was presumably running at the time was any different. -- Brett
On Mon, Dec 10, 2001 at 11:15:25AM -0800, Clayton Fiske wrote:
On Mon, Dec 10, 2001 at 10:18:48AM -0700, Joel Baker wrote:
I don't have a timeline to know which happened first; 2551 was down, or at least the majority of it was, for something on the order of 48-72 hours. The rendition I heard assigned the moniker because one of the major news outlets said that the mistake which triggered it was "like a misplaced ampersand". It was certainly the one on the wall beside his legacy Chevy's sombrero.
My oh my, how the versions differ.
As was recounted to me, the outage was about 19 hours, and was due to the semantics of Cisco config mode. Something like:
router ospf 1234 redistribute bgp subnets route-map blah
(everything fine, now let's turn it off...)
no redistribute bgp subnets route-map blah
So tell me, does this turn off the redistribution, or just remove the route-map... :)
And this is certainly worth remembering. Had I not known of this, I could likely have made the same mistake at some point.
-c
Ye gods. Guess I'd better keep a closer eye on NANOG. For the record: This version (Clay's) is more or less correct. :) The "oops" was pretty much *exactly* that... removing a route-map when intending to remove a redistribution. In my defense, however, we discovered the problem fairly quickly, but OSPF bug extant in IOS 10.whatever that we were running prevented us from cleaning it up in a timely manner. We basically wound up partitioning the network and rebooting *every* ospf-speaking device on it, because they were getting poisoned with gobs of LSA data that was never getting cleared, and was propagating the bogus information. Also for the record: I have *no idea* where Bob Metcalfe got that "ampersand" thing. It did make for an amusing engraving on a going-away gift I was given some years later: !!!!!!!!!!!!!!!!!!!!&................!!!!!!!!!!!!! :) +j -- Jeff Rizzo http://boogers.sf.ca.us/~riz
: : : business management is in more need of luminaries as the business failure : rate is orders of magnitude higher than the internet packet drop rate. But that's sooo not what we're talking about here... Two points: Business failure rates are not a function of the number of "luminaries" (I'm not even sure I like that word...) but of how well those sages are received and heeded. How many businesses fail because they didn't read Drucker? The failure rate, based on poor operational management, of networking and network based companies, is unknown but certainly many orders of magnitude higher than the internet packet drop rate... Peace, Petr -- Systems, Networks and Gadgets, done with Artful Intelligence -<>-<_>-<__>-<_>-<>- Policy: ASCII/text attchmnts alway read. PDF maybe read. Others, by necessity, may be ignored. Don't take it personally, it's a time issue.
On Fri, 7 Dec 2001, Petr Swedock wrote:
Business failure rates are not a function of the number of "luminaries" (I'm not even sure I like that word...) but of how well those sages are received and heeded. How many businesses fail because they didn't read Drucker?
Well... Drucker's classic text book defines the first job of top management to be "asking the question: What is our business and what should it be." I've seen an awful lot of businesses fail because they don't have an answer to that question. Miles ************************************************************************** The Center for Civic Networking PO Box 600618 Miles R. Fidelman, President & Newtonville, MA 02460-0006 Director, Municipal Telecommunications Strategies Program 617-558-3698 fax: 617-630-8946 mfidelman@civicnet.org http://civic.net/ccn.html Information Infrastructure: Public Spaces for the 21st Century Let's Start With: Internet Wall-Plugs Everywhere Say It Often, Say It Loud: "I Want My Internet!" **************************************************************************
At 07:12 AM 12/7/01 -0800, Randy Bush wrote:
business management is in more need of luminaries as the business failure rate is orders of magnitude higher than the internet packet drop rate.
You make it sound like the music industry. :-) Most labels are similar to business incubators in that they sign 10 new artists a month and end up dropping 9 of them because they didn't manufacture the "hit" they were looking for. Business failures are common. In the last few years, when the number of ".COM" start-ups drastically increased, there'll be at least a proportional increase in failures. This is normal. More people try, more people fail/succeed. I think the point I'm trying to get to here is that if you want industry "rock stars", you're asking for self-serving self-publicists who don't have the industry's best interests at heart (cough... ICANN). If you want that from the "geek" community, you should just put Kashpureff and Vixie in the ring on WWF and publish the results as a BCP. ;-) Best Regards, Simon -- (A)bort, (R)etry, (T)ake down entire network?
On Thu, 6 Dec 2001, Pete Kruckenberg wrote:
Though I can name off several 'credentialed' network engineering gurus and the 'bibles' of network engineering, a recent discussion about the source of network operations 'best practices' left me speechless, and curious.
Who is/are the network operations equivalents of people like Peter Drucker and Jack Welch--people who are looked at not only has role models for operations success, but as luminaries in the industry for having established and educated the masses about best practices?
A very intersting question. I too can think of people, books, and practices that lay out best practices for network design, but not operations. [One can argue that if those practices are followed, operations becomes simple - but, as we all know, that 's not the case]. I can also think of a few people who can be considered luminaries as regards network security - Cliff Stoll, Mark Eichin and Jon Rochlis, and more recently, the CERT organization. But for general operations, your statement seems to hold true:
Most network-oriented training seems to focus on the technology, not on operations (and those subtle but ever so critical differences between knowing how something is /supposed/ to work and how it /really/ works, and all of the effort it takes to create a smooth-running operations engine).
What are the network operations equivalents to business programs such as Six Sigma? What about something similar to
I recall a lot of very good people from my days at BBN, and we certainly developed a lot of internal policies and procedures - particularly for our government customers, who tend to insist on such things. I expect there ar also a lot of people, policies, and procedures hidden within various carriers, ISPs, and corporate IT/networking departments - but a lot of that doesn't get visibility, at least in part because carriers tend to consider such things to be proprietary information. We really do need such information. Miles Fidelman ************************************************************************** The Center for Civic Networking PO Box 600618 Miles R. Fidelman, President & Newtonville, MA 02460-0006 Director, Municipal Telecommunications Strategies Program 617-558-3698 fax: 617-630-8946 mfidelman@civicnet.org http://civic.net/ccn.html Information Infrastructure: Public Spaces for the 21st Century Let's Start With: Internet Wall-Plugs Everywhere Say It Often, Say It Loud: "I Want My Internet!" **************************************************************************
On Thu, 6 Dec 2001 20:40:36 -0700 (MST) Pete Kruckenberg wrote:
Who is/are the network operations equivalents of people like Peter Drucker and Jack Welch--people who are looked at not only has role models for operations success, but as luminaries in the industry for having established and educated the masses about best practices?
My idiosyncratic, incomplete, personal list is: Scott Bradner Vadim Antonov Randy Bush Sean Doran Sean Donelan Paul Vixie Steve Bellovin Tony Li Paul Ferguson Ted Lemon My criteria: 1) They have taken the time to regularly publish (via mailing list or more formal venues), 2) I have reading their opinions for a minimum of ten years, 3) Time has proven them right. There are many great engineers that I just haven't known long enough, don't regularly spout off. There are others who ought to be on this list but name just slipped my mind for a moment. regards, fletcher
: : Though I can name off several 'credentialed' network : engineering gurus and the 'bibles' of network engineering, a : recent discussion about the source of network operations : 'best practices' left me speechless, and curious. : : Who is/are the network operations equivalents of people like : Peter Drucker and Jack Welch--people who are looked at not : only has role models for operations success, but as : luminaries in the industry for having established and : educated the masses about best practices? With regards to Peter Drucker... his sagacity lies in a deep understanding of the history of his field and long observance of such (some sixty years!). Jack Welch is hype. I don't think we're in an equivalent place in the history of our field where any-one can, definitively, say, either from long study (Drucker) or manufactured hype (Welch), what works best, or -perhaps more importantly- what underlying misconceptions and mistakes fuel operations on a daily basis : What would be the network operations equivalents to revered : business tomes like "The Practice of Management", "Seven : Habits of Highly Successful People", "The G.E. Way" (and a : variety of others that populate the shelves of your friendly : local executive)? : : Most network-oriented training seems to focus on the : technology, not on operations (and those subtle but ever so : critical differences between knowing how something is : /supposed/ to work and how it /really/ works, and all of the : effort it takes to create a smooth-running operations : engine). There are three distinct possibilities that I sense: It is thought that the basic, already known, principles of engineering and management are a good fit, thus the training assumes this; No one sets out to be 'in operations', but rather drift or fall into the part-- or perhaps have the part fallen unto them... =-), and the training reflects this; The whole is never perceived as greater than the sum of the parts. That is to say, attention is paid to the individual components of a network and the bigger operational picture isn't tended except as a means of managing individual components... : What are the network operations equivalents to business : programs such as Six Sigma? What about something similar to : the various leading institutions of business management, : institutions that study of how networks are operated (and : used), and develop training and methodologies for better : operations practices? Don't know if you're aware: the IETF has two working groups in the Operations and Management Area, that might be tangent to what you're looking for: Benchmarking Methodology and Policy Framework. Might be worth a look, even if not exactly what you're thinking of... Though I've only glanced at them meself. : Pete. : : -- Systems, Networks and Gadgets, done with Artful Intelligence -<>-<_>-<__>-<_>-<>- Policy: ASCII/text attchmnts alway read. PDF maybe read. Others, by necessity, may be ignored. Don't take it personally, it's a time issue.
On Fri, 7 Dec 2001, Petr Swedock wrote:
With regards to Peter Drucker... his sagacity lies in a deep understanding of the history of his field and long observance of such (some sixty years!). Jack Welch is hype. I don't
True, re. Drucker. Jack Welch is a bit more than hype - General Electric is a heck of a business success story (even if some of its environmental practices have been questionable). Not too many companies have quite the track record of GE - and Welch was clearly behind a lot of that success. He's also trained a lot of people.
think we're in an equivalent place in the history of our field where any-one can, definitively, say, either from long study (Drucker) or manufactured hype (Welch), what works best, or -perhaps more importantly- what underlying misconceptions and mistakes fuel operations on a daily basis
there are probably a lot of lessons that can be extracted from other operational contexts - electric operations, air traffic control, emergency response, military command and control Miles ************************************************************************** The Center for Civic Networking PO Box 600618 Miles R. Fidelman, President & Newtonville, MA 02460-0006 Director, Municipal Telecommunications Strategies Program 617-558-3698 fax: 617-630-8946 mfidelman@civicnet.org http://civic.net/ccn.html Information Infrastructure: Public Spaces for the 21st Century Let's Start With: Internet Wall-Plugs Everywhere Say It Often, Say It Loud: "I Want My Internet!" **************************************************************************
On Thu, 6 Dec 2001, Pete Kruckenberg wrote:
What would be the network operations equivalents to revered business tomes like "The Practice of Management", "Seven Habits of Highly Successful People", "The G.E. Way" (and a variety of others that populate the shelves of your friendly local executive)?
Although Internet CEOs and CFOs have been running around screaming "The Internet is different, the old rules don't apply" most of the business of network operations is "Those who cannot remember the past are doomed to repeat it." Dilbert and Drucker are applicable as always. But I suspect that's not the answer you want. Here are a few suggestions for books which should be on every Internet manager's bookshelf. It wouldn't hurt to read a few of them too :-) Bell Labs ; Engineering and Operations in the Bell System Brooks, Frederick ; The Mythical Man-Month Economides, Nicholas ; The Economics of Networks (a paper which should be a book) Humphrey, Watts ; Managing Technical People Huston, Geoff ; ISP Survival Guide Kahin, Brian, editor ; Coordinating the Internet Knight, Christopher ; ISP Marketing Survival Guide Neumann, Peter ; Computer-Related Risks Schneider, Fred, editor ; Trust in Cyberspace Shapiro & Varian ; Information Rules
On Sat, 8 Dec 2001, Sean Donelan wrote:
Here are a few suggestions for books which should be on every Internet manager's bookshelf. It wouldn't hurt to read a few of them too :-)
I'm putting together a list of resources at http://inet-ops.stealthgeeks.net. If people would be interested in contributing their suggestions/links/etc., it would be most appreciated.
Hi Pete Would "old timers" that ran older networks (such as ARPANET or BITNET) count too ? from the answers it seems that no network operations existed before "the Internet" - of course the networks were _much_ smaller but IMHO 9.6Kbit/sec backbones must have had other challenges - right ? -- Rafi
On Sun, 9 Dec 2001, Rafi Sadowsky wrote:
Hi Pete
Would "old timers" that ran older networks (such as ARPANET or BITNET) count too ? from the answers it seems that no network operations existed before "the Internet" - of course the networks were _much_ smaller but IMHO 9.6Kbit/sec backbones must have had other challenges - right ?
There were operations centers just not *Internet* operations centers. Bellcore and IBM spring to mind.
Would "old timers" that ran older networks (such as ARPANET or BITNET) count too ? from the answers it seems that no network operations existed before "the Internet" - of course the networks were _much_ smaller but IMHO 9.6Kbit/sec backbones must have had other challenges - right ?
There was an ARPANET network operations center @ BBN. Craig
On Sat, 8 Dec 2001, Craig Partridge wrote:
Would "old timers" that ran older networks (such as ARPANET or BITNET) count too ? from the answers it seems that no network operations existed before "the Internet" - of course the networks were _much_ smaller but IMHO 9.6Kbit/sec backbones must have had other challenges - right ?
There was an ARPANET network operations center @ BBN.
Did any of the BBN managers publish a book on how to run a network operation center? I'm familar with Katie Hafner's "Where Wizards Stay Up Late." But it is more history than operations oriented.
In message <Pine.GSO.4.40.0112082015390.15168-100000@clifden.donelan.com>, Sean Donelan writes:
Did any of the BBN managers publish a book on how to run a network operation center?
Unfortunately not -- some of the BBN practices got incorporated in the network operations practices of CSNET and later NEARNET. But that's about it. Some day someone should convince Dan Long (head of CSNET and NEARNET operations) to write such a book. Craig
participants (19)
-
alex＠yuriev.com -
Bill Woodcock -
Brett Frankenberger -
Clayton Fiske -
Craig Partridge -
Fletcher E Kittredge -
Jeff Rizzo -
Joel Baker -
Joel Jaeggli -
Miles Fidelman -
Patrick Greenwell -
Pete Kruckenberg -
Petr Swedock -
Rafi Sadowsky -
Randy Bush -
Sean Donelan -
Simon Higgs -
Vadim Antonov -
Valdis.Kletnieks＠vt.edu