Are the days of the showpiece NOC office display gone forever?
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc. In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours. To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication. Are the days of such an environment gone forever?
We are still operating ours - 27 1080P projectors - but with a skeleton crew of just 3. Given the air volume, it’s almost like outside. Everyone else is WFH. —L.B. Lady Benjamin PD Cannon, ASCE 6x7 Networks & 6x7 Telecom, LLC CEO ben@6by7.net <mailto:ben@6by7.net> "The only fully end-to-end encrypted global telecommunications company in the world.” FCC License KJ6FJJ
On Dec 16, 2020, at 12:49 PM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
Peace, On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon <ben@6by7.net> wrote:
We are still operating ours - 27 1080P projectors - but with a skeleton crew of just 3. Given the air volume, it’s almost like outside.
A devil advocate here, First of all, COVID-19 is really serious. With that in mind, with all the necessary precautions office space *may* be managed safely to prevent the spread. Production plants had security measures preventing workforce injuries for a century already. Just a bit of that, with constant trainings, would get your monitoring room safe, especially with all the bars closed and everything operating on delivery. That is not to say that large monitoring rooms are a better choice over automation (which they are not). -- Töma
That is not to say that large monitoring rooms are a better choice over automation (which they are not).
I'm sure when the automation is perfect and widespread to the point that it catches and alerts on every network event, the monitoring rooms will disappear. But unless you have an entire organization dedicated to automation development or pay an incredibly large sum of money for pre-built packages, the business decision may still be made to actively monitor the network with eyeballs. Every failure mode is known until a new one pops up. Automation without any kind of ML secret sauce relies on known failure-modes. Not advocating one or the other, just playing Devil's advocate for the Devil's advocate. -Matt On Wed, Dec 16, 2020 at 2:28 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon <ben@6by7.net> wrote:
We are still operating ours - 27 1080P projectors - but with a skeleton crew of just 3. Given the air volume, it’s almost like outside.
A devil advocate here,
First of all, COVID-19 is really serious.
With that in mind, with all the necessary precautions office space *may* be managed safely to prevent the spread.
Production plants had security measures preventing workforce injuries for a century already. Just a bit of that, with constant trainings, would get your monitoring room safe, especially with all the bars closed and everything operating on delivery.
That is not to say that large monitoring rooms are a better choice over automation (which they are not).
-- Töma
-- Matt Erculiani ERCUL-ARIN
Peace, On Thu, Dec 17, 2020, 1:50 AM Matt Erculiani <merculiani@gmail.com> wrote:
I'm sure when the automation is perfect and widespread to the point that it catches and alerts on every network event, the monitoring rooms will disappear.
Which is never, but: With a proper RCA after each incident, not blaming customers or peers (which is common) but accepting the blame, and not personally, but as a fault of failover, security and notification mechanisms to be refined, and by employing third party tools trained on big chunks of data of yours and others — you can reduce the number of screens required for daily routine to a sustainable amount that a single onshift engineer can carry home. And that is around where the investment returns to you. -- Töma
On 12/17/20 00:50, Matt Erculiani wrote:
But unless you have an entire organization dedicated to automation development or pay an incredibly large sum of money for pre-built packages, the business decision may still be made to actively monitor the network with eyeballs.
Solutions do exist, or so the salesmen/women that peddle them tell me. I'm not convinced. Also, I'm a bit ancient. Mark.
I'm sure when the automation is perfect and widespread to the point that it catches and alerts on every network event, the monitoring rooms will disappear.
The chances of this happening are exactly 0%.
But unless you have an entire organization dedicated to automation development or pay an incredibly large sum of money for pre-built packages, the business decision may still be made to actively monitor the network with eyeballs.
Contrary to what salespeople will say, the answer is not 100% automation, or 100% humans. The proper answer is an often changing combination of the two. Every failure mode is known until a new one pops up. Automation without
any kind of ML secret sauce relies on known failure-modes.
ML is not the magical unicorn solution that solves everything, contrary to what many papers and salespeople tell you. Let's take a network interface that is randomly shitting on packets. What is more important operationally, identifying that the packets are being shat on, or having ML predict when the next shatting will occur? Clearly the first right? You want to find it and fix it as fast as possible. There could be a place for ML when it comes to diagnosing the reason for the shitting , but that's different. There may be some interesting applications for ML in tracking down really complicated operational anomalies, but it will never be a primary mode of detection for the same reason you said ; Every failure mode is known until a new one pops up. You can't train an ML model for a failure condition that you don't know exists. On Wed, Dec 16, 2020 at 5:51 PM Matt Erculiani <merculiani@gmail.com> wrote:
That is not to say that large monitoring rooms are a better choice over automation (which they are not).
I'm sure when the automation is perfect and widespread to the point that it catches and alerts on every network event, the monitoring rooms will disappear.
But unless you have an entire organization dedicated to automation development or pay an incredibly large sum of money for pre-built packages, the business decision may still be made to actively monitor the network with eyeballs.
Every failure mode is known until a new one pops up. Automation without any kind of ML secret sauce relies on known failure-modes.
Not advocating one or the other, just playing Devil's advocate for the Devil's advocate.
-Matt
On Wed, Dec 16, 2020 at 2:28 PM Töma Gavrichenkov <ximaera@gmail.com> wrote:
Peace,
On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon <ben@6by7.net> wrote:
We are still operating ours - 27 1080P projectors - but with a skeleton crew of just 3. Given the air volume, it’s almost like outside.
A devil advocate here,
First of all, COVID-19 is really serious.
With that in mind, with all the necessary precautions office space *may* be managed safely to prevent the spread.
Production plants had security measures preventing workforce injuries for a century already. Just a bit of that, with constant trainings, would get your monitoring room safe, especially with all the bars closed and everything operating on delivery.
That is not to say that large monitoring rooms are a better choice over automation (which they are not).
-- Töma
-- Matt Erculiani ERCUL-ARIN
On Thu, 17 Dec 2020, Tom Beecher wrote:
I'm sure when the automation is perfect and widespread to the point that it catches and alerts on every network event, the monitoring rooms will disappear.
The chances of this happening are exactly 0%.
Indeed. More broadly, a lot of people have tried to get rid of operations staff and suffered the consequences.
Contrary to what salespeople will say, the answer is not 100% automation, or 100% humans. The proper answer is an often changing combination of the two.
Exactly. There is an argument to be said that human operators are actually part of the computer system. This is implied in terms like 'wetware' but not often explicitely stated. If the last 50 years has shown us anything it is that humans and computers working together can achieve far more than either in isolation. Cheers, Rob
On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
On Thu, 17 Dec 2020, Tom Beecher wrote: If the last 50 years has shown us anything it is that humans and computers working together can achieve far more than either in isolation.
Cheers,
Rob
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal. (pet peeve) -Wayne --- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
On 12/23/20 07:41, Wayne Bouchard wrote:
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal.
(pet peeve)
The good news is that choice deals with this problem. The level of patience we've had to allow this type of customer interaction has been drastically reduced by our experiences with free or paid services we experience with apps on our phones. Without realizing it, our basic expectations rise from how we experience one app that has nothing to do with the other. It put more stock in choice. Either we are deleting an app 5 seconds after it doesn't meet our re-trained expectations, or we are cancelling a contract and moving on to another provider. I, like you, refuse to call a call centre and ask for support. The same goes for online services whose support is limited to bots or FAQ's. The moment I can't get a fix via an e-mail and I have to speak to someone waiting with a script, I cancel and move on. Mark.
On Tuesday, 22 December, 2020 22:42, Wayne Bouchard wrote:
On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
On Thu, 17 Dec 2020, Tom Beecher wrote:
If the last 50 years has shown us anything it is that humans and computers working together can achieve far more than either in isolation.
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal.
The correct term of art for these humans that execute "scripts" when answering inquiries is "flapper". Often there are multiple layers of "flapper" before one can communicate with someone who has any clue what they (or you) are speaking about. This is deliberate in an attempt to cull the wheat from the chaff of the support calls because 99.9999999% of callers are "chaff" and do not have a problem that is worthy of attention by someone who knows what they are doing. Many organizations which employ "flappers" have "flapper bypass systems" in place in which there are either "magical incantations" or perhaps an entry in the CRM system that identifies the probability that a caller is "wheat" vs "chaff" so that the multi-level flapper system can be bypassed. There are even a few organizations which do not employ flappers at all -- though they are few and far between. If an organization does not have a functional "flapper bypass system" then usually the most effective system to bypass multiple layers of flappers is what is called the "shit principle". The "shit principle" states that shit works best when flowing downhill and therefore the most effective "flapper bypass" is to direct the problem to the higest level executive in the organization with the least probability of being able to address the issue. That person will simply direct an under-thing to "take care of this and have them stop bothering me" which will result in you immediately having the issue resolved by a competent individual. Organizations without other functional "flapper bypass" procedures usually have a huge organization dedicated to addressing issues raised through the "shit principle" since this is, in reality, their chosen "flapper bypass" system. -- Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.
Can't resist two comments: For those who might otherwise wonder, "Flapper" has no connection to the US Roaring Twenties, but was a concept pioneered by Jonathan Swift in "Gulliver's Travels" particular to the land of Laputa (NOT to be confused with Lilliput, with which many are familiar..) - where certain people are so self-important that they are accompanied at all times by a dedicated servant who, when in the opinion of the servant the master would benefit by awareness of ome event or situation, would (gently) cudgel the master with a bladder of dried peas or pebbles, either on the ears or the eyes, so that the master's attention would be drawn from their intense interior environment to something occurring in the mundane world. (In my opinion, this was Swift's most trenchant critique of civil society.) For the second comment - no discussion of a flapper bypass system is complete without: https://xkcd.com/806 Oh, have I often wished... Blessings.. ..Allen Allen M. Kitchen 404 Franklin Street, Butler PA 16001 412-295-7763 (cell; text OK 24x7) allenmckinleykitchen@gmail.com On Wed, Dec 23, 2020 at 12:12 PM Keith Medcalf <kmedcalf@dessus.com> wrote:
On Tuesday, 22 December, 2020 22:42, Wayne Bouchard wrote:
On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
On Thu, 17 Dec 2020, Tom Beecher wrote:
If the last 50 years has shown us anything it is that humans and computers working together can achieve far more than either in isolation.
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal.
The correct term of art for these humans that execute "scripts" when answering inquiries is "flapper". Often there are multiple layers of "flapper" before one can communicate with someone who has any clue what they (or you) are speaking about. This is deliberate in an attempt to cull the wheat from the chaff of the support calls because 99.9999999% of callers are "chaff" and do not have a problem that is worthy of attention by someone who knows what they are doing.
Many organizations which employ "flappers" have "flapper bypass systems" in place in which there are either "magical incantations" or perhaps an entry in the CRM system that identifies the probability that a caller is "wheat" vs "chaff" so that the multi-level flapper system can be bypassed.
There are even a few organizations which do not employ flappers at all -- though they are few and far between.
If an organization does not have a functional "flapper bypass system" then usually the most effective system to bypass multiple layers of flappers is what is called the "shit principle". The "shit principle" states that shit works best when flowing downhill and therefore the most effective "flapper bypass" is to direct the problem to the higest level executive in the organization with the least probability of being able to address the issue. That person will simply direct an under-thing to "take care of this and have them stop bothering me" which will result in you immediately having the issue resolved by a competent individual.
Organizations without other functional "flapper bypass" procedures usually have a huge organization dedicated to addressing issues raised through the "shit principle" since this is, in reality, their chosen "flapper bypass" system.
-- Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.
On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote:
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal.
There are some easy methods for service/support organizations to decrease the pain that this inflicts on people reporting problems. For example, one thing that I've taught people to do is to make liberal use of procmail in order to sort incoming traffic to role accounts. It requires diligence, but that diligence is repaid many times over by how it expedites dealing with problems. A simple example of this is that when a problem report is received at the RFC 2142 security@ role address, and it's clueful, well-written, and important, a procmail rule gets created for the sending address so that all future messages from that address are prioritized...because it obviously came from someone who knows what the heck they're doing and did us a favor by telling us that we have a problem. Chances are that any future messages from them will be similarly helpful and that if we respond to those quickly we may be able to forestall a lot more messages that aren't going to be as clueful. The opposite thing is done with clueless/misdirected/etc. reports: they're not discarded, but they go into the low-priority queue. Everything else goes somewhere in the middle. Repeated hundreds or thousands of times over many years, this builds a ruleset that pre-sorts messages rather well. It's not perfect, it's not foolproof, but it helps us *and* it helps lower the frustration level of people sending clueful messages, because it better positions us to read, act on, and respond to those. Those people are catching our mistakes, the least we can do is try to pay attention. (Hint: a useful way to begin building such a ruleset is to grab all the addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset with them...because traffic received at role accounts from participants in these mailing lists is probably useful.) ---rsk
It’d be real interesting to open-source this somehow, produce a useable open or quasi open (maybe curated somehow) reputation score for email. Ms. Lady Benjamin PD Cannon, ASCE 6x7 Networks & 6x7 Telecom, LLC CEO ben@6by7.net "The only fully end-to-end encrypted global telecommunications company in the world.” FCC License KJ6FJJ Sent from my iPhone via RFC1149.
On Dec 30, 2020, at 3:04 PM, Rich Kulawiec <rsk@gsp.org> wrote:
On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote:
And if the last 15 years has shown us anything, it is that when you can't get past the auto-attendant and talk to a real human, and if that person can't talk to you like a person instead of reading scripts at you, your stress levels go way up as does your desire to break things. Automation in customer service (or excessive emphasis on procedures) is a really nice way of taking a five minute problem and turning it into an hour long ordeal.
There are some easy methods for service/support organizations to decrease the pain that this inflicts on people reporting problems.
For example, one thing that I've taught people to do is to make liberal use of procmail in order to sort incoming traffic to role accounts. It requires diligence, but that diligence is repaid many times over by how it expedites dealing with problems. A simple example of this is that when a problem report is received at the RFC 2142 security@ role address, and it's clueful, well-written, and important, a procmail rule gets created for the sending address so that all future messages from that address are prioritized...because it obviously came from someone who knows what the heck they're doing and did us a favor by telling us that we have a problem. Chances are that any future messages from them will be similarly helpful and that if we respond to those quickly we may be able to forestall a lot more messages that aren't going to be as clueful.
The opposite thing is done with clueless/misdirected/etc. reports: they're not discarded, but they go into the low-priority queue.
Everything else goes somewhere in the middle.
Repeated hundreds or thousands of times over many years, this builds a ruleset that pre-sorts messages rather well. It's not perfect, it's not foolproof, but it helps us *and* it helps lower the frustration level of people sending clueful messages, because it better positions us to read, act on, and respond to those. Those people are catching our mistakes, the least we can do is try to pay attention.
(Hint: a useful way to begin building such a ruleset is to grab all the addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset with them...because traffic received at role accounts from participants in these mailing lists is probably useful.)
---rsk
Frankly, I think the days of the "showpiece NOC" being relevant ended a while before COVID. I worked at a Tier 1 for >10 years in a customer-facing capacity, largely dealing with "serious" enterprise customers. The number of customers who toured the NOC in that time was less than 5 (i.e much less than 1%), despite generally offering it up to new and quickly-scaling customers. I actually spent more time visiting customer NOCs over that time than I did my own with customers (and some of theirs were more impressive anyway). Now, I did spend a lot of time coordinating calls between the NOC and customers, some of which was in the break-fix, mea culpa sort of vain, but also a lot of "spend time getting comfortable" types of conversations too, especially with customers considering their first service. So it wasn't that folks didn't care about what was going on there inasmuch as they recognized (quite rightfully IMO) that they weren't going to get any value being there that they couldn't get meeting the folks and learning about operational procedures, etc. over a conference call, so why waste the time/money to travel for that sort of thing? I don't know if I have a biased sample or if this is reflective of the norm these days, but the "NOC Tour" was something that didn't get executed on very often. On Wed, Dec 16, 2020 at 3:51 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
-- - Dave Cohen craetdave@gmail.com @dCoSays www.venicesunlight.com
Dave Cohen wrote:
Frankly, I think the days of the "showpiece NOC" being relevant ended a while before COVID. I worked at a Tier 1 for >10 years in a customer-facing capacity, largely dealing with "serious" enterprise customers. The number of customers who toured the NOC in that time was less than 5 (i.e much less than 1%), despite generally offering it up to new and quickly-scaling customers. I actually spent more time visiting customer NOCs over that time than I did my own with customers (and some of theirs were more impressive anyway). I just remember that, back in my BBN days (a LONG time ago), the various Defense Data Network NOCs, around the world, used to keep their big-screen monitors tuned to CNN.
Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
Peace, On Wed, Dec 16, 2020, 11:50 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays.
a) Without a change in processes you need to replicate some of that at home of every onshift engineer of your NOC. This might sound expensive but a single failure might cost you more. b) But to be honest, most of the monitoring done by "a hundred of blinking red eyes in a monitoring room" should be replaced with data analysis, correlation tools, and automated alerts. Those don't look that awesome on the Instagram accounts of your marketing team but work better. -- Töma
We used to have some CRTs with MRTG running in the late 90’s 😊 P From: NANOG <nanog-bounces+razor=meganet.net@nanog.org> On Behalf Of Eric Kuhnke Sent: Wednesday, December 16, 2020 3:50 PM To: nanog@nanog.org list <nanog@nanog.org> Subject: Are the days of the showpiece NOC office display gone forever? With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc. In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours. To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication. Are the days of such an environment gone forever?
I would not be surprised to see Cisco CTC (the alarm control panel/monitoring software for 15454s carrying TDM, SDH/SONET circuits) still being used in the year 2030 in some places. On Wed, Dec 16, 2020 at 1:10 PM Paul Amaral <razor@meganet.net> wrote:
We used to have some CRTs with MRTG running in the late 90’s 😊
P
*From:* NANOG <nanog-bounces+razor=meganet.net@nanog.org> *On Behalf Of *Eric Kuhnke *Sent:* Wednesday, December 16, 2020 3:50 PM *To:* nanog@nanog.org list <nanog@nanog.org> *Subject:* Are the days of the showpiece NOC office display gone forever?
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
Call me crazy, but I have never cared about the second half of that. On Wed, Dec 16, 2020 at 3:51 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
I remember working in the showpiece "Uncle Bernie" Ebbers had built in Ashburn, VA, for UUNET. You can even catch a glimpse of me in the American Greed episode dedicated to WorldCom's downfall. I wonder just what that place looks like now. Since then, I have seen NOCs with multiple displays for multiple customers, but only designed to be useful to NOC staff, not prospective customers or executives looking for something pretty to watch. -----Original Message----- From: Eric Kuhnke <eric.kuhnke@gmail.com> To: nanog@nanog.org list <nanog@nanog.org> Sent: Wed, Dec 16, 2020 3:49 pm Subject: Are the days of the showpiece NOC office display gone forever? With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc. In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours. To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication. Are the days of such an environment gone forever?
On 16 Dec 2020, at 15.49, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
It seems like the "something impressive looking" might just change. Add a bit of marketing copy about how "our monitoring is distributed, just like our servers" (or some more punchy variant) and show customers what the smaller monitoring stations WFH employees are using look like.
On 12/16/20 23:48, Max Harmony via NANOG wrote:
It seems like the "something impressive looking" might just change. Add a bit of marketing copy about how "our monitoring is distributed, just like our servers" (or some more punchy variant) and show customers what the smaller monitoring stations WFH employees are using look like.
Nowadays, customers tend to care about you watching their circuit, and service. Most still get surprised when they don't understand why the NOC never knew their circuit went down, and need help understanding the difference between monitoring the core, and monitoring customers who choose to purchase that service. Mark.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
though your message has a current date, its content seems to be at least 15 years old randy
On Dec 16, 2020, at 7:25 PM, Randy Bush <randy@psg.com> wrote:
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
though your message has a current date, its content seems to be at least 15 years old
randy
The EDS Network Management Center, or IMS, had multiple large screens, shadow-free lighting, and lots of console positions and a visitor gallery with curtained window overlooking the operations room. We finished it right around 1990. Right when the suburban sprawl ws starting to hit Plano. After so much there during construction, I was taken aback when I realized I did not recognize the inside the building with it’s finished walls. That implies “at least 15 years” could well be “30 years” . James R. Cutler James.cutler@consultant.com GPG keys: hkps://hkps.pool.sks-keyservers.net
Perhaps I should have clarified: "from the perspective of persons who have the word "Sales" in their job titles, considered to be impressive looking for customer tours" On Wed, Dec 16, 2020 at 4:25 PM Randy Bush <randy@psg.com> wrote:
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
though your message has a current date, its content seems to be at least 15 years old
randy
On 12/16/20 22:49, Eric Kuhnke wrote:
With the covid19 situation, obviously lots of ISPs have their NOC personnel working from home, with VPN (or remote desktop) access to all the internal tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the purpose of having large situational awareness displays on a wall, network weathermaps and charts, alerting systems, composed of four or more big flat panel displays. Ideally configured to be actually useful for NOC purposes and also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of seriousness on the part of an ISP, I suppose depends on what sort of customers they are, and their relative degree of technical sophistication.
Are the days of such an environment gone forever?
The customers we've been dealing with, particularly this year, fancy being listened and spoken to more than what our brand of NOC coffee is. Mark.
participants (21)
-
Allen Kitchen -
Ben Cannon -
Bryan Holloway -
Dave Cohen -
Eric Kuhnke -
James R Cutler -
Joe Provo -
Keith Medcalf -
Lady Benjamin PD Cannon
-
Mark Tinka -
Matt Erculiani -
Max Harmony -
Michael Perkins -
Miles Fidelman -
Paul Amaral -
Randy Bush -
Rich Kulawiec -
Robert Brockway -
Tom Beecher -
Töma Gavrichenkov -
Wayne Bouchard