Benefits (and Detriments) of Standardizing Network Equipment in a Global Organization
Hail NANOGers! A global hospitality organization with 100+ locations recently asked us how to weigh the importance of standardizing infrastructure across all their locations versus allowing each international location to select on their own kit. My first instinct was to jump on my favorite search engine and look for an authoritative document covering the topic. To my surprise I have not been able to find such a thing. So I've begun to write one myself, and as I start I've realized that: a) This is likely to be a document that will be helpful to the wider community, and b) This is likely a topic that many of you have a great deal of knowledge and personal experience. If you have pointers to an existing doc, please share. If you have a case study, lesson learned, data point, or even just a strong opinion; I'd love to hear it! My intention is to put this together BCOP style, but with more of a focus on why rather than how this time around. Feel free to reply on or off list. Thanks in advance for your input! Cheers, ~Chris PS - I won't use any direct quotes without advance permission and I'll provide attribution to all that contribute meaningfully. -- @ChrisGrundemann http://chrisgrundemann.com
On Fri, 23 Dec 2016 15:36:10 -0500, Chris Grundemann said:
A global hospitality organization with 100+ locations recently asked us how to weigh the importance of standardizing infrastructure across all their locations versus allowing each international location to select on their own kit.
The first question that comes to mind is: Does the organization have any centralized IT, or is *that* done by each location? The procurement directives need to be coming from the group that actually does day-to-day support of each location, or the resulting culture clash will cause issues....
System automation and life cycle management is exponentially easier when you have uniform environments. I am in the process of standardizing global infrastructure and developing the automation process now. Nolan ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> Sent: Monday, December 26, 2016 7:55 PM To: Chris Grundemann Cc: nanog@nanog.org Subject: Re: Benefits (and Detriments) of Standardizing Network Equipment in a Global Organization On Fri, 23 Dec 2016 15:36:10 -0500, Chris Grundemann said:
A global hospitality organization with 100+ locations recently asked us how to weigh the importance of standardizing infrastructure across all their locations versus allowing each international location to select on their own kit.
The first question that comes to mind is: Does the organization have any centralized IT, or is *that* done by each location? The procurement directives need to be coming from the group that actually does day-to-day support of each location, or the resulting culture clash will cause issues....
In a message written on Fri, Dec 23, 2016 at 03:36:10PM -0500, Chris Grundemann wrote:
If you have a case study, lesson learned, data point, or even just a strong opinion; I'd love to hear it!
I think the high level items are pretty clear here: 1 Vendor Quicker/easier to implement, staff only needs to learn/configure one platform, vendor can help end to end, usually fewer interop issues. Spend may get extra discounts or support bennies. However one bug can wipe out everything, no ability to compare real world performance with a competitor, vendor may think they "own" you come renewal or more sales. Hard to threaten to leave. 2 Vendor Can be implemented multiple ways, for instance 1 vendor per site alternating sites, or gear deployed in pairs with one from each vendor up and down the stack. Harder to implement, staff needs to know both, all configs must be done for both, vendors will always blame the other vendor for interop issues. Twice as much chance of needing to do emergency upgrades. More resilliance to a single bug, can compare real world performance of the two vendors. Both vendors will compete hard to get more of your business, but have a harder time justifing bennies internally due to lower spend. 3 or more Vendors Generally the same as two-vendors, just ++. That is more of the pros, and more of the cons. Limited additional upside to having 3 or more active vendors. Generally occurs as a vendor falls out of favor, two new ones get deployed moving forward, the old one sticks around for a while. Having worked places that were single vendor, 2 vendor, and "whatever we can buy" shops I'll say it basically doesn't matter. What matters is how you set up the org. Want to be lean on staff, go single vendor. Want maximum resilliance and/or negotiating power, go 2 vendor. Inherit a mess, learn to live in a 3+ vendor world. It's not that one is better than the other, it's just they require different approaches to get the same outcome. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
G'day Leo, On 28 December 2016 at 07:10, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Fri, Dec 23, 2016 at 03:36:10PM -0500, Chris
Grundemann wrote:
If you have a case study, lesson learned, data point, or even just a strong opinion; I'd love to hear it!
I think the high level items are pretty clear here:
[...]
2 Vendor
Can be implemented multiple ways, for instance 1 vendor per site alternating sites, or gear deployed in pairs with one from each vendor up and down the stack.
Harder to implement, staff needs to know both, all configs must be done for both, vendors will always blame the other vendor for interop issues. Twice as much chance of needing to do emergency upgrades.
More resilliance to a single bug, can compare real world performance of the two vendors. Both vendors will compete hard to get more of your business, but have a harder time justifing bennies internally due to lower spend. [...]
I agree with many of the points you made but here's another data point on the topic of bugs -- I watched a presentation [1] from a couple of guys from Facebook at AusNOG 2016 and one of the takeaways from their talk was that "multivendor is hard". When you have two vendors, you get Vendor A's bugs, Vendor B's bugs, and the Vendor A+B interop bugs. In theory this only gets worse with 3+ vendors. Cheers, Dale [1] < http://www.ausnog.net/sites/default/files/ausnog-2016/presentations/2.10.6_Z...
On Tue, Dec 27, 2016 at 3:10 PM, Leo Bicknell <bicknell@ufp.org> wrote:
2 Vendor
Can be implemented multiple ways, for instance 1 vendor per site alternating sites, or gear deployed in pairs with one from each vendor up and down the stack.
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
On Wed, Dec 28, 2016 at 1:39 PM, Chris Grundemann <cgrundemann@gmail.com> wrote:
On Tue, Dec 27, 2016 at 3:10 PM, Leo Bicknell <bicknell@ufp.org> wrote:
2 Vendor
Can be implemented multiple ways, for instance 1 vendor per site alternating sites, or gear deployed in pairs with one from each vendor up and down the stack.
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
It also doesn't get you to a place where you can 'require' similar functionality between vendors at each layer... you MAY (if not careful) get locked into a particular vendor at one/some levels of the stack ;( (which you noted as a problem still) if you choose poorly on 'features used' (oops! I love me some eigrp... crap) -chris
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
i think this is where i say that i hope my competitors do this. it is a recipe for a complex set of delicate dependencies and great fun debugging. randy
On Dec 28, 2016, at 5:34 PM, Randy Bush <randy@psg.com> wrote:
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
i think this is where i say that i hope my competitors do this. it is a recipe for a complex set of delicate dependencies and great fun debugging.
One of the more spectacular failures I've seen was a bug in a network core router that caused bad into to be carried by all of that same vendor's routers across the core to the edges (made by a different vendor) which promptly barfed and locked up. So I'd be cautious about saying "vendor X for one layer, vendor Y for adjacent layer" as a multi-vendor strategy. David Barak Sent from mobile device, please excuse autocorrection artifacts
In a message written on Wed, Dec 28, 2016 at 01:39:59PM -0500, Chris Grundemann wrote:
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
While a lot of people seem to be beating you up over this approach, many folks end up in it for various reasons. For instance the chances a vendor makes both a functional edge router and a high quality firewall are low, which means they often are sourced from different companies. But I think the question others are trying to ask is a different hyptothetical. Say there are two vendors, of of which makes perfectly good edge routers and core routers. What are the pros to buying all of the edge from one, and all of the core from the other? I have to admit I'm having trouble coming up with potential technical upsides to such a solution. There may well be a business up side, in that due to the way price structures are done that such a method saves captial. But in terms of technical resilliance, if there's a bug that takes out all cores or all edges the whole network is down, and there's actually 2x the risk as it could happen at either layer! -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
I'm not a fan of vendor lock-in, and have been bitten by it. I eschew vendor specific solutions unless it is essential to delivering a particular result. Keeping multiple players at the table, and making those players aware that you have other options that you can and do take advantage of seems to keep them on their toes when it's time for refresh. On Thu, Dec 29, 2016 at 9:44 AM, Leo Bicknell <bicknell@ufp.org> wrote:
In a message written on Wed, Dec 28, 2016 at 01:39:59PM -0500, Chris Grundemann wrote:
An alternative multi-vendor approach is to use 1 vendor per stack layer, but alternate layer to layer. That is; Vendor A edge router, Vendor B firewall, Vendor A/C switches, Vendor D anti-SPAM software, etc. This doesn't address the bug impact issue as well as it alleviates the vendor "ownership" issue though...
While a lot of people seem to be beating you up over this approach, many folks end up in it for various reasons. For instance the chances a vendor makes both a functional edge router and a high quality firewall are low, which means they often are sourced from different companies.
But I think the question others are trying to ask is a different hyptothetical. Say there are two vendors, of of which makes perfectly good edge routers and core routers. What are the pros to buying all of the edge from one, and all of the core from the other?
I have to admit I'm having trouble coming up with potential technical upsides to such a solution. There may well be a business up side, in that due to the way price structures are done that such a method saves captial. But in terms of technical resilliance, if there's a bug that takes out all cores or all edges the whole network is down, and there's actually 2x the risk as it could happen at either layer!
-- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
On Thu, 29 Dec 2016 07:44:45 -0800, Leo Bicknell said:
But I think the question others are trying to ask is a different hyptothetical. Say there are two vendors, of of which makes perfectly good edge routers and core routers. What are the pros to buying all of the edge from one, and all of the core from the other?
The *original* question, which seems to have gotten lost, was: Say you're doing business in 100 countries, with some stated level of possible autonomy for each business unit. Is it better for upper corporate to say "all 100 national business units will use vendor A for edge devices and vendor B for routing", or "all 100 business units shall choose, based on local conditions such as vendor support, a standard set of vendors for their operations"? Stated differently, "Which causes more trouble - a mix of Vendor A in Denmark talking to Vendor B in Finland, or corporate mandating the use of Vendor Q even if Q doesn't have a support office in Kazakhstan while vendor F has an office in the building next door"?
In a message written on Thu, Dec 29, 2016 at 01:22:28PM -0500, Valdis.Kletnieks@vt.edu wrote:
Say you're doing business in 100 countries, with some stated level of possible autonomy for each business unit.
In all honesty, the original question was a poor straw man for multiple reasons: * Basically nobody does business in 100 countries. Level 3 only claims 60. Verizon does claim 150, but a lot of those are rather arms-length deals. Apple has a "presense" in 97 countries. It's a question about perhaps not a unicorn, but a rare albino pony only seen a few times in the wild! * The companies that do business in these countries rarely have "100 national business units". The chance that all countries are wholy owned subsidiaries created by the corporate parent is zero. They are parterships, co-branding deals, buyouts of existing companies. All of which bring baggage that affects the question more than any any technical details. * Because of how these entities come to be, the chance that the network contains Vendor's A and B, and corporate gets to dictate anything is zero. The network will have Vendors A-Z, plus a few more. Legacy stuff hidden in corners from 50 different M&A activities. Multiple engineering teams, in multiple locations. * Technical people never get to decide the level of "autonomy". A mix of local laws, M&A terms, and other business interests will rule.
Is it better for upper corporate to say "all 100 national business units will use vendor A for edge devices and vendor B for routing", or "all 100 business units shall choose, based on local conditions such as vendor support, a standard set of vendors for their operations"?
Which leads to an easy answer. It's better for upper corporate to negotiate bulk deals (note I did not say one vendor) and offer standard solutions to each national BU, so that the engineering does not need to be repeated 100 times. Simple economies of scale. That said, some number of the national BU's will not follow that advice, for perhaps good and often bad reason. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
On 12/29/16 10:22 AM, Valdis.Kletnieks@vt.edu wrote:
On Thu, 29 Dec 2016 07:44:45 -0800, Leo Bicknell said:
But I think the question others are trying to ask is a different hyptothetical. Say there are two vendors, of of which makes perfectly good edge routers and core routers. What are the pros to buying all of the edge from one, and all of the core from the other?
The *original* question, which seems to have gotten lost, was:
Say you're doing business in 100 countries, with some stated level of possible autonomy for each business unit.
Is it better for upper corporate to say "all 100 national business units will use vendor A for edge devices and vendor B for routing", or "all 100 business units shall choose, based on local conditions such as vendor support, a standard set of vendors for their operations"?
Stated differently, "Which causes more trouble - a mix of Vendor A in Denmark talking to Vendor B in Finland, or corporate mandating the use of Vendor Q even if Q doesn't have a support office in Kazakhstan while vendor F has an office in the building next door"?
ok I'll bite. imho, repeatable patterns of deployment are a great economic and organizational simplification. imho that doesn't mean they are identical. There may be generational differences even in what otherwise would be cookie cutter deployments, e.g. because devices go end of sale, new ones become available, regional vendor preference or vendor diversification. If these are centrally organized and operated or coordinated, then the minimum number of variants possible considering the circumstances where variation is is desirable. It minimizes the effort and training required to deploy and operate the system. If I were to take CDN pops as an example. Inter-generational variation is necessary as are accommodations for varying scale. the system is centrally coordinated and common operating methods are necessary if these systems are to behave a cohesive whole. Systems where deployment is less centralized, and local autonomy is therefore necessary as for example occurs when local contractors use their own equipment may come to different conclusions. e.g. that the specification of the minimum necessary common elements becomes the only feasible approach. For example if you are an Uber driver your minim IT requirements are something like Uber cell phone requirements iPhone requirements to run the Uber driver app Must be iPhone 4S, 5, 5C, 5S, 6, 6 Plus, 6S, 6S Plus, SE, 7, or 7 Plus running iOS 8 or later versions For best results: Use iPhone 5 or newer Android requirements to run the Uber driver app Any smart phone from 2013 or newer, running Android version 4.0 or newer For best results: The phone should run Android 5.0 or newer
When doing business in 100 countries, what if vendor A has support in 80 of those countries, and vendor B has good presence in the last 20 ? What if you require a vendor that has presence in all countries and this limits your RFPs to a single vendor ? Does your company run semi autonomous subsidiaries in each country with its own IT/networking staff ? Who buys local connectivity, HQ in USA or the local subsidiary ? So, if you maintain links to 100 countries around the globe, do you want central management ? can it provide localised support in local languages and local times zones from head office ? Or would that stretch it beyond reasonable capacity and you start to need support from different locations anyways ? Does HQ staff have legal knowledge or all local regulations? Do they have experience bribing officials where bribing is part of business? What happens when country X has special legal requirements, and country Y has conflicting requirememnts that prevent uniform deployment ? It wasn't that long ago that US equipment with encryption couldn't be exported everywhere because encryption was considered a military secret. Consider that in today's environment, it isn't so ludicrous to suggest that a country may require that the equipment has a "backdoor". So it is best to allow that country to have its own separate equipment with minimal management abilities to/from that country to prevent that country's government from interfering with your opps in other countries. It may seem more efficient to manage everything centrally but... I'll use an airline analogy: Southwest airlines was quite succesful with a single plane type. Common training for pilots, all planes maintained from a central hangar, common spare parts etc. If it had 100 planes, and all were maintained from one hangar capable of maintaining 100 planes, this was much better than having 50 737s, and 50 A 320s, requiring 2 separate hangars, each used at only 50% capacity. BUT, if you have 200 planes, then the second batch of planes could be Airbus, and maintained in their own hangar, resulting in both hangars being used at 100% capacity. The point is that beyond a certain size, the advantages of having everything common are not as important, and having dfferent equipment gives you more leverage when negotiating, as well as isolates bugs/viruses to only part of your network. Another aspect is of innovation. When HQ standardizes with one vendor and it is all centralliy managed, it becomes really hard to introduce new technologies because your systems are cast into concrete. If you give each country some autonomy for local equipment, they may be experimenting with different vendors and could find that some new vendor is much better than the one used at HQ, and that experience could then percolate up to headquarters (instead of everything decided at HQ and percolating down to each branch office/subsidiary). At the end of the day, it all depends on how autonomous each subsidiary is around the world. This is quite different from having 100 branches in the USA, each getting physical connection from the same fibre vendor and each operating under same laws, and minimal time zones (still 5 hours between New York and Hawaii though).
Hey Chris, Size has a lot to do with policy. For very large organizations, regional models make sense. Orwell had his regional divisions[1] and Level3 has theirs[2]. TW had a domestic version of regions before things were centralized. I'd argue for the organizational affect of standardization. Is operations centralized? Deployments should be standardized. Is operations distributed? Deployments should be centralized to those operational distributions. Do you have the resources to centralize? Technology research and acquisitions teams are expensive. Engineering teams are expensive. Operations teams are generally cheaper per unit, but one of the largest teams and therefore expensive. The bureaucracy supports whatever model you're after in a top-down way for the proverbial 80%. So many networks grow organically until operations break down and leadership implements centralized policy. Centralizing operations and empowering the Ops team's voice is an effective way to get to a more standardized environment. When Ops bucks and won't support clever (read: retarded) stuff and they have strong leadership to support them, things have to change. Rather than standardizing on a vendor, I support standardized deployments. Deployment X gets bill of materials Y. A significant assumption supporting this model includes well known variables that distinguish one deployment type from another, defined by centralized technology research, acquisitions, and engineering teams. Jason [1]https://en.wikipedia.org/wiki/Nations_of_Nineteen_Eighty-Four#/media/File:19... [2]http://www.level3.com/en/global-reach/ On Fri, Dec 23, 2016 at 1:36 PM, Chris Grundemann <cgrundemann@gmail.com> wrote:
Hail NANOGers!
A global hospitality organization with 100+ locations recently asked us how to weigh the importance of standardizing infrastructure across all their locations versus allowing each international location to select on their own kit.
My first instinct was to jump on my favorite search engine and look for an authoritative document covering the topic. To my surprise I have not been able to find such a thing. So I've begun to write one myself, and as I start I've realized that: a) This is likely to be a document that will be helpful to the wider community, and b) This is likely a topic that many of you have a great deal of knowledge and personal experience.
If you have pointers to an existing doc, please share.
If you have a case study, lesson learned, data point, or even just a strong opinion; I'd love to hear it!
My intention is to put this together BCOP style, but with more of a focus on why rather than how this time around. Feel free to reply on or off list.
Thanks in advance for your input!
Cheers, ~Chris
PS - I won't use any direct quotes without advance permission and I'll provide attribution to all that contribute meaningfully.
-- @ChrisGrundemann http://chrisgrundemann.com
participants (12)
-
Chad Dailey -
Chris Grundemann -
Christopher Morrow -
Dale Shaw -
David Barak -
Jason Iannone -
Jean-Francois Mezei -
joel jaeggli -
Leo Bicknell -
Nolan Berry -
Randy Bush -
Valdis.Kletnieks＠vt.edu