Networking Pearl Harbor in the Making
< CRAPAGANDA > Which operating system, embedded in more than 80% of enterprise IT environments, represents one of the fastest-growing hacker targets and potentially the most-devastating information-security vulnerability? Hint: It ain't Windows. Cisco Systems' Internetwork Operating System now sits at the center of the information security vortex. Because IOS controls the routers that underpin most business networks as well as the Internet, anyone exploiting its flaws stands to wreak havoc on those networks and maybe even reach into the computer systems and databases connected to them. IOS is a highly sophisticated piece of software, but--as with Microsoft's Windows--that's a double-edged proposition. Software complexity can be a hacker's best friend. http://www.informationweek.com/story/showArticle.jhtml?articleID=173402976&tid=5979%2C5989 EOC =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo GPG Key ID 0x97B43D89 http://mo.fscker.com :: Obscurity through Insecurity "How can we account for our present situation unless we believe that men high in this government are concerting to deliver us to disaster?" Joseph McCarthy "America's Retreat from Victory"
On Mon, Nov 07, 2005 at 06:43:35AM -0500, J. Oquendo wrote:
the center of the information security vortex. Because IOS controls the routers that underpin most business networks as well as the Internet,
I think in general this is an argument against converged networks, the added complexity and outages may not be worth the gains.. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
At 08:52 AM 11/7/2005, you wrote:
On Mon, Nov 07, 2005 at 06:43:35AM -0500, J. Oquendo wrote:
the center of the information security vortex. Because IOS controls the routers that underpin most business networks as well as the Internet,
I think in general this is an argument against converged networks, the added complexity and outages may not be worth the gains..
It is an argument for proper patching policy and procedures. There is no zero day exploit for this exploit and to my knowledge, there hasn't been one yet which came out at the same time as the advisory for ANY major vendor although the window is shrinking. All worms and other exploits which have achieved press coverage and caused major network disruption would have been avoided by proper patching. All of our network is now patched for the latest Cisco advisory. We were already running fixed code on a few routers when the advisory came out so we knew the code was stable and moved to it on all other boxes. I understand that not everyone can act as quickly as we do, but to delay patching indefinitely until the problem occurs - for "stability" reasons is not the solution either. Better code is part of the solution and teaching and enforcing proper programming techniques to create secure code in the first place are just part of the solution. Getting people to install (so far) secure code is another bigger problem which can be solved today. I think all the major vendors are aware of the extent of the problem and are making their systems more secure by auditing their existing code more thoroughly as well as teaching their programmers to code securely in the first place. -Robert Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
Robert,
All of our network is now patched for the latest Cisco advisory. We were already running fixed code on a few routers when the advisory came out so we knew the code was stable and moved to it on all other boxes.
I'm not exactly "in the know" on this one, but the heap-overflow advisory that we've seen indicates that the IOS updates Cisco put out are not patches for this problem: "Cisco has devised counter-measures by implementing extra checks to enforce the proper integrity of system timers. This extra validation should reduce the possibility of heap-based overflow attack vectors achieving remote code execution." from http://www.cisco.com/warp/public/707/cisco-sa-20051102-timers.shtml We've asked Cisco for a better explanation - namely, are their recommended updates "patches" to the problem (i.e. repairs) or simply mitigating updates that make is harder to exploit. The wording of their advisory seems to indicate the latter. This latter case is what worries me since it implies that there is a fundamental problem in IOS, the problem still exists even after patching, and that Cisco can't readily repair it. Unfortunately, so far we've gotten the run-around and haven't been able to get a better answer, again leading me to believe the worst. Eric :)
On Nov 7, 2005, at 10:21 AM, Eric Gauthier wrote:
This latter case is what worries me since it implies that there is a fundamental problem in IOS, the problem still exists even after patching, and that Cisco can't readily repair it.
I would postulate that this is a fundamental problem in how machines represent and execute code. --- James Baldwin "Tolerance is for the insincere"
Looks like vendor J is going to benefit from the issues laid out for Vendor C. http://www.networkworld.com/news/2005/110405-juniper-cisco-hacker.html
At 08:52 AM 11/7/2005, you wrote:
the center of the information security vortex. Because IOS controls
On Mon, Nov 07, 2005 at 06:43:35AM -0500, J. Oquendo wrote: the
routers that underpin most business networks as well as the Internet,
I think in general this is an argument against converged networks, the added complexity and outages may not be worth the gains..
It is an argument for proper patching policy and procedures. There is no zero day exploit for this exploit and to my knowledge, there hasn't been one yet which came out at the same time as the advisory for ANY major vendor although the window is shrinking. All worms and other exploits which have achieved press coverage and caused major network disruption would have been avoided by proper patching. All of our network is now patched for the latest Cisco advisory. We were already running fixed code on a few routers when the advisory came out so we knew the code was stable and moved to it on all other boxes. I understand that not everyone can act as quickly as we do, but to delay patching indefinitely until the problem occurs - for "stability" reasons is not the solution either. Better code is part of the solution and teaching and enforcing proper programming techniques to create secure code in the first place are just part of the solution. Getting people to install (so far) secure code is another bigger problem which can be solved today. I think all the major vendors are aware of the extent of the problem and are making their systems more secure by auditing their existing code more thoroughly as well as teaching their programmers to code securely in the first place.
-Robert
Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin
On Nov 7, 2005, at 11:26 AM, Eric Germann wrote:
Looks like vendor J is going to benefit from the issues laid out for Vendor C.
http://www.networkworld.com/news/2005/110405-juniper-cisco-hacker.html
Cisco, Juniper, or vendor "X". We all benefit by having "genetic diversity" in our routing/switching systems. I have been bit hard, as many of us on this thread have been bit, by bugs in vendor software/hardware. Support your IETF! Don't use proprietary protocols and insist on interoperability. If you have the wherewithal install at least two different vendors for your critical services. Then make them play nice together! There, now I feel much better... glad to get that off my chest. And yes, I actually have put my money where my mouth is and built a stable and efficient dual core with Cisco and Juniper running MPLS together. To be fair I was a bit wimpy about installing some of the latest greatest tricks though <grin>. Regards, Blaine
On Mon, 7 Nov 2005, Blaine Christian wrote:
Looks like vendor J is going to benefit from the issues laid out for Vendor C.
http://www.networkworld.com/news/2005/110405-juniper-cisco-hacker.html Cisco, Juniper, or vendor "X". We all benefit by having "genetic
On Nov 7, 2005, at 11:26 AM, Eric Germann wrote: diversity" in our routing/switching systems. I have been bit hard, as many of us on this thread have been bit, by bugs in vendor software/hardware. Support your IETF! Don't use proprietary protocols and insist on interoperability. If you have the wherewithal install at least two different vendors for your critical services. Then make them play nice together!
How do the operators/engineers explain to 'management', or whomever asks, the 'training issues' that always crop up when more than one vendor are proposed? Has anyone had good luck with this arguement? (my answer is sort of along the lines of: "Its just a router, no matter the vendor and they all have command-line help" but that's not always recieved well :) ) Just curious as I'm sure there are folks stuck in an all vendor X shop who look over the electronic fence and see vendor Y with 'so much better' or 'so much faster' or 'so much more blinkly lighty'... and try to have their management agree to purchasing new devices :)
http://www.networkworld.com/news/2005/110405-juniper-cisco- hacker.html
Cisco, Juniper, or vendor "X". We all benefit by having "genetic diversity" in our routing/switching systems. I have been bit hard, as many of us on this thread have been bit, by bugs in vendor software/hardware. Support your IETF! Don't use proprietary protocols and insist on interoperability. If you have the wherewithal install at least two different vendors for your critical services. Then make them play nice together!
How do the operators/engineers explain to 'management', or whomever asks, the 'training issues' that always crop up when more than one vendor are proposed? Has anyone had good luck with this arguement? (my answer is sort of along the lines of: "Its just a router, no matter the vendor and they all have command-line help" but that's not always recieved well :) )
Just curious as I'm sure there are folks stuck in an all vendor X shop who look over the electronic fence and see vendor Y with 'so much better' or 'so much faster' or 'so much more blinkly lighty'... and try to have their management agree to purchasing new devices :)
Well, the last time I just whined a lot ? <grin> Seriously, we actually put together a presentation that described a series of major events that have occurred through the use of monoculture networks/systems and stated that for many financial/ security reasons it is best to maintain at least two vendors. We covered the following o Security Implications: How monoculture networks/operating systems are prone to attack. o Financial Impact: How managing multiple vendors can reduce long term expense. o Stability: How monoculture networks/systems are prone to network/ system wide failures. o Viability: How existing technology is capable of interop and how we, the engineering team, were capable of making it happen. o Customer demand: How our customers actually "felt better" about our service as a result of it's genetic diversity. Regards, Blaine
How do the operators/engineers explain to 'management', or whomever asks, the 'training issues' that always crop up when more than one vendor are proposed? Has anyone had good luck with this arguement? (my answer is sort of along the lines of: "Its just a router, no matter the vendor and they all have command-line help" but that's not always recieved well :) )
Just curious as I'm sure there are folks stuck in an all vendor X shop who look over the electronic fence and see vendor Y with 'so much better' or 'so much faster' or 'so much more blinkly lighty'... and try to have their management agree to purchasing new devices :)
We have that exact problem, if one considers it to be a problem. We have only vendor X, and we've often wanted to try out others. However, the management arguments and opinions are often difficult to sway. For one, we have very few problems that are ever seen by customers or management. So, convincing them that diversity could buy us better reliability is near impossible. Then the additional cost of training and spare equipment. We also have the customer opinions/perception to deal with (accompanied by marketing), where they think having a "Cisco Powered Network" would automatically mean it's the best. Despite not having service impacting problems, we do have a number of "bugs", however would those issues get better or worse when having to deal with multiple vendors, various platforms per vendor, and inter-operability?
Well, the last time I just whined a lot ? <grin>
Seriously, we actually put together a presentation that described a series of major events that have occurred through the use of monoculture networks/systems and stated that for many financial/ security reasons it is best to maintain at least two vendors.
We covered the following
o Security Implications: How monoculture networks/operating systems are prone to attack. o Financial Impact: How managing multiple vendors can reduce long term expense. o Stability: How monoculture networks/systems are prone to network/ system wide failures. o Viability: How existing technology is capable of interop and how we, the engineering team, were capable of making it happen. o Customer demand: How our customers actually "felt better" about our service as a result of it's genetic diversity.
We covered only a couple of these areas, and maybe addressing some of the others might make a better case.
Regards,
Blaine
-- ------------------------------------------------------ Tom Sands Chief Network Engineer Rackspace Managed Hosting (210)447-4065 ------------------------------------------------------
On Nov 7, 2005, at 11:15 AM, Tom Sands wrote:
We have that exact problem, if one considers it to be a problem. We have only vendor X, and we've often wanted to try out others. However, the management arguments and opinions are often difficult to sway.
Something that often makes management understand the benefit is explaining how much more attentive your current vendor will become once you start implementing a competing vendor. I have had vendor that would barely give me the time of day suddenly give me 2-3 full time onsite SEs, a large pile of onsite spares, an additional 5-7% blanket discount, free training credits, T-shirts and and around $750k of "free" hardware to try and stop me moving to their competitor (as tempting as it may be, don't just cry wolf to get free stuff though!)
For one, we have very few problems that are ever seen by customers or management.
Well, in that case you should make sure that you are keeping management informed as to the deficiencies of the current vendor - as long as you know that your replacement vendor will not have similar issues :-)
So, convincing them that diversity could buy us better reliability is near impossible. Then the additional cost of training and spare equipment.
Talk to the new vendor - most vendors will be more than happy to give you free training credits to get you as a new customer, same goes for spares. Play up the free training to management - explain the importance of ongoing education, how eager the network engineers are to learn more, etc. This way they can give everyone training with no out of pocket expense!
We also have the customer opinions/perception to deal with (accompanied by marketing), where they think having a "Cisco Powered Network" would automatically mean it's the best.
I think that you might be pleasantly surprised by the increasing clue factor of your customers - they will understand that best-of-breed means that you get to choose the best device for the purpose. Your customers are involved in technology - while some of them like Linux and some like Windows, they all understand that different technologies have different strengths and weaknesses (ok, so maybe a true Linux zealot won't agreee that Windows is useful for anything - maybe emacs vs vi... hmmm, maybe not that either...)
Despite not having service impacting problems, we do have a number of "bugs", however would those issues get better or worse when having to deal with multiple vendors, various platforms per vendor, and inter-operability?
With multiple vendors you get to choose the device that best fits the requirements - if devices from multiple vendors fit equally well, you get to choose the one with the fewest bugs. You suddenly also have a lot more pull with your vendor to get your current "bugs" addressed. I personally haven't found interoperability to be a major issue (providing you are not relying on any proprietary protocols) - the ability to choose the best device for the task more than pays for any interoperability concerns - you also get to call up the vendors and say "Fix this or I'm replacing it with a $other_vendor box". There are also some things that certain vendors just don't do well - being limited to just choosing devices from a single vendor limits what you can do. Warren
Well, the last time I just whined a lot ? <grin> Seriously, we actually put together a presentation that described a series of major events that have occurred through the use of monoculture networks/systems and stated that for many financial/ security reasons it is best to maintain at least two vendors. We covered the following o Security Implications: How monoculture networks/operating systems are prone to attack. o Financial Impact: How managing multiple vendors can reduce long term expense. o Stability: How monoculture networks/systems are prone to network/ system wide failures. o Viability: How existing technology is capable of interop and how we, the engineering team, were capable of making it happen. o Customer demand: How our customers actually "felt better" about our service as a result of it's genetic diversity.
We covered only a couple of these areas, and maybe addressing some of the others might make a better case.
Regards, Blaine
-- ------------------------------------------------------ Tom Sands Chief Network Engineer Rackspace Managed Hosting (210)447-4065 ------------------------------------------------------
-- "He who laughs last, thinks slowest." -- Anonymous
How do the operators/engineers explain to 'management', or whomever asks, the 'training issues' that always crop up when more than one vendor are proposed? Has anyone had good luck with this arguement? (my answer is sort of along the lines of: "Its just a router, no matter the vendor and they all have command-line help" but that's not always recieved well :) )
Just curious as I'm sure there are folks stuck in an all vendor X shop who look over the electronic fence and see vendor Y with 'so much better' or 'so much faster' or 'so much more blinkly lighty'... and try to have
It's a variation of the classic insource/outsource question. Do you do the training in-house because it is mission critical to the business or do you outsource it to a vendor focused training organization. If you outsource training then you hire CCIEs to run a Cisco network and JNCIEs to run a Juniper network. But it is not the only way. If you choose to run a dual-vendor network such as Juniper and Cisco then you could still outsource training but since the cost of training is based on a model of one *CIE per person, it probably is not cost-effective for each person to get both the CCIE and JNCIE. Someone who considers training to be an inhouse responsibility would not pay for employees to get their *CIE but would instead pay for an internal training program. I suspect that this inhouse training model will work best in small and midsize companies but that larger networks simply can't afford the complexity because they are less able to hire and maintain clueful managers. their
management agree to purchasing new devices :)
Management has probably made the vendor decision based on high-level business reasons that involve strategy, stock price, and marketing in addition to the low-level reasons of fitness for purpose and cost. I don't expect that any large company will change unless some crisis hits them that forces them to review their decisions. --Michael Dillon
participants (11)
-
Blaine Christian
-
Christopher L. Morrow
-
Eric Gauthier
-
Eric Germann
-
J. Oquendo
-
James Baldwin
-
Jared Mauch
-
Michael.Dillonļ¼ btradianz.com
-
Robert Boyle
-
Tom Sands
-
Warren Kumari