On Wed, 7 Jul 2010, Patrick Giagnocavo wrote:

andrew.wallace wrote:

...

Article: http://online.wsj.com/article/SB10001424052748704545004575352983850463108.ht...

Why does it cost $100 million to install and configure OpenBSD on a bunch of old systems?

A) it's being done for the government B) it's being done by a defense contractor C) regardless of what they install, somebody's got to manage it and be managed by multiple layers of managers D) other Pick three or more answers. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

You forgot -It is carrier grade, ISO certified and other certification program not worth the paper it is printed on. ----- Original Message ----- From: "Jon Lewis" <jlewis@lewis.org> To: "Patrick Giagnocavo" <patrick@zill.net> Cc: nanog@nanog.org Sent: Thursday, 8 July, 2010 2:14:53 PM Subject: Re: U.S. Plans Cyber Shield for Utilities, Companies On Wed, 7 Jul 2010, Patrick Giagnocavo wrote:

andrew.wallace wrote:

...

Article: http://online.wsj.com/article/SB10001424052748704545004575352983850463108.ht...

Why does it cost $100 million to install and configure OpenBSD on a bunch of old systems?

A) it's being done for the government B) it's being done by a defense contractor C) regardless of what they install, somebody's got to manage it and be managed by multiple layers of managers D) other

On Wed, Jul 07, 2010, Patrick Giagnocavo wrote:

Why does it cost $100 million to install and configure OpenBSD on a bunch of old systems?

I think you've misunderstood the question if you think "openbsd on old systems" is the answer. :) Adrian

Because no-one who could do it for less can afford to respond to government contracts, and make sure they comply with all the applicable laws and regulations, and keep the sort of records, and be prepared for the audits of said records, required. As soon as you do business with the govt, the overhead goes through the roof.

-----Original Message----- From: Patrick Giagnocavo [mailto:patrick@zill.net] Sent: Wednesday, July 07, 2010 7:02 PM To: nanog@nanog.org Subject: Re: U.S. Plans Cyber Shield for Utilities, Companies

andrew.wallace wrote:

...

Article:

http://online.wsj.com/article/SB100014240527487045450045753529838504631 08.html

...

Why does it cost $100 million to install and configure OpenBSD on a bunch of old systems?

--Patrick

On Wed, 7 Jul 2010, Michael Painter wrote:

Have we all gone mad? I find it hard to understand that a nuclear power plant, air-traffic control network, or electrical grid would be 'linked' to the Internet in the interest of 'efficiency'. Air gap them all and let them apply for "Inefficiency Relief" from the $100 million relief fund.

Absolutely! For example, those thousands of flight plans filed every day by airlines across the globe, not to mention private flights, should be done manually the old fashioned way, with a paper form and stopping by your local FAA office where a human keys them into the ATC computer. Oh wait, we closed all of those offices when we moved all of those functions to the Internet. I guess we'll just have to re-open them. And flight tracking data that airlines and freight companies use to track their aircraft, yea, let's cut those off too. If they want to know where their plane is, just have them call the FAA. Surely the government can staff some huge call centers to handle the load of each airline calling about each flight every few minutes. Heck, removing all of these functions from the Internet will create jobs, too, right? And no one would mind paying for all of this out of their airline tickets, it should only increase fares by a third or so. -- Brandon Ross AIM: BrandonNRoss ICQ: 2269442 Skype: brandonross Yahoo: BrandonNRoss

On Thu, 8 Jul 2010, Joe Greco wrote:

There's a happy medium in there somewhere; it's not clear that having (to use the examples given) air traffic control computers directly on the Internet has sufficient value to outweigh the risks. However, it seems that being able to securely gateway appropriate information between the two networks should be manageable, certainly a lot more manageable than the NxM complexity involved if you try to do it by securing each and every Internet-connected ATC PC individually.

What makes you think that isn't exactly what this "Cyber Shield" project is supposed to do? Heck, what makes you think that's not the way most of these systems already work today? Do people really think the guy in the airport control tower is really surfing Facebook while he's controlling aircraft on the same computer, or that capability is even what is under consideration? -- Brandon Ross AIM: BrandonNRoss ICQ: 2269442 Skype: brandonross Yahoo: BrandonNRoss

On Thu, 8 Jul 2010, J. Oquendo wrote:

Brandon Ross wrote:

...

Do people really think the guy in the airport control tower is really surfing Facebook while he's controlling aircraft on the same computer, or that capability is even what is under consideration?

"Air traffic controller suspended for allowing son to radio instructions to pilots at New York's Kennedy Airport" http://www.dallasnews.com/sharedcontent/dws/dn/latestnews/stories/030410dnna...

Please read critically before replying. My exact quote included the words "on the same computer". The article that started this thread is about protecting critical systems, not preventing people from making stupid mistakes. If you want to talk about ATC procedures or the misbehavior of controllers using unapproved devices, there's a whole separate mailing list for that. -- Brandon Ross AIM: BrandonNRoss ICQ: 2269442 Skype: brandonross Yahoo: BrandonNRoss

Owen DeLong wrote: [snip]

I know this from being a pilot, and, also from having toured the following ATC facilities:

Towers: TRACONs: ARTCCs:

Ditto to absolutely EVERYTHING that Owen said, and I can guarantee this further, having had experience with various east coast and southeastern Towers, TRACONs, and ARTCCs (and having fond memories of it all). Personally, I'm more concerned about "Cisco Security Advisory: Hard-Coded SNMP Community Names in Cisco Industrial Ethernet 3000 Series Switches Vulnerability" than worrying about whatever the latest scary headline from n3td3v (aka andrew wallace) is. -- All men whilst they are awake are in one common world: but each of them, when he is asleep, is in a world of his own. Plutarch

On Thu, Jul 08, 2010 at 09:51:52AM -0400, Brandon Ross wrote:

On Wed, 7 Jul 2010, Michael Painter wrote:

...

Have we all gone mad?

Absolutely! For example, those thousands of flight plans filed every day by airlines across the globe, not to mention private flights, should be done manually the old fashioned way, with a paper form and stopping by your local FAA office where a human keys them into the ATC computer. Oh wait, we closed all of those offices when we moved all of those functions to the Internet. I guess we'll just have to re-open them.

yeah! jobs for americans! actually, the interesting questions raised are along the lines of "what is your contingency plan?" ... the big EMP is coming.

Heck, removing all of these functions from the Internet will create jobs, too, right? And no one would mind paying for all of this out of their airline tickets, it should only increase fares by a third or so.

-- Brandon Ross AIM: BrandonNRoss ICQ: 2269442 Skype: brandonross Yahoo: BrandonNRoss

On Jul 8, 2010, at 10:13 AM, George Bonser wrote:

...

-----Original Message----- From: Brandon Ross Sent: Thursday, July 08, 2010 6:52 AM To: Michael Painter Cc: nanog@nanog.org Subject: Re: U.S. Plans Cyber Shield for Utilities, Companies

On Wed, 7 Jul 2010, Michael Painter wrote:

...

Have we all gone mad? I find it hard to understand that a nuclear power plant, air-traffic control network, or electrical grid would be 'linked' to the Internet in the interest of 'efficiency'. Air gap them all and let them apply for "Inefficiency Relief" from the $100 million relief fund.

Absolutely! For example, those thousands of flight plans filed every day by airlines across the globe, not to mention private flights, should be done manually the old fashioned way, with a paper form and stopping by your local FAA office where a human keys them into the ATC computer. Oh wait, we closed all of those offices when we moved all of those functions to the Internet. I guess we'll just have to re-open them.

I believe the point was in response to:

"control systems that were often designed without Internet connectivity or security in mind. Many of those systems-which run everything from subway systems to air-traffic control networks-have since been linked to the Internet"

If something was designed without network security "in mind" and then connected to the internet as-is, then yeah, that pretty much is not only "madness" but is just asking for trouble. So I am torn between this being another exercise in treating the symptoms while ignoring the underlying cause and at least having SOMEONE watching the front door if the owners aren't paying any attention themselves. But I would think the cost of the program could be scaled back somewhat if certain basic security practices were mandated prior to the system being installed.

I think part of the problem comes from interrelationships between the transitive property of trust (if A trusts B and B trusts C, then A trusts C whether A knows it or not) and the perceived vs. actual nature of linkage. For example, it would seem madness to put an HTTP server directly on the primary Air Traffic Scheduling System at "FAA Central" and have it collect flight plans directly from the internet. However, what happens is that FAA contracts Lockheed out to run several Automated Flight Service Stations and also contracts two other companies (GTE and CONTEL last I looked at who had the contracts) to run a service known as "Direct User Access Terminals" or DUATS. Lockheed runs their own systems and interacts with pilots by telephone and radio. Flight Plans and Pilot Reports collected by Lockheed are put into Lockheed systems which are then linked into the FAA systems. I do not know if any of those links involve internet connectivity or not. LIkely some do. The DUATS systems also link into the FAA computers for uploading flight plans and pilot reports and for getting weather and NOTAM information from the FAA. As such, at least on some level, the FAA systems are linked to systems that are linked to the internet and there definitely isn't an air-gap. I suspect it is a full enough form of proxy that only data can traverse from one to the other. I think the design of the systems is probably relatively sane on that level. However, I doubt anyone on this list really knows for sure how the systems were designed or the exact nature of their linkages and I suspect there are many many other examples of such indirect linkages that have grown organically over time as the internet has moved from scientific novelty to a place to distribute web access and now starts to become the fundamental basis for communication among humans, machines, and others throughout the world. There used to be a clear line between telecom and datacom. It used to be that the internet was clearly datacom. Today, it's almost as if telecom as a separate discipline is going away and instead voice is becoming an application on the datacom network. It used to be that datacom was many disparate specialized networks each serving a particular datacom purpose. Today, the internet has become the generic low-level building block upon which virtually every datacom application, including the new telecom (voice as an application on a data network) is being built. With these changes and their relationships to legacy systems come new security concerns. Some known, many likely not even noticed as things move forward. Owen

On Wed, 07 Jul 2010 19:16:27 -1000, Michael Painter said:

I find it hard to understand that a nuclear power plant, air-traffic control network, or electrical grid would be 'linked' to the Internet in the interest of 'efficiency'. Air gap them all and let them apply for "Inefficiency Relief" from the $100 million relief fund.

OK, so you airgap the whole thing, and apply for "Inefficiency Relief" to help pay for those 2,397 separate dark fiber dedicated links you need to contact your 2,397 remote sensing stations and control points. And of course, since you end up burning a *lot* of dark fiber pairs when every utility starts doing that, the provider gets to go back and put a whole lot more 96-pair or whatever alongside the previous bundle, driving prices back up after our long-term fiber glut. And then you discover that your actual network reliability goes *down*, because getting your provider to troubleshoot your measly 64K channel is a pain and takes a long time to get results - whereas if you went commodity Internet your packets are now mixed in with everybody else's on a important 10GE link. Sure, that 10GE link may be just 2 fibers over in the same bundle - but guess which one will probably be spliced first after the backhoe hits? (Plus of course, if 37 of those 2,397 links were in the bundle, it's going to take 37 splices to get you 100% back up, instead of just one splice....) What's the going rate these days that you have to pay to make sure your fiber gets spliced first rather than that other customer's 10GE? And what's it cost to do it for all 2,397 links? And if your electrical-grid fiber is in the same cable as the other customer's ATC cable, who gets spliced first? If you have a single point of failure in your design, you really want to make sure that the point is heavily fate-shared with enough other customers that the provider will feel *really* motivated to fix your problem. ;)

On Jul 8, 2010, at 10:12 AM, Valdis.Kletnieks@vt.edu wrote:

On Wed, 07 Jul 2010 19:16:27 -1000, Michael Painter said:

...

I find it hard to understand that a nuclear power plant, air- traffic control network, or electrical grid would be 'linked' to the Internet in the interest of 'efficiency'. Air gap them all and let them apply for "Inefficiency Relief" from the $100 million relief fund.

OK, so you airgap the whole thing, and apply for "Inefficiency Relief" to help pay for those 2,397 separate dark fiber dedicated links you need to contact your 2,397 remote sensing stations and control points. And of course, since you end up burning a *lot* of dark fiber pairs when every utility starts doing that, the provider gets to go back and put a whole lot more 96-pair or whatever alongside the previous bundle, driving prices back up after our long- term fiber glut.

I think that there needs to be a balance. There is no Internet access to certain military systems, for example, but that doesn't mean that the base housing them has no Internet access. I would expect the same to be true for, e.g., nuclear power systems. If this has never been thought through by someone, it would not be a bad idea to start now. On the other hand, my friends in military networking tend to be cynical about these kinds of exercises. They may or may not actually increase security, in fact they sometimes degrade it, but they tend to be very good at sending money to politically well connected contractors. Regards Marshall

And then you discover that your actual network reliability goes *down*, because getting your provider to troubleshoot your measly 64K channel is a pain and takes a long time to get results - whereas if you went commodity Internet your packets are now mixed in with everybody else's on a important 10GE link. Sure, that 10GE link may be just 2 fibers over in the same bundle - but guess which one will probably be spliced first after the backhoe hits? (Plus of course, if 37 of those 2,397 links were in the bundle, it's going to take 37 splices to get you 100% back up, instead of just one splice....)

What's the going rate these days that you have to pay to make sure your fiber gets spliced first rather than that other customer's 10GE? And what's it cost to do it for all 2,397 links? And if your electrical-grid fiber is in the same cable as the other customer's ATC cable, who gets spliced first?

If you have a single point of failure in your design, you really want to make sure that the point is heavily fate-shared with enough other customers that the provider will feel *really* motivated to fix your problem. ;)

Valdis.Kletnieks@vt.edu wrote:

What's the going rate these days that you have to pay to make sure your fiber gets spliced first rather than that other customer's 10GE?

I'm not familiar with cable break splicing procedures, but is it even possible to pay extra to have your splice done first? I would think that the logistics of splicing are such that the guy down in the hole doesn't know whose traffic is on each strand in the bundle, and his job is just to splice them as he matches them (using color codes or similar on the sheaths of the individual strands) as fast as he can. Trying to identify a specific strand and then splicing it first would greatly slow down the task of splicing them all. If you have more than 1 strand that needs to get spliced "first" it would likely take longer to identify these "special" customers and get them done first than to just splice with no priority and get the whole bundle done. jc

On Jul 8, 2010, at 9:26 AM, Valdis.Kletnieks@vt.edu wrote:

...

I'm not familiar with cable break splicing procedures, but is it even possible to pay extra to have your splice done first? I would think that the logistics of splicing are such that the guy down in the hole doesn't know whose traffic is on each strand in the bundle

Exactly - which is a case for just having everybody's traffic mingled on a very busy 12-pair rather than several 96-pair with lots of dedicated links, *everybody* ends up back in service a lot faster...

And remember - this industry has more trouble with backhoes and would-be copper thieves than terrorists. Anybody who is defending against terrorists by increasing their vulnerability to backhoes is, well...

Having done a good bit of manual copper and [old school fusion] fiber splicing for a few years as an outside plant monkey in the Army Signal Corp and a short stint thereafter as a contractor, I assure you that prioritization can make a significant different with large cable damage, in particular when single wire/pair splicing is done. Copper multi-pair splicing still allows specific bundles to be prioritized as well, sorta the same as fiber. Given that cuts and other damage usually requires splicing on two ends, some bit of coordination is required but mostly trivial, in particular with large copper cable (e.g., 2400 pair). Of course, in fairness to Valdis's comment, setup time on both ends is often the dominating factor, although bundle 1 to bundle 96 is an 2400 pair copper cable could be several hours or more. Of course, physical plant prioritization is only the dominating factor when last mile damage occurs. It's more useful and commonly employed when intermediate facility failures happen - prioritized regrooming of critical services is sometimes even automated, and often results in, err.. less critical services being booted until full restoration has occurred. -danny

I find it hard to understand that a nuclear power plant, air-traffic control network, or electrical grid would be 'linked' to the Internet in the interest of 'efficiency'.

The Davis-Besse nuclear generating station computers were hit by the SQL Slammer / Saphire worm back in 2003. http://www.theregister.co.uk/2003/08/20/slammer_worm_crashed_ohio_nuke/ The utility claims that there was never a risk, but take that with a grain of salt since this is the same utility that conveniently managed to overlook a hole the size of a book in the reactor head for several years. Cheers, Michael Holstein Cleveland State University

On Jul 8, 2010, at 11:56 AM, J. Oquendo wrote:

@Jared's TSP link... Wonder how this will affect VoIP ITSP's etal, e.g., how many local NS/EP's have swapped over to VoIP. Logically, anyone with a network running a managed VoIP service, trunk, etc., could qualify.

This certainly is a frequent discussion point in some circles. A lot of carriers take your POTS/PRI and turn them into VoIP internally so they get the advantages of multiplexing the data on IP. This isn't universal, but a good question to ask your carrier. If you care about your call going through, you may actually want to pay a bit more and be on that more expensive TDM gear. Of course, whomever you call may also need to be on that same carrier. Take some time and research and test things out so you don't end up having trouble when you need your communications the most. You may also want to find your local HAM operator and buddy up with them, or get your no-code license today :) - Jared