Re: Telco's write best practices for packet switching networks
Subject: Telco's write best practices for packet switching networks
Sean, On Mar 6, 7:56am, Sean Donelan wrote: * *After the SNMP excitement I asked if anyone had suggestions on how *to architect or design a backbone network to be less suspectible to *problems. It turns out the telephone industry has written a set of *best practices for the Internet. Uh huh... The report is 122 pages long and contains a review of how they took existing Best Practices, collected mostly from the previous NRIC, and refined them (based on criteria outlined on p.22). Then performed a SURVEY of what people thought about the 256 BP recommendations they had, covering Service Providers of multiple services and equipment suppliers, all somehow related to telecommunications. The BPs themselves range from: 5-501 "Network Operators should report problems discovered from their testing to the Equipment Supplier whose equipment was found to be the cause of problem." ...to... 5-758 "If 911 call completion is affected, test calls should be made by the Server Provider to the PSAP(s) to assess the impact. Once service is restored, the Service Provider should make multiple 911 test calls to ensure they complete properly." (See Appendix A) I don't think they were focusing on BPs for the Internet per se so much as believing that many BPs that work for PSTN should also work for the Internet. Whether or not they are correct is a (if not THE) challenging matter for debate. *Focus Group 2.A.2: Best Practices on Packet Switching. Karl Rauscher, *Lucent Technologies *http://www.nric.org/pubs/index.html * * Mr. Rauscher gave an example of the kind of information to be found * there. The best practice used in the example states that critical * packet network elements such as control elements, access in signaling * gateways and DNS servers, should have firewall protection such as * screening and filtering. One hundred percent of the respondents * indicated they were implementing this best practice. * *Cool, who has an OC-192 firewall on their control elements? What is *a control element, is that the same as a router or is that a signaling *gateway? Damned flat internet... can't tell if you're kidding or not. I'm guessing that you are, but just in case... Control elements are the router-like bits or computer-like bits that actually "control" management of the switch (i.e. you connect to that part in order to reconfigure, upgrade, etc. the element). Most ISPs have a comparable set-up wrt modems/terminal servers for managing their network elements - same dealy, but ISPs can choose between inband & OOB whereas the telcos can't. (Or couldn't, til recently, when Net/Bell convergence started urging the market toward big damn fiber switches with in-band mgmt tools.) So - in the world of telco, the control elements are JUST OOB. Since you literally can't reach them inband, the OOB element mgmt can be done through modems or a separate network which is firewalled off from the rest of the Internet. That's what they're talking about in your excerpt. What I find interesting is that I've heard a lot of cage rattling to take the Internet in this direction, i.e. stop managing it in-band where all the kiddies and the terrorists can get at it and start managing it OOB. Hide it, shut it away, don't route it, etc. nevermind what a pain it is to manage TWO networks... nevermind how much flexibility you lose. (Sorry, my bias is showing.) And I'm hearing this from BOTH NetHeads AND BellHeads. Kelly J. -- Kelly J. Cooper - Security Engineer, CISSP GENUITY - Main # - 800-632-7638 3 Van de Graaff Drive - Fax - 781-262-2744 Burlington, MA 01803 - http://www.genuity.net
Most ISPs have a comparable set-up wrt modems/terminal servers for managing their network elements - same dealy, but ISPs can choose between inband & OOB whereas the telcos can't. (Or couldn't, til recently, when Net/Bell convergence started urging the market toward big damn fiber switches with in-band mgmt tools.)
The inband/OOB debate is always squirrely. Things like BGP/OSPF are in-band, and ISPs can't really choose an out of band way to exchange routing information. Its true that console access has a choice of accessing the management port through different paths. The router will continue to route, even if the operator can't access the console port. The telephone world thinks of the debate in terms of 260Hz and tone signalling versus SS7 control channels. If you disrupt the SS7 control channel, the telephone switch won't complete new calls even if the trunk groups still work. The management or craft ports are a different matter. Physical attacks make it more interesting. Because the telephone network uses seperate signalling channels, you can disrupt a lot of calls by destroying a relatively few control points/links. Since the Internet uses in-band control, as long as there is some physical connectivity, you can use it for both control and user traffic. Everytime Illuminet has a glitch, a dozen states have problems completing calls between ILECs and CLECs. This affects a lot of dialup access to the Internet.
So - in the world of telco, the control elements are JUST OOB. Since you literally can't reach them inband, the OOB element mgmt can be done through modems or a separate network which is firewalled off from the rest of the Internet. That's what they're talking about in your excerpt.
Where it gets interesting is when the assumptions about what is "outside" or "inside" is violated. I think the Internet is actually much more secure now because its so open, we don't make assumptions about who we trust. The telephone network is built on a house of trust, and if you can get on the "inside" the world is yours.
What I find interesting is that I've heard a lot of cage rattling to take the Internet in this direction, i.e. stop managing it in-band where all the kiddies and the terrorists can get at it and start managing it OOB. Hide it, shut it away, don't route it, etc. nevermind what a pain it is to manage TWO networks... nevermind how much flexibility you lose. (Sorry, my bias is showing.)
Having a seperate network didn't stop Mitnick :-) I think some of it is "the grass is always greener on the other side of the fence." Reserving bandwidth for specific purposes tends to make your network more brittle, and less responsive to unexpected events. I try to explain it's like car pool lanes on the highway making traffic jams worse. I happen to believe you need both in-band and out-of-band control access, and you need the same level of security on both. But I tend to order my goals with availability first. Having your network down may be "secure" but it isn't very useful.
Kelly J. Cooper - Security Engineer, CISSP
So why did you get the CISSP? I just received my CISSP certificate, but I needed to get it for resume padding purposes.
There are four different issues with IB vs OOB signalling & control, actually -- 1) isolation of control traffic from payload traffic to eliminate possible security breaches. 2) isolation of control traffic from payload traffic to prevent starvation of control traffic by (possibly misbehaving) payload traffic. 3) having an alternative, isolated, routing infrastructure for control traffic which can function even when primary routing is hosed. 4) having a physically separate network for control traffic which can concievably survive when primary network is broken. On #1, Internet routing protocols are notoriously weak. Using globally routable frames to carry neighbour-to-neighbour routing information is a recipe for disaster (i think everyone on this list can think of few not-yet-plugged holes arising from this approach). In most IGPs and eBGP there's simply no need to use routable IP packets, period. The only exception is iBGP hack (and consequent route-reflector kludgery), and this can only be cured by a better-designed IGP which can carry all exterior routing information. I hope someone's doing something to make it happen. Using non-IP packets does not always bring isolation; OSI stack is even more vulnerable. So a cheap fix would be to design routing hardware in a way forcing some reserved IP addresses to be non-routable. (127/8 seems to be a good candidate :). Even better is to start using non-IP frame types altogether. With all its weakness, outside attacks on ARP are unheard of. Another (weaker) option is to use cryptography. Besides inevitable bugs (like numerous problems in SSH), crypto is slow and hard to do right. #2 is also a known pitfall (hello, OFRV :) Although, in theory, just jacking up priority on packets carrying routing protocols & network monitoring traffic could take care of this problem, the reality is quite hairier. Most hardware doesn't prioritize generation of ICMPs (so a lot of looped or misdirected packets can swamp the routing processor which is incidentally used to separate control traffic from transit traffic). Usually, there are cross-interface dependencies resulting from shared buffer memory, the supposedly "non-blocking" switching fabrics being anything but, confusion between queueing and drop priorities, plain broken design of packet classifiers (hard to do it right at high speeds :), and simply network admins being lazy to configure interface ToS processing appropriately (and/or failing to filter out packets with ToS similar to the routing protocols' at all ingress points!) Given the practical problems of getting ToS for control traffic being set up properly, the option of guaranteeing bandwidth and processing capacity to control traffic by using separate, non-configurable forwarding/queueing for non-IP traffic seems to be quite reasonable. Problems arising from control traffic starvation are numerous and can easily lead to prolonged network-wide failures. Therefore, making control traffic "OOB" is definitely worth-while. #3 is somewhat muddled by the fact that having valid routing information while having no functioning payload pathway is somewhat irrelevant (in theory, haiving such information may let network to use unidirectionally-broken paths, or allow faster recovery from network fractures by eliminating the need to send updates about _working_ parts of the split-off networks). In practice, the only real gain from a redundant control network is the ability to better diagnose problems, particularly routing problems in the primary network (i.e. the "OOB" network is used to carry diagnostic traffic only). A dedicated OOB network for console access to various pieces of equipment is a lot more useful. The horrible weakness of SNMP makes separate control network somewhat more resistant to attacks; however, it also requires zealous filtering of SNMP and other control protocols (such as telnet :) packets coming to the router's control unit(s) from the primary network. If such filtering is broken or glossed over, there are no security gains. #4 is hardly useful in any situation (with the exception of diagnostic network). In fact, telco-issue "OOB" is usually muxed over the same wires. So, i would say i'm pro-OOB where it concerns clean confinement of control traffic into a non-routable, unconditionally-prioritized frames, and contra-OOB when it comes to making separate networks for control traffic. Your definition of "separate network" may vary :) --vadim
In a message written on Fri, Mar 08, 2002 at 05:52:46PM -0800, Vadim Antonov wrote:
1) isolation of control traffic from payload traffic to eliminate possible security breaches. [snip] On #1, Internet routing protocols are notoriously weak. Using globally routable frames to carry neighbour-to-neighbour routing information is a recipe for disaster (i think everyone on this list can think of few not-yet-plugged holes arising from this approach).
This is an area of interest of mine when looking at IPv6. IPv6 has the notion of link local IP addresses, that can't (for some definition of can't) be accessed unless you are on that link. This could go a long way to fixing the problems you mention, but it introduces some additional configuration issues. In particular, the current practice of using the same link local addresses on every link means you would need to configure both the address and the port. In any event, I wonder if there is an opportunity here for additional security. Although any changes are clearly years off. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Fri, 8 Mar 2002, Vadim Antonov wrote:
So, i would say i'm pro-OOB where it concerns clean confinement of control traffic into a non-routable, unconditionally-prioritized frames, and contra-OOB when it comes to making separate networks for control traffic. Your definition of "separate network" may vary :)
Of course, like many things security looks easy until you have to do it yourself. So I don't mean to suggest there are any really any easy answers. But I've been wondering about simple structural changes which would improve the intrinsic security of the net. For example, remember when BARRNET had the problem with people stealing passwords on their backbone. One simple change was removing general purpose computers which could be used as sniffers from their core router LANs. My simple question is why do exchange point prefixes or backbone network prefixes need to be announced to peers or customers? If no one announced IXP prefixes, it would be more difficult (modulo LSSR/SSSR) to send bogus packets at distant routing gateways. The attacker would need to be directly connected, or compromise something. This has been something which has bugged me ever since I connected a router to mae-east. There is no "true" ASN for inter-provider network prefixes, yet the prefixes show up in the BGP tables via multiple providers. Private inter-ISP links aren't any better. They are frequently taken from some provider's internal space, and announced by a combination of providers. This isn't really OOB, but similar to your idea of not using a globally routable network to exchange routing information. Its not as difficult as making a 127/8 kludge. Its a small matter of not announcing prefixes used for BGP to your BGP peers (next-hop-self).
### On Mon, 11 Mar 2002 04:49:46 -0500 (EST), Sean Donelan ### <sean@donelan.com> casually decided to expound upon Vadim Antonov ### <avg@exigengroup.com> the following thoughts about "Re: Telco's write ### best practices for packet switching networks": SD> My simple question is why do exchange point prefixes or backbone SD> network prefixes need to be announced to peers or customers? SD> SD> This has been something which has bugged me ever since I connected SD> a router to mae-east. I think the main justification one could use (and I don't necessarily agree with this) is to aide in troubleshooting. Announcing the space may make it easier for multiple parties to troubleshoot through their backbone. On the flipside of this argument of course is why not filter that space to only your NOCs and engineers? Now the counter-argument to that might be that the space starts to add up in terms of bloating ACLs and such. One could go back and forth on this all day I suppose including arguments for and against troubleshooting from production devices vs troubleshooting from a backend system. Another reason mae-east was announced at least historically may have been to help support activities such as the Routing Arbiter Project. I know from experience that due to the nature of how they were positioned within exchange points, the routeservers needed to be reachable by Merit personnel. However, the proper solution there should have been for only Merit's primary transit provider to carry those routes and possibly filter as much as possible the space except to Merit. There were workable solutions even back then. I think we all just chose the path of least resistance because it was easier and the risk factours were perceived to be low. We all know that was a false assumption. I remember the first smurf attack against mae-east and how it knocked out quite a few peers. -- /*===================[ Jake Khuon <khuon@NEEBU.Net> ]======================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | --------------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| N E T W O R K S | +=========================================================================*/
On Mon, 11 Mar 2002, Jake Khuon wrote:
There were workable solutions even back then. I think we all just chose the path of least resistance because it was easier and the risk factours were perceived to be low. We all know that was a false assumption. I remember the first smurf attack against mae-east and how it knocked out quite a few peers.
Yep, I understand. History is never as neat as we would like. It may have been suitable in the past. Is it time to change? I'm not suggesting RFC1918 space for internal backbone routers and IXPs, but not announcing your internal-only nets would (slightly) increase the difficulty of attacking the core. It doesn't even require ISPs to agree on a best practice. A provider can choose to implement it themselves to protect their own core network. Perhaps the attacks on core routers aren't bad enough to justify such a drastic step yet. I get conflicting signals from engineers still working. Some say they see attacks all the time, others say they've never seen one on their core routers.
On Mon, 11 Mar 2002, Jake Khuon wrote:
There were workable solutions even back then. I think we all just chose the path of least resistance because it was easier and the risk factours were perceived to be low. We all know that was a false assumption. I remember the first smurf attack against mae-east and how it knocked out quite a few peers.
Yep, I understand. History is never as neat as we would like. It may have been suitable in the past. Is it time to change?
I'm not suggesting RFC1918 space for internal backbone routers and IXPs, but not announcing your internal-only nets would (slightly) increase the difficulty of attacking the core. It doesn't even require ISPs to agree on a best practice. A provider can choose to implement it themselves to protect their own core network.
Perhaps the attacks on core routers aren't bad enough to justify such a drastic step yet. I get conflicting signals from engineers still working. Some say they see attacks all the time, others say they've never seen one on their core routers.
On the downside -- this is yet another instance of conflict between research and operations. Being able to address the (core) routers directly is an important capability researchers use for tasks like topology discovery and path/routing characterization. Of course, if researchers can talk to the routers, so can the attackers ..... -- Ratul
On Tuesday, March 12, 2002, at 03:23 , Ratul Mahajan wrote:
Perhaps the attacks on core routers aren't bad enough to justify such a drastic step yet. I get conflicting signals from engineers still working. Some say they see attacks all the time, others say they've never seen one on their core routers.
On the downside -- this is yet another instance of conflict between research and operations. Being able to address the (core) routers directly is an important capability researchers use for tasks like topology discovery and path/routing characterization. Of course, if researchers can talk to the routers, so can the attackers .....
Just because routing protocols use addressing or protocols which are not globally routable doesn't mean that core routers can't be addressed directly. IS-IS neighbours use NSAP addressing and OSI transport to exchange routing information, for example, but traceroute still works. Joe
### On Tue, 12 Mar 2002 12:23:51 -0800 (PST), Ratul Mahajan ### <ratul@cs.washington.edu> casually decided to expound upon Sean Donelan ### <sean@donelan.com> the following thoughts about "Re: Telco's write best ### practices for packet switching networks ": RM> On the downside -- this is yet another instance of conflict between RM> research and operations. Being able to address the (core) routers This may be a repeat discussion but I also wonder if there are some other social level conflicts derived from how one structures their management network. For instance, many providers have a seperate group which handles the corporate IT which is different from the group which handles the production provider network. One could take the stance that the production network should only be reachable from the corporate network and that the management network become an extension of the corporate network. I imagine that many network engineers on the side of the production network might take issue with that (I probably would). For better or worse, many of us have gotten used to managing our backbones under a single umbrella including control over how we design and run our management network. I'd be interested in hearing about some of the practises of bigger providers (assuming I'm not asking anyone to violate security) on how they let their emloyees access their infrastrcture. Do you seperate and outsource your management infrastructure to your corporate IT support? Do you seperate but control it within your production network engineering groups? If so, do you have a special group within network engineering concentrating specifically on management or do you have the same people designing the network also do the management design? -- /*===================[ Jake Khuon <khuon@NEEBU.Net> ]======================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | --------------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| N E T W O R K S | +=========================================================================*/
On Wed, 13 Mar 2002, Jake Khuon wrote:
emloyees access their infrastrcture. Do you seperate and outsource your management infrastructure to your corporate IT support? Do you seperate but control it within your production network engineering groups? If so, do you have a special group within network engineering concentrating specifically on management or do you have the same people designing the network also do the management design?
Although many of the principles are the same, there are differences between running a corporate network and a public network. You can have the same people doing both. In small ISPs its likely the same people will be doing both. A larger company will have seperate groups because they serve different masters and have different measures of success. A company may not want to pay for the same levels of reliablity and survivability for their corporate network as their public IP network.
On Wed, 13 Mar 2002, Sean Donelan wrote:
Although many of the principles are the same, there are differences between running a corporate network and a public network. You can have the same people doing both. In small ISPs its likely the same people will be doing both. A larger company will have seperate groups because they serve different masters and have different measures of success. A company may not want to pay for the same levels of reliablity and survivability for their corporate network as their public IP network.
The goals of the corporate network and the public IP network are often different, at best. The corporate network is inevitably focused around the needs of the business, including such irritations as file sharing, printing, calender services, video conferencing, and other notoriously secure (heh!) services. The public IP network is focused by and large on providing a limited number of services, and flinging packets around as fast as possible. The clue behind the public IP network is almost always focused on the network (often to the point of considering any systems involved to be second class citizens "Why should I care if there's a system down? It only matters if the network's down"[1]) The clue on the corporate network often doesn't care at all about the network (beyond "is it running") - but really cares that their services are deployed and accessible. I think that Sean's right about the goals being different - but it's more than just "reliability and survivability". The network and enterprise markets are notably different, with different goals and requirements. Most vendors seem to have a very clear grasp on that - and I suspect that it'll be another 5-10 years before we see any form of true convergance (if not longer). [1] This ignoring the fact that a down'd network monitoring system may cause all sorts of interesting side effects in viewing the network... ========================================================================== "A cat spends her life conflicted between a deep, passionate and profound desire for fish and an equally deep, passionate and profound desire to avoid getting wet. This is the defining metaphor of my life right now."
On Mon, 11 Mar 2002, Sean Donelan wrote:
So, i would say i'm pro-OOB where it concerns clean confinement of control traffic into a non-routable, unconditionally-prioritized frames, and contra-OOB when it comes to making separate networks for control traffic. Your definition of "separate network" may vary :)
Of course, like many things security looks easy until you have to do it yourself. So I don't mean to suggest there are any really any easy answers.
It would help if equipment vendors made it easier to enable management on a per-interface basis, so you could just disable it on "in band" interfaces.
My simple question is why do exchange point prefixes or backbone network prefixes need to be announced to peers or customers? If no one announced IXP prefixes, it would be more difficult (modulo LSSR/SSSR) to send bogus packets at distant routing gateways. The attacker would need to be directly connected, or compromise something.
Define "need". It is extremely helpful to receive ICMP messages from within the IP address range of an exchange. If the routes aren't announced, packets from these addresses would be dropped by routers performing unicast RPF checks. Too bad for traceroute, but potientally much more serious for path MTU discovery. Nearly all implementations are broken to the degree they can't recover from the situation where they don't receive "datagram too big" messages, and exchange points are typically the places where networks with different MTUs come together. I think I would prefer to just announce the prefixes, but route them to the null interface somewhere. This doesn't get in the way of unicast RPF elsewhere, but protects the interconnect addresses equally well, and it allows a more fine grained approach. Anyway, I feel the effort needed to educate networks about this problem would be better spent in trying to get them to filter out outbound packets with bogus source addresses. I still see lots of 192.168/16 source addresses in packets received from peers.
participants (9)
-
Gwendolynn ferch Elydyr -
Iljitsch van Beijnum -
Jake Khuon -
Joe Abley -
Kelly J. Cooper -
Leo Bicknell -
Ratul Mahajan -
Sean Donelan -
Vadim Antonov