Hello, I'd like everyones 2 cents on the BCP for network management of an ISP PoPs, with a non-security oriented NOC, Most of my routers doesnt have crypto IOS images, couldnt agree with core members to do a major upgrade, just a promise of doign that when other needs to an IOS upgrade come up, So i need to workaround it and secure management traffic somehow, Usually the NOC logs to the PoPs 24x7, so i definitely need to hit a balance between encryption/security and usability, thats why i excluded OTP, My homework concluded: 1) Establishing an ipsec tunnel from each NOC Pc to a VPN concentrator, and of course on every PC, there would be static routes injected to take management traffic through the tunnel, Major advantage is usability and transperancy to the user, One major pitfall here is when ipsec tunnels break, my presence would be needed to troubleshoot that, 2) An OpenBSD bastion host(s), where the NOC would ssh in, get authenticated from TACACS+ or ssh certs, and then just telnet from there all day, One major advantage here is the heavy monitoring/limiting i can do on a *nix box, systrace their login shell to a policy (telnet/ping/traceroute only) 3) Or just an IOS based bastion router that also runs ssh, This has the advantage of IOS limitations in a way, not much maintaining is needed but being limited with 16 vtys is a problem, also vtys may get stuck and all these ssh sessions would kill the memory of the router, I would of course have multiple setups one at the Datacenter, another at some PoP, redundant solutions incase one fails, and For the record, I do run rancid, syslogging and we do AAA, so its just down to whats others experiences/ideas about secure management?
Kim, Its terribly important that your routers' management traffic be encrypted all the way to the device. For this reason, the best practice is to use ssh2. There are some other hacks that can be used, but they are hacks, and are not scalable. Bastion hosts are a good thing and can be a great place to put in checks for multi-factor authentication (another must-have), but they do not replace the need for end-to-end encryption. Turn off telnet and web administration today. While you are at it, look at your SNMP setup. You want your SNMP management to have the same characteristics as your vty management - strong authentication and encryption. Cleartext community strings don't cut it, as an example. Also, you need a secure Out of Band management network. You may want to check out the NSP-Security mail list. - Dan On 1/11/05 4:17 AM, "Kim Onnel" <karim.adel@gmail.com> wrote:
Hello, I'd like everyones 2 cents on the BCP for network management of an ISP PoPs, with a non-security oriented NOC,
Most of my routers doesnt have crypto IOS images, couldnt agree with core members to do a major upgrade, just a promise of doign that when other needs to an IOS upgrade come up, So i need to workaround it and secure management traffic somehow,
Usually the NOC logs to the PoPs 24x7, so i definitely need to hit a balance between encryption/security and usability, thats why i excluded OTP,
My homework concluded:
1) Establishing an ipsec tunnel from each NOC Pc to a VPN concentrator, and of course on every PC, there would be static routes injected to take management traffic through the tunnel,
Major advantage is usability and transperancy to the user, One major pitfall here is when ipsec tunnels break, my presence would be needed to troubleshoot that,
2) An OpenBSD bastion host(s), where the NOC would ssh in, get authenticated from TACACS+ or ssh certs, and then just telnet from there all day,
One major advantage here is the heavy monitoring/limiting i can do on a *nix box, systrace their login shell to a policy (telnet/ping/traceroute only)
3) Or just an IOS based bastion router that also runs ssh,
This has the advantage of IOS limitations in a way, not much maintaining is needed but being limited with 16 vtys is a problem, also vtys may get stuck and all these ssh sessions would kill the memory of the router,
I would of course have multiple setups one at the Datacenter, another at some PoP, redundant solutions incase one fails,
and For the record, I do run rancid, syslogging and we do AAA, so its just down to whats others experiences/ideas about secure management?
On 11-jan-05, at 18:48, Daniel Golding wrote:
Its terribly important that your routers' management traffic be encrypted all the way to the device.
Why "terribly important"? If this stuff runs over your own network then others aren't going to be able to sniff it without physically getting at your stuff. And if they can do that crypto won't buy you anything. That said, being able to connect to your stuff with crypto is better than without crypto, of course.
Bastion hosts are a good thing and can be a great place to put in checks for multi-factor authentication (another must-have),
Just make sure that when half your routers are dead you can still connect to the remaining half. A single bastion host isn't good enough.
While you are at it, look at your SNMP setup. You want your SNMP management to have the same characteristics as your vty management - strong authentication and encryption. Cleartext community strings don't cut it, as an example.
Not for write access, anyway. For read access you can get away with being slightly less paranoid.
Also, you need a secure Out of Band management network.
True out of band management networks are very hard to build and very hard to use, and you run the risk that you can't get at your stuff because the management network is down.
Iljitsch van Beijnum wrote:
On 11-jan-05, at 18:48, Daniel Golding wrote:
True out of band management networks are very hard to build and very hard to use, and you run the risk that you can't get at your stuff because the management network is down.
IS-IS can be highly recommended for true out of band management, it is reachable when IP goes down the drain entirely. Gernot Schmied
On 12-jan-05, at 11:30, Gernot W. Schmied wrote:
True out of band management networks are very hard to build and very hard to use, and you run the risk that you can't get at your stuff because the management network is down.
IS-IS can be highly recommended for true out of band management, it is reachable when IP goes down the drain entirely.
To me, true "out of band management" means that the management traffic doesn't flow over production links. You are right that IS-IS can continue to function when IP is confused (although with integrated IS-IS OSI will probably be just as confused as IP). But IS-IS isn't a management protocol, of course. :-) IPv6 is also very useful in providing non-IPv4 management.
On Wed, 2005-01-12 at 12:37, David Gethings wrote:
IPv6 is also very useful in providing non-IPv4 management. Well if we're offering protocols other than IP(v4) for OOB management
On Wed, 2005-01-12 at 12:25 +0100, Iljitsch van Beijnum wrote: then might I chip in with MPLS?
What ever happened to simple ISND or analogue dial-up with a small router or modem attached...? Not very hi-tech en often quite slow, but usually suffices for emergency maintenance and prolly as far apart from the operational network as possible (provided your not using transmission from the same telco that supplies the phone lines that is ;-) -- --- Erik Haagsman Network Architect We Dare BV tel: +31(0)10 7507008 fax:+31(0)10 7507005 http://www.we-dare.nl
On 1/12/05 8:46 AM, "Erik Haagsman" <erik@we-dare.net> wrote:
On Wed, 2005-01-12 at 12:37, David Gethings wrote:
IPv6 is also very useful in providing non-IPv4 management. Well if we're offering protocols other than IP(v4) for OOB management
On Wed, 2005-01-12 at 12:25 +0100, Iljitsch van Beijnum wrote: then might I chip in with MPLS?
What ever happened to simple ISND or analogue dial-up with a small router or modem attached...? Not very hi-tech en often quite slow, but usually suffices for emergency maintenance and prolly as far apart from the operational network as possible (provided your not using transmission from the same telco that supplies the phone lines that is ;-)
The biggest problem I've seen with dial-up OOB is reliability. You really need you really need to have a good series of testing scripts to ensure that all the phone lines are working, modems have reset properly, serial ports are ok, etc. Without this, reliability is low. - Dan
On Wed, 2005-01-12 at 20:12, Daniel Golding wrote:
The biggest problem I've seen with dial-up OOB is reliability. You really need you really need to have a good series of testing scripts to ensure that all the phone lines are working, modems have reset properly, serial ports are ok, etc. Without this, reliability is low.
Although it's perhaps not as reliable as a series of dedicated cicruits to connect various locations, I don't consider an ISDN router with it's Ethernet port connected to a management ethernet port as an unreliable solution. Modems and TA's perhaps, but a series of 2600's or similar devices with basic rate interfaces on each location shouldn't be your biggest worry at the moment you actually need them. CHeers, -- --- Erik Haagsman Network Architect We Dare BV tel: +31(0)10 7507008 fax:+31(0)10 7507005 http://www.we-dare.nl
Iljitsch van Beijnum wrote:
On 12-jan-05, at 11:30, Gernot W. Schmied wrote:
True out of band management networks are very hard to build and very hard to use, and you run the risk that you can't get at your stuff because the management network is down.
IS-IS can be highly recommended for true out of band management, it is reachable when IP goes down the drain entirely.
To me, true "out of band management" means that the management traffic doesn't flow over production links. You are right that IS-IS can continue to function when IP is confused (although with integrated IS-IS OSI will probably be just as confused as IP). But IS-IS isn't a management protocol, of course. :-)
IPv6 is also very useful in providing non-IPv4 management.
True, but integrated IS-IS is not true IS-IS strictly speaking. I am referring to ISO CLNS/CLNP, who actually needs IP if you have other fine network layer protocols alt your disposal ,-)? I used to recommend this measure in combination with BRI ISDN management lines, it's affordable and works without constantly testing analog dialin. A dedicated infrastructure beyond that measure simply is not justifiable economically. Besides, SDH and DWDM use separate management approaches as well, so does SS7 infrastructure. It is always a combination. Some people also use management VCIs/DLCIs which does not buy you much. my 0.02$, Gernot
On Tue, 11 Jan 2005 11:17:55 +0200, Kim Onnel <karim.adel@gmail.com> wrote:
Hello, I'd like everyones 2 cents on the BCP for network management of an ISP PoPs, with a non-security oriented NOC,
. . .
2) An OpenBSD bastion host(s), where the NOC would ssh in, get authenticated from TACACS+ or ssh certs, and then just telnet from there all day,
If the OpenBSD host is located in the same physical site as the Cisco products, you have the additional option of providing serial console access to the console port on the Cisco devices through the OpenBSD bastion host. To take this a step further, you can log all serial port I/O to disk. Using the serial console as your management port has one major drawback (some would call it a feature), you can only have one person (two with the AUX port) logged into a given router or switch at a time. This is very much "out of band" management, and having remote serial access is great for fault diagnosis and recovery (Not just for Cisco, Sun, etc). I've resolved more than a few Cisco switch "halt and catch fire" failures based on the last gasp fault message dumped to the console port.
On 11 Jan 2005, at 15:28, Kevin wrote:
On Tue, 11 Jan 2005 11:17:55 +0200, Kim Onnel <karim.adel@gmail.com> wrote:
Hello, I'd like everyones 2 cents on the BCP for network management of an ISP PoPs, with a non-security oriented NOC, . . . 2) An OpenBSD bastion host(s), where the NOC would ssh in, get authenticated from TACACS+ or ssh certs, and then just telnet from there all day,
If the OpenBSD host is located in the same physical site as the Cisco products, you have the additional option of providing serial console access to the console port on the Cisco devices through the OpenBSD bastion host. To take this a step further, you can log all serial port I/O to disk.
Using the serial console as your management port has one major drawback (some would call it a feature), you can only have one person (two with the AUX port) logged into a given router or switch at a time.
To do both serial console access and continuous logging of console output (and to allow multiple users to simultaneously access the same console port) try rtty. It's old, and it hasn't been updated in ages, and it turns out that's ok because it Just Works. At ISC, we've used rtty with PCI-based multi-port serial cards, and also with USB-based multi-port serial cards. It'll work with anything that can present a character device in /dev. ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz Joe
[...]
2) An OpenBSD bastion host(s), where the NOC would ssh in, get authenticated from TACACS+ or ssh certs, and then just telnet from there all day,
[...] (and s/telnet/ssh as has been suggested already)
3) Or just an IOS based bastion router that also runs ssh,
[...] When crafting the ACL that restricts what source IP{,v6} addresses may ssh to the router, you may want to include each router's neighbors by both their loopback and any interface addresses that might source a packet (if your security policy permits it). Having all your loopbacks and internal interfaces in a small number of prefixes dedicated to the task can help you craft a more-maintainable ACL. The motivation for doing this is that if dynamic routing melts down, you may find that using PMR to ssh from router to router is helpful. If you find yourself in a situation where you're using PMR, you may also need to turn off "ip ssh source-interface Loopback0" if you have it turned on - if dynamic routing has melted to the point where routers don't know each others' loopbacks, sourcing an ssh packet from a loopback won't get you far. If you use TACACS for AAA, plan in advance to have at least one login on the router with local credentials so that you can get in when TACACS is broken. Stephen
When crafting the ACL that restricts what source IP{,v6} addresses may ssh to the router, you may want to include each router's neighbors by both their loopback and any interface addresses that might source a packet (if your security policy permits it).
I forgot a phrase: [that might source a packet] headed for another router. Stephen
participants (9)
-
Daniel Golding -
David Gethings -
Erik Haagsman -
Gernot W. Schmied -
Iljitsch van Beijnum -
Joe Abley -
Kevin -
Kim Onnel -
Stephen Stuart