A couple of times during NANOG25, from the floor and from the podium, it was identified that the tools available for managing networks were garbage. I was surprised to hear that even real basics, such as change control and configuration management, weren't widely adopted. There definitely seemed to be an acceptance (and perhaps this is only true at some carriers) that many problems facing providers today are as a result of a dearth of decent tools to configure 'best common practices' into the routers - and as a result of this, the 'problems' with the networks were not with the h/w and/or the protocols they support, but with the people, and their lack of experience and/or ability to properly configure the boxes. A couple of comments that I heard over the last few days: 1) User interfaces are horrible and counter intuitive - I want 'xyz' out of my GUI 2) Systems blindly apply bad configurations to routers - they should be able to do 'some' verification before crashing my network - and can't roll back after they wreck things 3) Change control either doesn't exist, isn't usable, or isn't granular enough 4) There isn't anything to track non sanctioned changes to the network (i.e.: hacker induced re-configurations) I would very much like to hear about "specific" needs for (provisioning) tools that would satisfy your needs - needs that are either being poorly met to today, or not at all. In the hopes of preventing a vendor-bash extravaganza, I would suggest as a point of reference, that the NMS recommendations presented by Avi Freedman during the conference ("Industry/Government Infrastructure Vulnerability Assessment: Background and Recommendations". Of the recommendations pertinent to network management, many refer to future-features. As an additional attempt to constraint the discussion, I would recommend that the needs identified be realistic (i.e.: supportable on current equipment, the cost of the solution would be less than the cost of the problem, etc). Cheers, David - David Daley +1.905.922.6560 (global) daley@montagueriver.com www.montagueriver.com Montague River Networks Inc.
On Wed, 12 Jun 2002, David Daley wrote: > I would very much like to hear about "specific" needs for (provisioning) > tools that would satisfy your needs http://www.ietf.org/internet-drafts/draft-ops-operator-req-mgmt-02.txt -Bill
David, Almost all of what you're talking about is network device configuration file management -- there are several solutions out there today that do this. The rest is template-based configuration provisioning tools, which typically have no operational model of the network -- so it should be no surprise that they generate the wrong configurations. So there are two questions: The first is why aren't operators using even simple config management tools (Is every single one lacking somehow, or is it operational intertia?) The more interesting one, IMHO, concerns operational complexity. It seems that complexity is really what makes it hard to operate an IP network -- even with highly skilled engineers -- and is also the barrier to writing useful network provisioning and configuration software. What abstractions would make it easier to understand the network and hence figure out the right configuration changes to make, so software wouldn't generate config changes that are broken? Regards, Mathew At 01:38 PM 6/12/2002 -0400, David Daley wrote:
A couple of times during NANOG25, from the floor and from the podium, it was identified that the tools available for managing networks were garbage. I was surprised to hear that even real basics, such as change control and configuration management, weren't widely adopted. There definitely seemed to be an acceptance (and perhaps this is only true at some carriers) that many problems facing providers today are as a result of a dearth of decent tools to configure 'best common practices' into the routers - and as a result of this, the 'problems' with the networks were not with the h/w and/or the protocols they support, but with the people, and their lack of experience and/or ability to properly configure the boxes.
A couple of comments that I heard over the last few days: 1) User interfaces are horrible and counter intuitive - I want 'xyz' out of my GUI 2) Systems blindly apply bad configurations to routers - they should be able to do 'some' verification before crashing my network - and can't roll back after they wreck things 3) Change control either doesn't exist, isn't usable, or isn't granular enough 4) There isn't anything to track non sanctioned changes to the network (i.e.: hacker induced re-configurations)
I would very much like to hear about "specific" needs for (provisioning) tools that would satisfy your needs - needs that are either being poorly met to today, or not at all. In the hopes of preventing a vendor-bash extravaganza, I would suggest as a point of reference, that the NMS recommendations presented by Avi Freedman during the conference ("Industry/Government Infrastructure Vulnerability Assessment: Background and Recommendations". Of the recommendations pertinent to network management, many refer to future-features. As an additional attempt to constraint the discussion, I would recommend that the needs identified be realistic (i.e.: supportable on current equipment, the cost of the solution would be less than the cost of the problem, etc).
Cheers, David
- David Daley +1.905.922.6560 (global) daley@montagueriver.com www.montagueriver.com Montague River Networks Inc.
In the referenced message, David Daley said: <snip>
4) There isn't anything to track non sanctioned changes to the network (i.e.: hacker induced re-configurations)
I would be really surprised if anything other than mom-and-pop shops didn't have _at least_ this. rtrmon or rancid can do great config archiving and provide difference output.
On Wed, 12 Jun 2002, Stephen Griffin wrote: :: I would be really surprised if anything other than mom-and-pop shops :: didn't have _at least_ this. :: :: rtrmon or rancid can do great config archiving and provide difference :: output. :: I don't think the issue is detecting change as much as it is associating change to specific goals/tickets, etc.. If an ACL changes on a router, rancid will pick it up, but right now there is no automated way to tell whether that was as a result of a customer request or a security breach. -jba __ [jba@analogue.net] :: analogue.networks.nyc :: http://analogue.net
### On Wed, 12 Jun 2002 18:37:07 -0400 (EDT), jeffrey arnold ### <jba@analogue.net> casually decided to expound upon <nanog@merit.edu> ### the following thoughts about "Re: What's wrong with provisioning ### tools?": ja> On Wed, 12 Jun 2002, Stephen Griffin wrote: ja> ja> :: I would be really surprised if anything other than mom-and-pop shops ja> :: didn't have _at least_ this. ja> :: ja> :: rtrmon or rancid can do great config archiving and provide difference ja> :: output. ja> ja> I don't think the issue is detecting change as much as it is associating ja> change to specific goals/tickets, etc.. If an ACL changes on a router, ja> rancid will pick it up, but right now there is no automated way to tell ja> whether that was as a result of a customer request or a security breach. I've had quite a bit of experience with config management tools and have written some myself many years ago as did probably others due to the at the time lack of such things. However, many vendors are providing thrid-party solutions. The one I've seen that seems most suited to an ISP environment is GoldWire although to be honest, I have not really looked in-depth into such products for almost a year now so there might be others. -- /*===================[ Jake Khuon <khuon@NEEBU.Net> ]======================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | --------------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| N E T W O R K S | +=========================================================================*/
On Wed, 12 Jun 2002, Stephen Griffin wrote:
In the referenced message, David Daley said: <snip>
4) There isn't anything to track non sanctioned changes to the network (i.e.: hacker induced re-configurations)
I would be really surprised if anything other than mom-and-pop shops didn't have _at least_ this.
rtrmon or rancid can do great config archiving and provide difference output.
I didn't find anything that really suited my needs at the time (late 2000/early 2001), so I ended up writing my own archiver. From time to time I've thought about adding it to the COSI-NMS project on Sourceforge, but never gotten around to it. I've also other similar tools outside of Sourceforce, such as Pancho (http://pancho.lunarmedia.net/). I wrote the code behind mine to be fairly modular, so that adding a module to back up a config from a new device is pretty easy. It currently backs up these devices using either SNMP or Expect scripts for devices that require it: Cisco IOS <12.0 Cisco IOS >=12.0 Cisco CatOS Cisco 5000 VPN concentrators (the Compatible Systems ones, not Altiga) Cisco LocalDirectors Lucent TAOS (Max TNTs) Alteon WebOS (ACEdirectors) Redback AOS Nortel BayRS (Bay Networks nee Wellfleet) <-config is binary other odds and ends as they come up, like Netopia routers, etc. I haven't written anything to back up Junipers yet because I don't have any to test against. Aside from the Nortel routers, I support versioning on everything else. Keep in mind this is only one piece of the puzzle - backing up what's already out there. I intentionally left out the functionality to allow a config to be uploaded to one of the devices above for reasons already specified in this thread - it's just too dangerous. You can melt down a whole network really quickly if you're not careful. jms
participants (7)
-
Bill Woodcock -
David Daley -
Jake Khuon -
jeffrey arnold -
Mathew Lodge -
Stephen Griffin -
Streiner, Justin