Network Configuration Management Practices
Hi, I am doing some independent research on Network Configuration Management Practices. I am trying to get information from service providers and enterprises on how they handle this function. I have the following specific questions: 1) What configuration issues most affect the performance and reliability of your network? 2) What are you doing to alleviate these problems? 3) What software are you using to manage network configurations for your routers, switches, firewalls, load balancers, servers, etc? Do you use home grown scripts? Or do you use commercial applications (if so, which ones) 4) Does your configuration management software integrate with your network fault and performance management system? If so, what level of integration do you have? If not, is this integration something you desire? 5) What functions are you most lacking in your configuration management software tools? Any response and input will be greatly appreciated. Thanks, Carl -- Carl W. Kalbfleisch www.kalbfleisch.us
On Tue, 14 Sep 2004, Carl W.Kalbfleisch wrote: : I am doing some independent research on Network Configuration : Management Practices. I am trying to get information from service : providers and enterprises on how they handle this function. I have the : following specific questions: : : 1) What configuration issues most affect the performance and : reliability of your network? Fingers... >;-) scott
Hmm, there are many approaches, starting with _what is primary_ (in Moscow's ISP files was primary, in enterprise here configs are primary). In my case, I use some hard rules: - no matter what is primary, configurations should be stored into CVS or simular system, and made available (for network engineers) on the internal web (with restricted access); - system should collect all changes automatically (or update configs from files automatically), make diffs and send change reports. - In any case, I must be able to see real configuration and see all changes, applying for last few weeks, without telnetting to the box. Without such things, I am blind ( I feel myself blind, when I come to the new network, and they have not such things in their system, making changes _on live servers_ and making 'telnet' to evaluate configuration). Few tools (opensource and commercial) allows to automate this job. One more thing. We tried to review _proposed changes_ and _changed applied_. Practice showed, that it is impossible to see errors in proposed updates, even if 3 - 4 engineers review it (not design flaws, but syntac and semantics errors), so we did not got many use from pre-change reviews (except design ones). But we got extremely high profit from post-change reviews (verifying, what really changed on the router / firewall after maintanance window) - it allows to see some unwanted changes and avoid few possible service disruptions. ----- Original Message ----- From: "Scott Weeks" <surfer@mauigateway.com> To: "Carl W.Kalbfleisch" <c.kalbfleisch@comcast.net> Cc: <nanog@merit.edu> Sent: Tuesday, September 14, 2004 3:08 PM Subject: Re: Network Configuration Management Practices
On Tue, 14 Sep 2004, Carl W.Kalbfleisch wrote:
: I am doing some independent research on Network Configuration : Management Practices. I am trying to get information from service : providers and enterprises on how they handle this function. I have the : following specific questions: : : 1) What configuration issues most affect the performance and : reliability of your network?
Fingers... >;-)
scott
There has been some public available software for backing up Cisco router configuration. The backup is not in CVS but in plain file. Joe --- Alexei Roudnev <alex@relcom.net> wrote:
Hmm, there are many approaches, starting with _what is primary_ (in Moscow's ISP files was primary, in enterprise here configs are primary).
In my case, I use some hard rules: - no matter what is primary, configurations should be stored into CVS or simular system, and made available (for network engineers) on the internal web (with restricted access); - system should collect all changes automatically (or update configs from files automatically), make diffs and send change reports. - In any case, I must be able to see real configuration and see all changes, applying for last few weeks, without telnetting to the box.
Without such things, I am blind ( I feel myself blind, when I come to the new network, and they have not such things in their system, making changes _on live servers_ and making 'telnet' to evaluate configuration).
Few tools (opensource and commercial) allows to automate this job.
One more thing. We tried to review _proposed changes_ and _changed applied_. Practice showed, that it is impossible to see errors in proposed updates, even if 3 - 4 engineers review it (not design flaws, but syntac and semantics errors), so we did not got many use from pre-change reviews (except design ones). But we got extremely high profit from post-change reviews (verifying, what really changed on the router / firewall after maintanance window) - it allows to see some unwanted changes and avoid few possible service disruptions.
----- Original Message ----- From: "Scott Weeks" <surfer@mauigateway.com> To: "Carl W.Kalbfleisch" <c.kalbfleisch@comcast.net> Cc: <nanog@merit.edu> Sent: Tuesday, September 14, 2004 3:08 PM Subject: Re: Network Configuration Management Practices
On Tue, 14 Sep 2004, Carl W.Kalbfleisch wrote:
: I am doing some independent research on Network
: Management Practices. I am trying to get information from service : providers and enterprises on how they handle
: following specific questions: : : 1) What configuration issues most affect the
Configuration this function. I have the performance and
: reliability of your network?
Fingers... >;-)
scott
__________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com
I posted our software (doing this) onto http://snmpstat.sf.net (named as CCR - Cisco Configuration Repository). It is 100% WEB configured and supports IOS, CatOS, PIX and some old VPN devices (they all have different commands to save config). ----- Original Message ----- From: "Joe Shen" <joe_hznm@yahoo.com.sg> To: "Alexei Roudnev" <alex@relcom.net>; "Scott Weeks" <surfer@mauigateway.com>; "Carl W.Kalbfleisch" <c.kalbfleisch@comcast.net> Cc: <nanog@merit.edu> Sent: Wednesday, September 15, 2004 1:59 AM Subject: Re: Network Configuration Management Practices
There has been some public available software for backing up Cisco router configuration.
The backup is not in CVS but in plain file.
Joe
--- Alexei Roudnev <alex@relcom.net> wrote:
Hmm, there are many approaches, starting with _what is primary_ (in Moscow's ISP files was primary, in enterprise here configs are primary).
In my case, I use some hard rules: - no matter what is primary, configurations should be stored into CVS or simular system, and made available (for network engineers) on the internal web (with restricted access); - system should collect all changes automatically (or update configs from files automatically), make diffs and send change reports. - In any case, I must be able to see real configuration and see all changes, applying for last few weeks, without telnetting to the box.
Without such things, I am blind ( I feel myself blind, when I come to the new network, and they have not such things in their system, making changes _on live servers_ and making 'telnet' to evaluate configuration).
Few tools (opensource and commercial) allows to automate this job.
One more thing. We tried to review _proposed changes_ and _changed applied_. Practice showed, that it is impossible to see errors in proposed updates, even if 3 - 4 engineers review it (not design flaws, but syntac and semantics errors), so we did not got many use from pre-change reviews (except design ones). But we got extremely high profit from post-change reviews (verifying, what really changed on the router / firewall after maintanance window) - it allows to see some unwanted changes and avoid few possible service disruptions.
----- Original Message ----- From: "Scott Weeks" <surfer@mauigateway.com> To: "Carl W.Kalbfleisch" <c.kalbfleisch@comcast.net> Cc: <nanog@merit.edu> Sent: Tuesday, September 14, 2004 3:08 PM Subject: Re: Network Configuration Management Practices
On Tue, 14 Sep 2004, Carl W.Kalbfleisch wrote:
: I am doing some independent research on Network
: Management Practices. I am trying to get information from service : providers and enterprises on how they handle
: following specific questions: : : 1) What configuration issues most affect the
Configuration this function. I have the performance and
: reliability of your network?
Fingers... >;-)
scott
__________________________________________________ Do You Yahoo!? Download the latest ringtones, games, and more! http://sg.mobile.yahoo.com
On Wed, Sep 15, 2004 at 12:27:20AM -0700, Alexei Roudnev wrote:
One more thing. We tried to review _proposed changes_ and _changed applied_. Practice showed, that it is impossible to see errors in proposed updates, even if 3 - 4 engineers review it (not design flaws, but syntac and semantics errors), so we did not got many use from pre-change reviews (except design ones). But we got extremely high profit from post-change reviews (verifying, what really changed on the router / firewall after maintanance window) - it allows to see some unwanted changes and avoid few possible service disruptions.
This doesn't seem to scale too well. When you have frequent changes (i.e. many access devices) the diff load becomes unmanageably large. My ideal would be to have a network monitoring tool which compares the actual network against a configured baseline. The presumption would be that if the network matches what have been set forth as engineering rules, I don't really care what the specific settings are. Currently we do something sort of halfway: archive the actual configs and then run audit scripts against them, which parse the configs. Definitely not ideal but it helps catch simpler errors. One of these days when I have extra cycles.. (yeah, right) Austin
On Wed, Sep 15, 2004 at 12:27:20AM -0700, Alexei Roudnev wrote:
One more thing. We tried to review _proposed changes_ and _changed
applied_.
Practice showed, that it is impossible to see errors in proposed updates, even if 3 - 4 engineers review it (not design flaws, but syntac and semantics errors), so we did not got many use from pre-change reviews (except design ones). But we got extremely high profit from post-change reviews (verifying, what really changed on the router / firewall after maintanance window) - it allows to see some unwanted changes and avoid few possible service disruptions.
This doesn't seem to scale too well. When you have frequent changes (i.e. many access devices) the diff load becomes unmanageably large. My ideal would be to have a network monitoring tool which compares the actual network against a configured baseline. The presumption would be
It I have frequent changes, I always automate them so that: - operator enter data into the database; - operator click 'UPDATE' - operator review proposed update and click APPLY - tier-3 receive change report and review it. We did such thing (analyzing configs, creating schemas and posting it all @ internal WEB) when I woprked in ISP in MOscow, but we never (!) allowed anyone to configure routers manually, except very unusual changes. Everything other (interfaces, E1 channels, access lists, BGP filters, route maps and so on) was generated and updated automatically. When I saw tier-1 people doing 'conf t' here in USA, I think _oh, they have so many money that they can allow people to touch configs manually' -:). /Unfortunately, Cisco is not old Cisco now, with a lot of skilled and helpful developers, so no one hope that they will help in such automation/. ----- Original Message ----- From: "Austin Schutz" <tex@off.org> To: "Alexei Roudnev" <alex@relcom.net> Cc: "Scott Weeks" <surfer@mauigateway.com>; "Carl W.Kalbfleisch" <c.kalbfleisch@comcast.net>; <nanog@merit.edu> Sent: Wednesday, September 15, 2004 2:25 AM Subject: Re: Network Configuration Management Practices that
if the network matches what have been set forth as engineering rules, I don't really care what the specific settings are. Currently we do something sort of halfway: archive the actual configs and then run audit scripts against them, which parse the configs. Definitely not ideal but it helps catch simpler errors. One of these days when I have extra cycles.. (yeah, right)
Austin
participants (5)
-
Alexei Roudnev -
Austin Schutz -
Carl W.Kalbfleisch -
Joe Shen -
Scott Weeks