Managing IOS Configuration Snippets
Howdy network operator cognoscenti, I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame: You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier. Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
On 2/26/2014 4:22 PM, Ryan Shea wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
I started a long email that really should just be a blog post. I need to get a blog or something. Short story is this: NETCONF is probably the future of change management on all types of routers and switches. It's not supported everywhere yet and is missing lots of features but they're working on it. Look at the talk given at NANOG60 for more information. There is a puppet module that is also incomplete. I'm not sure this is the right way to go (http://puppetlabs.com/blog/puppet-network-device-management) Most people roll their own solution. If you're looking to do that consider using augeas for parsing the configuration files. It can be really useful for documenting changes, and probably to diff parts of the config. You might also consider rabbitmq or another message queue to handle scheduling and deploying the changes. It can retry failed updates. You should work towards all or nothing commits (not all cisco gear supports this, but you can fake it in a couple of ways. Ultimately you want to rollback to a known good configuration if things go wrong) If you have money and want this right now: Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :) Cheers, Robert
On 2/26/2014 5:37 PM, Robert Drake wrote:
Most people roll their own solution. If you're looking to do that consider using augeas for parsing the configuration files. It can be really useful for documenting changes, and probably to diff parts of the config. You might also consider rabbitmq or another message queue to handle scheduling and deploying the changes. It can retry failed updates. You should work towards all or nothing commits (not all cisco gear supports this, but you can fake it in a couple of ways. Ultimately you want to rollback to a known good configuration if things go wrong)
I should amend that even though I recommend all this I haven't used any of it for networking. I guess those are more shiny ball ideas than actual things I've used. We have perl scripts that wrap an in-house API to access our IPAM which generates initial configuration. The template files are a mix of m4 and Template::Toolkit. We use basically one-off perl scripts for auditing sections of the configs to find discrepancies. We use rancid to collect configs. We just started using netdot which is nice for topology discovery. TACACS and DHCP logs are parsed and stored in logstash. All of those tools provide the who, what, where and when but not the why. The why would require a bit more custom stuff and forcing people to use a frontend interface instead of directly touching the routers. We aren't ready for that yet.
Robert - all great suggestions. Big cross-vendor configuration generation and deployment is outside the scope of what I was hoping for here. The goal is to have the version information somehow encoded into the configuration, and I'm not sure that NETCONF has anything to say about that matter. Certainly the same problem of which-versions-are-where exists in the puppet/chef world and there are platform specific ways to answer those questions. Deep analysis of the router configuration itself can give pretty strong hints about which version are deployed, but lets assume full config digestion and comparison is out of the question. From some off-list responses I am hearing that some folks do similar kludges with other text fields, wether they be remark/banner/snmp-foo/interface descriptions. On Wed, Feb 26, 2014 at 5:59 PM, Robert Drake <rdrake@direcpath.com> wrote:
On 2/26/2014 5:37 PM, Robert Drake wrote:
Most people roll their own solution. If you're looking to do that consider using augeas for parsing the configuration files. It can be really useful for documenting changes, and probably to diff parts of the config. You might also consider rabbitmq or another message queue to handle scheduling and deploying the changes. It can retry failed updates. You should work towards all or nothing commits (not all cisco gear supports this, but you can fake it in a couple of ways. Ultimately you want to rollback to a known good configuration if things go wrong)
I should amend that even though I recommend all this I haven't used any of it for networking. I guess those are more shiny ball ideas than actual things I've used. We have perl scripts that wrap an in-house API to access our IPAM which generates initial configuration. The template files are a mix of m4 and Template::Toolkit.
We use basically one-off perl scripts for auditing sections of the configs to find discrepancies. We use rancid to collect configs. We just started using netdot which is nice for topology discovery. TACACS and DHCP logs are parsed and stored in logstash. All of those tools provide the who, what, where and when but not the why. The why would require a bit more custom stuff and forcing people to use a frontend interface instead of directly touching the routers. We aren't ready for that yet.
On Wed, Feb 26, 2014 at 6:27 PM, Ryan Shea <ryanshea@google.com> wrote:
Robert - all great suggestions. Big cross-vendor configuration generation and deployment is outside the scope of what I was hoping for here. The goal is to have the version information somehow encoded into the configuration, and I'm not sure that NETCONF has anything to say about that matter. Certainly the same problem of which-versions-are-where exists in the puppet/chef world and there are platform specific ways to answer those
puppet solves this by comparing a complete md5(file) with deployed md5(file)... not as simple to do that on: access-list 150 permit icmp any any access-list 150 permit tcp any eq 80 any access-list 150 deny ip any any it'd be super nice if you could grab out just the hermetic bit of config you care about, and md5sum() that, eh? provided your stored config was written out in the IOS version (specific?) spacing/etc manner, of course.
questions. Deep analysis of the router configuration itself can give pretty strong hints about which version are deployed, but lets assume full config digestion and comparison is out of the question. From some off-list responses I am hearing that some folks do similar kludges with other text fields, wether they be remark/banner/snmp-foo/interface descriptions.
this makes me sad... but go 'state of the art network equipment!' is it time to start asking vendors for more operable configuration storage and access?
We are evaluating a piece of software called Skybox: http://www.skyboxsecurity.com/ It's geared to security analytics, but it does allow you to define configurations that are expected on a device, what software version it is running, whether commands that aren't there are, and those that should be there aren't, e.t.c. It supports all major network equipment vendors, and also allows for simple or complex regular expressions that can be used to search configuration files more easily. It is an offline system, so all you do is regularly present it with a text file of the device's running configuration, and it will do the necessary checks per the policy you have defined. Based on the configuration files it has, it can also create a visual model of your network. Not something you'd rely on given you have other tools for that, but kind of cool, nonetheless. Worth a look, I'd say. Mark.
On (2014-02-26 17:37 -0500), Robert Drake wrote:
Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :)
Tail-F is probably least bad option out there. In configuration management, this is super easy: DB => Template => Network This is super hard: Network => DB The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB. The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it. You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude. But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone. -- ++ytti
On Thu, Feb 27, 2014 at 8:58 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2014-02-26 17:37 -0500), Robert Drake wrote:
Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :)
Tail-F is probably least bad option out there.
In configuration management, this is super easy:
DB => Template => Network
This is super hard:
Network => DB
The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB.
The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it.
You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude.
But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone.
-- ++ytti
Agree with this. We started out with rancid, quickly moved to a homebrew scp and git backed system with webgit/cgit as the user interface. If you are lucky your network equipment supports "advanced features" like ssh keys. If not, you might be stuck using sshpass to ease config collection. Built a config parsing system that would decompose monolithic configs into configlet files. Md5sum the file and use as part of the filename. You can then see "version" information for parts of the config tree. Quickly realized that maintaining this system is a full time job, due to the advanced status of network equipment software... Now looking at Tail-F NCS. Demo is impressive. I'm hopeful. Stating the obvious: the software running on most network equipment is of poor quality. The tools to manage this are a combination of high quality engineers and homebrew tools. Vendor tools are of a similar quality to the equipment software. I'd like to think "SDN" is an attempt to improve this, but I have my doubts. -- Tim:>
A couple more thoughts, regarding Network => DB I completely agree that trying to use the network config itself as the authority for what we intend to be on a device is not the right long-term approach. There is still a problem with Network => DB that I see. Assuming you have *many* devices, that may or may not be up at a given time, or may be in various stages of turn-up / burn-in / decom it is expected that a config change will not successfully make it to all devices. There are other timing issues, like a config built for a device being turned up, followed by a push of an update to all devices that "succeeds", followed by the final turn-up of this device. Even if you have a fancy config pushing engine, let's just take as a given that you'll need to scrub through your rancid-git backups to determine what needs to be updated. Regarding the MD5 approach, let's also think that configlets could have "no" commands in them. In the NTP example I had before, if we wanted to remove an NTP server the configlet would need the "no" version, but the rancid backup obviously would not have this. I'm not trying to work a unit test assertion framework here either. Some vendors have more robust commenting, and this can be quite convenient for explicitly stating what was pushed to the device. What are you using in your network... banner, snmp-location, hope, prayer? On Thu, Feb 27, 2014 at 9:36 AM, Tim Durack <tdurack@gmail.com> wrote:
On Thu, Feb 27, 2014 at 8:58 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2014-02-26 17:37 -0500), Robert Drake wrote:
Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :)
Tail-F is probably least bad option out there.
In configuration management, this is super easy:
DB => Template => Network
This is super hard:
Network => DB
The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB.
The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it.
You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude.
But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone.
-- ++ytti
Agree with this.
We started out with rancid, quickly moved to a homebrew scp and git backed system with webgit/cgit as the user interface. If you are lucky your network equipment supports "advanced features" like ssh keys. If not, you might be stuck using sshpass to ease config collection.
Built a config parsing system that would decompose monolithic configs into configlet files. Md5sum the file and use as part of the filename. You can then see "version" information for parts of the config tree. Quickly realized that maintaining this system is a full time job, due to the advanced status of network equipment software...
Now looking at Tail-F NCS. Demo is impressive. I'm hopeful.
Stating the obvious: the software running on most network equipment is of poor quality. The tools to manage this are a combination of high quality engineers and homebrew tools. Vendor tools are of a similar quality to the equipment software. I'd like to think "SDN" is an attempt to improve this, but I have my doubts.
-- Tim:>
On Thu, Feb 27, 2014 at 9:50 AM, Ryan Shea <ryanshea@google.com> wrote:
A couple more thoughts, regarding
Network => DB
I completely agree that trying to use the network config itself as the authority for what we intend to be on a device is not the right long-term approach. There is still a problem with Network => DB that I see. Assuming you have *many* devices, that may or may not be up at a given time, or may be in various stages of turn-up / burn-in / decom it is expected that a config change will not successfully make it to all devices. There are other timing issues, like a config built for a device being turned up, followed by a push of an update to all devices that "succeeds", followed by the final turn-up of this device. Even if you have a fancy config pushing engine, let's just take as a given that you'll need to scrub through your rancid-git backups to determine what needs to be updated.
Regarding the MD5 approach, let's also think that configlets could have "no" commands in them. In the NTP example I had before, if we wanted to remove an NTP server the configlet would need the "no" version, but the rancid backup obviously would not have this. I'm not trying to work a unit test assertion framework here either. Some vendors have more robust commenting, and this can be quite convenient for explicitly stating what was pushed to the device. What are you using in your network... banner, snmp-location, hope, prayer?
We don't do this, but the only flexible commenting in IOS style configs is ACLs. You could have an ACL that contains remarks only, and include version information: ip access-list CFG-VER remark CFG-VER-NTP 1.0.3 remark CFG-VER-VTY 4.3.2 end You could break this into individual ACLs if you prefer: ip access-list CFG-VER-NTP remark CFG-VER-NTP 1.0.3 end ip access-list CFG-VER-VTY remark CFG-VER-VTY 4.3.2 end Seems ridiculous, but that is the sorry state of the network OS. -- Tim:>
Along those same lines, we've been using alias exec for the same thing for a while: Alias exec NTP 6500_NTP_V1.0.1 Alias exec bgp 6500_peer_V2.0.0 Thanks, Chuck -----Original Message----- From: Tim Durack [mailto:tdurack@gmail.com] Sent: Thursday, February 27, 2014 11:50 AM To: Ryan Shea Cc: nanog@nanog.org Subject: Re: Managing IOS Configuration Snippets On Thu, Feb 27, 2014 at 9:50 AM, Ryan Shea <ryanshea@google.com> wrote:
A couple more thoughts, regarding
Network => DB
I completely agree that trying to use the network config itself as the authority for what we intend to be on a device is not the right long-term approach. There is still a problem with Network => DB that I see. Assuming you have *many* devices, that may or may not be up at a given time, or may be in various stages of turn-up / burn-in / decom it is expected that a config change will not successfully make it to all devices. There are other timing issues, like a config built for a device being turned up, followed by a push of an update to all devices that "succeeds", followed by the final turn-up of this device. Even if you have a fancy config pushing engine, let's just take as a given that you'll need to scrub through your rancid-git backups to determine what needs to be updated.
Regarding the MD5 approach, let's also think that configlets could have "no" commands in them. In the NTP example I had before, if we wanted to remove an NTP server the configlet would need the "no" version, but the rancid backup obviously would not have this. I'm not trying to work a unit test assertion framework here either. Some vendors have more robust commenting, and this can be quite convenient for explicitly stating what was pushed to the device. What are you using in your network... banner, snmp-location, hope, prayer?
We don't do this, but the only flexible commenting in IOS style configs is ACLs. You could have an ACL that contains remarks only, and include version information: ip access-list CFG-VER remark CFG-VER-NTP 1.0.3 remark CFG-VER-VTY 4.3.2 end You could break this into individual ACLs if you prefer: ip access-list CFG-VER-NTP remark CFG-VER-NTP 1.0.3 end ip access-list CFG-VER-VTY remark CFG-VER-VTY 4.3.2 end Seems ridiculous, but that is the sorry state of the network OS. -- Tim:>
On (2014-02-27 09:50 -0500), Ryan Shea wrote:
Regarding the MD5 approach, let's also think that configlets could have "no" commands in them. In the NTP example I had before, if we wanted to
For DB => Template => Network it's to me very easy, but yes, each template you make must have anti-template version. So let's say you have NTP model, which may contain some access restriction information, NTP version, NTP peers. When you apply this model to device, then some platform specific ntp template is called. If you remove this from device, you need to call 'anti' version of the template. Very simple and easy. You also wondered how do I know which version of config network device has, this is hard problem. To know exactly what is wrong and how to address just that. If you can relax requirement to know if configuration is correct or incorrect it becomes trivial. But fixing incorrect is either full reprovision of new config (at least in IOS and JunOS not a problem, won't break the unchanged bits). Or you have human resolve it (of course as custom dictates first you punish the responsible severely but swiftly) -- ++ytti
We've gone off the rails a bit here. The 'in-line' bit was really at the heart of my question. With the number of responses so far it's starting to sound like there is not an answer other than kludge. "workable solutions for a way to track *in-line* the configuration revisions you have on your cisco-like devices" This could be on brocade/hp/arista/ios, so "ask the vendor to..." is not a (short term) solution. Assume we've got rancid-git, super awesome network engineers who write configuration bits and test and review them, and a super dooper tempating config pushy tool with retries and a blinky dashboard. Now, I hand you the 'show run' output and ask you if version 77 of the vty config is on this device. Can you answer the question? Now I hand you the 'show run' from 10,000 more device configs - and 100 more configuration chunks from revision control. Can you still answer the question? Assume a magical revision-history-aware configuration cross reference parser (while a noble and lovely goal) is not available. On Thu, Feb 27, 2014 at 8:58 AM, Saku Ytti <saku@ytti.fi> wrote:
On (2014-02-26 17:37 -0500), Robert Drake wrote:
Consider looking at Tail-F's NCS, which according to marketing presentations appears to do everything I want right now. I'd like to believe them but I don't have any money so I can't test it out. :)
Tail-F is probably least bad option out there.
In configuration management, this is super easy:
DB => Template => Network
This is super hard:
Network => DB
The first one keeps all platform specific logic in flat ascii files filled with variables from template. When you introduce new product, feature, vendor to network, you only add new ascii templates, extremely easy, no platform-specific logic in DB.
The second one every little change in network, requires parser changes trying to model it back to DB. This is not sustainable. We can kid ourselves that NetCONF/YANG will solve this, but they won't. SNMP is old technology, when new feature comes to vendor, it may take _years_ before MIB comes. There is no reason to suspect you will be able to get feature out via NetCONF just because it is there. And if you can't do it 100% then you have to write parser which can understand it.
You only need the second one, in case 100% is not from DB. But it is actually trivial to produce 100% from DB. You don't want DB to model base configuration, that's lot of work for no gain, that'll come from template or at most DB vendor-specific-blob. Then after you push configuration from DB to network, you immediately collect configuration and create relation of DB-config 2 network-config, now you can keep ensuring network has correct config. If it does not have, you don't know why not, you can't fix the error itself, but you can repovision whole box, so you do get configuration conformance check, it's just very crude.
But the alternative, trying to understand network config, is just never ending path to to pain. If someone is going to do it, model it to python or ruby ORM and put it in github so others can contribute and we don't need to do it alone.
-- ++ytti
Thus spake Ryan Shea (ryanshea@google.com) on Thu, Feb 27, 2014 at 09:38:33AM -0500:
Now, I hand you the 'show run' output and ask you if version 77 of the vty config is on this device. Can you answer the question? Now I hand you the 'show run' from 10,000 more device configs - and 100 more configuration chunks from revision control. Can you still answer the question? Assume a magical revision-history-aware configuration cross reference parser (while a noble and lovely goal) is not available.
sgrep -s "vty" r-cssc-b280c-1-core.conf Found stanza in r-cssc-b280c-1-core.conf size:9
Hi Ryan, If I'm understanding what you're trying to do, you could script around our rather unsophisticated 'sgrep' (stanza grep) tool combined with scripting around rancid & rcs to do what I think you are looking for. http://net.doit.wisc.edu/~dwcarder/scripts/sgrep sgrep can dump out a "stanza" of ios-like config, then you can rcsdiff that to your master, per 'chunk' of config. ex: line vty 0 4 access-class G-A-AdminAccess in exec-timeout 30 0 ipv6 access-class G-A-v6AdminAccess in line vty 5 24 access-class G-A-AdminAccess in exec-timeout 30 0 ipv6 access-class G-A-v6AdminAccess in ! See the -s and -e options for our sgrep. Add 'xargs -P' around your glue, and I think you'd be in luck. If you were building configs around this model, you could use m4. Dale
On 2/28/2014 9:19 PM, Dale W. Carder wrote:
If I'm understanding what you're trying to do, you could script around our rather unsophisticated 'sgrep' (stanza grep) tool combined with scripting around rancid & rcs to do what I think you are looking for.
http://net.doit.wisc.edu/~dwcarder/scripts/sgrep
sgrep can dump out a "stanza" of ios-like config, then you can rcsdiff that to your master, per 'chunk' of config. Dale
I'm digging the idea of your command. Along the same lines I've got this awk snippet that I made and then forgot about. It functions like the cisco pipe begin/end commands: #!/bin/sh if [ "x${2}" = "x" ]; then awk "/${1}/{temp=1}; temp==1{print}" else awk "/${1}/{temp=1};/${2}/{temp=0}; temp==1{print}" fi Usage: cat router-confg | begin_regex 'line vty' '!' If you omit the second argument it just shows you from your match until the end of file.
For a large install I set up a solution that might help. I utilized a Mediawiki install and its API to create, update and pull the configuration on many IOS devices. A wiki page for the host name was dynamically created and the configuration was placed there daily or hourly. This allowed support to review the configuration and advise customers quicker. Additional hacks for updating the devices via the wiki were used. The goal was transparency for the support team and the side effect was wiki page history showing what day and what lines changed. As mentioned the answer to your question would likely make a good article. On Wed, Feb 26, 2014 at 3:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
-- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
To clarify a bit, systems to grab or store the running config or keep track of intent. Let's assume that comparing the deployed configuration of an individual device to intent derived from a bunch of configuration bits from an RCS system is *hard*. For example, let's say you have a vty configuration which has a couple sections, line vty 0 2 and line vty 3 5. Someone updates this configuration in your RCS which removes the access-class from line vty 0 2 and adds it to the access-class for line vty 3 5. Let's also assume that you have *lots* of devices and *lots* of configurations and you cannot reasonably egrep/regexp your way to success here. I thank you all for your responses. I was hoping that someone trick I was not seeing and would say "oh, you just need to do..." On Thu, Feb 27, 2014 at 6:52 AM, Andrew Latham <lathama@gmail.com> wrote:
For a large install I set up a solution that might help. I utilized a Mediawiki install and its API to create, update and pull the configuration on many IOS devices. A wiki page for the host name was dynamically created and the configuration was placed there daily or hourly. This allowed support to review the configuration and advise customers quicker. Additional hacks for updating the devices via the wiki were used. The goal was transparency for the support team and the side effect was wiki page history showing what day and what lines changed. As mentioned the answer to your question would likely make a good article.
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical
for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make
On Wed, Feb 26, 2014 at 3:22 PM, Ryan Shea <ryanshea@google.com> wrote: sections, this a
bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
-- ~ Andrew "lathama" Latham lathama@gmail.com http://lathama.net ~
On 2/26/14, 16:22 , Ryan Shea wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. ...
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. So at login you'd see something like: blah blah authorized users blah blah $Id: routername-confg,v 1.23 2013/08/20 03:07:16 username Exp INCR: 1.2,1.3,1.4,1.5,1.6 and that version tracking made its way through to rancid for easy offline auditing. This made it nice and easy to tell when and what had been updated, though it still would take a couple steps to identify what exact subsections had been changed over time (since the incremental version tags were specific deltas in per-device configurations. You could probably do it in a more global way too - git commit ids, maybe? - but you also don't want to make the version string too wordy either). -e * based on ftp://ftp.cac.washington.edu/pub/config-generator/, but substantially enhanced beyond the last public domain version. I know I'd be really happy if the current version was ever open-sourced...
On Thu, Feb 27, 2014 at 10:34 PM, Erik Muller <erikm@buh.org> wrote:
At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner.
This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/ -- Suresh Ramasubramanian (ops.lists@gmail.com)
On 2/27/14, 12:21 , Suresh Ramasubramanian wrote:
This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/
But that's just archiving, like rancid, right? Still doesn't have any correlation to the template-management side of things. While having the backups makes it easy to check for simple things ("do all my routers have the right syslog host set?"), OP's question is about tracking what versions of templates may have been applied to routers; if there's any complex logic (like, "are all active customer routes on this device included in the bcp38 acl on the upstream interface") or site-specific things, that can get a lot harder to audit without the metadata on how the configuration got there. -e
A lot of template management discussion focusses on using the network configs as the canonical model of the network. Storing the network model in the DB (whatever form that takes) is much more sane. There is the brownfields issue of populating that database and then building device state from there, but once that's done a lot of the problems go away. A solution like rancid/tail-f then simply becomes the mechanics to push your device state to the devices. Some good stuff here: https://www.nanog.org/meetings/nanog44/presentations/Monday/Gill_programatic... --Simon On 27 February 2014 09:46, Erik Muller <erikm@buh.org> wrote:
On 2/27/14, 12:21 , Suresh Ramasubramanian wrote:
This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/
But that's just archiving, like rancid, right? Still doesn't have any correlation to the template-management side of things. While having the backups makes it easy to check for simple things ("do all my routers have the right syslog host set?"), OP's question is about tracking what versions of templates may have been applied to routers; if there's any complex logic (like, "are all active customer routes on this device included in the bcp38 acl on the upstream interface") or site-specific things, that can get a lot harder to audit without the metadata on how the configuration got there. -e
On 27 Feb 2014, at 12:46, Erik Muller <erikm@buh.org> wrote:
On 2/27/14, 12:21 , Suresh Ramasubramanian wrote:
This has been around for several years now - http://sourceforge.net/projects/cisco-conf-rep/
But that's just archiving, like rancid, right?
This is not any kind of sensible answer to the original question, but the general approach “give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff” has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a “production” system. Examples, as ever: http://www.nanog.org/meetings/nanog26/presentations/stephen.pdf ftp://ftp.isc.org/isc/toolmakers/ (It’s not pretty. But it’s not supposed to be pretty.) Joe
On 2/27/14, 15:52 , Joe Abley wrote:
This is not any kind of sensible answer to the original question, but the general approach “give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff” has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a “production” system.
+1000. And that applies equally to the backend. I have yet to meet a fancy, integrated, database-driven configuration management system that can beat a bunch of flat files and a few perl scripts. Hackability of a system can be a definite virtue here. -e
Definitely. Depends what form the database takes - I don't think SQL is the right answer here. Sticking with flat files and perl scripts as much as possible is good guidance. I'm biased, but I'd go with Python: http://www.youtube.com/watch?v=EGK5jjyUBCQ --Simon On 27 February 2014 13:05, Erik Muller <erikm@buh.org> wrote:
On 2/27/14, 15:52 , Joe Abley wrote:
This is not any kind of sensible answer to the original question, but the general approach "give ops people a shell on a box with a rancid repository, encourage them to write scripts that do stuff" has the potential to cause all kinds of good things to happen faster than the time taken to organise a conference call to discuss requirements gathering for a "production" system.
+1000. And that applies equally to the backend. I have yet to meet a fancy, integrated, database-driven configuration management system that can beat a bunch of flat files and a few perl scripts. Hackability of a system can be a definite virtue here. -e
Very cool, thanks Erik. I can think of many ways to encode version metadata. Probably best to be somewhere in between overly verbose (full version $Id / date / author for every config chunk) and being unreadable (base64 encoded gzip of unique configlet identifiers and versions). Updating a banner feels a bit easier when you are pushing a full device-specific configuration from a templating system. Regardless of where it is stored, keeping the metadata in one of these fields (banner for example) means that checking the contents of the banner configlet now requires slightly more logic - which is fine. Chuck, interesting use of alias. Simon, completely agree that the network itself should not be the intent store. The real focus here is when your intent is in a DB/templating system thingy, how do you operationally ensure that intent matches reality. Again, with many devices going through upgrades, disabled/unreachable devices, new devices, pre-configured devices. The intent pusher is not blindly and constantly pushing to all devices, and it's likely not safe to do that. On Thu, Feb 27, 2014 at 12:04 PM, Erik Muller <erikm@buh.org> wrote:
On 2/26/14, 16:22 , Ryan Shea wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices.
...
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. So at login you'd see something like:
blah blah authorized users blah blah $Id: routername-confg,v 1.23 2013/08/20 03:07:16 username Exp INCR: 1.2,1.3,1.4,1.5,1.6
and that version tracking made its way through to rancid for easy offline auditing. This made it nice and easy to tell when and what had been updated, though it still would take a couple steps to identify what exact subsections had been changed over time (since the incremental version tags were specific deltas in per-device configurations. You could probably do it in a more global way too - git commit ids, maybe? - but you also don't want to make the version string too wordy either).
-e
* based on ftp://ftp.cac.washington.edu/pub/config-generator/, but substantially enhanced beyond the last public domain version. I know I'd be really happy if the current version was ever open-sourced...
On 27 February 2014 10:39, Ryan Shea <ryanshea@google.com> wrote:
Very cool, thanks Erik. I can think of many ways to encode version metadata. Probably best to be somewhere in between overly verbose (full version $Id / date / author for every config chunk) and being unreadable (base64 encoded gzip of unique configlet identifiers and versions). Updating a banner feels a bit easier when you are pushing a full device-specific configuration from a templating system. Regardless of where it is stored, keeping the metadata in one of these fields (banner for example) means that checking the contents of the banner configlet now requires slightly more logic - which is fine.
Chuck, interesting use of alias.
Simon, completely agree that the network itself should not be the intent store. The real focus here is when your intent is in a DB/templating system thingy, how do you operationally ensure that intent matches reality. Again, with many devices going through upgrades, disabled/unreachable devices, new devices, pre-configured devices. The intent pusher is not blindly and constantly pushing to all devices, and it's likely not safe to do that.
Absolutely. Expect/SNMP/config scraping is a solution here. It's tedious and boring to write the hooks, but it's not impossible. A solution like tail-f is a much better solution here. My personal preference would be to just push/pull JSON off the devices. I think there are two separate components here (which are often conflated): the mechanics to push/pull from devices into a data structure, and the network database to work with those data structures. There's a place for both device level models and network level models. --Simon
On Thu, Feb 27, 2014 at 12:04 PM, Erik Muller <erikm@buh.org> wrote:
On 2/26/14, 16:22 , Ryan Shea wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices.
...
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
At a previous job, our roll-your-own solution was a template based system(*) generating full configs; all the version history for template sections, per-router local tweaks, and generated results was kept in RCS, and the actual last-configured version, plus any incremental changes, was kept in the login banner. So at login you'd see something like:
blah blah authorized users blah blah $Id: routername-confg,v 1.23 2013/08/20 03:07:16 username Exp INCR: 1.2,1.3,1.4,1.5,1.6
and that version tracking made its way through to rancid for easy offline auditing. This made it nice and easy to tell when and what had been updated, though it still would take a couple steps to identify what exact subsections had been changed over time (since the incremental version tags were specific deltas in per-device configurations. You could probably do it in a more global way too - git commit ids, maybe? - but you also don't want to make the version string too wordy either).
-e
* based on ftp://ftp.cac.washington.edu/pub/config-generator/, but substantially enhanced beyond the last public domain version. I know I'd be really happy if the current version was ever open-sourced...
Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my network I would want to audit it and fix not version control. What if some of the configs don’t match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found. On Feb 26, 2014, at 4:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley <no.spam@comcast.net> wrote:
Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my
sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'. sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ? It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no fair comparing contracted work where you bill for time + materials, that's the wrong incentive set)... I bet that'd get them to fix stuff up right quick. network I would want to audit it and fix not version control. What if some of the configs don't match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found.
On Feb 26, 2014, at 4:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
Keegan, don't get me wrong, I am not suggesting that even if version numbers were happily encoded in robust comments that this would be the same as actually digesting the configuration. If the function of checking using 'fancy versioning' is not an operational best practice, what do you suggest (all-knowing/singing/dancing tool which understands the configuration and your intent aside)? You say IF NTP or CPP were configured differently - with a large enough network there will always be configurations which have differences. With that as an operational constant, how do you determine which devices have the latest iteration of your line vty configuration. How often will a change not be rolled out to every router. This is again related to the size and churn of the network, but my practical estimation is that once you get into thousands of routers there will almost always be some that get missed. Comprehensive auditing is very important, and arguably more useful than version checking - but it requires that you make knowledgeable and complete assertions. I assert the my snmp config should look like the snmp snippet version 77 is easier to grok than "make sure our community string is not set to public" (and repeat hand-crafted audit logic for every segment of the config). What if some of the configs don't match the defined versions? This is why it may make sense to break snippets into functional areas. "Just fix it" might be sane for a banner, but squashing an interface mtu change that was put there for a reason could end in tears. I consider this bit out of the scope of the question, but yes it is another important problem. On Thu, Feb 27, 2014 at 10:03 PM, Christopher Morrow < morrowc.lists@gmail.com> wrote:
Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way
On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley <no.spam@comcast.net> wrote: than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my
sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'.
sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ?
It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no fair comparing contracted work where you bill for time + materials, that's the wrong incentive set)... I bet that'd get them to fix stuff up right quick.
network I would want to audit it and fix not version control. What if some of the configs don't match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found.
On Feb 26, 2014, at 4:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which
IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical
for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make
use sections, this a
bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
On Feb 28, 2014, at 9:11 AM, Ryan Shea <ryanshea@google.com> wrote:
Keegan, don't get me wrong, I am not suggesting that even if version numbers were happily encoded in robust comments that this would be the same as actually digesting the configuration. If the function of checking using 'fancy versioning' is not an operational best practice, what do you suggest (all-knowing/singing/dancing tool which understands the configuration and your intent aside)? You say IF NTP or CPP were configured differently - with a large enough network there will always be configurations which have differences. With that as an operational constant, how do you determine which devices have the latest iteration of your line vty configuration.
That’s what I mean. The things that lend well to to version tracking don’t tend to change much. How many ways are there to configure VTY lines, or NTP, or CPP, or even OSPF and if we’re talking about an access ACL why not just audit the configs to make sure that all the entries are there. Am I really going to care that one router has version 1.0 versus another router that has version 2.2.12 build9? It’s not source code..
How often will a change not be rolled out to every router. This is again related to the size and churn of the network, but my practical estimation is that once you get into thousands of routers there will almost always be some that get missed.
Again, a router that was missed is a reason for audit and remediation not versioning. If you find a router with config missing does it really matter what version it’s on and when that version was valid? Not in my experience.
Comprehensive auditing is very important, and arguably more useful than version checking - but it requires that you make knowledgeable and complete assertions. I assert the my snmp config should look like the snmp snippet version 77 is easier to grok than "make sure our community string is not set to public" (and repeat hand-crafted audit logic for every segment of the config).
There may be some differences, but those are normally due to equipment lifecycle, mergers/consolidations and such. It’s easy to refer to something as the config for a particular platform or company than a version number. This can be tracked in GIT or SVN. Even then there will not be constant changes. I’d lean towards standardization. So the equipment that cannot adhere to the defined standards probably won’t evolve much on it’s own.
What if some of the configs don't match the defined versions? This is why it may make sense to break snippets into functional areas. "Just fix it" might be sane for a banner, but squashing an interface mtu change that was put there for a reason could end in tears. I consider this bit out of the scope of the question, but yes it is another important problem.
I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. With software for example, if something is different it might be a different version of that application with compatibility issues, dependencies, library issues, etc. When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. Again, this is for things that lend themselves to snippets. ACL’s, NTP, SNMP, CPP, even Spanning-tree. Not for things like interface IP’s or static routes that may be different across different boxes or location. If you’re referring to the latter I may have misunderstood your question..
On Thu, Feb 27, 2014 at 10:03 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote: On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley <no.spam@comcast.net> wrote:
Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my
sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'.
sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ?
It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no fair comparing contracted work where you bill for time + materials, that's the wrong incentive set)... I bet that'd get them to fix stuff up right quick.
network I would want to audit it and fix not version control. What if some of the configs don't match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found.
On Feb 26, 2014, at 4:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to track in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical sections, for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put aside for the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make this a bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
It sounds like your suggestion is to not check version numbers, write smart audits and react. Thinking of ACLs for example, version checking from a remark is a limted shortcut (but still very useful way) to validate distribution of an ACL. These things may change ~continuously. Grabbing a config and jamming that into some rancid rcs thing is different than intent version tracking. Our friends at Juniper (or the IOS-XR folks) seem to think the version control paradigm lends itself well to router configs, albeit at the whole-config level. Oh, and I think we're crossing streams here regarding versioning. I am not suggesting to take the configured state of the network, consider that proper intent, record it in a revision and then check for it in the future. I am assuming that the intent versions represent reviewed and tested configuration states by ops/neteng. Different across the fleet does not necessarily mean non-deterministic, so intent versioning can still be useful. I'm sure (not first-hand sure) that some of the other suggested templating systems could build parts of the expected config based on attributes of a device. On Fri, Feb 28, 2014 at 9:49 AM, Keegan Holley <no.spam@comcast.net> wrote:
On Feb 28, 2014, at 9:11 AM, Ryan Shea <ryanshea@google.com> wrote:
Keegan, don't get me wrong, I am not suggesting that even if version numbers were happily encoded in robust comments that this would be the same as actually digesting the configuration. If the function of checking using 'fancy versioning' is not an operational best practice, what do you suggest (all-knowing/singing/dancing tool which understands the configuration and your intent aside)? You say IF NTP or CPP were configured differently - with a large enough network there will always be configurations which have differences. With that as an operational constant, how do you determine which devices have the latest iteration of your line vty configuration.
That’s what I mean. The things that lend well to to version tracking don’t tend to change much. How many ways are there to configure VTY lines, or NTP, or CPP, or even OSPF and if we’re talking about an access ACL why not just audit the configs to make sure that all the entries are there. Am I really going to care that one router has version 1.0 versus another router that has version 2.2.12 build9? It’s not source code..
How often will a change not be rolled out to every router. This is again related to the size and churn of the network, but my practical estimation is that once you get into thousands of routers there will almost always be some that get missed.
Again, a router that was missed is a reason for audit and remediation not versioning. If you find a router with config missing does it really matter what version it’s on and when that version was valid? Not in my experience.
Comprehensive auditing is very important, and arguably more useful than version checking - but it requires that you make knowledgeable and complete assertions. I assert the my snmp config should look like the snmp snippet version 77 is easier to grok than "make sure our community string is not set to public" (and repeat hand-crafted audit logic for every segment of the config).
There may be some differences, but those are normally due to equipment lifecycle, mergers/consolidations and such. It’s easy to refer to something as the config for a particular platform or company than a version number. This can be tracked in GIT or SVN. Even then there will not be constant changes. I’d lean towards standardization. So the equipment that cannot adhere to the defined standards probably won’t evolve much on it’s own.
What if some of the configs don't match the defined versions? This is why it may make sense to break snippets into functional areas. "Just fix it" might be sane for a banner, but squashing an interface mtu change that was put there for a reason could end in tears. I consider this bit out of the scope of the question, but yes it is another important problem.
I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning. With software for example, if something is different it might be a different version of that application with compatibility issues, dependencies, library issues, etc. When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version. Again, this is for things that lend themselves to snippets. ACL’s, NTP, SNMP, CPP, even Spanning-tree. Not for things like interface IP’s or static routes that may be different across different boxes or location. If you’re referring to the latter I may have misunderstood your question..
On Thu, Feb 27, 2014 at 10:03 PM, Christopher Morrow < morrowc.lists@gmail.com> wrote:
Putting aside the fact that snippets aren't a good way to conceptualize deployed router code, my gut still tells me to question the question here. The first is does this stuff change often enough to warrant a fancy versioning solution? I have yet to see NTP deployed in a different way
On Thu, Feb 27, 2014 at 8:38 PM, Keegan Holley <no.spam@comcast.net> wrote: than when I first learned to configure it. Next, when it does change how often is it not rolled out to every router. If NTP or CPP or SNMP or some other administrative option were configured differently across my
sure, so you're saying that a large bit (maybe) of the router config is 'one size fits all' and 'never changes' where 'never' is really 'very infrequently'.
sure, agreed... but there are parts of the config that do change more frequently (depending on the network perhaps)... how do you go about seeing which version / setup is deployed EXCEPT by building a home-grown 'config parser' and seeing that 'what is deployed matches mostly what I have in my config store for this router/class-of-router/network' ?
It's a shame that vendors of network equipment don't have to manage large networks of their own equipment under constrained opex environments (no fair comparing contracted work where you bill for time + materials, that's the wrong incentive set)... I bet that'd get them to fix stuff up right quick.
network I would want to audit it and fix not version control. What if some of the configs don't match the defined versions? It may be better to create standard templates and version them in SVN or GIT and then use config backups to track which devices have the standard configs. There are some for pay tools that can search for certain statements on various boxes and either alert or remediate when differences are found.
On Feb 26, 2014, at 4:22 PM, Ryan Shea <ryanshea@google.com> wrote:
Howdy network operator cognoscenti,
I'd love to hear your creative and workable solutions for a way to
in-line the configuration revisions you have on your cisco-like devices. Let me clearify/frame:
You have a set of tested/approved configurations for your routers which use IOS style configuration. These configurations of course are always refined and updated. You break these pieces of configuration into logical
for example a configuration file for NTP configuration, a file for control plane filter and store these in some revision control system. Put
the moment whether this is a reasonable way to comprehend deployed configurations. What methods do some of you use to know which version of a configuration you have deployed to a given router for auditing and update purposes? Remarks are a convenient way to do this for ACLs - but I don't have similar mechanics for top level configurations. About a decade ago I thought I'd be super clever and encode versioning information into the snmp location - but that is just awful and there is a much better way everyone is using, right? Flexible commenting on other vendors/platforms make
track sections, aside for this a
bit easier.
Assume that this version encoding perfectly captures what is on the router and that no person is monkeying with the config... version 77 of the control plane filter is the same everywhere.
Thus spake Keegan Holley (no.spam@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500:
I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning.
Um, what? $> rlog r-cssc-b280c-1-core.conf | grep 'total revision' total revisions: 2009; selected revisions: 2009
When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version.
We have our operators manually check in revisions (think in rcs terms: co -l router, go do work, verify it, ci -u router) rather than unsolicited / cron-triggered checkins. Then the check-in message contains the operator's description text of the change and often a ticket number. So there slightly fewer fat-finger configs checked in. Dale
On Feb 28, 2014, at 9:35 PM, Dale W. Carder <dwcarder@wisc.edu> wrote:
Thus spake Keegan Holley (no.spam@comcast.net) on Fri, Feb 28, 2014 at 09:49:19AM -0500:
I wasn’t saying just fix it. I was saying that router configs don’t lend well to versioning.
Um, what?
$> rlog r-cssc-b280c-1-core.conf | grep 'total revision' total revisions: 2009; selected revisions: 2009
I wish you were here to see my eyes rolling.. 2009 versions of something are no more grok-able than one current version. Congrats, you have a config backup system.
When it’s a router config chances are someone fat-fingered something. Most of the time the best thing to do is to fix or at least alert on the error, not to record it as a valid config version.
We have our operators manually check in revisions (think in rcs terms: co -l router, go do work, verify it, ci -u router) rather than unsolicited / cron-triggered checkins. Then the check-in message contains the operator's description text of the change and often a ticket number. So there slightly fewer fat-finger configs checked in.
That’s not what the OP was looking for AFAIK. This is just change management.
Dale
On Feb 27, 2014, at 7:38 PM, Keegan Holley <no.spam@comcast.net> wrote:
Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here.
What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view. For instance: customer AAAAA interface GigabitEthernet1/2/3.10 description AAAAA ip address 10.0.1.1 255.255.255.0 router bgp 1 neighbor 10.0.1.2 prefix-list AAAAA-in in ip prefix-list AAAAA-in 10.1.0.0/24 end customer BBBBB interface GigabitEthernet1/2/3.11 description BBBBB ip address 10.0.2.1 255.255.255.0 router bgp 1 neighbor 10.0.2.2 prefix-list BBBBB-in in ip prefix-list BBBBB-in 10.2.0.0/24 end Then I should be able to do: show run - Normal output like we see today, the "device" view. customer AAAAA show run - Same format as I have above, just config relevant to customer AAAAA. I can even see extending the tag to work with some other commands: customer AAAAA show int customer AAAAA show bgp ipv4 uni sum customer AAAAA show ip prefix-list The same functionality would work for snippets: customer ntp-servers-v1.0 ntp server 1.2.3.4 ntp server 1.2.3.5 ntp server 1.2.3.6 end Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of "no customer AAAAA", that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, "show run" to see who's configure on it sort of operations. I don't know of any platform that has implemented this sort of config framework though. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Isn't that framework called Juniper Op scripts? Run command which takes xml output of config (or other RPC) applies xslt, produces the output you describe. On Fri, Feb 28, 2014 at 10:24 AM, Leo Bicknell <bicknell@ufp.org> wrote:
On Feb 27, 2014, at 7:38 PM, Keegan Holley <no.spam@comcast.net> wrote:
Putting aside the fact that snippets aren’t a good way to conceptualize deployed router code, my gut still tells me to question the question here.
What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view.
For instance:
customer AAAAA interface GigabitEthernet1/2/3.10 description AAAAA ip address 10.0.1.1 255.255.255.0 router bgp 1 neighbor 10.0.1.2 prefix-list AAAAA-in in ip prefix-list AAAAA-in 10.1.0.0/24 end
customer BBBBB interface GigabitEthernet1/2/3.11 description BBBBB ip address 10.0.2.1 255.255.255.0 router bgp 1 neighbor 10.0.2.2 prefix-list BBBBB-in in ip prefix-list BBBBB-in 10.2.0.0/24 end
Then I should be able to do:
show run - Normal output like we see today, the "device" view. customer AAAAA show run - Same format as I have above, just config relevant to customer AAAAA.
I can even see extending the tag to work with some other commands:
customer AAAAA show int customer AAAAA show bgp ipv4 uni sum customer AAAAA show ip prefix-list
The same functionality would work for snippets:
customer ntp-servers-v1.0 ntp server 1.2.3.4 ntp server 1.2.3.5 ntp server 1.2.3.6 end
Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of "no customer AAAAA", that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, "show run" to see who's configure on it sort of operations.
I don't know of any platform that has implemented this sort of config framework though.
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On 2/28/14, 10:24 , Leo Bicknell wrote:
What I have always wanted is a way to group configuration, in particular by customer. Ideally with the ability to see it both as a unified view, and also as a per-customer view.
For instance:
customer AAAAA interface GigabitEthernet1/2/3.10 description AAAAA ip address 10.0.1.1 255.255.255.0 router bgp 1 neighbor 10.0.1.2 prefix-list AAAAA-in in ip prefix-list AAAAA-in 10.1.0.0/24 end ...
Basically this follows the two modes in which engineers look at a device. Most of the time is configuring a specific customer, and wanting to be sure they are configured right; including the hard case of "no customer AAAAA", that is making sure all configuration for a specific customer is removed. The rest of the time is typically troubleshooting a network level problem where you want the device view we have today, I see interface Gig1/2/3 is dropping packets, "show run" to see who's configure on it sort of operations.
I don't know of any platform that has implemented this sort of config framework though.
It's not the cleanest, but in junos you can pretty much get this by defining a configuration group per customer and applying them all. Your RE may hate you at commit time, but I've seen this approach work quite well. -e
participants (15)
-
Andrew Latham -
Christopher Morrow -
Chuck Church -
Dale W. Carder -
Erik Muller -
Joe Abley -
Keegan Holley -
Leo Bicknell -
Mark Tinka -
Robert Drake -
Ryan Shea -
Saku Ytti -
Simon Knight -
Suresh Ramasubramanian -
Tim Durack