Re: Templating/automating configuration
Because we had different sources of truth which were written in-house, we wound up rolling our own template engine in Python. It took about 3 weeks to write the engine and adapt existing templates. Given a circuit ID, it generates the full config for copy and paste into a terminal session. It also hooks into a configuration parser tool, written in-house, that tracks configured interfaces, so it is easy to see whether the template would overwrite an existing interface. I used the Jinja2 template engine, along with pyodbc/unixODBC/FreeTDS for access to a Microsoft SQL backend. The keys for us are: * extracting information from a source of truth * validating the information for correctness * making sure you don't overwrite existing config * outputting the right templates for the circuit features It made more sense to write a tool than it did to try to adapt something for our environment. If I had a free hand and unlimited budget, I would find a single app that functions as a source of truth for all circuits and products, which includes a templating engine that hooks in easily. -Brian ---- On Tue, 06 Jun 2017 08:22:59 -0500 Graham Johnston <johnstong@westmancom.com> wrote ---- Short of complete SDN, for those of you that have some degree of configuration templating and/or automation tools what is it that you run? I'm envisioning some sort of tool that let's me define template snippets of configuration and aids in their deployment to devices. I'm okay doing the heaving lifting in defining everything, I'm just looking for the tool that stitches it together and hopefully makes things a little less error prone for those who aren't as adept. Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com>
Hi, Here are some extra pointers: https://youtube.com/watch?v=C7pkab8n7ys https://www.nanog.org/sites/default/files/dosdontsnetworkautomation.pdf https://github.com/coloclue/kees Kind regards, Job On Tue, 6 Jun 2017 at 13:49, Brian Knight <ml@knight-networks.com> wrote:
Because we had different sources of truth which were written in-house, we wound up rolling our own template engine in Python. It took about 3 weeks to write the engine and adapt existing templates. Given a circuit ID, it generates the full config for copy and paste into a terminal session. It also hooks into a configuration parser tool, written in-house, that tracks configured interfaces, so it is easy to see whether the template would overwrite an existing interface.
I used the Jinja2 template engine, along with pyodbc/unixODBC/FreeTDS for access to a Microsoft SQL backend.
The keys for us are:
* extracting information from a source of truth
* validating the information for correctness
* making sure you don't overwrite existing config
* outputting the right templates for the circuit features
It made more sense to write a tool than it did to try to adapt something for our environment.
If I had a free hand and unlimited budget, I would find a single app that functions as a source of truth for all circuits and products, which includes a templating engine that hooks in easily.
-Brian
---- On Tue, 06 Jun 2017 08:22:59 -0500 Graham Johnston & lt;johnstong@westmancom.com> wrote ----
Short of complete SDN, for those of you that have some degree of configuration templating and/or automation tools what is it that you run? I'm envisioning some sort of tool that let's me define template snippets of configuration and aids in their deployment to devices. I'm okay doing the heaving lifting in defining everything, I'm just looking for the tool that stitches it together and hopefully makes things a little less error prone for those who aren't as adept.
Graham Johnston
Network Planner
Westman Communications Group
204.717.2829
johnstong@westmancom.com<mailto:johnstong@westmancom.com>
Job, Would you be able to provide any further insight into your Don’t #5 – “Don’t agree to change management. Managers are rarely engineers and should not be making technical decisions. (nor should sales)“. Thanks, Graham From: Job Snijders [mailto:job@ntt.net] Sent: Tuesday, June 6, 2017 4:03 PM To: Brian Knight <ml@knight-networks.com>; Graham Johnston <johnstong@westmancom.com> Cc: nanog@nanog.org Subject: Re: Templating/automating configuration Hi, Here are some extra pointers: https://youtube.com/watch?v=C7pkab8n7ys https://www.nanog.org/sites/default/files/dosdontsnetworkautomation.pdf https://github.com/coloclue/kees Kind regards, Job On Tue, 6 Jun 2017 at 13:49, Brian Knight <ml@knight-networks.com<mailto:ml@knight-networks.com>> wrote: Because we had different sources of truth which were written in-house, we wound up rolling our own template engine in Python. It took about 3 weeks to write the engine and adapt existing templates. Given a circuit ID, it generates the full config for copy and paste into a terminal session. It also hooks into a configuration parser tool, written in-house, that tracks configured interfaces, so it is easy to see whether the template would overwrite an existing interface. I used the Jinja2 template engine, along with pyodbc/unixODBC/FreeTDS for access to a Microsoft SQL backend. The keys for us are: * extracting information from a source of truth * validating the information for correctness * making sure you don't overwrite existing config * outputting the right templates for the circuit features It made more sense to write a tool than it did to try to adapt something for our environment. If I had a free hand and unlimited budget, I would find a single app that functions as a source of truth for all circuits and products, which includes a templating engine that hooks in easily. -Brian ---- On Tue, 06 Jun 2017 08:22:59 -0500 Graham Johnston <johnstong@westmancom.com<mailto:lt%3Bjohnstong@westmancom.com>> wrote ---- Short of complete SDN, for those of you that have some degree of configuration templating and/or automation tools what is it that you run? I'm envisioning some sort of tool that let's me define template snippets of configuration and aids in their deployment to devices. I'm okay doing the heaving lifting in defining everything, I'm just looking for the tool that stitches it together and hopefully makes things a little less error prone for those who aren't as adept. Graham Johnston Network Planner Westman Communications Group 204.717.2829 johnstong@westmancom.com<mailto:johnstong@westmancom.com><mailto:johnstong@westmancom.com<mailto:johnstong@westmancom.com>>
Hi Graham, The talk was giving in context of motivating people to start with network automation and help them go from 'no automation' to a step further 'some automation'. On Wed, Jun 14, 2017 at 07:50:05PM +0000, Graham Johnston wrote:
Would you be able to provide any further insight into your Don’t #5 – “Don’t agree to change management.
I think the development team of the network automation software should define their own process around change management. If you want to use kanban? great! if you want to use simple fifo model applied to issues filed on your private github project? great! My point was: don't let someone from higher up dictate how, and when you do software releases. Another aspect is that you most likely will have proceses that should run without any human intervention: such as the nightly update for all EBGP prefix-filters. You don't want to end up in a situation where a computer generates those configs and has to hand them over to a human for some additional checks and subsequently pushing it out to the network. Imagine having the computer print out your automatically generated configs, a human pick them up, review them, and type them back into a computer for the changes to take effect! That would be terrible.
Managers are rarely engineers and should not be making technical decisions. (nor should sales)“.
That was a simple point: ideally a manager enables you to do your work, and trusts you to do the work. If you have a manager who opinionatedly argues with you on tabs vs spaces or how to push a configuration to a device, you might find that you don't have enough freedom of movement to succesfully bootstrap the automation project. In other words: don't roll over and blindly accept what other (inexperienced) folks within the organisation tell you, try to find your own path. However, do make sure you steal the good ideas from the sysadmins: they often are ahead of netops in terms of automation and understanding idempotency. Kind regards, Job
Graham Johnston wrote:
Would you be able to provide any further insight into your Don’t #5 – “Don’t agree to change management. Managers are rarely engineers and should not be making technical decisions. (nor should sales)“.
What do you think the purpose of change control / management is? Nick
On Wed, Jun 14, 2017 at 09:35:59PM +0100, Nick Hilliard wrote:
Graham Johnston wrote:
Would you be able to provide any further insight into your Don’t #5 – “Don’t agree to change management. Managers are rarely engineers and should not be making technical decisions. (nor should sales)“.
What do you think the purpose of change control / management is?
On Wed, Jun 14, 2017 at 3:55 PM, Job Snijders <job@ntt.net> wrote:
On Wed, Jun 14, 2017 at 09:35:59PM +0100, Nick Hilliard wrote:
Graham Johnston wrote:
Would you be able to provide any further insight into your Don’t #5 – “Don’t agree to change management. Managers are rarely engineers and should not be making technical decisions. (nor should sales)“.
What do you think the purpose of change control / management is?
On Wed, Jun 14, 2017 at 09:35:59PM +0100, Nick Hilliard wrote:
What do you think the purpose of change control / management is?
Bureaucratic change control implementations using the ITIL view of change control with a formal CAB are likely an (over)reaction to human mistakes causing outages, most of which could probably be avoided with a simpler less-formal process such as peer or team review. Change control functions as a risk transfer away from operations teams to CAB board members, since if things go wrong b/c of a change: it is now the CAB's fault. There may also be bias towards change-aversity if the CAB cannot be held accountable for issues that come from delaying or rejecting important maintenance. Overall purpose for change control / management, when applied to substantial modifications to an operating environment or configuration of business-critical network/applications is To mitigate possibility of damage/outages from high-impact / high-risk changes made by humans to systems and network-devices by requiring standards of formal written documentation and planning, combined with peer review And approvals by business and technical stakeholders for the maintenance time, including evaluation of exact proposed configuration changes, implementation plans, and backout/contingency plan: for possible errors or omissions. But as with most things can be taken to an unreasonable extreme. The use of change management procedures has a high associated cost, b/c the time and labor to implement even simple relatively low-risk changes can be dramatically increased with an unreasonable delay, and extensive test labs may be necessary. There may actually be increases in various risks, if any kind of maintenance is delayed or lost in the paperwork. -- -JH
On Wed, 14 Jun 2017 21:35:59 +0100, Nick Hilliard <nick@foobar.org> may have written:
What do you think the purpose of change control / management is?
To provide employment for change managers of course. -- Mike Meredith, University of Portsmouth Chief Systems Engineer, Hostmaster, Security, and Timelord!
Hi Brian, On Tuesday, 6 June, 2017 21:48, "Brian Knight" <ml@knight-networks.com> said:
Because we had different sources of truth which were written in-house, we wound up rolling our own template engine in Python. It took about 3 weeks to write the engine and adapt existing templates. Given a circuit ID, it generates the full config for copy and paste into a terminal session. It also hooks into a configuration parser tool, written in-house, that tracks configured interfaces, so it is easy to see whether the template would overwrite an existing interface.
Interesting. I'm going through much the same process at the moment, due to similar requirements - multiple sources of truth, validation that there's no clash with existing configs, but also with a requirement for network-wide atomic operations. The latter has been a strong driver for a custom tool - it's now grabbing an exclusive lock on all the devices, making all the checks, pushing all the config, commit check everywhere, commit everywhere, and only once all the commits succeed, release the locks. If any of those steps fail anywhere, we get to roll back everywhere. (Obviously with appropriate timeouts / back-offs / deadlock prevention, and specific to platforms with sane config management - no vanilla IOS). Did you find anything to give you a leg-up on config parsing, or did you have to do that completely from scratch? At the moment, I'm working with PyEZ (I know, vendor lock-in, but we're firmly a Juniper shop, and going in eyes-open to the lock-in) to build a limited model of just the parts of the config I'm interested in validating, and it seems to be working.
If I had a free hand and unlimited budget, I would find a single app that functions as a source of truth for all circuits and products, which includes a templating engine that hooks in easily.
Plus the business buy-in and the resource to go back and standardise all the existing configs, so the application can fully parse and understand the network before it starts. That, and a pony :) Regards, Tim.
---- On Wed, 07 Jun 2017 04:23:33 -0500 <tim@pelican.org> wrote ---- Hi Brian, On Tuesday, 6 June, 2017 21:48, "Brian Knight" <ml@knight-networks.com> said: > Because we had different sources of truth which were written in-house, we wound up > rolling our own template engine in Python. It took about 3 weeks to write the > engine and adapt existing templates. Given a circuit ID, it generates the full > config for copy and paste into a terminal session. It also hooks into a > configuration parser tool, written in-house, that tracks configured interfaces, so > it is easy to see whether the template would overwrite an existing interface. Interesting. I'm going through much the same process at the moment, due to similar requirements - multiple sources of truth, validation that there's no clash with existing configs, but also with a requirement for network-wide atomic operations. The latter has been a strong driver for a custom tool - it's now grabbing an exclusive lock on all the devices, making all the checks, pushing all the config, commit check everywhere, commit everywhere, and only once all the commits succeed, release the locks. If any of those steps fail anywhere, we get to roll back everywhere. (Obviously with appropriate timeouts / back-offs / deadlock prevention, and specific to platforms with sane config management - no vanilla IOS). Did you find anything to give you a leg-up on config parsing, or did you have to do that completely from scratch? At the moment, I'm working with PyEZ (I know, vendor lock-in, but we're firmly a Juniper shop, and going in eyes-open to the lock-in) to build a limited model of just the parts of the config I'm interested in validating, and it seems to be working. The import process to the database runs directly on our rancid server, reading the downloaded configs out of the appropriate directory within rancid. Most of our gear is Cisco, so the ciscoconfparse module for Python helps a lot with organizing and querying the config. From there, the config is parsed for key items like interface name, description, configured bandwidth, etc., and that info is then added or updated as necessary in the database. Because it's dependent on rancid, there is some lag time between when a change is made and when the database gets updated, so we still strongly encourage running the pre-config checks for new circuits. But with PyEZ, it looks like you easily could, after grabbing that lock, validate the existing config before pushing down new config. Lots of possibilities there. I'm envious that you have a vendor-written Python module as a choice! > If I had a free hand and unlimited budget, I would find a single app that > functions as a source of truth for all circuits and products, which includes a > templating engine that hooks in easily. Plus the business buy-in and the resource to go back and standardise all the existing configs, so the application can fully parse and understand the network before it starts. That, and a pony :) Or, at least, rebuild the existing configs based on the new source of truth, so that subsequent config parsing conforms to a single standard. Regards, Tim. Cheers, -Brian
On 7 June 2017 at 19:52, Brian Knight <ml@knight-networks.com> wrote:
The import process to the database runs directly on our rancid server, reading the downloaded configs out of the appropriate directory within rancid. Most of our gear is Cisco, so the ciscoconfparse module for Python helps a lot with organizing and querying the config. From there, the config is parsed for key items like interface name, description, configured bandwidth, etc., and that info is then added or updated as necessary in the database.
Because it's dependent on rancid, there is some lag time between when a change is made and when the database gets updated, so we still strongly encourage running the pre-config checks for new circuits. But with PyEZ, it looks like you easily could, after grabbing that lock, validate the existing config before pushing down new config. Lots of possibilities there. I'm envious that you have a vendor-written Python module as a choice! ...
Or, at least, rebuild the existing configs based on the new source of truth, so that subsequent config parsing conforms to a single standard.
Cisco IOS and IOS-XE have config locks on the CLI, as well as automatic configuration rollbacks and the ability to generate a config diff on deice. For some reason lots of people seem to forget/ignore this. If you are using NAPALM then I believe you can also implement this through NAPALM. Cheers, James.
participants (9)
-
'Job Snijders' -
Brian Knight -
Graham Johnston -
James Bensley -
Jimmy Hess -
Job Snijders
-
Mike Meredith -
Nick Hilliard -
tim＠pelican.org