Linux Centralized Administration
Hey folks. just curious what people are using for automating updates to Linux boxes? Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network? Cheers, Paul
On Thu, 12 Jan 2012 16:02:49 EST, Paul Stewart said:
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
You can configure yum-updatesd to download and/or apply new updates automagically. Whether that's a good idea is a different question.
On Thu, Jan 12, 2012 at 04:02:49PM -0500, Paul Stewart wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
yum
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
yum install yum-cron chkconfig yum-cron on service yum-cron start
We are using Security Blanket. It's a COTs product that works really well.... -----Original Message----- From: Chuck Anderson [mailto:cra@WPI.EDU] Sent: Thursday, January 12, 2012 4:10 PM To: nanog@nanog.org Subject: Re: Linux Centralized Administration On Thu, Jan 12, 2012 at 04:02:49PM -0500, Paul Stewart wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
yum
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
yum install yum-cron chkconfig yum-cron on service yum-cron start
On 12 January 2012 21:02, Paul Stewart <paul@paulstewart.org> wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
It so happens that just yesterday I stumbled across Spacewalk (http://spacewalk.redhat.com) - which is the open source version of RHN Satellite. I ran into a few problems setting the server up - but nothing too difficult to solve, and client installation is a breeze. Dan
On Thu, Jan 12, 2012 at 1:02 PM, Paul Stewart <paul@paulstewart.org> wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
There's no tool I could recommend that would be very close to RHN. However, for solving the problem of keeping packages up to date and systems in a known-state, I would recommend checking out some configuration management tools. There are several popular ones nowadays, though I personally prefer Puppet or Chef. Both are tools that allow administrators to declare what a system should look like, and abstract away the hard work of making that happen on a variety of platforms. In both cases, it's possible to monitor how well those tools are working and what they're doing in the background so that you can get an idea of what's up to date and what's not. Are you just trying to solve for making sure that packages are up to date? Making sure that running daemons are also up to date? Cheers, jof
We use puppet - http://puppetlabs.com/. Works good for us. Nitin ----- Original Message ----- From: "Paul Stewart" <paul@paulstewart.org> To: nanog@nanog.org Sent: Thursday, January 12, 2012 4:02:49 PM Subject: Linux Centralized Administration Hey folks. just curious what people are using for automating updates to Linux boxes? Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network? Cheers, Paul
We use SALT, written in python and setup in 10 minutes. Seriously easy! Wickedly fast! http://saltstack.org/ -Bret On Jan 12, 2012, at 2:13 PM, Nitin Mehrotra wrote:
We use puppet - http://puppetlabs.com/.
Works good for us.
Nitin
----- Original Message ----- From: "Paul Stewart" <paul@paulstewart.org> To: nanog@nanog.org Sent: Thursday, January 12, 2012 4:02:49 PM Subject: Linux Centralized Administration
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Cheers,
Paul
Fabric is also a fine one, if you *don't* want abstraction of what you're doing: http://fabfile.org On Thu, Jan 12, 2012 at 1:16 PM, Bret Palsson <bret@getjive.com> wrote:
We use SALT, written in python and setup in 10 minutes. Seriously easy! Wickedly fast! http://saltstack.org/
-Bret On Jan 12, 2012, at 2:13 PM, Nitin Mehrotra wrote:
We use puppet - http://puppetlabs.com/.
Works good for us.
Nitin
----- Original Message ----- From: "Paul Stewart" <paul@paulstewart.org> To: nanog@nanog.org Sent: Thursday, January 12, 2012 4:02:49 PM Subject: Linux Centralized Administration
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Cheers,
Paul
I run spacewalk (as mentioned above), and have for some time. Once you get the errata importing set up, it's pretty much full RHN. -Blake
On Thu, Jan 12, 2012 at 04:02:49PM -0500, Paul Stewart wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
At work, we use (and built) a tool called 'tingle' (https://github.com/anchor/tingle), which handles it all for us across our internal and managed-for-customers infrastructures. Personally, I don't run CentOS, but I use unattended-upgrades on my personal herd of Debian machines, which works well enough. - Matt -- A woman in liquor production / Owns a still of exquisite construction. The alcohol boils / Through magnetic coils. She says that it's "proof by induction." -- http://limerickdb.com/?34
Here at Twitter we make extensive use of Puppet. It's great, but we had a hard learning curve and much customization to get it to work the way we wanted to. I'd also recommend Chef, which is like Puppet but includes more tools (like a machine database) out of the box. -j On Thu, Jan 12, 2012 at 2:27 PM, Matthew Palmer <mpalmer@hezmatt.org> wrote:
On Thu, Jan 12, 2012 at 04:02:49PM -0500, Paul Stewart wrote:
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
At work, we use (and built) a tool called 'tingle' (https://github.com/anchor/tingle), which handles it all for us across our internal and managed-for-customers infrastructures.
Personally, I don't run CentOS, but I use unattended-upgrades on my personal herd of Debian machines, which works well enough.
- Matt
-- A woman in liquor production / Owns a still of exquisite construction. The alcohol boils / Through magnetic coils. She says that it's "proof by induction." -- http://limerickdb.com/?34
On Thu, Jan 12, 2012 at 3:02 PM, Paul Stewart <paul@paulstewart.org> wrote:
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Something to think about before attempting to centrally manage, your systems actually have to be centrally manageable -- that doesn't happen automatically and requires extra work. The just run yum update strategy is only reliable when all packages on the system were installed from RPM and all software RPMs installed are properly maintained by the vendor using Yum. Some packages have updates that are distributed with Yum, but yum updating "breaks" the application, until a manual update procedure is completed. Sometimes an updated kernel won't boot. Sometimes, a third-party driver for RAID card X won't load in the patched kernel, and after a reboot, the OS never comes back up because it's sitting at a kernel panic message indicating no hard drive found. Cacti/OpenNMS are good examples -- after a yum update to a new version, you must manually invoke, a potentially dangerous "installer" program or web page has to be used, after a new update, config files, or database schema have to be edited or patched by hand; until you manually take some action to "fix" the config, the application is broken after update. As soon as you attempt to restart the application it will shutdown OK, but not come back up. Occassionally, there is a library update that breaks binary compatibility with existing applications, for example a certain update to net-snmp-libs in Centos 5.something. yum-updatesd surely doesn't know when auto-applying an update will cause an important service to suddenly break To centrally manage effectively, you basically need a homogenous environment with a configuration that is very close to stock config, so that effective testing is possible; homogenous meaning an identical list of installed packages and software all installed the same way on every system centrally managed as a group, identical SKUs for every hardware component in every installation configured identically, same hw revisions, etc. No "extra" applications or files floating around on a one-off server. So yum-updatesd would be a bad idea for production systems that have any third-party packages; even if YUM maintained. And even if YUM maintained, third party YUM repos may become neglected, or change into 404 errors, causing yum to break entirely. Often commercial third-party software used on CentOS systems will be distributed in another format, such as .tar.gz. Yum cannot do much with that; the third party package will likely get neglected and not updated. Often various applications you require may need versions of libraries or applications that are not yet available in RPM format, or they're part of Fedora instead. In any case, if you wind up rebuilding the RPM for CentOS using rpmbuild or installing from source, Yum update won't help you with those packages, and may break their dependencies later. That might just be a testament to how poor the available packaged software selections are in CentOS, that commonly needed packages aren't part of the distribution; and commonly outdated versions of libraries are present. But YUM-updatesd's usefulness certainly applies to less than 100% of systems. -- -JH
On 1/12/2012 4:43 PM, Jimmy Hess wrote:
On Thu, Jan 12, 2012 at 3:02 PM, Paul Stewart<paul@paulstewart.org> wrote:
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Something to think about before attempting to centrally manage, your systems actually have to be centrally manageable -- that doesn't happen automatically and requires extra work.
this is why i never update. i would rather build a new image and deploy it to the thousands of servers than worry about updates. be it an openssh security notice, or new ntp configuration, for me it is easier to rebuild servers than update config files.
On 01/12/2012 03:51 PM, chaim.rieger@gmail.com wrote:
On 1/12/2012 4:43 PM, Jimmy Hess wrote:
On Thu, Jan 12, 2012 at 3:02 PM, Paul Stewart<paul@paulstewart.org> wrote:
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Something to think about before attempting to centrally manage, your systems actually have to be centrally manageable -- that doesn't happen automatically and requires extra work.
this is why i never update. i would rather build a new image and deploy it to the thousands of servers than worry about updates. be it an openssh security notice, or new ntp configuration, for me it is easier to rebuild servers than update config files.
.. you never update? How frequently do you rebuild your entire server stack, weekly? Paul
On 01/12/2012 03:51 PM, chaim.rieger@gmail.com wrote:
On 1/12/2012 4:43 PM, Jimmy Hess wrote:
On Thu, Jan 12, 2012 at 3:02 PM, Paul Stewart<paul@paulstewart.org> wrote:
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
Something to think about before attempting to centrally manage, your systems actually have to be centrally manageable -- that doesn't happen automatically and requires extra work.
this is why i never update. i would rather build a new image and deploy it to the thousands of servers than worry about updates. be it an openssh security notice, or new ntp configuration, for me it is easier to rebuild servers than update config files.
For that matter, imaging is a bad way to go about handling this, you'd be better served by setting up something like Puppet or Chef and have them handle configuration management for you centrally, along with necessary software packages. Paul
On 13 January 2012 01:57, Paul Graydon <paul@paulgraydon.co.uk> wrote:
On 01/12/2012 03:51 PM, chaim.rieger@gmail.com wrote:
On 1/12/2012 4:43 PM, Jimmy Hess wrote:
Something to think about before attempting to centrally manage, your systems actually have to be centrally manageable -- that doesn't happen automatically and requires extra work.
this is why i never update. i would rather build a new image and deploy it to the thousands of servers than worry about updates. be it an openssh security notice, or new ntp configuration, for me it is easier to rebuild servers than update config files.
For that matter, imaging is a bad way to go about handling this, you'd be better served by setting up something like Puppet or Chef and have them handle configuration management for you centrally, along with necessary software packages.
Paul
I looked into Puppet and though I've got it managing parts of our infrastructure it seems quite difficult to bolt on to an existing setup. There are also some things that I can't see how to do easily with Puppet ("Don't upgrade packages on the live environment until we've tested them in staging" being a big one.) I'm starting to look at Blueprint (http://devstructure.com) to help build the Puppet manifests so that we can deploy Puppet without breaking any existing machines, Puppet for configuration management and Spacewalk to audit what is up-to-date and help schedule security updates. Dan
On Fri, 13 Jan 2012, Daniel Ankers wrote:
I looked into Puppet and though I've got it managing parts of our infrastructure it seems quite difficult to bolt on to an existing setup. There are also some things that I can't see how to do easily with Puppet ("Don't upgrade packages on the live environment until we've tested them in staging" being a big one.)
Has anyone mentioned cluster ssh yet? Depending on your scale, cluster ssh and a "really big screen" may be a suitable way to manage N servers and do things like apply updates or make identical changes to all at once (or in groups). It also gives you the flexibility to apply commands to all or single out a system and do things just in the one window, then to back to talking to all. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Hello, On Fri, 13 Jan 2012 12:42:30 -0500 (EST) Jon Lewis <jlewis@lewis.org> wrote:
On Fri, 13 Jan 2012, Daniel Ankers wrote:
I looked into Puppet and though I've got it managing parts of our infrastructure it seems quite difficult to bolt on to an existing setup. There are also some things that I can't see how to do easily with Puppet ("Don't upgrade packages on the live environment until we've tested them in staging" being a big one.)
Has anyone mentioned cluster ssh yet? Depending on your scale, cluster ssh and a "really big screen" may be a suitable way to manage N servers and do things like apply updates or make identical changes to all at once (or in groups). It also gives you the flexibility to apply commands to all or single out a system and do things just in the one window, then to back to talking to all.
Continuing that line of tools, I'm using parallel-ssh (http://code.google.com/p/parallel-ssh/) with great success for managing several hundred servers, spread all over the world. -- Best regards, Nickola Kolev
Sounds like a poorly designed package. Wordpress does a good job of allowing back end updates without impacting the services provided, even with database changes. Part of a well designed and maintained system is the ability to do painless upgrades. Jared Mauch On Jan 12, 2012, at 7:43 PM, Jimmy Hess <mysidia@gmail.com> wrote:
Cacti/OpenNMS are good examples -- after a yum update to a new version, you must manually invoke, a potentially dangerous "installer" program or web page has to be used, after a new update, config files, or database schema have to be edited or patched by hand; until you manually take some action to "fix" the config, the application is broken after update. As soon as you attempt to restart the application it will shutdown OK, but not come back up.
Hey folks. just curious what people are using for automating updates to Linux boxes?
Today, we manually do YUM updates to all the CentOS servers . just an example but a good one. I have heard there are some open source solutions similar to that of Red Hat Network?
We did create our own solution and are still expanding it. Currently we set what a server should look like at the servers, we want to change it to the central system. This would make it easier to deploy extra servers (only entering a MAC address, selecting software and starting a server should be enough to auto-deploy it). Our current solution is designed for Debian/Ubuntu, but should also work on other Linux distributions. A working copy might be available; please contact me offlist and I'll look what I can do. Kind regards, Mark
participants (19)
-
Blake Dunlap -
Bret Palsson -
chaim.rieger＠gmail.com -
Chuck Anderson -
Daniel Ankers -
Green, Timothy -
Jacob Taylor -
Jared Mauch -
Jimmy Hess -
John Adams -
Jon Lewis -
Jonathan Lassoff -
Mark Scholten -
Matthew Palmer -
Nickola Kolev -
Nitin Mehrotra -
Paul Graydon -
Paul Stewart -
Valdis.Kletnieks＠vt.edu