On Tue, 21 Oct 2014 14:44:57 +0900, Randy Bush said:

systemd is insanity. one would have hoped that deb and others would know better. sigh.

It started as a replacement init system. I suspected it had jumped the shark when it sprouted an entirely new DHCP and NTP service. And this was confirmed when I saw this: "Leading up to this has been cursor rendering support, keyboard mapping support, screen renderer, DRM back-end, input interface, and dozens of other commits." http://www.phoronix.com/scan.php?page=news_item&px=MTgwNzQ When your init system is worrying about cursor rendering, you have truly fallen victim to severe feature bloat. I guess Jamie Zawinski was right: "Every program attempts to expand until it can read mail."

I was actually not aware of this. I've been told that systemd also includes fsck's functionality (or is planning to?). That just seems absurd to me. I didn't really have a strong opinion on either side of this yet. Seeing the replies from other people here, though, and reading some more about it, this seems to be a very bad idea. The binary logs for example worry me, especially corruption issues: http://www.reddit.com/r/linux/comments/1y6q0l/systemds_binary_logs_and_corru... https://bbs.archlinux.org/viewtopic.php?id=169966 On 21-10-2014 14:40, Valdis.Kletnieks@vt.edu wrote:

...

systemd is insanity. one would have hoped that deb and others would know better. sigh. It started as a replacement init system. I suspected it had jumped

On Tue, 21 Oct 2014 14:44:57 +0900, Randy Bush said: the shark when it sprouted an entirely new DHCP and NTP service. And this was confirmed when I saw this:

"Leading up to this has been cursor rendering support, keyboard mapping support, screen renderer, DRM back-end, input interface, and dozens of other commits."

http://www.phoronix.com/scan.php?page=news_item&px=MTgwNzQ

When your init system is worrying about cursor rendering, you have truly fallen victim to severe feature bloat. I guess Jamie Zawinski was right: "Every program attempts to expand until it can read mail."

I have been working with developing systems that boot with Linux for a number of years on a multitude of distributions and I never saw a problem with the tools or the process. Purely the lack of standards. It seems stubborn at the least to propose an opaque software solution when a simple standards organization for how to structure the init system is all that is required. Why not write high level code to manage standardized scripts rather than replace them with binary darkness. The only reason we have desktop Linux is due to the flexibility of the Unix architecture, seems silly to abandon that now. The pure fact of it is that developers hate messing with text because its unpredictable and prone to bugs. However with standards and possibly a decent meta format (I look towards YML, XML, JSON) that can be consumed and produced with scripts there should be no issues. The final fact is that bash itself is a dirty language that developers hate and system administrators love. Its a gross blend of programming functionality mixed with command line awareness and its unpredictable, especially at the code generation level. Its also too sensitive to text formatting and line endings. In conclusion, we will continue to deploy our scripts to a number of Linux distributions and have came to the conclusion that it is simply cheaper to have a human deal with the actual deployment of the system rather than write host of deployment code to cover every system. End users know how to use their system and we rely on that. Finally, why not focus on creating and maintaining collaborative unbiased standards rather than parading egos and hurting communities. On Tue, Oct 21, 2014 at 1:55 PM, Owen DeLong <owen@delong.com> wrote:

Wait… Let me see if I understand this correctly…

1. Move fsck functionality into systemd 2. Have it generate opaque binary logs 3. If your filesystem is corrupted in a way that systems can’t repair, you can’t even read the logs of what systemd saw or did?

Yeah, that sounds like a very definite “bad thing”.

Owen

On Oct 21, 2014, at 10:17 AM, Israel G. Lugo <israel.lugo@lugosys.com> wrote:

...

I was actually not aware of this. I've been told that systemd also includes fsck's functionality (or is planning to?). That just seems absurd to me.

I didn't really have a strong opinion on either side of this yet. Seeing the replies from other people here, though, and reading some more about it, this seems to be a very bad idea.

The binary logs for example worry me, especially corruption issues:

...

https://bbs.archlinux.org/viewtopic.php?id=169966

On 21-10-2014 14:40, Valdis.Kletnieks@vt.edu wrote:

...

...

systemd is insanity. one would have hoped that deb and others would know better. sigh. It started as a replacement init system. I suspected it had jumped

On Tue, 21 Oct 2014 14:44:57 +0900, Randy Bush said: the shark when it sprouted an entirely new DHCP and NTP service. And

http://www.reddit.com/r/linux/comments/1y6q0l/systemds_binary_logs_and_corru... this

...

...

was confirmed when I saw this:

"Leading up to this has been cursor rendering support, keyboard mapping support, screen renderer, DRM back-end, input interface, and dozens of other commits."

http://www.phoronix.com/scan.php?page=news_item&px=MTgwNzQ

When your init system is worrying about cursor rendering, you have truly fallen victim to severe feature bloat. I guess Jamie Zawinski was right: "Every program attempts to expand until it can read mail."

-- eSited LLC (701) 390-9638

On October 22, 2014 at 11:36 jamie.s.bowden@raytheon.com (Jamie Bowden) wrote:

...

From: Bryan Tong

...

The final fact is that bash itself is a dirty language that developers hate and system administrators love.

Excuse me? I've been administering systems for over twenty years now and I can't say that I've ever even once chosen to use bash over any alternative; no matter how much that alternative might suck, bash sucks more. Your Linux addicts who've never used another flavor of Unix may be addicted to bash, but there's no helping some people.

I wish I had a nickel for every time I started to implement something in bash/sh, used it a while, and quickly realized I needed something like perl and had to rewrite the whole thing. Sure, one can insist on charging forward in sh but at some point it becomes, as Ken Thompson so eloquently put it on another topic entirely, like kicking a dead whale down the beach. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

This is a good point, but it is perfectly possible to have a sysvinit system not written in shell scripts. I had rewritten most of the init system in python at one point for example. And its only been made easier to do over time now that #! Interpertation was moved kernelside. A system like this could solve all the problems of both sides since it is now a proper programming language so we can do more config style initscripts without the blob that is systemd. And for parallel init, you can do it in bash, iirc Gentoo does parallel init in bash. On Oct 23, 2014 1:45 PM, "Lamar Owen" <lowen@pari.edu> wrote:

On 10/22/2014 03:51 PM, Barry Shein wrote:

...

I wish I had a nickel for every time I started to implement something in bash/sh, used it a while, and quickly realized I needed something like perl and had to rewrite the whole thing.

Barry, you've been around a long time, and these words are pearls of wisdom.

This seems to be the reasoning behind replacing the spaghetti known as 'initscripts' currently written in sh with something written in a real programming language. Upstart and systemd are both responses to the inflexible spaghetti now lurking in the initscript system, of which the pile steaming in /etc/init.d is but a part.

...

becomes, as Ken Thompson so eloquently put it on another topic entirely, like kicking a dead whale down the beach.

This seems to be the attitude of the systemd developers, although

Sure, one can insist on charging forward in sh but at some point it they're not as eloquent as Thompson in saying it. But I remember being just as abrasive, just on a smaller scale, myself.......

Now, I've read the arguments, and I am squarely in the 'do one thing and do it well' camp. But, let's turn that on its head, shall we? Why oh why do we want every single package to implement its own initscript and possibly do it poorly? Wouldn't it be a case of 'do one thing well' if one could do away with executable shell code in each individual package that is going to run as root? Wouldn't it be more 'do one thing well' if you had a 'super' inetd setup that can start services in a better way than with individually packaged (by different packagers in most cases) shell scripts that are going to run as root? Shouldn't the initialization logic, which is 'one thing that needs doing' be in one container and not thousands?

Now, I say that having been a packager for a largish RPM package several years ago. I waded the morass of the various packages' initscripts; each packager was responsible for their own script, and it was a big mess with initscripts doing potentially dangerous things (mine included; to clear it up, I maintained the PostgreSQL packages for the PostgreSQL upstream project from 1999 to 2004). Ever since 1999 there have been issues with distributed initialization logic (that runs as *root* no less) under hundreds of different packagers' control. It was and is a kludge of the kludgiest sort.

Having a single executable program interpret a thousand config files written by a hundred packagers is orders of magnitude better, security-wise, than having thousands of executable (as *root*) scripts written by hundreds of different packagers, in my experienced opinion. If anything, having all initialization executable code in one monolithic package very closely monitored by several developers (and, well, for this purpose 'developers with attitudes' might not be the worst thing in the world) is more secure. It *is* a smaller attack surface than the feeping creaturism found in the typical /etc/init.d directory. And Barry's pearl of wisdom above most definitely applies to /etc/rc.sysinit and its cousin /etc/rc.local.

Now, as much as I dislike this magnitude of change, it seems to me that systemd actually is more inline with the traditional Unix philosophy than the current initialization system is. But I always reserve the right to be wrong. And I am definitely not fond of the attitudes of the various systemd developers; systemd assuredly has its shortcomings. But it *is* here to stay, at least in RHEL-land, for at least the next ten years.

Having said that, if you want to use Upstart, by all means use Upstart; RHEL6 (and rebuilds) will have Upstart until 2020. So you're covered for quite a while yet, if you use CentOS, Scientific Linux, or another RHEL6 rebuild (or actual RHEL6).

And for those who bugle that systemd will be the 'end of unix as we know it' I just have one thing to trumpet:

"Death of Internet Predicted. Film at Eleven."

On 10/23/2014 02:22 PM, Valdis.Kletnieks@vt.edu wrote:

On Thu, 23 Oct 2014 13:43:03 -0400, Lamar Owen said:

...

Now, I've read the arguments, and I am squarely in the 'do one thing and do it well' camp. But, let's turn that on its head, shall we? Why oh why do we want every single package to implement its own initscript and possibly do it poorly? Umm.. because maybe, just maybe, the package maintainers know more about the ugly details of what it takes to start a given package than the init system authors know?

Speaking from my own experience, the actually relevant and package-specific guts of the typical initscript could be easily replaced by a simple text configuration that simply gives: 1.) What to start 2.) When to start it (traditional initscripts work on a linear timeline of priority slots; systemd units have more flexibility) 3.) How to start it (command line options) This should not need to be an executable script. This is what systemd brings to the table (Upstart brought some of this, too). It allows the packager to declare those details that the packager knows about the package and eliminates the boilerplate (that is different between versions of the same distribution; I for one maintained initscripts across multiple versions of multiple distributions, all of which had different boilerplate and different syntax). I should not have needed to learn all the different boilerplate, as that was a distraction from the real meat of packaging (it could be argued that the presence of syntactically arcane boilerplate is a problem in and of itself: as a for instance, the nice 'daemon' function most RPM-based distributions supply in /etc/init.d/functions works for some initscripts and not for others; PostgreSQL is one for which it doesn't work and it's not obvious at first glance why it doesn't); I should simply have been able to tell the init system in a declarative syntax that I needed to start the program '/usr/bin/postmaster' with the command line options for database directory and listening port (among some other items). And that includes 99% of what the various initscripts do (yeah, the PostgreSQL script of which I was one author did one thing that in hindsight should simply not have been in the initscript at all). Many of the 1% exceptions perhaps don't belong in code that is run as root every single time that particular daemon needs to start or stop. The perhaps 0.5% remaining that absolutely must be run before starting or stopping, well, yes, there should be an option in that declarative syntax to say, for instance, 'before starting /usr/bin/postmaster, check the version of the database and fail if it's older with a message to the log telling the admin they need to DUMP/RESTORE the database prior to trying to start again' ...... (the systemd syntax does allow this). I have personally compared the current PostgreSQL systemd unit (/usr/lib/systemd/system/postgresql.service on my CentOS 7 system) to the old initscript. I wish it had been that simple years ago; the .service file is much cleaner, clearer, and easier to understand; no funky shell quote escapes needed. And it doesn't execute as root, nor does it source arcane boilerplate functions, the source of some of which will make your eyes bleed. And to do a customized version you just drop your edited copy in /etc/systemd/system/ and you're done, since it won't get overwritten when you update packages. When configuring Cisco IOS, we use a declarative syntax; the systemd .service file's syntax is as readable as the typical startup-confg is. Imagine if we used the typical bash initscript syntax to bring up interfaces and services on our routers.

On 10/23/2014 10:43 AM, Lamar Owen wrote:

Wouldn't it be more 'do one thing well' if you had a 'super' inetd setup that can start services in a better way than with individually packaged (by different packagers in most cases) shell scripts that are going to run as root?

inetd versus xinetd -- I thought that was an excellent change. It was closely targeted, well documented, and did indeed do one thing well: accept connections and pass them off to a handler. I'm not so sure about systemd yet. I've not tried to create a daemon of my own to run under the setup. I have tried to use upstart, and found the execution of upstart (CentOS 6) didn't match the documentation. Further, there was no "here's how you go from sysvinit to upstart" that made sense. I did see that systemd *does* have a paper for people needing to move from the old sysvinit to systemd, which is a plus on the surface. I'll be trying that in the future.

I pled the Linux people to stay inside the unix philosophy to use text files. Low newbies like me learn from reading config files, and fix thing by reading log files, tryiing to make some sense of the error messages there, and using the most suspicious line as the handle to google for a solution (that is often some stackoverflow article, or some forum posts). I dismay after the idea of somebody replacing all that text by a binary that spouts "the service stoped running" or that corrupt, because some buffer was not flush when the kerfukle happened. Even if going to binary gives a extra 20% speed, I think speed is important but not that important. I plead save the discoverability, learn-bility, debug-ness of text (even text scripts) over mysterious binary blobs elfs generating mysterious binary blobs journals. If they nerf text files, is like they nerf Google for me, and my ability to maintain and configure systems. -- -- ℱin del ℳensaje.

On Fri, Oct 24, 2014 at 12:41:39AM -0400, Barry Shein wrote:

All those init.d scripts do about 95% the same thing, all hacked together in shell. Most of them are probably just slightly edited versions of some few paleo-scripts.

in FreeBSD, the bulk of the rc.d scripts are basically the same format, inhaling a standardized library of functions, populating some variables, maybe adding a few custom functions, then jumping to "main". all largely driven off variables which have preset defaults in /etc/default/rc.conf, and over-ridden by /etc/rc.conf. something i saw in various linux systems, which i now see moving into FreeBSD, is the concept of the .d config directory, where the files in /etc/myapp.d are effectively concatenated to /etc/myapp.conf. this is quite nice from a sysadmin perspective, as you can set defaults in /etc/myapp.conf, and then over-ride them with server/environmental specific (and appropriately named) files in /etc/myapp.d/.... --jim -- Jim Mercer Reptilian Research jim@reptiles.org +1 416 410-5633 "He who dies with the most toys is nonetheless dead"

On 10/25/2014 08:12 AM, Jeffrey Ollie wrote:

If all of the scripts are cut'n'paste copes of each other, wouldn't it be better to figure out a way to stop cutting and pasting? I can't count the number of times I've run into problems with my code because of that, never mind how many times it's happened in other people's code.

I suggest that the corruption and rot was introduced when we left /etc/inittab as the sole way to run permanent daemons. We had telinit(8) already to change run levels, so expanding on telinit would have been more sensible -- telinit itself to change temporarily things temporarily (start, stop, restart), and something that modified the inittab (editor? script? GUI?) to make changes that last across boots. The first change would be to expand the daemon identifier from two characters to proper labels...but I digress. The whole rc script thing strikes me as an interim solution that required a minimum of code changes (graduate student project?) that went viral. Bad as it was, it worked. Duct tape and bailing wire.... Why did we have copy-and-paste for so long? The answer is in the form of another question: who was willing to take the time to re-factor the Sys-V way of doing things? Was "re-factor" even in the vocabulary in those days? Systemd is not a re-factoring. It's a from-scratch do-over. What it does that is good is that it provides dependency capability not available in the original inittab. It makes dependency resolution named-based (named semaphores) instead of number-based, which is what the Sys-V method used. The result is far more parallelism in system start-up than afforded by ethier of the previous methods. (I can't speak to Upstart -- I've never really understood it, or how to go from SysV to Upstart. I even filed a bug on this last point against Upstart, and saw not even "you stupid ****". Will it die?) Much of the heartburn I've been observing isn't about core systemd itself, or the goals of systemd itself, but about the culture that had grown around this replacement for the gaggle of shell scripts and the one-line-per-daemon table. Like a gun, the weapon doesn't kill people; it's the operator behind the trigger who is aiming the business end in bad directions. Much of the belly-aching has been about ripple effects that are tangential to systemd itself -- it's unfortunate that these tangents are tied to systemd, it's been said time and time again "you don't like it, you can do something about it." It means you can't use the entire package "out of the box" without changing more than a set of configuration files. My particular bone is NTP substitution -- I'd like to use my cesium clock on my Stratum 1 time server without having to spend an afternoon trying to hork everything together. I'd like for my Stratum II time server to work "out of the box" with the time servers *I* select, using the University of Delaware software. Oh, and I hate binary logs. Period. If you can't stand plain text, then try XML. At least humans have a *chance* to read it without having to make fancy reader tools.

On Sat, Oct 25, 2014 at 1:55 PM, Matthew Petach <mpetach@netflight.com> wrote:

Why can't systemd have a --text flag to tell it to output in ascii text mode for those of us who prefer it that way?

It does. --no-pager -- Pete

On Sat, Oct 25, 2014 at 01:55:43PM -0700, Matthew Petach wrote:

On Sat, Oct 25, 2014 at 10:22 AM, Stephen Satchell <list@satchell.net> wrote:

...

Oh, and I hate binary logs. Period. If you can't stand plain text, then try XML. At least humans have a *chance* to read it without having to make fancy reader tools.

Completely agree on this point--but I fail to see why it has to be one or the other? Why can't systemd have a --text flag to tell it to output in ascii text mode for those of us who prefer it that way?

Because Lennart knows better than you. - Matt -- <Igloo> I remember going to my first tutorial in room 404. I was most upset when I found it.

On Sat, Oct 25, 2014 at 12:22 PM, Stephen Satchell <list@satchell.net> wrote:

The whole rc script thing strikes me as an interim solution that required a minimum of code changes (graduate student project?) that went viral. Bad as it was, it worked. Duct tape and bailing wire.... [snip] Systemd is not a re-factoring. It's a from-scratch do-over. What it does that is good is that it provides dependency capability not available in the original inittab. It makes dependency resolution [snip]

The trouble is not that systemd is a re-factoring or that it is a do-over. The problem is that the scope of the systemd project is way too large and is ever-expanding! The next thing you know, SystemD will add package management, ISO building, and eliminate the need for Debian, Ubuntu, SuSE, Redhat, Etc to even exist. In the extreme case, at that point, we can rename GNU/Linux to GNU/SystemD, because hey, the Linux kernel is really just a little wrapper around the hardware to support the SystemD userland. The introduction of dependency support is solving issues with SysV init that are problems; I will give you that, but copy and paste init scripts and sequence-based dependencies are largely an aesthetic issue. SysV Init also has the advantage of more deterministic system startup behavior. What do you think happens when you have SystemD, but one of the critical packages has a service dependency incorrectly defined? If the scope of SystemD were appropriately constrained, it would not be also trying to replace the standard SYSLOG facilities with a program-specific logging format for everything. I'm not saying that Syslog is great; perhaps there should be new binary logging formats, or Libc built-in logging to a RDBMs, Redis database, or ElasticSearch cluster, but a distribution's choice of INIT program should not be forcing a choice on you in itself. Also.... since there are NTP daemons, DHCP, etc, all shipping with SystemD, chances are using something different will be along the path of greatest resistance. If history will be any guide; having all these extra services under the same package init, will likely mean that the maintenance will leave much to be desired ---- and you will be forced to upgrade other things and probably a reboot just to get a bug fix or security patch for your NTP client daemon. It doesn't really matter if they are not all running as PID #1. The problem is really that these services have been lumped into the scope of the same project. Proper integration into a software system does not mean expanding your scope duplicating other programs' functionality into your program, while introducing your own quirks. -- -JH

All those init.d scripts do about 95% the same thing, all hacked together in shell. Most of them are probably just slightly edited versions of some few paleo-scripts.

Set the location of the pid file, set the path of the executable, set the command line flags/options, maybe change some flags/options based on some options in another file like /etc/sysconfig/daemon_name (also shell commands which are just executed inline), then the start/stop/reload/restart/status case statements. And the dependencies of course.

It really could just be config files like xinetd or logrotate except for a few hard cases where you could have a "run this script" attribute.

Replacing "run these scripts in the right order" with a config-driven "service manager" sounds like a sensible development. (Not necessarily the One True Way, but certainly an option). Pulling complicated things like chroot, capabilities etc into one place, getting them right, and then letting services specify what they want, rather than everyone having to re-invent the same shell scripts sounds like it would encourage use of those features. I can even see some more advanced functionality to specify checks / frequencies for "is this service still running / alive" that effectively moves a lot of watchdog functionality into the "service manager". I'm somewhat confused (without reading the implementation details, just conceptually) as to why the "service manager" is also providing DHCP client, SNTP client, virtual consoles, session management... I can completely understand why the "do one thing" crowd are taking objection to that. Regards, Tim.

On 10/22/2014 04:04 AM, Rich Kulawiec wrote:

I've seen similar tactical mistakes when developers insist that information *must* be stored in a relational database -- even though plain old ordinary text files are perfectly adequate for the task, are easier to debug, are easier to fix, and easier to maintain.

Right on, bro. I'm in the middle of a "system" where the architect insists on just that. Never mind that I brought the relational database server to its knees (even flat on its back) because of the sheer number of processes I'm running, and the load on the engine that causes. My *ONLY* option was to replicate the information in the database and store it in files on the servers I'm running on -- semi-permanent data on the hard drives, working data in a RAM disk system. "But updates don't happen instantly, we have to wait for your replication daemon to run!" Yep. Amazing how that makes the entire system considerably more stable when you don't change horses mid-flight. Not to mention the considerable reduction in inter-server network traffic. How did this discussion get into NANOG? :)

On October 22, 2014 at 07:04 rsk@gsp.org (Rich Kulawiec) wrote:

I've seen similar tactical mistakes when developers insist that information *must* be stored in a relational database -- even though plain old ordinary text files are perfectly adequate for the task, are easier to debug, are easier to fix, and easier to maintain. There is an unfortunate tendency among many developers to attempt to wring the very last bit of performance out of systems and not to take into consideration that the scarcest and most expensive resource is the system administrator. Saving a few microseconds or a handful of bytes here and there is a horribly bad idea if it chews up an extra hour or week of SA time.

Obviously it depends on the application, generalities are dangerous. But one advantage of DBs are that you automatically get all the mechanics of failover, distribution, backup and recovery, atomicity, consistency, integrity, security, etc. that come with the DB essentially for "free". There is a tendency that one starts with this idea of keeping it simple, such as text files, and then proceeds to build all these mechanisms themselves, usually poorly. Look at how many different formats of configuration files we have on a typical *ix system, nearly one per application/daemon that needs a config file. Why do I have to know how to properly modify a passwd file, named config, logrotate, tcp wrappers, mail daemon configs, anti-spam configs, etc etc etc (usually in /etc!) down to what they will each take for a comment or separator or stanza syntax. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On Tue, Oct 21, 2014 at 3:41 PM, Eugeniu Patrascu <eugen@imacandi.net> wrote:

I think systemd wants to become the next Emacs ;))

Or the next user activity collection point. Systemd really is a black hole to 99.9% of the people who will use/deploy it... seems perfect for lots of things. -Jim P.

Why not write it in Emacs? Joe McGuckin ViaNet Communications joe@via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax On Oct 21, 2014, at 12:41 PM, Eugeniu Patrascu <eugen@imacandi.net> wrote:

On Tue, Oct 21, 2014 at 4:40 PM, <Valdis.Kletnieks@vt.edu> wrote:

...

http://www.phoronix.com/scan.php?page=news_item&px=MTgwNzQ

When your init system is worrying about cursor rendering, you have truly fallen victim to severe feature bloat. I guess Jamie Zawinski was right: "Every program attempts to expand until it can read mail."

I think systemd wants to become the next Emacs ;))

On Tue, Oct 21, 2014 at 07:20:12PM -0500, Jimmy Hess wrote:

On Tue, Oct 21, 2014 at 8:40 AM, <Valdis.Kletnieks@vt.edu> wrote: [snip]

...

It started as a replacement init system. I suspected it had jumped the shark when it sprouted an entirely new DHCP and NTP service. And this

Yikes. What's next? Built-in DNS server + LDAP/Hesiod + Kerberos + SMB/Active Directory client and server + Solitaire + Network Neighborhood functionality built into the program ?

You missed "font renderer". https://technet.microsoft.com/library/security/ms14-058 - Matt -- A friend is someone you can call to help you move. A best friend is someone you can call to help you move a body.

On 10/21/2014 05:20 PM, Jimmy Hess wrote:

The all-in-one approach of systemd might have a place on some specialized desktop distros, but outside that niche its' IMO a terrible idea.

The proper fix is probably a go back to Upstart or SysVInit and rewrite systemd, so all the pieces are separated and exist as a higher layer on top of init.

That is how systemd works, there are many parts and "systemd" is the sum of all those parts. It has a PID1 that replaces sysvinit/upstart, but afaik it doesn't do a whole lot extra. I don't use systemd, and I don't know a lot about it, but it seems lots of people don't get that it's not all lumped in PID1, there are a lot of processes that do different things that are mostly tightly held together (I think only udev and a couple other things still work without the rest of systemd.) Then again, the systemd people do spread FUD about sysvinit as well, Poettering's own blog for example even misleads on how systemd and sysvinit work http://0pointer.de/blog/projects/why.html A lot of things in the comparison chart have sysvinit listed as "no", when it's obviously not init's job directly, but a subprocess/script, except it lists "yes" for systemd on some, where systemd still passes it off to a subprocess! (They really are taking advantage of the PID1 and the entire bundle of software both being named "systemd" I guess.) [Insert obligatory "No I don't think sysvinit is perfect, but..." here] ps. What's with all the fear/hate of shell scripts? I realize the init scripts on most Linux distros are messy (200 LOC just to start sshd? Come on debian.) but the solution isn't to run away from them; rewrite scripts to have saner logic and not do a dozen redundant/unnecessary checks.

On Wed, Oct 22, 2014 at 12:57 PM, * <turmoil@privacyrequired.com> wrote:

Poettering's own blog for example even misleads on how systemd and sysvinit work http://0pointer.de/blog/projects/why.html

Oh look... he's related to PulseAudio and Avahi ..... If you've ever tried above average audio on Linux, then you know all about PulseAudio issues. If you've ever secured a subnet, then you know all about Avahi (aka mDNS). -Jim P.

On Tue, Oct 21, 2014 at 09:40:30AM -0400, Valdis.Kletnieks@vt.edu wrote:

On Tue, 21 Oct 2014 14:44:57 +0900, Randy Bush said:

...

systemd is insanity. one would have hoped that deb and others would know better. sigh.

It started as a replacement init system. I suspected it had jumped the shark when it sprouted an entirely new DHCP and NTP service. And this was confirmed when I saw this:

"Leading up to this has been cursor rendering support, keyboard mapping support, screen renderer, DRM back-end, input interface, and dozens of other commits."

http://www.phoronix.com/scan.php?page=news_item&px=MTgwNzQ

When your init system is worrying about cursor rendering, you have truly fallen victim to severe feature bloat. I guess Jamie Zawinski was right: "Every program attempts to expand until it can read mail."

Ooooh... /me submits a patch to systemd to provide localhost:25 and //usr/sbin/sendmail emulation... - Matt -- The real art of conversation is not only to say the right thing at the right place but to leave unsaid the wrong thing at the tempting moment. -- Dorothy Nevill

Perhaps they could make a "desktop" version with systemd if the devs want it that bad, but it'd be a shame if they ruined it for everyone that uses Debian as a server as well. Haven't installed x on Debian since Etch... o.O On Tue, Oct 21, 2014 at 12:44 AM, Randy Bush <randy@psg.com> wrote:

systemd is insanity. one would have hoped that deb and others would know better. sigh.

vmlinux.el here we come!

randy

On Tue, Oct 21, 2014 at 4:10 PM, Andrew Sullivan <asullivan@dyn.com> wrote:

On Tue, Oct 21, 2014 at 03:11:55PM -0400, Barry Shein wrote:

...

But for example some of my servers boot in seconds.

One is reminded of a mail, included in the Preface to _The UNIX-HATERS Handbook_, available at

it's really not clear to me that 'reboots in seconds' is a thing to optimize... I suppose the win is: "Is the startup/shutdown process clear, conscise and understandable at 3am local time?" followed by: "Can I adjust my startup processes to meet my needs easily and without finding a phd in unix?" If systemd is simply a change in how I think about /etc/init.d/* and /etc/rc?.d/* cool, if it's more complexity and less EASY flexibility then it's a fail. -chris

On October 21, 2014 at 16:43 morrowc.lists@gmail.com (Christopher Morrow) wrote:

On Tue, Oct 21, 2014 at 4:10 PM, Andrew Sullivan <asullivan@dyn.com> wrote:

...

On Tue, Oct 21, 2014 at 03:11:55PM -0400, Barry Shein wrote:

...

But for example some of my servers boot in seconds.

One is reminded of a mail, included in the Preface to _The UNIX-HATERS Handbook_, available at

it's really not clear to me that 'reboots in seconds' is a thing to optimize...

The unix community has exerted great amounts of effort over the decades to speed up reboot, particularly after crashes but also planned. Perhaps you don't remember the days when an fsck was basically mandatory and could take 15-20 minutes on a large disk. Then we added the clean bit (disk unmounted cleanly, no need for fsck), reorg'd the file system layout to speed up fsck considerably and make it more reliable/recoverable, added journaled file systems which really sped things up often eliminating the need to fsck after a crash entirely and recovering in seconds, various attempts to figure out the dependency graph of servers and services which need to be started so they could be started in parallel where dependencies are met, etc. And learned how to do hot failover and master/slave servers etc. And you whisk all that away with "it's not really clear to me that 'reboots in seconds' is a think to be optimized"???? To me that's like saying it's not important to try to design so one can recover from a network outage in seconds. Anyhow, if it's not clear: I disagree.

I suppose the win is: "Is the startup/shutdown process clear, conscise and understandable at 3am local time?"

followed by: "Can I adjust my startup processes to meet my needs easily and without finding a phd in unix?"

If systemd is simply a change in how I think about /etc/init.d/* and /etc/rc?.d/* cool, if it's more complexity and less EASY flexibility then it's a fail.

Actually, much of that is less important except perhaps to a hobbyist. You only have to get the startup/shutdown process etc right once in a while and generally during a planned outage. Recovering from a failure or going back into service quickly after a planned outage is critical and can be critical at any time. Obviously one can appeal to extremum but what you say doesn't make sense to me. At any rate, you are disputing a huge, decades long, and widely fought battle. It's certainly not my opinion. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On October 22, 2014 at 15:31 jfbeam@gmail.com (Ricky Beam) wrote:

On Wed, 22 Oct 2014 14:31:02 -0400, Barry Shein <bzs@world.std.com> wrote:

...

Perhaps you don't remember the days when an fsck was basically mandatory and could take 15-20 minutes on a large disk.

Journaling has all but done away with fsck. You'd have to go *way* back to have systems that ran a full fsck on every boot -- and in my experience, you absolutely wanted that fsck.

That was my point, it was a very brief and concise 30 year history. That's why I mentioned the introduction of the clean bit which was when we began recording (there may have been earlier experiments) the clean unmounting of a file system in the superblock so no need to fsck.

...

And you whisk all that away with "it's not really clear to me that 'reboots in seconds' is a think to be optimized"????

(I hope it's clear I meant "thing to be opt...")

Your efforts are better spent avoiding an outage in the first place. If outages are common enough to be something that needs to be "sped up", then you've already failed.

One important tool is failover. But once a system fails over you'd like to see the failed component back in service as quickly as possible unless you have an infinite number of redundant systems. Your advice doesn't ring true to me. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On Wed, Oct 22, 2014 at 4:48 PM, <nanog@jack.fr.eu.org> wrote:

Bah, boot speed;

On my server, boot is slow down by hardware initialization. The soft side is quite low.

But the point is not "makes things faster from 15 to 14 sec is useless". The point is : it's good, but at what price ?

I agree that "boot speed" is a red herring. Booting in a highly dynamic environment is really one of systemd's key achievements. True, this is most apparent in systems like laptops that can be docked/undocked, tend to have a lot of USB devices added/removed, etc. But servers have these same issues, if only in lesser degree. sysvinit generally had to wait for a fixed interval for all hardware to finish initializing. It was a little more sophisticated than one init script having a "sleep X" command in it, but not much. systemd is able to start services as soon as all of the hardware it needs is ready, rather than waiting arbitrary amounts of time and then possibly failing anyway because it didn't wait long enough. One main example of this is a large RAID array or LVM volume that's used for data storage (not the OS). systemd can let parts of the system boot that don't require the data storage to be present start up while waiting for all of the data storage drives to spin up and get assembled.

As you said, there were many improvements over the past. What was the "clean bit" cost ? None but benefits, right ? What about fs logs ? Does it have a cost ?

If systemd is just about time, it will be fine. But why trying to recreate (ans thus, squeeze) some old daemons like cron or syslog ? Both of them are doing a perfect job.

Syslog didn't capture the stdout/stderr from daemon start up. Syslog did only a mediocre job of capturing dmesg from early boot. If you have access to a system running systemd/journald run "journalctl -k -b". That'll show you all of the kernel messages since the last boot. At least on the RHEL systems that I have access to, /var/log/dmesg doesn't timestamp the lines in the file, making it difficult to impossible to correlate with other log lines. The kernel messages in /var/log/messages are timestamped with the time that rsyslog was able to (finally) start and pull the messages out of the kernel message buffer.

Can I use systemd without any of journald stuff ?

I wouldn't want to.

If not, then the 1sec speedup is far too expensive.

The boot speed up is a nice benefit, but not the only (or even the best) reason to use systemd. -- Jeff Ollie

On Wed, Oct 22, 2014 at 1:31 PM, Barry Shein <bzs@world.std.com> wrote: [snip]

The unix community has exerted great amounts of effort over the decades to speed up reboot, particularly after crashes but also planned. Perhaps you don't remember the days when an fsck was basically mandatory and could take 15-20 minutes on a large disk.

Then we added the clean bit (disk unmounted cleanly, no need for [snip] And you whisk all that away with "it's not really clear to me that 'reboots in seconds' is a think to be optimized"????

False dilemma. Optimizing reboot time down from 20 minutes to 1 minute is a significantly meaningful improvement; it's literally a 85% reduction in time spent during each boot process from the original time. Reducing boot time from 20 minutes to 10 seconds is not significantly better than reducing it to 1 minute. A different choice of tradeoffs is more appropriate to different kinds of systems, depending on their use case (Desktop vs Server)! Especially, when the method of reduction is subject to diminishing returns and increasing fragility or increasing complexity -- greater risk that something is breaking or more potential for unreliability is introduced into the startup process. Also, you may very well spend more time booting your system in order to troubleshoot, the fact that some applications are starting up in an unexpected order resulting in some issue.

To me that's like saying it's not important to try to design so one can recover from a network outage in seconds.

If you need to ensure that a service is not disrupted for more than seconds, then reboot is not the answer. It is some form of clustering. Reboot as a troubleshooting procedure is for desktops. 10 seconds from power on to user interface for desktops, will meaningfully improve the user experience, but not for servers. For servers, you ideally want to take the misbehaving node out of service and let its failover partner takeover. -- -JH

---- Original Message -----

From: "Jeffrey Ollie" <jeff@ocjtech.us>

On Wed, Oct 22, 2014 at 9:48 PM, Jimmy Hess <mysidia@gmail.com> wrote:

...

On Wed, Oct 22, 2014 at 1:31 PM, Barry Shein <bzs@world.std.com> wrote:

...

And you whisk all that away with "it's not really clear to me that 'reboots in seconds' is a think to be optimized"????

False dilemma. [ snip ] 10 seconds from power on to user interface for desktops, will meaningfully improve the user experience, but not for servers.

It's a false dilemma only if you're thinking about traditional physical servers. Consider:

1) What if you're spinning up several thousand Hadoop nodes on AWS or GCE so that you can do some sort of "big data" operation.

2) What if PewDiePie just mentioned one of your products in a video and you need to quickly scale up the number of backend servers to handle the load.

I'm sure that there are many other scenarios that I could devise where a fast "server" boot time was important.

I will stipulate this use case. I will counter with "you wouldn't be running a "real" distro in that case anyway; you'd be running something super trimmed down, and possibly custom built, or based on something like CoreOS, that only does one job. Well. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On Mon, Oct 27, 2014 at 10:35 AM, Jay Ashworth <jra@baylink.com> wrote:

I will stipulate this use case.

I will counter with "you wouldn't be running a "real" distro in that case anyway; you'd be running something super trimmed down, and possibly custom built, or based on something like CoreOS, that only does one job.

Well. :-)

From: https://coreos.com/using-coreos/systemd/ "CoreOS uses systemd as the core of its distributed init system, fleet. Systemd is well supported in many Linux distros, making it familiar to most engineers. Every aspect of CoreOS is deeply integrated with systemd." -- Jeff Ollie

On 10/27/2014 11:35 AM, Jay Ashworth wrote:

I will counter with "you wouldn't be running a "real" distro in that case anyway; you'd be running something super trimmed down, and possibly custom built, or based on something like CoreOS, that only does one job. Well.

Hmm, now this one I wasn't aware of.... this tidbit here has made this thread worthwhile to me, as we work on developing some clustered 'things' for use here..... CoreOS wasn't even on the 'look at this at some point in time' list before, but it is now..... Thanks, Jay.

----- Original Message -----

From: "Miles Fidelman" <mfidelman@meetinghouse.net>

...

Hmm, now this one I wasn't aware of.... this tidbit here has made this thread worthwhile to me, as we work on developing some clustered 'things' for use here..... CoreOS wasn't even on the 'look at this at some point in time' list before, but it is now..... Thanks, Jay.

Funny, and here my reaction is just the opposite - to remove CoreOS from my list of things to look at.

I am happy and sad for you both. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

I'm reminded of the remark often attributed to DEC CEO Ken Olson, roughly: With VMS (their big complex OS) it might take hours searching through manuals to find a feature you need while with Unix you can determine in seconds that it is not available. On October 21, 2014 at 16:10 asullivan@dyn.com (Andrew Sullivan) wrote:

On Tue, Oct 21, 2014 at 03:11:55PM -0400, Barry Shein wrote:

...

But for example some of my servers boot in seconds.

One is reminded of a mail, included in the Preface to _The UNIX-HATERS Handbook_, available at http://www.art.net/~hopkins/Don/unix-haters/preface.html. Apparently, things really are going to get a lot worse before they get worse.

Best regards,

A

-- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448

-- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On October 23, 2014 at 04:42 randy@psg.com (Randy Bush) wrote:

Barry Schein:

Interesting you went to the trouble to add a 'c' to my name! You need better quoting tools.

...

I'm reminded of the remark often attributed to DEC CEO Ken Olson, roughly:

With VMS (their big complex OS) it might take hours searching through manuals to find a feature you need while with Unix you can determine in seconds that it is not available.

and how did that work out for vms? and digital?

A few people made billions, a few more made many millions, hundreds of thousands (or thereabouts) had pretty good jobs for upwards of 20 years, and then the second largest computer company in the world vaporized almost mysteriously. The VAX hardware was important. It was for the time relatively inexpensive and very capable, the 32-bit address space (ok, technically four 30 bit addr spaces) and VM hardware at those prices were revolutionary. You had many of the capabilities of a multi-million dollar mainframe for about 1/10th the cost. Ran Unix great! VMS not so much. Mostly re-warmed over RSX (an earlier DEC OS) with a few new ideas to take advantage of the platform, and some cobbling from their TOPS-10 and TOPS-20 OS's (e.g. galaxy.) IMHO DEC desparately wanted to go head on with IBM's 370 line but just didn't seem to "get" why companies bought IBM mainframes, or found those parts too expensive to compete on. But they did ok financially anyhow so who's to criticize? VMS even had PIP! And sometimes you needed it. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

Probably a lot of it has to do with: - we're merging udev and a bunch of other things into systemd - you want GNOME to work, you'd better use systemd - Canonical (Ubuntu) DIDN'T commit to udev until Debian made the decision - they would have kept going with upstart, but when Debian committed, they decided they didn't want to support a now-orphaned init system - Gentoo supports systemd as an option, it's fork funtoo doesn't - Slackware doesn't Miles Fidelman Jay Ashworth wrote:

The thing that I don't understand about systemd is how it managed to get *EVERY SINGLE DISTRIBUTION'S RELEASE MANAGER* on board in less than a year, given how thoroughly it violates the Unix philosophy, and how poorly documented it is -- to the point where you can't even run sysvinit anymore unless you're willing to build initscripts by hand, since packages don't even include them anymore.

Does Poettering have compromising photographs of all these guys in a puppy pile at a Linuxcon somewhere?

Cheers, -- jra

----- Original Message -----

...

From: "Barry Shein" <bzs@world.std.com> To: "Israel G. Lugo" <israel.lugo@lugosys.com> Cc: nanog@nanog.org Sent: Tuesday, October 21, 2014 3:11:55 PM Subject: Re: Linux: concerns over systemd [OT] I've done a fair amount of hand-to-hand combat with systemd.

When it's good it's good, tho not always apparent why it's good. But for example some of my servers boot in seconds.

When it's bad it can be painful and incredibly opaque and a huge time sink.

Googling for suggestions I've found several threads where the co-author (Poettering) jumps in usually to be annoyingly arrogant (I'm sure he's very bright and good to children and pets and overworked) responding with comments like why don't you just read your logs and not bother this list or similar (that was paraphrased.) The logs are, in my experience, almost always useless or nearly so, "mumble failed to start" basically.

I'm not the only one:

http://www.muktware.com/2014/04/linus-torvalds-happy-systemd-author-kay-siev...

It also resists tools like strace because it tends to do things by IPC. In one extreme case I just reworked an /etc/init.d script to avoid systemd (not use the various /etc/rc.foo files), mostly just hit it with a sledgehammer and put fixing that on my TODO list. Unfortunately I am mortal and have limited time on this earth.

My experience as I said is mixed, hard cases are very hard where they really seem like they shouldn't be (just tell me roughly what you're trying to do rather than just fail, eg, via some debug enable), most are just your usual oops it wants this or that situations.

I don't think I'd want to revert to sysvinit, systemd seems architecturally superior.

But it needs a lot more transparency and some attempt to gather common problems -- like why is it hanging asking for a password on the console when I can't see why it thinks it needs one? -- and FAQ them with real answers or add some code/configuration to fix that (never ask for a password in this script OK? And no --no-ask-password isn't fixing this so stop repeating that answer!)

-- -Barry Shein

The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

----- Original Message -----

From: "Ricky Beam" <jfbeam@gmail.com>

On Tue, 21 Oct 2014 18:29:44 -0400, Jay Ashworth <jra@baylink.com> wrote:

...

The thing that I don't understand about systemd is how it managed to get *EVERY SINGLE DISTRIBUTION'S RELEASE MANAGER* on board...

It's spelled "Red Hat". Add in GNOME and debian (et. al.) is backed into a corner. Red Hat is soo f'ing big, pretty much every project under the sun is going to stop maintaining scripts in favor of systemd.

GNOME is probably the linchpin. But it's not just RH. It's Debian, and by extension *buntu, and SuSE, and at least one other major independent parent distro that I can't think of just now... And as far as I know, it's done; SuSE packages already largely don't even include initscripts. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

George Herbert wrote:

...

On Oct 21, 2014, at 6:03 PM, Jay Ashworth <jra@baylink.com> wrote:

GNOME is probably the linchpin.

But it's not just RH. It's Debian, and by extension *buntu, and SuSE, and at least one other major independent parent distro that I can't think of just now...

And as far as I know, it's done; SuSE packages already largely don't even include initscripts. Enough to make a grown man fork RHEL (or, CentOS).

Which leads me to ask - those of you running server farms - what distros are popular these days, for server-side operations? We've been running Debian like forever (by way of Solaris and redhat) - but this systemd thing is making me rethink things. Seems like an awful lot of folks are now designing for the desktop, and it might be time to migrate to a BSD or Solaris derivative. What are others doing? Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

Before leaving Debian, things to think: - will systemd be officialy the only system available ? - if so, won't we get a way to bypass that ? I'm not gonna throw Debian away due to such a mess, without fighting hard, and I think you should do the same: talk, patch if needed, show you're here If all good people who laugh insanely about systemd leave Debian alone, who's left ? gnome-people ? systemd-fanatics (heretics?) ? On 22/10/2014 12:09, Tom Hill wrote:

On 22/10/14 10:41, Miles Fidelman wrote:

...

Which leads me to ask - those of you running server farms - what distros are popular these days, for server-side operations? We've been running Debian like forever (by way of Solaris and redhat) - but this systemd thing is making me rethink things. Seems like an awful lot of folks are now designing for the desktop, and it might be time to migrate to a BSD or Solaris derivative. What are others doing?

Not making rash decisions.

Debian and CentOS are still the 'asked for' distributions of Linux. Once in a blue moon, someone asks about something else.

Those that care are outnumbered greatly by those that just want a known platform to develop upon. (The irony of this is not lost on me).

I'd take systemd over ditching apt/yum in a mad panic. And I'm certainly no fan of systemd myself.

On 22 October 2014 11:34, <nanog@jack.fr.eu.org> wrote:

Before leaving Debian, things to think: - will systemd be officialy the only system available ? - if so, won't we get a way to bypass that ?

And one other thought... is it really that bad? Personally I like it a lot better than sysV plus inittab plus daemontools. Dan

When it's working, no doupt, I'll be fine I don't care (or just a few) about when it's working. The point is: what about it's failure ? On the ethical point of view, systemd is killed anyway On 22/10/2014 13:00, Daniel Ankers wrote:

On 22 October 2014 11:34, <nanog@jack.fr.eu.org <mailto:nanog@jack.fr.eu.org>> wrote:

Before leaving Debian, things to think: - will systemd be officialy the only system available ? - if so, won't we get a way to bypass that ?

And one other thought... is it really that bad?

Personally I like it a lot better than sysV plus inittab plus daemontools.

Dan

On Wed, 22 Oct 2014 12:34:17 +0200, nanog@jack.fr.eu.org said:

Before leaving Debian, things to think: - will systemd be officialy the only system available ? - if so, won't we get a way to bypass that ?

Somebody already forked systemd at a point before it completely lost the plot. http://uselessd.darknedgy.net/

Which leads me to ask - those of you running server farms - what distros are popular these days, for server-side operations?

been running bsd forever. but moving to debian and ganeti, as bsd does not host virtualization. would love it if debian ditched this systemd monstrosity and provided solid zfs. randy

...

...

Which leads me to ask - those of you running server farms - what distros are popular these days, for server-side operations? been running bsd forever. but moving to debian and ganeti, as bsd does not host virtualization. Simply not true; http://bhyve.org/ It is a bit immature compared to Xen+Ganeti or something like that.

apologies. i thought we were talking about production systems. my mistake. randy

On Wed, Oct 22, 2014 at 6:12 AM, Joe Greco <jgreco@ns.sol.net> wrote:

when so much effort has been put into that very issue, specifically so that we could gain the advantages of a BSD hypervisor that supported ZFS natively... [snip]

If you want native ZFS support, then Solaris x86-64+Zones+KVM or SmartOS. Now Solaris/Illumos' process supervision and fault management systems, SMF and FMA are pretty complex, but they aren't stuffing 1000 random tasks into the init program. ^_^

... JG

-- -JH

David Ford wrote:

...

Which leads me to ask - those of you running server farms - what distros are popular these days, for server-side operations? We've been running Debian like forever (by way of Solaris and redhat) - but this systemd thing is making me rethink things. Seems like an awful lot of folks are now designing for the desktop, and it might be time to migrate to a BSD or Solaris derivative. What are others doing?

<snip>

before making ill informed rash decisions, study what you're talking about and then decide whether or not you like and can/can't use said subject.

Is that not the point of discussions like this one? -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

the vast majority of negative tongue wagging regarding systemd is ill informed.

can we skip the ad homina and leave that for the systemd dev fora?

does systemd have growing pains? definitely. are some egos involved? sure. can systemd be far reaching? yes, is such reach mandated? no. use the units you want and disregard the rest.

how does this work out in practice? at install, can i choose whether systemd is used for X as opposed for the separate component? can i template such choices for cluster deployment with the usual tools? as barry said, we get to drive this car. randy

On Wed, Oct 22, 2014 at 3:34 PM, Randy Bush <randy@psg.com> wrote:

...

the vast majority of negative tongue wagging regarding systemd is ill informed.

can we skip the ad homina and leave that for the systemd dev fora?

I don't think that it's an ad homina attack, as it's pretty clear that many of the people commenting have not spent a significant time using systemd so many of their comments are based on what they've read on the Internet, not from practical experience with systemd.

...

does systemd have growing pains? definitely. are some egos involved? sure. can systemd be far reaching? yes, is such reach mandated? no. use the units you want and disregard the rest.

how does this work out in practice? at install, can i choose whether systemd is used for X as opposed for the separate component? can i template such choices for cluster deployment with the usual tools?

I think that Debian's plan to allow multiple init systems (irregardless of which one is default) is a bad plan. The non-default ones won't get any love - at some point they'll just stop working (or indeed, work at all). Allowing choice of components is a good thing at one level (e.g. sendmail vs. postfix vs. exim). I really don't care (and don't really even remeber) which SMTP server is installed by default on my systems because my configuration management system makes sure that the SMTP server that I prefer is installed and configured the way I want it once the system is up and running. For something like PID 1, each distribution should make a choice and stick with it. I really couldn't care what Debian's init system is, as I don't use Debian (never have, at least not when I have had a choice). If Debian goes through with the switch to systemd, they won't gain me as a user as there are a host of other reasons that I prefer something other than Debian (or Debian-derived) distributions. If a group of people fork Debian because of systemd, more power to them. -- Jeff Ollie

On Wed, 22 Oct 2014 19:35:51 -0000, David Ford said:

into a common bus. not everything in systemd is a requirement to run it. just because a unit is offered for dhcp or ntp, doesn't mean you are required to use it.

Actually, systemd 216 will cram systemd-timesyncd down your throat even if you had ntpd installed. https://bugzilla.redhat.com/show_bug.cgi?id=1136905 https://mail.gnome.org/archives/distributor-list/2014-September/msg00002.htm... Lennart's attitude was pretty much "why would anybody want to run ntpd when they have our SNTP implementation": http://lists.freedesktop.org/archives/systemd-devel/2014-August/022537.html There's been similar issues with their dhcp.

On Wed, Oct 22, 2014 at 3:48 PM, <Valdis.Kletnieks@vt.edu> wrote:

On Wed, 22 Oct 2014 19:35:51 -0000, David Ford said:

...

into a common bus. not everything in systemd is a requirement to run it. just because a unit is offered for dhcp or ntp, doesn't mean you are required to use it.

Actually, systemd 216 will cram systemd-timesyncd down your throat even if you had ntpd installed.

Oh really? From my Fedora 21 (alpha) system at work: # rpm -q systemd systemd-216-5.fc21.x86_64 # systemctl status systemd-timesyncd.service ● systemd-timesyncd.service - Network Time Synchronization Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; disabled) Active: inactive (dead) Docs: man:systemd-timesyncd.service(8) # systemctl status ntpd.service ● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled) Active: active (running) since Thu 2014-10-16 16:06:22 CDT; 6 days ago Main PID: 1438 (ntpd) CGroup: /system.slice/ntpd.service └─1438 /usr/sbin/ntpd -u ntp:ntp -g I'm not sure what NTP service is installed by default in Fedora 21+ (which will ship with systemd 216), as this system has been upgraded from previous Fedora versions, but as you can see it's perfectly possible to run ntpd on a system that used systemd 216.

https://bugzilla.redhat.com/show_bug.cgi?id=1136905 https://mail.gnome.org/archives/distributor-list/2014-September/msg00002.htm...

Lennart's attitude was pretty much "why would anybody want to run ntpd when they have our SNTP implementation":

A vast oversimplification of Lennart's point. Basically you left off "if you really want to run chronyd or ntpd service go right ahead, we're just not going to have code in systemd to do that for you".

http://lists.freedesktop.org/archives/systemd-devel/2014-August/022537.html

There's been similar issues with their dhcp.

The "bug" here is really timedatectl (a component of systemd) isn't going to manage chronyd or ntpd (third party packages), and that made a Gnome control panel (which used timedatectl under the hood) report that network time synchronization wasn't enabled. If you don't want timedated then it's perfectly possible to disable timedated and use something else. If someone cares enough, I'm sure that the Gnome control panel will get updated so that I can manage chronyd or ntpd itself. -- Jeff Ollie

Often presented with an alternate spelling from those of us who had to live with it. On Tue, Oct 21, 2014 at 01:44:17PM -0700, Eric Brunner-Williams wrote:

...

systemd is insanity.

see also smit.

Philip Dorr wrote:

Could someone comment on why they chose systemd over upstart (other than the Canonical CLA)? Or point to an article on it?

If you want to wade into the mess, the archives of the Debian Tech. Committee (https://lists.debian.org/debian-ctte/), for Dec 2013-March 2014, make for some interesting reading (if you have a Baroque sense of what's interesting). The voting is documented here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708 (I think - Debian decision making is incredibly convoluted.) This article has some background that's reasonably accurate: http://www.zdnet.com/debian-inches-towards-new-init-system-decision-amid-fal... I seem to recall a summary document comparing the various available init systems, and how systemd came out on top - but I'll be damned if I can find it now. Some of the position statements are linked from here: https://wiki.debian.org/Debate/initsystem Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

On October 21, 2014 at 13:44 brunner@nic-naa.net (Eric Brunner-Williams) wrote:

...

systemd is insanity.

see also smit.

SMIT! Rhymes with .... -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On Wed, Oct 22, 2014 at 11:12 AM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:

Seems to me, this has been a very rational discussion,

Hardly. The discussion so far has been weighted very heavily on the side of Dana Carvey's "Grumpy Old Man"-style whining. "That's the way it was and we liked it!". The people that like systemd (like myself) have wisely learned that the people that hate systemd, hate it mostly because it's different from what came before and don't want to change. There's no way to argue rationally with that.

confined to one very identifiable thread,

Thank goodness!

containing what at least this reader finds very useful (operational impacts of systemd in server-side environments, and what alternatives people are looking at).

Just because it's useful, doesn't mean that it isn't off-topic for NANOG. At least until Cisco starts using systemd as pid 1 in IOS XE I suppose...

If you're not interested, you have a delete key. Please use it, and don't turn this into a flame war.

Sure, I have a delete key, but it'd still be better if people moved this discussion somewhere more appropriate. -- Jeff Ollie

On Wed, Oct 22, 2014 at 12:43:53PM -0400, C. Jon Larsen wrote:

Incorrect assumption. systemd is a massive security hole waiting to happen and it does not follow the unix philosophy of done 1 thing and do it well/correct.

It does seem to me that this angle, at least, is on-topic for NANOG, and I hope someone has suggestions for how to mitigate it. It seems that we've had two or, arguably, three recent examples (heartbleed, shellshock, arguably poodle) of complicated code that too few people understood and that led to widespread, late-night-inducing emergency action once a serious vulnerability was discovered. Surely that direction of development in a process that runs as PID 1 is something that has significant follow-on effects for network security. But I have no clue what one can do about it. For many years, I liked to keep some Linux and some BSD systems around, because it seemed to me that the different styles tended to encourage diversity and that was a good thing. But management of BSD systems -- particularly the nonsense of rebuilding things from source all the time -- started to look mighty onerous compared to apt-get update; apt-get upgrade. Others apparently agreed, and now there are enough things that work well on Linux but not as well (or not at all) on BSD that the diversity argument isn't as strong. (Also, of course, certain kinds of things, like some kinds of database replication, don't work well across platforms, so there's another reason to converge on a single system.) Debian was always the Linux platform that seemed most insistent on having more than one way to do it, but in recent years that philosophy has made it more work to use than the alternatives; and the alternatives have often gotten good enough that one doesn't care (Ubuntu is the obvious example here). So, now we have an encroaching monoculture, and no real option to do anything about it. Maybe this is just the way the Internet is, now. A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448

On Wed, Oct 22, 2014 at 01:35:26PM -0400, Daniel Corbe wrote:

Not to get even further off topic here but when was the last time you maintained a BSD system? FreeBSD (at least) adopted binary package management as its preferred interface to ports through pkg-ng somewhere in the 9-RELEASE cycle.

Yeah, it was some time ago (fortunately, people keep me far away from systems administration now). But that doesn't exactly help: since others drew the same conclusion, _even if BSD is now a perfectly rational choice_, I've lost functionality I desire. Docker is a great example of this. For many use cases, it's obviously better than the alternatives. So now I _can't_ pick FreeBSD. And so on.

This systemd debacle is an excellent reason to look into stuff that isn't Linux. The Linux camp all too often become victims of "not invented here"

I don't think the latter is restricted to the Linux world! But as I suggested, the network security implications of all that stuff hidden in one critical system sure seem to require some thinking. A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448

On 10/22/14, 9:01 AM, "Jeffrey Ollie" <jeff@ocjtech.us> wrote:

Bull. If you've been around the FOSS community even for a short while, you'd know that systemd has become a religious topic akin to

On 10/22/14, 10:41 AM, "Jeffrey Ollie" <jeff@ocjtech.us> wrote:

sums up my thoughts on the "unix philosophy". It's not the be-all-end-all that you make it out to be. Again, this sounds a lot like "Grumpy Old Man" complaining.

You appear to be the only one here having difficulties being civil on the topic. The rest of us are having an interesting, pleasant discussion about a rather huge operational shift. You like systemd. Great. More power to you. You apparently don’t like it that many others have differing views or are still reserving judgment. Also great, you can ignore the thread. That said, my take on the mess. I need to see systemd running for while in large-scale production before I’m moving anything to it. I do rather hate the NIH-itis, the svchost.exe-ish nature of the technical approach, the core devs' attitudes, the tightly-coupled nature of the approach to doing the various things it wants to do, and the devs' stated goal of forcing every linux distro in to using it. And fu’ing binary logs. OTOH, it does offer some nice things that can become annoying with other tools. The single-sourced APIs for userspace apps might be appreciated by some developers, especially refugees from MSFT-land. It is immature compared to the competition with a radically un-unix approach developed by people with a track record of being extremely difficult to collaborate with, and it does several specific things I really don’t like. That adds up to a hard sell for me. Some may see me as a "grumpy old man" for that. I see it as a technical conservatism for my production environments borne of having been burned by shiny-cool-new before one too many times, and a tired dislike of being paged out of bed over some chrome plated new hotness that crapped itself again. There is a deeper argument about "the unix way" here as well. I do see a lot of people become frustrated by what they see as impedance mismatched between tools as they learn ("why should I have to care about what the IFS is?"), or because a tool fails to react in accordance with their expectations, or because they are using the wrong tool ("I want to match these HTML tags with a regex..."), or because they realize a problem they thought should be easy, isn't. I think that’s natural, to some extent. I did (and still do, sometimes - there seems to be a hard limit on the number of awk implementation differences that fit in my brain, for instance). And there are things that could be made a lot better. But when I start wondering if my init system has a flight simulator easter egg, well, there’s a problem, at least for me. It is funny to see people use "but that approach is 40 years old!" on both sides of the argument. I do love playing with new toys, and think the systemd folks should write whatever they want. I have major issues with the monoculture they want to establish. (And yes, a core developer clearly stated that as a goal[1].) Most of my systems are Linux these days, and I am mostly distro-agnostic, although my default does happen to be Debian. I do think I’m going to have to get back up to speed on FreeBSD again. -j [1] https://lists.fedoraproject.org/pipermail/devel/2011-June/152672.html : "[...]we want to gently push the distros to standardize on the same components for the base system[...]"

On 10/22/2014 10:43 AM, C. Jon Larsen wrote:

...

Hardly. The discussion so far has been weighted very heavily on the side of Dana Carvey's "Grumpy Old Man"-style whining. "That's the way it was and we liked it!".

The people that like systemd (like myself) have wisely learned that the people that hate systemd, hate it mostly because it's different from what came before and don't want to change. There's no way to argue rationally with that.

Incorrect assumption. systemd is a massive security hole waiting to happen and it does not follow the unix philosophy of done 1 thing and do it well/correct.

i was beginning to wonder how secure systemd is also. --John

Its basically ignoring 40 years of best practices. Thats why folks that have been there, done that, dont want any part of it. Not because its new, but because its a flawed concept.

You are free to use it, but it would be a poor choice for system that has hopes of being secure.

...

-- Jeff Ollie

On Wed, 22 Oct 2014 13:13:29 -0600, John Schiel said:

i was beginning to wonder how secure systemd is also.

One of the 3 CIA pillars of security is "availability". And if it's oh-dark-30, figuring out what symlink is supposed to be where for a given failed systemd unit can be a tad challenging. At least under sysvinit, either /etc/rc5.d/S50foobar is there or it isn't(*). And if they carry through on their systemd-console threat, that could get even worse - that introduces a whole new pile of risks for being unable to diagnose early boot bugs So yeah, there's security issues other than "can it be hacked because it's got a huge surface area". (*) Unless you're really having a bad night and it's a hard link to /dev/sda1 or something. :)

On Wed, Oct 22, 2014 at 3:22 PM, John Schiel <jschiel@flowtools.net> wrote:

On 10/22/2014 01:30 PM, Valdis.Kletnieks@vt.edu wrote:

...

On Wed, 22 Oct 2014 13:13:29 -0600, John Schiel said:

...

i was beginning to wonder how secure systemd is also.

One of the 3 CIA pillars of security is "availability". And if it's oh-dark-30, figuring out what symlink is supposed to be where for a given failed systemd unit can be a tad challenging. At least under sysvinit, either /etc/rc5.d/S50foobar is there or it isn't(*).

Agreed, the "oh-dark-thirty" call outs will be harder to resolve but I'm sure some folks will learn to deal with it. It's new and changes the job but as was noted earlier, there is always change.

I disagree. I believe that the features of systemd will make "oh-dark-thirty" call outs easier to resolve, but only if you take the time to familiarize yourself with the tools at hand *before* problems happen. But really, there's nothing new here. *Of course* systems that are unfamiliar to you will be more difficult to fix. It'd take *me* *forever* to fix a problem on the HP-UX systems at work, mainly because I'd spend too much time figuring out where everything was. However the guy in the cube next to me wouldn't have that problem... To borrow Barry's automotive metaphor, this is like saying that electric motors are bad because I only know how to fix gasoline engines. -- Jeff Ollie

4. More fundamentally, 0-dark-30 events are almost always unexpected (other than in the sense of Murphy's Law), and tricky to resolve - one has hopefully prepared for the expected.

"If it was part of the 'plan' it’s an event; if it is not then it’s a 'disaster'"

On Wed, Oct 22, 2014 at 2:30 PM, <Valdis.Kletnieks@vt.edu> wrote:

On Wed, 22 Oct 2014 13:13:29 -0600, John Schiel said:

...

i was beginning to wonder how secure systemd is also.

One of the 3 CIA pillars of security is "availability". And if it's oh-dark-30, figuring out what symlink is supposed to be where for a given failed systemd unit can be a tad challenging. At least under sysvinit, either /etc/rc5.d/S50foobar is there or it isn't(*).

How long has it been since you've looked at/used systemd? Manual management of symlinks to control what services are started at boot time are a thing of the past. "systemctl enable|disable|status <service>" will handle all of that for you. I agree, managing symlinks was annoying in the early systemd days, but I haven't messed with those symlinks manually in a long time, except to remove orphan symlinks caused by services that weren't removed cleanly from the system. I think your fear comes more from a lack of familiarity with systemd, rather than a true technical deficit. Personally I find systemd services much easier to debug, especially since stderr and stdout from all services is captured, rather than being lost to /dev/null. I find it VERY annoying that many init.d scripts eventually boil down to "/usr/sbin/program --someflags 2>&1 > /dev/null &".

And if they carry through on their systemd-console threat, that could get even worse - that introduces a whole new pile of risks for being unable to diagnose early boot bugs

I haven't seen much about systemd-console, but I thought that this reddit thread was interesting. http://www.reddit.com/r/linux/comments/1rmj9e/systemd_is_about_to_gain_a_sys... It seems to me that moving VTs out of the kernel and into a userspace process would be a good thing. I guess one could argue about whether systemd is the right place for it, but as you can see from that reddit thread there have been a number of other projects that have attempted to do that but have failed to gain traction for one reason or another. One thing that the systemd developers are undeniably good at is getting their work adopted by the distributions.

So yeah, there's security issues other than "can it be hacked because it's got a huge surface area".

Nothing new here. And before you get started, Bash and OpenSSL prove that old code isn't necessarily secure code. -- Jeff Ollie

----- Original Message -----

On 22-10-2014 17:30, Jeffrey Ollie wrote:

...

Hardly. The discussion so far has been weighted very heavily on the side of Dana Carvey's "Grumpy Old Man"-style whining. "That's the way it was and we liked it!". The people that like systemd (like myself) have wisely learned that the people that hate systemd, hate it mostly because it's different from what came before and don't want to change. There's no way to argue rationally with that.

That's not true, Jeff. Nearly all the anti-systemd arguments I have seen have itemized the design features of the package which we don't like, and provided what seem to us to be good and cogent reasons why we don't like those, in many cases backed up with examples of how earlier similar designs failed that way. "Grumpy Old Man" is a strawman argument, and there's no way to argue with that at all: you merely identify it as fallacious and ignore it. And if the same proponent keeps doing it, you demonstrate the sound an idiot makes falling to the bottom of your killfile. Please don't. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On Wed, Oct 22, 2014 at 12:35 PM, George Herbert <george.herbert@gmail.com> wrote:

...

On Oct 22, 2014, at 9:30 AM, Jeffrey Ollie <jeff@ocjtech.us> wrote:

The people that like systemd (like myself) have wisely learned that the people that hate systemd, hate it mostly because it's different from what came before and don't want to change. There's no way to argue rationally with that.

I think you are monumentally misreading the situation.

A) Change is the constant in IT. Staying relevant and employable has put me through five or more generational shifts in enterprise OS, plus diversions to Mach, Plan 9, MacOS, etc. Change is normal.

I agree with you about change, but there are a number of people that vocally resist any kind of change, no matter what. There's little that can change their mind other than time. It's a useful skill to be able to recognize these people and not to let them get you down.

B) Systemd and the Solaris SMF that it conceptually followed have a number of technical flaws, ranging from obscure interfaces (sometimes requiring source code to understand) to lack of human readable configs to (at least with SMF, and as far as I can tell systemd) a lack of ability to even print/dump out the current dependencies and ordering tree.

I have no experience with Solaris SMF so I can't comment on it specifically, but in my opinion systemd: 1) has excellent documentation, especially when you compare it to other open source documentation. 2) What's more readable than .ini files? They even avoided the temptation to use XML. I can't tell you how much time I've wasted reading shell script spaghetti trying to figure out exactly how a service is started up. Services implemented in Java and Ruby seem to be especially bad at that. 3) systemctl list-dependencies, although since service start-up in systemd is highly dynamic it's difficult to predict what order services will start up.

C) In both systemd and SMF a tremendous unpreparedness of training and documentation accompanied rollout. These were not reasonably enterprise ready at launch, or now.

In 2010/2011 when systemd was first being integrated into Fedora (and I believe SuSE) I would have agreed with you. However this is four years later and I couldn't disagree with you more. More to the point, Red Hat disagrees with you, given that they have put their money where their mouth is and integrated systemd into RHEL7.

D) The architectural case that the services adopted in systemd over time belong there or are safe there is not proven, and not that I see well argued or documented. Conglomerated services are at least to be eyed skeptically.

I did not closely follow systemd's development but it is evident from a distance that operator feedback in the community and to Sun regarding SMF flaws was somehow missed in systemd's development as they did the same wrong things.

A change this big deserves architectural clarity and justification.

I don't follow systemd development closely either, but Lennart Poettering, Kay Sievers, et. al. have made extraordinary attempts to communicating about systemd. They've appeared at numerous conferences and given talks about systemd, written blog posts, documentation, etc. I don't know what else they can do other than to be invited over to everyone's home for tea and a systemd discussion.

We get snide comments about change being good and core developers Linus evidently feels are unsafe.

We also get snide comments about change being bad. As for Linus, other than the fact that he has an extraordinary amount of influence over what goes into the kernel, I've learned to take his comments on other matters with a very large grain of salt. I have a lot of respect for the guy, but his attitude and behavior towards other people is appalling. What's really surprising about some of Linus' comments WRT to systemd is that one of systemd's main goals is to make it easier to use some of the advanced functionality of the Linux kernel that were little used or ignored before. -- Jeff Ollie

Well said, Rich. Rich Kulawiec wrote:

On Wed, Oct 22, 2014 at 11:30:57AM -0500, Jeffrey Ollie wrote:

...

The people that like systemd (like myself) have wisely learned that the people that hate systemd, hate it mostly because it's different from what came before and don't want to change. That's an entirely unfair characterization.

Some of us, including a lot of people on this list, have been pushing the envelope on change for decades. We've run alpha software on beta hardware, cobbled together networks with duct tape, and burned a lot of midnight oil while making innumerable mistakes and learning from them.

Speaking for myself, after migrating through far too many Unix and Linux distributions to count, starting with Research Unix v6, my entire professional career has been *constant* change. I suspect that the same is true of everyone else who's been doing this for a while. Anyone who doesn't embrace change as a way of life probably isn't on this list; they're somewhere else, maintaining Cobol written during the Nixon administration.

My problem with systemd is not that it's a change: it's just one more out of tens of thousands and so, in that sense, it's really not a big deal. My problem with it is that I think it's a bad change, one not in keeping with the "software tools" philosophy that has served us ridiculously well for a very long time and -- so far -- has not been shown itself to be in need of replacement.

A Leatherman pocket multitool is highly useful: I've had one for years. It's great. Until you need two screwdrivers at the same time...at which point it becomes obvious why serious mechanics/craftsmen carry around a toolbox with dozens of tools and why glomming all of those into one supertool would be A Very Bad Move.

Similarly, the monolithic (and ever-expanding) nature of systemd is a strategic design error. That's probably not obvious to people who measure their experience in years instead of decades -- it wouldn't have been obvious to me back in the day either. But it's pretty clear from here, and dismissing it as "hey you kids get off my lawn" geezer whining is not going to advance the discussion. It would be better, I think, to pull out a copy of Kernighan and Plauger's book -- which is rather brief, actually -- read it cover-to-cover, and then consider carefully whether the myriad-and-steadily-increasing number of functions being subsumed by systemd should actually be in one program.

If that doesn't suffice, then I suspect it will only require waiting a little while until a demonstration of why monolithic integration is a bad idea will be provided by someone who is at this moment studying the large-and-growing attack surface presented by systemd.

I hope I'm wrong about that. I'm probably not.

(By the way, this should not be read to express unabashed support for *any* of the various init systems that have been present in SysV or BSD or AIX or Debian or Dynix or Red Hat or HPUX or Mint or Ultrix or Solaris or or or or. They all have their issues, some of which were or are sporadically annoying.)

---rsk

-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

On 10/23/2014 12:05 AM, Jeffrey Ollie wrote:

systemd is a tool designed to get the system to a state where "real work" can be done. NTP servers, DHCP clients, consoles, aren't the real work of a system, or at least I hope not, because that would be boring to me.

That idea sounds interesting. I can see where you're coming from. Basically, these things are "details", that shouldn't really be all that important, so systemd is supposed to take care of it all and leave you to worry about the actual bread and butter. That about it? Legitimate question, not trying to be sarcastic: would you concede that the amount to which something is a "detail" may vary significantly per the use case, and requirements? On my desktop I might not care about whatever the DHCP client is doing, or the NTP server, but on a server that may very well be a different story. For example, I'm very interested in NTP and accurate timekeeping -- mostly as a personal hobby, but it's been useful at work as well. I for one would definitely not consider NTP one of those "details" that just "come with the bootup process".

[regarding the Leatherman] Software doesn't have mass so your analogy doesn't quite work.

Analogies can only be taken so far before they cease to make sense. However, in the software world you could consider a "mass" equivalent: code size / complexity. Building on that, just as the Leatherman can't be as good as all the individual tools combined without weighing too much, one could say systemd can't be as good as all the individual tools it aims to replace without being too big and complex. Program "mass", in this case, may impact anything from performance, to ease of configuration, to complicated dependencies, to security problems. I did find it interesting, however, what you mentioned in another email, about systemd implementing certain isolation features such as separate filesystem namespaces and so on. That may be very useful. I think the main point that we could hopefully all agree on here, is that it would be very difficult to have a single "one size fits all" solution. The requirements and concerns of the desktop, for example, are simply too different from those of the server or router space. systemd, for better or for worse, can't be the one magic bullet. Great or terrible as though it may be, I don't much like the total break in compatibility (correct me if I'm wrong). I'm not saying SysV is all that good, but there are other replacements, and new ones can be designed, but don't make it so that everyone-has-to-use-yours-or-else! I guess we have GNOME to thank for that... And that's what troubles me the most: the lack of choice that seems to be creeping up, with everyone just ganging up and jumping to systemd like the floor is on fire. I'm with Jay Ashworth on this one: what gives?? Also, for the person who considered this OT for not being about Cisco, keep in mind all the world is not a VAX^H^H^HCisco ;)

Israel G. Lugo wrote:

On 10/23/2014 12:05 AM, Jeffrey Ollie wrote:

...

systemd is a tool designed to get the system to a state where "real work" can be done. NTP servers, DHCP clients, consoles, aren't the real work of a system, or at least I hope not, because that would be boring to me.

Depends on the system.

Legitimate question, not trying to be sarcastic: would you concede that the amount to which something is a "detail" may vary significantly per the use case, and requirements? On my desktop I might not care about whatever the DHCP client is doing, or the NTP server, but on a server that may very well be a different story.

Re. NTP: Timekeeping is rather essential in lots of applications - like, for example, transit operations, where I currently spend my work life. An accurate, accessible central clock tends to be a rather important system component. And we're talking concerns in the range of seconds. When you start getting into serious real-time systems (laboratory instrumentation, utility operations, warfighting, ....) - yeah, NTP servers start getting really interesting, to a lot of people. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

Hey... how about not using selective editing to change the thread of discussion (see below) Jeffrey Ollie wrote:

...

Re. NTP: Timekeeping is rather essential in lots of applications - like, for example, transit operations, where I currently spend my work life. An accurate, accessible central clock tends to be a rather important system component. And we're talking concerns in the range of seconds. When you start getting into serious real-time systems (laboratory instrumentation, utility operations, warfighting, ....) - yeah, NTP servers start getting really interesting, to a lot of people. As I've already said a couple of times, systemd does not force a

On Wed, Oct 22, 2014 at 8:35 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote: particular NTP implementation on you. It comes with one (timedated), and has a utility to manage it (timedatectl) but the admin can install and use a different one if they like.

The only thing that has changed recently with respect to that is that timedatectl can no longer be used to manage chronyd or ntpd.

What you snipped out of the message was YOUR previous statement, to which I was responding:

systemd is a tool designed to get the system to a state where "real work" can be done. NTP servers, DHCP clients, consoles, aren't the real work of a system, or at least I hope not, because that would be boring to me.

If you're going to simply keep repeating "I like systemd, everything is copacetic" - maybe you should take your fanboy attitude elsewhere, and let those of us concerned with operational impacts have a meaningful conversation here. And maybe, you should check out some of the upstream bug reports re. systemd interactions with NTP. Plonk. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

"NANOG" <nanog-bounces@nanog.org> wrote on 10/22/2014 10:47:46 PM:

The arguments against systemd that I've seen so far:

1) It's different so it's bad. 2) There's a lot of code, there must be some really bad security problems just waiting to happen, so it's bad. 3) It doesn't do things the way we've always done them, so it's bad. 4) The systemd developers are jerks, so it's bad.

Hmmm. It seems that list is missing its most important item. As an impartial lurker, the primary objection I've seen is: 1. "Try to do everything" software is not optimal, and will lead to heartache. Joe

Once upon a time, Jay Ashworth <jra@baylink.com> said:

"Try to do everything *inside PID 1*" is the real problem.

And that is not what systemd is doing; make sure you know what you are complaining about. systemd-the-project != systemd-the-pid-1. PID 1 is responsible for managing services/daemons, and AFAIK that's all systemd's PID 1 does. -- Chris Adams <cma@cmadams.net>

Jay Ashworth wrote:

----- Original Message -----

...

From: "Chris Adams" <cma@cmadams.net> Once upon a time, Jay Ashworth <jra@baylink.com> said:

...

"Try to do everything *inside PID 1*" is the real problem. And that is not what systemd is doing; make sure you know what you are complaining about. systemd-the-project != systemd-the-pid-1. PID 1 is responsible for managing services/daemons, and AFAIK that's all systemd's PID 1 does. Indeed. I was quoting (I thought) better read people than me. If that's the case, I retract about 25% of my distaste for it.

Well, yeah... but there's this huge (and growing) network of interlocking dependencies, and, at least at the moment, no way to avoid installing systemd as PID1 during a fresh Debian install (no installer choice, and a bug in debootstrap that prevents preseeding). So, yes, you can uninstall systemd as PID1, but it sure seems to be rather difficult. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra

On 10/22/2014 08:20 PM, Simon Lyall wrote:

On Wed, 22 Oct 2014, Miles Fidelman wrote:

...

And maybe, you should check out some of the upstream bug reports re. systemd interactions with NTP.

If you think the current situation is all good then maybe you should look at other bugs for ntp. eg this one I that affected me with Ubuntu Disktop. They only run time syncing when the network is bounced so if you have a stable network then your machine will never sync:

https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1178933

Note that the importance of this has been set to "wishlist".

Looking at the ticket you submitted, I see this statement:

On Ubuntu 12.04 desktop, in System Settings > Time & Date, when the time is set to update Automatically from Internet it syncs once but then drifts out over a period of days.

In Syslog I can see that ntpdate is called on boot but is never called again.

I'm a long-time user of NTP, and what you are asking for is a no-good way of doing things. What you are supposed to do is use the ntpdate(8) utility *ONCE* on boot to initially set the system clock, then you have ntpd(8) running to do two things for you: sync up to one or more time sources, and discipline the local clock. What is "discipline the local clock?" This is the process of determining the *exact* frequency of the crystal clock in your local computer, and tuning your local clock hardware so that local real time is in sync with the world. This is because the accuracy of the crystal is specified to be rather loose (hundreds of parts per million), but the relative accuracy of a crystal is actually within tens of parts or even single-digit parts per million. So if you can measure the drift, you can in software compensate for it so you get a true-chimer. That's what the ntpd daemon does. Now, what is supposed to start the ntpd daemon in Ubuntu? DHCP has no significant effects on the filters used in ntpd(8). I didn't have any problem getting this to work "the right way" on Slackware, Red Hat, and Debian. It works "out of the box" with Fedora and CentOS, to the point that I don't even think much about it other than in the GUI to point to my local time source on system install. It even works with the Ubuntu 8.04.4 LTS server that was forced down my throat a few years back, which I'm in the process of trying to retire in favor of a CentOS 6.5 server implementation. (At least THAT Ubuntu-loving sysadmin is gone now.) What happens on your box when you do "/etc/init.d/ntp start"? And how frequently do you reboot your box? It sounds like it's up all the time, which means the local clock can be trimmed very accurately indeed. That's the SERVER way of running a time synchronization. So it would appear that you have a quarrel with GUI support, not with NTP itself.

On 10/23/14 4:01 PM, Simon Lyall wrote:

On Wed, 22 Oct 2014, Stephen Satchell wrote:

...

...

On Wed, 22 Oct 2014, Miles Fidelman wrote:

...

And maybe, you should check out some of the upstream bug reports re. systemd interactions with NTP.

If you think the current situation is all good then maybe you should look at other bugs for ntp. eg this one I that affected me with Ubuntu Disktop. They only run time syncing when the network is bounced so if you have a stable network then your machine will never sync:

https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1178933 [..] I'm a long-time user of NTP, and what you are asking for is a no-good way of doing things. What you are supposed to do is use the ntpdate(8) utility *ONCE* on boot to initially set the system clock, then you have ntpd(8) running to do two things for you: sync up to one or more time

On 10/22/2014 08:20 PM, Simon Lyall wrote: sources, and discipline the local clock. [..] That's the SERVER way of running a time synchronization. So it would appear that you have a quarrel with GUI support, not with NTP itself.

What my point was is that the "simple default for end users" [1] is already significantly broken in Ubuntu (that is just one bug that bit me, there are plenty of others).

The systemd system seems to offer and improvement on the existing "simple default" setup while still enabling experts to run a full ntpd install if they wish.

[1] - I know how to setup and run ntpd, I didn't expect to need to do it on my workstation however.

If you are actually arguing that because Ubuntu made a mistake on how the "Internet time synchronization" option is configured, therefore we need systemd, you need to rethink your position. :) FWIW, the problem you're describing with that option is real, and was revisited in later versions. As of 14.04 it was still broken, but for a different reason having to do with permissions on the ntpd install. However, fixing that problem doesn't require systemd, it requires fixing *that* problem. I am not against systemd per se, I honestly don't know enough about it to form an intelligent opinion. The line of reasoning (I believe to be) espoused here is quite concerning, "If there is a problem, we need to bring the solution into systemd." To the extent that's accurate, it's overwhelmingly likely to be wrong. I could say a lot more about Unix system design philosophies from my time in the FreeBSD project, but this thread started off-topic, and has only gotten worse. :) Doug

On Wed, Oct 22, 2014 at 9:18 PM, <Valdis.Kletnieks@vt.edu> wrote:

On Wed, 22 Oct 2014 20:45:01 -0500, Jeffrey Ollie said:

...

As I've already said a couple of times, systemd does not force a particular NTP implementation on you. It comes with one (timedated), and has a utility to manage it (timedatectl) but the admin can install and use a different one if they like.

Yeah, and if you want anything else to integrate in with systemd other than the supplied SNTP, you can pretty much go pound sand, according to Marcel Holtmann:

"So this means that you get a really good basis where clocks are synchronized. If you want something better, you can install it. It is a waste of time trying to integrate anything else than timesyncd since if you use anything else, you will manually configure it and use its advanced options. If you do not need the advanced features, then why install other NTP clients in the first place."

http://lists.freedesktop.org/archives/systemd-devel/2014-August/022572.html

You should read the whole thread at freedesktop - it's pretty obvious that there's a really heavy bias towards "we have a solution that's good for desktops, and if you have a different use case, go pound sand".

Yes, I've read the thread, and I think "go pound sand" is an unfair characterization of what *I* saw in that thread. To achieve the level of integration that timedated has with the rest of systemd would require more than just putting code into timedatectl to write out /etc/ntpd.conf and starting a service. timedated talks to networkd (that DHCP server that everyone is hating on as well) in real-time to determine the state of the network and to get any NTP servers that were sent in DHCP packets. To do that for chronyd or ntpd in the same way would require code changes and the systemd developers didn't want to do the work, as far as I know no one else has done the work, and I'm skeptical of the chances that such patches would get accepted upstream. So, if you want/need to run chronyd or ntpd you can, it'll "integrate" into the system just like any other service would, and you'll be no better/worse off than before timedated came into existence. -- Jeff Ollie

On 2014-10-23 9:15, Matt Palmer wrote:

On Wed, Oct 22, 2014 at 10:05:30PM -0500, Jeffrey Ollie wrote:

...

To achieve the level of integration that timedated has with the rest of systemd would require more than just putting code into timedatectl to write out /etc/ntpd.conf and starting a service. timedated talks to networkd (that DHCP server that everyone is hating on as well) in real-time to determine the state of the network and to get any NTP servers that were sent in DHCP packets. To do that for chronyd or ntpd in the same way would require code changes and the systemd developers didn't want to do the work,

This is the core problem with systemd, in my mind -- and what has gotten Linus, amongst other people, so thoroughly cheesed off with the systemd devs. They don't play well with other children. They don't appear particularly interested in reusing any existing code, because it's a lot more fun to write new code. I'm a strong proponent of Joel Spolsky's views on rewrites (sorry, no URL, I'm on the train) and I don't doubt that all the

http://www.joelonsoftware.com/articles/fog0000000069.html

same problems will come to haunt systemd on its way from being the new kid on the block to being legacy code[1].

- Matt

[1] A computer industry term which means, "it works".

On 10/22/2014 02:43 PM, Rich Kulawiec wrote:

A Leatherman pocket multitool is highly useful: I've had one for years. It's great. Until you need two screwdrivers at the same time...at which point it becomes obvious why serious mechanics/craftsmen carry around a toolbox with dozens of tools and why glomming all of those into one supertool would be A Very Bad Move.

I find this analogy very, very interesting. I'm looking at my Leatherman tool, and see that every single tool is based on a standard; nothing "invented here just for the Leatherman." The Phillips screwdriver is the standard P2 shape, not some off-the-wall grinding. And so forth.

Similarly, the monolithic (and ever-expanding) nature of systemd is a strategic design error. That's probably not obvious to people who measure their experience in years instead of decades -- it wouldn't have been obvious to me back in the day either. But it's pretty clear from here, and dismissing it as "hey you kids get off my lawn" geezer whining is not going to advance the discussion.

It's one thing to integrate a bridge to existing functions into a system. It's another to re-invent things for the sake of re-inventing them. If you have a perfectly good DHCP package, work *with* it instead of going through the trial of building and debugging your own version. Why duplicate effort? Why have two of something that has to be maintained instead of concentrating on improving the one you have? Now, sometimes you do have to sit down and say "this is hopeless, let's start from scratch" -- particularly when it was done to a bad spec and tried to do too much without step-wise refinement. (healthcare.gov, anyone?) But even when HeartBleed-SSH was forked, it wasn't abandoned and started from scratch; the BSD people just pruned all the non-POSIX guck and stripped it down to essentials. And did a proper security audit. We don't need systemd to turn into a hydra. If there is an issue with NTP or DHCP or whatever, systemd developers should go to the upstream and fix the issue in that package, not clone its own version and, perhaps, make the same mistakes all over again.

On 10/23/2014 at 10:56 AM Randy wrote: |I've enjoyed kernel hot patches (ksplice) until now. | |So my primary concern is that updates to systemd appears to require a |full reboot: | |http://forums.fedoraforum.org/showthread.php?t=300166 | |Is systemd really like a 2nd 'kernel' -- demanding mass reboots every |time a security issue is discovered? | |I hope not! ============= GNU/Linux is morphing into GNU/systemd ....

On Thu, Oct 23, 2014 at 12:12:13PM -0400, Jim Popovitch wrote:

On Thu, Oct 23, 2014 at 12:04 PM, Mike. <the.lists@mgm51.com> wrote:

...

GNU/Linux is morphing into GNU/systemd ....

Let's start calling it Systemd/Linux... that will get RMS on their case real fast. :-)

I don't think they've done anything to deserve *that*... - Matt -- I don't do veggies if I can help it. -- stevo If you could see your colon, you'd be horrified. -- Iain Broadfoot If he could see his colon, he'd be management. -- David Scheidt

On Thu, Oct 23, 2014 at 04:17:14PM -0400, Miles Fidelman wrote:

Matt Palmer wrote:

...

On Thu, Oct 23, 2014 at 12:12:13PM -0400, Jim Popovitch wrote:

...

On Thu, Oct 23, 2014 at 12:04 PM, Mike. <the.lists@mgm51.com> wrote:

...

GNU/Linux is morphing into GNU/systemd .... Let's start calling it Systemd/Linux... that will get RMS on their case real fast. :-) I don't think they've done anything to deserve *that*...

PulseAudio, systemd - yeah... maybe they have :-)

Of all the PoetteringStuff out there, PA's the one that's caused me the least problems. I know that I'm in some vanishingly small minority there, because it seems to cause problems for everyone around me. One of my colleagues needs to "jiggle the wires" every time he wants to use gHangouts because PA gets itself in a tizzy. Now *Avahi*, on the other hand... FML. - Matt

Randy wrote:

I've enjoyed kernel hot patches (ksplice) until now.

So my primary concern is that updates to systemd appears to require a full reboot:

http://forums.fedoraforum.org/showthread.php?t=300166

Is systemd really like a 2nd 'kernel' -- demanding mass reboots every time a security issue is discovered?

I hope not!

-- ~Randy

Given that their focus is on reducing boot-time, why wouldn't they want to highlight that point by making you do it often?? It is clear that the systemd developers are on a solid track to catch up with Windows-9x/ME. At the timeframe of Win9x/ME development, Gates was hammering "boot-time : boot-time : boot-time" on a regular basis. My feedback to him and his direct reports was that getting rid of reasons to reboot was much more important than making an all too common activity "less unpleasant". Clearly a monolithic super ker-daemon will be at least as easy to use and maintain as a Win9x machine ... ;) Tony