I've used this on my personal machines for years, and I really appreciate how small, comprehensible, and rigorous the code base of each tool is. Not including the libraries, the most important bits, s6-supervise (the process monitor) and s6-svscan (pid 1) are under 1000 lines of code.
The coding style is a little terse and is clearly influenced by djb. The author has some (justified) mistrust of the C standard library, so you'll need to get used to the alternatives in their "skalibs" library, but I'd feel much more comfortable debugging a problem with this suite than I would with systemd.
I really like s6. I've been using it for a couple of years now as it's how HAProxy runs all of their containers, and modifying it has been pretty simple. The docs are excellent, and it's in 'just works' territory. I'd place it alongside redis and beanstalkd for 'just works' territory.
I've gone through this phase a while back where I had to choose an init process (PID 1) to run multiple-processes in a container. Here's my write up here. s6 is in there as well. https://ahmet.im/blog/minimal-init-process-for-containers/
Runit seems way simpler. It's both good and bad. I like its minimalism, but it involves more imperative shell script style, less declarative.
The documentation is very terse, too; I learned a lot from reading Void init scripts which I wish was reflected in the docs. (Maybe I should send a patch.)
I've found that s6 and s6-rc are correct to a fault. It works as documented, but you can't take shortcuts like you can with other init systems that don't have a validating compiler compilation step.
"Aggressive" processes that for one reason or another escape well-mannered foreground supervision; for example, often those that daemonize themselves incorrectly. Either they don't generate PID files correctly, or they don't cleanup after old PID files leaving them dangling, or they have useless PID files that point to the incorrect child processes that can't do useful work when they receive signals to reload/shutdown themselves safely.
Handling those daemons by jailing them with Linux-centric cgroups is the one thing systemd allows you to "get away with".
Are there any good how-tos for how to set up a non-trivial container with s6 and s6-rc? Last time I looked at this the documentation was pretty sparse, and more of a reference and design document than a set of how-tos.
There's a huge community https://www.linuxserver.io/ for people building "home server" containers that all use the s6-overlay, hundreds of examples there. They have a lot of tutorials and a very busy Discord, Reddit, etc with all levels of experience from container developers to people who don't program and are just getting into Docker. I run a bunch of these containers myself and am pretty happy with how adaptable they are.
I've been using execline to build sandboxes. This requires passing around several file descriptors between processes, which was a pain in posix shell or python.
I'm still missing some corner bits, but I'm quite happy with it for this rather exotic use case.
Does it have a concept of user services? That's a very important feature with a lot of pitfalls and OpenRC is only beginning to get it (https://github.com/OpenRC/openrc/pull/723).
The way s6 is designed, it doesn't really do "system" services or "user" services, it just does services. I guess this is also why it's a great fit inside containers.
Your services might want to change user (if they're system services), or save data to $HOME (if they're user services). That's purely determined by the service definition itself.
I've been using s6 for user services for over a year now quite happily. I'm using my frontend tool to manage them, and it basically just invokes s6 commands and provides a high-level interface for me to use.
No, that's not about changing users, any init system can do that. It's about having an interface where any unprivileged user can setup services (that could depend on systems services); like cron, basically.
I believe they are talking about a situation where S6 is pid1 / system init, starting "system" services (I know the distinction is not in S6), aka things configured by root. I'd expect DNS, cron, NTP, SSH to be in that list.
Then, also, separately from the pid1 S6, unprivileged users can have their own supervision trees independent of the pid1 instance, but that can depend upon things started by pid1.
Example:
pid1 S6 launches root-configured service, S as part of its supervision tree.
Unprivileged user, U, has a discrete instance of S6, which I'll refer to as pidN S6 with a supervision tree that U controls (e.g. in each user's $HOME). pidN S6 launches a service T which depends on S.
Is there a way for pidN S6 to wait on pid1 S6 to note that S is ready, blocking the launch of T until that notification.
The separate supervision trees part seems trivial based on my understanding of S6.
I also see in the docs that a daemon can signal its own supervising S6 that it is ready, so that S6 can manage the dependencies among multiple services in its supervision tree. The part that seems unclear to me (and the original question-asker) is if there can be cross-supervision-tree dependencies.
The brute-force way to do this would be to have the PID1 S6 launch a pidN S6 process for each user, with the supervision tree defined in that user's $HOME. Then the pid1 S6 can ensure that each pidN instance is dependent upon all PID1 services, so that the user instances only launch after all possible dependencies are initialized. This would still depend on an implicit contract between root and users so that root does in fact launch all upstream dependencies.
Alternatively, I guess you could give fine-grained sudo privileges to each user that gets a pidN S6, so that they can check the status of running daemons under the pid1 S6.
I don't know enough about how s6 work, so I'll probably write stupid things, but how do you start/enable services? By running sh/execline scripts and symlinking them in specific dirs? Well, you need privileges for that, no?
I can "crontab -e" as any user without privilege and edit my own crontab, separate from the root one, that's what I meant.
> On the service manager side, however, switching service managers is huge work, and I don't think you realize the amount of work that what you're suggesting (supporting several service managers) represents. It may seem simple, but it's really not, because there are so many small details, small differences in the way various service managers operate, that you cannot gloss over if you want things to work.
This was interesting to read. I wonder if Bercot wrote anything in detail about what this entails; if this is referring to per-package edge-cases, or the complexity of the logic required to facilitate interactions between services.
I wonder if anyone remembers the bad old says of System V init scripts without process supervision and with sequential runlevel-based startup and shutdown. Database server crashed (and didn't restart), and now you're looking for a log entry or cron email to troubleshoot what went wrong and having to manually restart it like an animal.
I pray that, in the future, *nix/POSIX adds process management programmable<->declarative (congruent) supervisor coprocesses and lifecycle hooks such that running a system becomes more like the Erlang BEAM runtime.
I love how easy it is to integrate the tiny, standalone tools that make up s6 without having to commit to an entire init system. It's a great example of the benefits of the UNIX philosophy.
Having tried similar things myself, I concur. One way to add a bit of sanity is to mandate running dnsmasq which has reasonable control interfaces, but it's overkill.
I use it in most of my containers. Very handy if you need to run some kind of pre-script before you main program, or need to run multiple programs in the same container.
Personally, I find the opinionation of s6 to be infuriating. As a process executor inside of a Linux container, I find myself spending more time reading source code to figure out why my application has failed (hint: it's probably due to goofy permission and environment state passing) than I would prefer.
I'm happy that s6 is so beloved, but I am not amongst that crowd.
I've used this on my personal machines for years, and I really appreciate how small, comprehensible, and rigorous the code base of each tool is. Not including the libraries, the most important bits, s6-supervise (the process monitor) and s6-svscan (pid 1) are under 1000 lines of code.
The coding style is a little terse and is clearly influenced by djb. The author has some (justified) mistrust of the C standard library, so you'll need to get used to the alternatives in their "skalibs" library, but I'd feel much more comfortable debugging a problem with this suite than I would with systemd.
Also, the related tool, execline, is a trip: https://skarnet.org/software/execline/ .
I really like s6. I've been using it for a couple of years now as it's how HAProxy runs all of their containers, and modifying it has been pretty simple. The docs are excellent, and it's in 'just works' territory. I'd place it alongside redis and beanstalkd for 'just works' territory.
> as it's how HAProxy runs all of their containers
Could you elaborate on this? I'm curious about using s6 as a container runner.
I've gone through this phase a while back where I had to choose an init process (PID 1) to run multiple-processes in a container. Here's my write up here. s6 is in there as well. https://ahmet.im/blog/minimal-init-process-for-containers/
I personally am a fan of runit
https://smarden.org/runit/
But s6 is excellent as well.
This was an interesting comparison: https://skarnet.org/software/s6/why.html
I've also found this useful: https://ahmet.im/blog/minimal-init-process-for-containers/
And this looks really good: https://wiki.gentoo.org/wiki/Comparison_of_init_systems
Runit seems way simpler. It's both good and bad. I like its minimalism, but it involves more imperative shell script style, less declarative. The documentation is very terse, too; I learned a lot from reading Void init scripts which I wish was reflected in the docs. (Maybe I should send a patch.)
If you're curious where these tools might be used, one place is it's a popular init system for containers, notably https://www.linuxserver.io/
I've found that s6 and s6-rc are correct to a fault. It works as documented, but you can't take shortcuts like you can with other init systems that don't have a validating compiler compilation step.
What shortcuts do others let you get away with?
"Aggressive" processes that for one reason or another escape well-mannered foreground supervision; for example, often those that daemonize themselves incorrectly. Either they don't generate PID files correctly, or they don't cleanup after old PID files leaving them dangling, or they have useless PID files that point to the incorrect child processes that can't do useful work when they receive signals to reload/shutdown themselves safely.
Handling those daemons by jailing them with Linux-centric cgroups is the one thing systemd allows you to "get away with".
I use s6 in my product. It’s the standard for every app deployed using our platform.
Couldn’t be happier with the decision.
You can see how we provision s6 files here https://github.com/upmaru/pakman
Are there any good how-tos for how to set up a non-trivial container with s6 and s6-rc? Last time I looked at this the documentation was pretty sparse, and more of a reference and design document than a set of how-tos.
This is well thought-out and quite well documented: https://github.com/just-containers/s6-overlay
There's a huge community https://www.linuxserver.io/ for people building "home server" containers that all use the s6-overlay, hundreds of examples there. They have a lot of tutorials and a very busy Discord, Reddit, etc with all levels of experience from container developers to people who don't program and are just getting into Docker. I run a bunch of these containers myself and am pretty happy with how adaptable they are.
The associated execline project is such a joy to use, compared to writing bash scripts, especially for the purposes of process management.
I've been using execline to build sandboxes. This requires passing around several file descriptors between processes, which was a pain in posix shell or python.
I'm still missing some corner bits, but I'm quite happy with it for this rather exotic use case.
Does it have a concept of user services? That's a very important feature with a lot of pitfalls and OpenRC is only beginning to get it (https://github.com/OpenRC/openrc/pull/723).
The way s6 is designed, it doesn't really do "system" services or "user" services, it just does services. I guess this is also why it's a great fit inside containers.
Your services might want to change user (if they're system services), or save data to $HOME (if they're user services). That's purely determined by the service definition itself.
I've been using s6 for user services for over a year now quite happily. I'm using my frontend tool to manage them, and it basically just invokes s6 commands and provides a high-level interface for me to use.
No, that's not about changing users, any init system can do that. It's about having an interface where any unprivileged user can setup services (that could depend on systems services); like cron, basically.
I've no idea what you're trying to say.
I believe they are talking about a situation where S6 is pid1 / system init, starting "system" services (I know the distinction is not in S6), aka things configured by root. I'd expect DNS, cron, NTP, SSH to be in that list.
Then, also, separately from the pid1 S6, unprivileged users can have their own supervision trees independent of the pid1 instance, but that can depend upon things started by pid1.
Example:
pid1 S6 launches root-configured service, S as part of its supervision tree.
Unprivileged user, U, has a discrete instance of S6, which I'll refer to as pidN S6 with a supervision tree that U controls (e.g. in each user's $HOME). pidN S6 launches a service T which depends on S.
Is there a way for pidN S6 to wait on pid1 S6 to note that S is ready, blocking the launch of T until that notification.
The separate supervision trees part seems trivial based on my understanding of S6.
I also see in the docs that a daemon can signal its own supervising S6 that it is ready, so that S6 can manage the dependencies among multiple services in its supervision tree. The part that seems unclear to me (and the original question-asker) is if there can be cross-supervision-tree dependencies.
The brute-force way to do this would be to have the PID1 S6 launch a pidN S6 process for each user, with the supervision tree defined in that user's $HOME. Then the pid1 S6 can ensure that each pidN instance is dependent upon all PID1 services, so that the user instances only launch after all possible dependencies are initialized. This would still depend on an implicit contract between root and users so that root does in fact launch all upstream dependencies.
Alternatively, I guess you could give fine-grained sudo privileges to each user that gets a pidN S6, so that they can check the status of running daemons under the pid1 S6.
I don't know enough about how s6 work, so I'll probably write stupid things, but how do you start/enable services? By running sh/execline scripts and symlinking them in specific dirs? Well, you need privileges for that, no?
I can "crontab -e" as any user without privilege and edit my own crontab, separate from the root one, that's what I meant.
Can't wait until this is integrated with alpine linux. Has there been recent progress on this? Been watching it for years.
There's some detailed status updates on this relevant issue: https://gitlab.alpinelinux.org/alpine/tsc/-/issues/78
> On the service manager side, however, switching service managers is huge work, and I don't think you realize the amount of work that what you're suggesting (supporting several service managers) represents. It may seem simple, but it's really not, because there are so many small details, small differences in the way various service managers operate, that you cannot gloss over if you want things to work.
This was interesting to read. I wonder if Bercot wrote anything in detail about what this entails; if this is referring to per-package edge-cases, or the complexity of the logic required to facilitate interactions between services.
feels pretty integrated in our containers; just works. what's there to integrate?
I think they (and me!) were looking for news on
https://skarnet.com/projects/service-manager.html
I wonder if anyone remembers the bad old says of System V init scripts without process supervision and with sequential runlevel-based startup and shutdown. Database server crashed (and didn't restart), and now you're looking for a log entry or cron email to troubleshoot what went wrong and having to manually restart it like an animal.
I pray that, in the future, *nix/POSIX adds process management programmable<->declarative (congruent) supervisor coprocesses and lifecycle hooks such that running a system becomes more like the Erlang BEAM runtime.
I love how easy it is to integrate the tiny, standalone tools that make up s6 without having to commit to an entire init system. It's a great example of the benefits of the UNIX philosophy.
Now we need a DNS resolver with these properties.
I'm pretty sure any of them except systemd-resolved satisfy
> without having to commit to an entire init system.
And many are small modular tools that integrate with other pieces.
Creators of Tailscale think differently: https://tailscale.com/blog/sisyphean-dns-client-linux
Having tried similar things myself, I concur. One way to add a bit of sanity is to mandate running dnsmasq which has reasonable control interfaces, but it's overkill.
https://skarnet.org/software/s6-networking/ & https://skarnet.org/software/s6-dns/ won't do?
I use it in most of my containers. Very handy if you need to run some kind of pre-script before you main program, or need to run multiple programs in the same container.
S6 and really all skarnet code is kind of "djb c" which makes reasoning about it a bit convoluted.
used it in containers. it replaced some python script solution, so i could reduce the image size as i otherwise did not need python
We use https://github.com/Yelp/pgctl heavily, which uses s6!
It’s good.
Personally, I find the opinionation of s6 to be infuriating. As a process executor inside of a Linux container, I find myself spending more time reading source code to figure out why my application has failed (hint: it's probably due to goofy permission and environment state passing) than I would prefer.
I'm happy that s6 is so beloved, but I am not amongst that crowd.