systemd for Administrators, Part III

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/systemd-for-admins-3.html

Here’s the third installment of my ongoing
series about
systemd
for administrators
.

How Do I Convert A SysV Init Script Into A systemd Service File?

Traditionally, Unix and Linux services (daemons) are started
via SysV init scripts. These are Bourne Shell scripts, usually
residing in a directory such as /etc/rc.d/init.d/ which when
called with one of a few standardized arguments (verbs) such as
start, stop or restart controls,
i.e. starts, stops or restarts the service in question. For starts
this usually involves invoking the daemon binary, which then forks a
background process (more precisely daemonizes). Shell scripts
tend to be slow, needlessly hard to read, very verbose and
fragile. Although they are immensly flexible (after all, they are just
code) some things are very hard to do properly with shell scripts,
such as ordering parallized execution, correctly supervising processes
or just configuring execution contexts in all detail. systemd provides
compatibility with these shell scripts, but due to the shortcomings
pointed out it is recommended to install native systemd service files
for all daemons installed. Also, in contrast to SysV init scripts
which have to be adjusted to the distribution systemd service files
are compatible with any kind of distribution running systemd (which
become more and more these days…). What follows is a terse guide how
to take a SysV init script and translate it into a native systemd
service file. Ideally, upstream projects should ship and install
systemd service files in their tarballs. If you have successfully
converted a SysV script according to the guidelines it might hence be
a good idea to submit the file as patch to upstream. How to prepare a
patch like that will be discussed in a later installment, suffice to
say at this point that the daemon(7)
manual page shipping with systemd contains a lot of useful information
regarding this.

So, let’s jump right in. As an example we’ll convert the init
script of the ABRT daemon into a systemd service file. ABRT is a
standard component of every Fedora install, and is an acronym for
Automatic Bug Reporting Tool, which pretty much describes what it
does, i.e. it is a service for collecting crash dumps. Its SysV script I have uploaded
here.

The first step when converting such a script is to read it
(surprise surprise!) and distill the useful information from the
usually pretty long script. In almost all cases the script consists of
mostly boilerplate code that is identical or at least very similar in
all init scripts, and usually copied and pasted from one to the
other. So, let’s extract the interesting information from the script
linked above:

  • A description string for the service is “Daemon to detect
    crashing apps
    “. As it turns out, the header comments include a
    redundant number of description strings, some of them describing less
    the actual service but the init script to start it. systemd services
    include a description too, and it should describe the service and not
    the service file.
  • The LSB header[1] contains dependency
    information. systemd due to its design around socket-based activation
    usually needs no (or very little) manually configured
    dependencies. (For details regarding socket activation see the original
    announcement blog post.
    ) In this case the dependency on
    $syslog (which encodes that abrtd requires a syslog daemon),
    is the only valuable information. While the header lists another
    dependency ($local_fs) this one is redundant with systemd as
    normal system services are always started with all local file systems
    available.
  • The LSB header suggests that this service should be started in
    runlevels 3 (multi-user) and 5 (graphical).
  • The daemon binary is /usr/sbin/abrtd

And that’s already it. The entire remaining content of this
115-line shell script is simply boilerplate or otherwise redundant
code: code that deals with synchronizing and serializing startup
(i.e. the code regarding lock files) or that outputs status messages
(i.e. the code calling echo), or simply parsing of the verbs (i.e. the
big case block).

From the information extracted above we can now write our systemd service file:

[Unit]
Description=Daemon to detect crashing apps
After=syslog.target

[Service]
ExecStart=/usr/sbin/abrtd
Type=forking

[Install]
WantedBy=multi-user.target

A little explanation of the contents of this file: The
[Unit] section contains generic information about the
service. systemd not only manages system services, but also devices,
mount points, timer, and other components of the system. The generic
term for all these objects in systemd is a unit, and the
[Unit] section encodes information about it that might be
applicable not only to services but also in to the other unit types
systemd maintains. In this case we set the following unit settings: we
set the description string and configure that the daemon shall be
started after Syslog[2], similar to what is encoded in the
LSB header of the original init script. For this Syslog dependency we
create a dependency of type After= on a systemd unit
syslog.target. The latter is a special target unit in systemd
and is the standardized name to pull in a syslog implementation. For
more information about these standardized names see the systemd.special(7). Note
that a dependency of type After= only encodes the suggested
ordering, but does not actually cause syslog to be started when abrtd
is — and this is exactly what we want, since abrtd actually works
fine even without syslog being around. However, if both are started
(and usually they are) then the order in which they are is controlled
with this dependency.

The next section is [Service] which encodes information
about the service itself. It contains all those settings that apply
only to services, and not the other kinds of units systemd maintains
(mount points, devices, timers, …). Two settings are used here:
ExecStart= takes the path to the binary to execute when the
service shall be started up. And with Type= we configure how
the service notifies the init system that it finished starting up. Since
traditional Unix daemons do this by returning to the parent process
after having forked off and initialized the background daemon we set
the type to forking here. That tells systemd to wait until
the start-up binary returns and then consider the processes still
running afterwards the daemon processes.

The final section is [Install]. It encodes information
about how the suggested installation should look like, i.e. under
which circumstances and by which triggers the service shall be
started. In this case we simply say that this service shall be started
when the multi-user.target unit is activated. This is a
special unit (see above) that basically takes the role of the classic
SysV Runlevel 3[3]. The setting WantedBy= has
little effect on the daemon during runtime. It is only read by the
systemctl enable command, which is the recommended way to
enable a service in systemd. This command will simply ensure that our
little service gets automatically activated as soon as
multi-user.target is requested, which it is on all normal
boots[4].

And that’s it. Now we already have a minimal working systemd
service file. To test it we copy it to
/etc/systemd/system/abrtd.service and invoke systemctl
daemon-reload
. This will make systemd take notice of it, and now
we can start the service with it: systemctl start
abrtd.service
. We can verify the status via systemctl status
abrtd.service
. And we can stop it again via systemctl stop
abrtd.service
. Finally, we can enable it, so that it is activated
by default on future boots with systemctl enable
abrtd.service
.

The service file above, while sufficient and basically a 1:1
translation (feature- and otherwise) of the SysV init script still has room for
improvement. Here it is a little bit updated:

[Unit]
Description=ABRT Automated Bug Reporting Tool
After=syslog.target

[Service]
Type=dbus
BusName=com.redhat.abrt
ExecStart=/usr/sbin/abrtd -d -s

[Install]
WantedBy=multi-user.target

So, what did we change? Two things: we improved the description
string a bit. More importantly however, we changed the type of the
service to dbus and configured the D-Bus bus name of the
service. Why did we do this? As mentioned classic SysV services
daemonize after startup, which usually involves double forking
and detaching from any terminal. While this is useful and necessary
when daemons are invoked via a script, this is unnecessary (and slow)
as well as counterproductive when a proper process babysitter such as
systemd is used. The reason for that is that the forked off daemon
process usually has little relation to the original process started by
systemd (after all the daemonizing scheme’s whole idea is to remove
this relation), and hence it is difficult for systemd to figure out
after the fork is finished which process belonging to the service is
actually the main process and which processes might just be
auxiliary. But that information is crucial to implement advanced
babysitting, i.e. supervising the process, automatic respawning on
abnormal termination, collectig crash and exit code information and
suchlike. In order to make it easier for systemd to figure out the
main process of the daemon we changed the service type to
dbus. The semantics of this service type are appropriate for
all services that take a name on the D-Bus system bus as last step of
their initialization[5]. ABRT is one of those. With this setting systemd
will spawn the ABRT process, which will no longer fork (this is
configured via the -d -s switches to the daemon), and systemd
will consider the service fully started up as soon as
com.redhat.abrt appears on the bus. This way the process
spawned by systemd is the main process of the daemon, systemd has a
reliable way to figure out when the daemon is fully started up and
systemd can easily supervise it.

And that’s all there is to it. We have a simple systemd service
file now that encodes in 10 lines more information than the original
SysV init script encoded in 115. And even now there’s a lot of room
left for further improvement utilizing more features systemd
offers. For example, we could set Restart=restart-always to
tell systemd to automatically restart this service when it dies. Or,
we could use OOMScoreAdjust=-500 to ask the kernel to please
leave this process around when the OOM killer wreaks havoc. Or, we
could use CPUSchedulingPolicy=idle to ensure that abrtd
processes crash dumps in background only, always allowing the kernel
to give preference to whatever else might be running and needing CPU
time.

For more information about the configuration options mentioned
here, see the respective man pages systemd.unit(5),
systemd.service(5),
systemd.exec(5). Or,
browse all of
systemd’s man pages
.

Of course, not all SysV scripts are as easy to convert as this
one. But gladly, as it turns out the vast majority actually are.

That’s it for today, come back soon for the next installment in our series.

Footnotes

[1] The LSB header of init scripts is a convention of
including meta data about the service in comment blocks at the top of
SysV init scripts and is
defined by the Linux Standard Base
. This was intended to
standardize init scripts between distributions. While most
distributions have adopted this scheme, the handling of the headers
varies greatly between the distributions, and in fact still makes it
necessary to adjust init scripts for every distribution. As such the LSB spec
never kept the promise it made.

[2] Strictly speaking, this dependency does not even have to
be encoded here, as it is redundant in a system where the Syslog
daemon is socket activatable. Modern syslog systems (for example
rsyslog v5) have been patched upstream to be socket-activatable. If
such a init system is used configuration of the
After=syslog.target dependency is redundant and
implicit. However, to maintain compatibility with syslog services that
have not been updated we include this dependency here.

[3] At least how it used to be defined on Fedora.

[4] Note that in systemd the graphical bootup
(graphical.target, taking the role of SysV runlevel 5) is an
implicit superset of the console-only bootup
(multi-user.target, i.e. like runlevel 3). That means hooking
a service into the latter will also hook it into the
former.

[5] Actually the majority of services of the default Fedora
install now take a name on the bus after startup.