Post Syndicated from Lennart Poettering original https://0pointer.net/blog/walkthrough-for-portable-services.html

Portable Services with systemd v239

systemd
v239
contains a great number of new features. One of them is first class
support for Portable
Services. In this blog story
I’d like to shed some light on what they are and why they might be
interesting for your application.

What are “Portable Services”?

The “Portable Service” concept takes inspiration from classic
chroot() environments as well as container management and brings a
number of their features to more regular system service management.

While the definition of what a “container” really is is hotly debated,
I figure people can generally agree that the “container” concept
primarily provides two major features:

Resource bundling: a container generally brings its own file system
tree along, bundling any shared libraries and other resources it
might need along with the main service executables.
Isolation and sand-boxing: a container operates in a name-spaced
environment that is relatively detached from the host. Besides
living in its own file system namespace it usually also has its own
user database, process tree and so on. Access from the container to
the host is limited with various security technologies.
Trending
Mutt 2.0 released

Of these two concepts the first one is also what traditional UNIX
chroot() environments are about.

Both resource bundling and isolation/sand-boxing are concepts systemd
has implemented to varying degrees for a longer time. Specifically,
RootDirectory=
and
RootImage=
have been around for a long time, and so have been the various
sand-boxing
features
systemd provides. The Portable Services concept builds on that,
putting these features together in a new, integrated way to make them
more accessible and usable.

OK, so what precisely is a “Portable Service”?

Much like a container image, a portable service on disk can be just a
directory tree that contains service executables and all their
dependencies, in a hierarchy resembling the normal Linux directory
hierarchy. A portable service can also be a raw disk image, containing
a file system containing such a tree (which can be mounted via a
loop-back block device), or multiple file systems (in which case they
need to follow the Discoverable Partitions
Specification
and be located within a GPT partition table). Regardless whether the
portable service on disk is a simple directory tree or a raw disk
image, let’s call this concept the portable service image.

Such images can be generated with any tool typically used for the
purpose of installing OSes inside some directory, for example dnf --installroot= or debootstrap. There are very few requirements made
on these trees, except the following two:

The tree should carry systemd unit
files
for relevant services in them.
The tree should carry
/usr/lib/os-release
(or /etc/os-release) OS release information.

Of course, as you might notice, OS trees generated from any of today’s
big distributions generally qualify for these two requirements without
any further modification, as pretty much all of them adopted
/usr/lib/os-release and tend to ship their major services with
systemd unit files.

A portable service image generated like this can be “attached” or
“detached” from a host:

“Attaching” an image to a host is done through the new
portablectl attach
command. This command dissects the image, reading the os-release
information, and searching for unit files in them. It then copies
relevant unit files out of the images and into
/etc/systemd/system/. After that it augments any copied service
unit files in two ways: a drop-in adding a RootDirectory= or
RootImage= line is added in so that even though the unit files
are now available on the host when started they run the referenced
binaries from the image. It also symlinks in a second drop-in which
is called a “profile”, which is supposed to carry additional
security settings to enforce on the attached services, to ensure
the right amount of sand-boxing.
“Detaching” an image from the host is done through portable detach. It reverses the steps above: the unit files copied out are
removed again, and so are the two drop-in files generated for them.

While a portable service is attached its relevant unit files are made
available on the host like any others: they will appear in systemctl list-unit-files, you can enable and disable them, you can start them
and stop them. You can extend them with systemctl edit. You can
introspect them. You can apply resource management to them like to any
other service, and you can process their logs like any other service
and so on. That’s because they really are native systemd services,
except that they have ‘twist’ if you so will: they have tougher
security by default and store their resources in a root directory or
image.

And that’s already the essence of what Portable Services are.

A couple of interesting points:

Even though the focus is on shipping service unit files in
portable service images, you can actually ship timer units, socket
units, target units, path units in portable services too. This
means you can very naturally do time, socket and path based
activation. It’s also entirely fine to ship multiple service units
in the same image, in case you have more complex applications.
This concept introduces zero new metadata. Unit files are an
existing concept, as are os-release files, and — in case you opt
for raw disk images — GPT partition tables are already established
too. This also means existing tools to generate images can be
reused for building portable service images to a large degree as no
completely new artifact types need to be generated.
Because the Portable Service concepts introduces zero new metadata
and just builds on existing security and resource bundling
features of systemd it’s implemented in a set of distinct tools,
relatively disconnected from the rest of systemd. Specifically, the
main user-facing command is
portablectl,
and the actual operations are implemented in
systemd-portabled.service. If
you so will, portable services are a true add-on to systemd, just
making a specific work-flow nicer to use than with the basic
operations systemd otherwise provides. Also note that
systemd-portabled provides bus APIs accessible to any program
that wants to interface with it, portablectl is just one tool
that happens to be shipped along with systemd.
Since Portable Services are a feature we only added very recently
we wanted to keep some freedom to make changes still. Due to that
we decided to install the portablectl command into
/usr/lib/systemd/ for now, so that it does not appear in $PATH
by default. This means, for now you have to invoke it with a full
path: /usr/lib/systemd/portablectl. We expect to move it into
/usr/bin/ very soon though, and make it a fully supported
interface of systemd.
You may wonder which unit files contained in a portable service
image are the ones considered “relevant” and are actually copied
out by the portablectl attach operation. Currently, this is
derived from the image name. Let’s say you have an image stored in
a directory /var/lib/portables/foobar_4711/ (or alternatively in
a raw image /var/lib/portables/foobar_4711.raw). In that case the
unit files copied out match the pattern foobar*.service,
foobar*.socket, foobar*.target, foobar*.path,
foobar*.timer.
The Portable Services concept does not define any specific method
how images get on the deployment machines, that’s entirely up to
administrators. You can just scp them there, or wget them. You
could even package them as RPMs and then deploy them with dnf if
you feel adventurous.
Portable service images can reside in any directory you
like. However, if you place them in /var/lib/portables/ then
portablectl will find them easily and can show you a list of
images you can attach and suchlike.
Attaching a portable service image can be done persistently, so
that it remains attached on subsequent boots (which is the default),
or it can be attached only until the next reboot, by passing
--runtime to portablectl.
Because portable service images are ultimately just regular OS
images, it’s natural and easy to build a single image that can be
used in three different ways:
1. It can be attached to any host as a portable service image.
2. It can be booted as OS container, for example in a container
  manager like systemd-nspawn.
3. It can be booted as host system, for example on bare metal or
  in a VM manager.
Of course, to qualify for the latter two the image needs to
contain more than just the service binaries, the os-release file
and the unit files. To be bootable an OS container manager such as
systemd-nspawn the image needs to contain an init system of some
form, for example
systemd. To
be bootable on bare metal or as VM it also needs a boot loader of
some form, for example
systemd-boot.

Profiles

In the previous section the “profile” concept was briefly
mentioned. Since they are a major feature of the Portable Services
concept, they deserve some focus. A “profile” is ultimately just a
pre-defined drop-in file for unit files that are attached to a
host. They are supposed to mostly contain sand-boxing and security
settings, but may actually contain any other settings, too. When a
portable service is attached a suitable profile has to be selected. If
none is selected explicitly, the default profile called default is
used. systemd ships with four different profiles out of the box:

The
default
profile provides a medium level of security. It contains settings to
drop capabilities, enforce system call filters, restrict many kernel
interfaces and mount various file systems read-only.
The
strict
profile is similar to the default profile, but generally uses the
most restrictive sand-boxing settings. For example networking is turned
off and access to AF_NETLINK sockets is prohibited.
The
trusted
profile is the least strict of them all. In fact it makes almost no
restrictions at all. A service run with this profile has basically
full access to the host system.
The
nonetwork
profile is mostly identical to default, but also turns off network access.

Note that the profile is selected at the time the portable service
image is attached, and it applies to all service files attached, in
case multiple are shipped in the same image. Thus, the sand-boxing
restriction to enforce are selected by the administrator attaching the
image and not the image vendor.

Additional profiles can be defined easily by the administrator, if
needed. We might also add additional profiles sooner or later to be
shipped with systemd out of the box.

What’s the use-case for this? If I have containers, why should I bother?

Portable Services are primarily intended to cover use-cases where code
should more feel like “extensions” to the host system rather than live
in disconnected, separate worlds. The profile concept is
supposed to be tunable to the exact right amount of integration or
isolation needed for an application.

In the container world the concept of “super-privileged containers”
has been touted a lot, i.e. containers that run with full
privileges. It’s precisely that use-case that portable services are
intended for: extensions to the host OS, that default to isolation,
but can optionally get as much access to the host as needed, and can
naturally take benefit of the full functionality of the host. The
concept should hence be useful for all kinds of low-level system
software that isn’t shipped with the OS itself but needs varying
degrees of integration with it. Besides servers and appliances this
should be particularly interesting for IoT and embedded devices.

Because portable services are just a relatively small extension to the
way system services are otherwise managed, they can be treated like
regular service for almost all use-cases: they will appear along
regular services in all tools that can introspect systemd unit data,
and can be managed the same way when it comes to logging, resource
management, runtime life-cycles and so on.

Portable services are a very generic concept. While the original
use-case is OS extensions, it’s of course entirely up to you and other
users to use them in a suitable way of your choice.

Walkthrough

Let’s have a look how this all can be used. We’ll start with building
a portable service image from scratch, before we attach, enable and
start it on a host.

Building a Portable Service image

As mentioned, you can use any tool you like that can create OS trees
or raw images for building Portable Service images, for example
debootstrap or dnf --installroot=. For this example walkthrough
run we’ll use mkosi, which is
ultimately just a fancy wrapper around dnf and debootstrap but
makes a number of things particularly easy when repetitively building
images from source trees.

I have pushed everything necessary to reproduce this walkthrough
locally to a GitHub
repository. Let’s check it out:

$ git clone https://github.com/systemd/portable-walkthrough.git

Let’s have a look in the repository:

First of all,
walkthroughd.c
is the main source file of our little service. To keep things
simple it’s written in C, but it could be in any language of your
choice. The daemon as implemented won’t do much: it just starts up
and waits for SIGTERM, at which point it will shut down. It’s
ultimately useless, but hopefully illustrates how this all fits
together. The C code has no dependencies besides libc.
walkthroughd.service
is a systemd unit file that starts our little daemon. It’s a simple
service, hence the unit file is trivial.
Makefile
is a short make build script to build the daemon binary. It’s
pretty trivial, too: it just takes the C file and builds a binary
from it. It can also install the daemon. It places the binary in
/usr/local/lib/walkthroughd/walkthroughd (why not in
/usr/local/bin? because it’s not a user-facing binary but a system
service binary), and its unit file in
/usr/local/lib/systemd/walkthroughd.service. If you want to test
the daemon on the host we can now simply run make and then
./walkthroughd in order to check everything works.
mkosi.default
is file that tells mkosi how to build the image. We opt for a
Fedora-based image here (but we might as well have used Debian
here, or any other supported distribution). We need no particular
packages during runtime (after all we only depend on libc), but
during the build phase we need gcc and make, hence these are the
only packages we list in BuildPackages=.
mkosi.build
is a shell script that is invoked during mkosi’s build logic. All
it does is invoke make and make install to build and install
our little daemon, and afterwards it extends the
distribution-supplied /etc/os-release file with an additional
field that describes our portable service a bit.

Let’s now use this to build the portable service image. For that we
use the mkosi tool. It’s
sufficient to invoke it without parameter to build the first image: it
will automatically discover mkosi.default and mkosi.build which
tells it what to do. (Note that if you work on a project like this for
a longer time, mkosi -if is probably the better command to use, as
it that speeds up building substantially by using an incremental build
mode). mkosi will download the necessary RPMs, and put them all
together. It will build our little daemon inside the image and after
all that’s done it will output the resulting image:
walkthroughd_1.raw.

Because we opted to build a GPT raw disk image in mkosi.default this
file is actually a raw disk image containing a GPT partition
table. You can use fdisk -l walkthroughd_1.raw to enumerate the
partition table. You can also use systemd-nspawn -i walkthroughd_1.raw to explore the image quickly if you need.

Using the Portable Service Image

Now that we have a portable service image, let’s see how we can
attach, enable and start the service included within it.

First, let’s attach the image:

# /usr/lib/systemd/portablectl attach ./walkthroughd_1.raw
(Matching unit files with prefix 'walkthroughd'.)
Created directory /etc/systemd/system/walkthroughd.service.d.
Written /etc/systemd/system/walkthroughd.service.d/20-portable.conf.
Created symlink /etc/systemd/system/walkthroughd.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
Copied /etc/systemd/system/walkthroughd.service.
Created symlink /etc/portables/walkthroughd_1.raw → /home/lennart/projects/portable-walkthrough/walkthroughd_1.raw.

The command will show you exactly what is has been doing: it just
copied the main service file out, and added the two drop-ins, as
expected.

Let’s see if the unit is now available on the host, just like a regular unit, as promised:

# systemctl status walkthroughd.service
● walkthroughd.service - A simple example service
   Loaded: loaded (/etc/systemd/system/walkthroughd.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/walkthroughd.service.d
           └─10-profile.conf, 20-portable.conf
   Active: inactive (dead)

Nice, it worked. We see that the unit file is available and that
systemd correctly discovered the two drop-ins. The unit is neither
enabled nor started however. Yes, attaching a portable service image
doesn’t imply enabling nor starting. It just means the unit files
contained in the image are made available to the host. It’s up to the
administrator to then enable them (so that they are automatically
started when needed, for example at boot), and/or start them (in case
they shall run right-away).

Let’s now enable and start the service in one step:

# systemctl enable --now walkthroughd.service
Created symlink /etc/systemd/system/multi-user.target.wants/walkthroughd.service → /etc/systemd/system/walkthroughd.service.

Let’s check if it’s running:

# systemctl status walkthroughd.service
● walkthroughd.service - A simple example service
   Loaded: loaded (/etc/systemd/system/walkthroughd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/walkthroughd.service.d
           └─10-profile.conf, 20-portable.conf
   Active: active (running) since Wed 2018-06-27 17:55:30 CEST; 4s ago
 Main PID: 45003 (walkthroughd)
    Tasks: 1 (limit: 4915)
   Memory: 4.3M
   CGroup: /system.slice/walkthroughd.service
           └─45003 /usr/local/lib/walkthroughd/walkthroughd

Jun 27 17:55:30 sigma walkthroughd[45003]: Initializing.

Perfect! We can see that the service is now enabled and running. The daemon is running as PID 45003.

Now that we verified that all is good, let’s stop, disable and detach the service again:

# systemctl disable --now walkthroughd.service
Removed /etc/systemd/system/multi-user.target.wants/walkthroughd.service.
# /usr/lib/systemd/portablectl detach ./walkthroughd_1.raw
Removed /etc/systemd/system/walkthroughd.service.
Removed /etc/systemd/system/walkthroughd.service.d/10-profile.conf.
Removed /etc/systemd/system/walkthroughd.service.d/20-portable.conf.
Removed /etc/systemd/system/walkthroughd.service.d.
Removed /etc/portables/walkthroughd_1.raw.

And finally, let’s see that it’s really gone:

# systemctl status walkthroughd
Unit walkthroughd.service could not be found.

Perfect! It worked!

I hope the above gets you started with Portable Services. If you have
further questions, please contact our mailing
list.

Noise

Walkthrough for Portable Services