Post Syndicated from Lennart Poettering original http://0pointer.net/blog/mkosi-a-tool-for-generating-os-images.html

Introducing mkosi

After blogging about
casync
I realized I never blogged about the
mkosi tool that combines nicely
with it. mkosi has been around for a while already, and its time to
make it a bit better known. mkosi stands for Make Operating System
Image, and is a tool for precisely that: generating an OS tree or
image that can be booted.

Yes, there are many tools like mkosi, and a number of them are quite
well known and popular. But mkosi has a number of features that I
think make it interesting for a variety of use-cases that other tools
don’t cover that well.

What is mkosi?

What are those use-cases, and what does mkosi precisely set apart?
mkosi is definitely a tool with a focus on developer’s needs for
building OS images, for testing and debugging, but also for generating
production images with cryptographic protection. A typical use-case
would be to add a mkosi.default file to an existing project (for
example, one written in C or Python), and thus making it easy to
generate an OS image for it. mkosi will put together the image with
development headers and tools, compile your code in it, run your test
suite, then throw away the image again, and build a new one, this time
without development headers and tools, and install your build
artifacts in it. This final image is then “production-ready”, and only
contains your built program and the minimal set of packages you
configured otherwise. Such an image could then be deployed with
casync (or any other tool of course) to be delivered to your set of
servers, or IoT devices or whatever you are building.

mkosi is supposed to be legacy-free: the focus is clearly on
today’s technology, not yesteryear’s. Specifically this means that
we’ll generate GPT partition tables, not MBR/DOS ones. When you tell
mkosi to generate a bootable image for you, it will make it bootable
on EFI, not on legacy BIOS. The GPT images generated follow
specifications such as the Discoverable Partitions
Specification,
so that /etc/fstab can remain unpopulated and tools such as
systemd-nspawn can automatically dissect the image and boot from
them.

So, let’s have a look on the specific images it can generate:

1. Raw GPT disk image, with ext4 as root
2. Raw GPT disk image, with btrfs as root
3. Raw GPT disk image, with a read-only squashfs as root
4. A plain directory on disk containing the OS tree directly (this is useful for creating generic container images)
5. A btrfs subvolume on disk, similar to the plain directory
6. A tarball of a plain directory

When any of the GPT choices above are selected, a couple of additional
options are available:

1. A swap partition may be added in
2. The system may be made bootable on EFI systems
3. Separate partitions for /home and /srv may be added in
4. The root, /home and /srv partitions may be optionally encrypted with LUKS
5. The root partition may be protected using dm-verity, thus making offline attacks on the generated system hard
6. If the image is made bootable, the dm-verity root hash is automatically added to the kernel command line, and the kernel together with its initial RAM disk and the kernel command line is optionally cryptographically signed for UEFI SecureBoot

Note that mkosi is distribution-agnostic. It currently can build
images based on the following Linux distributions:

1. Fedora
2. Debian
3. Ubuntu
4. ArchLinux
5. openSUSE

Note though that not all distributions are supported at the same
feature level currently. Also, as mkosi is based on dnf
--installroot, debootstrap, pacstrap and zypper, and those
packages are not packaged universally on all distributions, you might
not be able to build images for all those distributions on arbitrary
host distributions. For example, Fedora doesn’t package zypper,
hence you cannot build an openSUSE image easily on Fedora, but you can
still build Fedora (obviously…), Debian, Ubuntu and ArchLinux images
on it just fine.

The GPT images are put together in a way that they aren’t just
compatible with UEFI systems, but also with VM and container managers
(that is, at least the smart ones, i.e. VM managers that know UEFI,
and container managers that grok GPT disk images) to a large
degree. In fact, the idea is that you can use mkosi to build a
single GPT image that may be used to:

1. Boot on bare-metal boxes
2. Boot in a VM
3. Boot in a systemd-nspawn container
4. Directly run a systemd service off, using systemd’s RootImage= unit file setting

Note that in all four cases the dm-verity data is automatically used
if available to ensure the image is not tempered with (yes, you read
that right, systemd-nspawn and systemd’s RootImage= setting
automatically do dm-verity these days if the image has it.)

Mode of Operation

The simplest usage of mkosi is by simply invoking it without
parameters (as root):

# mkosi

Without any configuration this will create a GPT disk image for you,
will call it image.raw and drop it in the current directory. The
distribution used will be the same one as your host runs.

Of course in most cases you want more control about how the image is
put together, i.e. select package sets, select the distribution, size
partitions and so on. Most of that you can actually specify on the
command line, but it is recommended to instead create a couple of
mkosi.$SOMETHING files and directories in some directory. Then,
simply change to that directory and run mkosi without any further
arguments. The tool will then look in the current working directory
for these files and directories and make use of them (similar to how
make looks for a Makefile…). Every single file/directory is
optional, but if they exist they are honored. Here’s a list of the
files/directories mkosi currently looks for:

1. mkosi.default — This is the main configuration file, here you
   can configure what kind of image you want, which distribution, which
   packages and so on.

2. mkosi.extra/ — If this directory exists, then mkosi will copy
   everything inside it into the images built. You can place arbitrary
   directory hierarchies in here, and they’ll be copied over whatever is
   already in the image, after it was put together by the distribution’s
   package manager. This is the best way to drop additional static files
   into the image, or override distribution-supplied ones.

3. mkosi.build — This executable file is supposed to be a build
   script. When it exists, mkosi will build two images, one after the
   other in the mode already mentioned above: the first version is the
   build image, and may include various build-time dependencies such as
   a compiler or development headers. The build script is also copied
   into it, and then run inside it. The script should then build
   whatever shall be built and place the result in $DESTDIR (don’t
   worry, popular build tools such as Automake or Meson all honor
   $DESTDIR anyway, so there’s not much to do here explicitly). It may
   also run a test suite, or anything else you like. After the script
   finished, the build image is removed again, and a second image (the
   final image) is built. This time, no development packages are
   included, and the build script is not copied into the image again —
   however, the build artifacts from the first run (i.e. those placed in
   $DESTDIR) are copied into the image.

4. mkosi.postinst — If this executable script exists, it is invoked
   inside the image (inside a systemd-nspawn invocation) and can
   adjust the image as it likes at a very late point in the image
   preparation. If mkosi.build exists, i.e. the dual-phased
   development build process used, then this script will be invoked
   twice: once inside the build image and once inside the final
   image. The first parameter passed to the script clarifies which phase
   it is run in.

5. mkosi.nspawn — If this file exists, it should contain a
   container configuration file for systemd-nspawn (see
   systemd.nspawn(5)
   for details), which shall be shipped along with the final image and
   shall be included in the check-sum calculations (see below).

6. mkosi.cache/ — If this directory exists, it is used as package
   cache directory for the builds. This directory is effectively bind
   mounted into the image at build time, in order to speed up building
   images. The package installers of the various distributions will
   place their package files here, so that subsequent runs can reuse
   them.

7. mkosi.passphrase — If this file exists, it should contain a
   pass-phrase to use for the LUKS encryption (if that’s enabled for the
   image built). This file should not be readable to other users.

8. mkosi.secure-boot.crt and mkosi.secure-boot.key should be an
   X.509 key pair to use for signing the kernel and initrd for UEFI
   SecureBoot, if that’s enabled.

How to use it

So, let’s come back to our most trivial example, without any of the
mkosi.$SOMETHING files around:

# mkosi

As mentioned, this will create a build file image.raw in the current
directory. How do we use it? Of course, we could dd it onto some USB
stick and boot it on a bare-metal device. However, it’s much simpler
to first run it in a container for testing:

# systemd-nspawn -bi image.raw

And there you go: the image should boot up, and just work for you.

Now, let’s make things more interesting. Let’s still not use any of
the mkosi.$SOMETHING files around:

# mkosi -t raw_btrfs --bootable -o foobar.raw
# systemd-nspawn -bi foobar.raw

This is similar as the above, but we made three changes: it’s no
longer GPT + ext4, but GPT + btrfs. Moreover, the system is made
bootable on UEFI systems, and finally, the output is now called
foobar.raw.

Because this system is bootable on UEFI systems, we can run it in KVM:

qemu-kvm -m 512 -smp 2 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -drive format=raw,file=foobar.raw

This will look very similar to the systemd-nspawn invocation, except
that this uses full VM virtualization rather than container
virtualization. (Note that the way to run a UEFI qemu/kvm instance
appears to change all the time and is different on the various
distributions. It’s quite annoying, and I can’t really tell you what
the right qemu command line is to make this work on your system.)

Of course, it’s not all raw GPT disk images with mkosi. Let’s try
a plain directory image:

# mkosi -d fedora -t directory -o quux
# systemd-nspawn -bD quux

Of course, if you generate the image as plain directory you can’t boot
it on bare-metal just like that, nor run it in a VM.

A more complex command line is the following:

# mkosi -d fedora -t raw_squashfs --checksum --xz --package=openssh-clients --package=emacs

In this mode we explicitly pick Fedora as the distribution to use, ask
mkosi to generate a compressed GPT image with a root squashfs,
compress the result with xz, and generate a SHA256SUMS file with
the hashes of the generated artifacts. The package will contain the
SSH client as well as everybody’s favorite editor.

Now, let’s make use of the various mkosi.$SOMETHING files. Let’s
say we are working on some Automake-based project and want to make it
easy to generate a disk image off the development tree with the
version you are hacking on. Create a configuration file:

# cat > mkosi.default <<EOF
[Distribution]
Distribution=fedora
Release=24

[Output]
Format=raw_btrfs
Bootable=yes

[Packages]
# The packages to appear in both the build and the final image
Packages=openssh-clients httpd
# The packages to appear in the build image, but absent from the final image
BuildPackages=make gcc libcurl-devel
EOF

And let’s add a build script:

# cat > mkosi.build <<EOF
#!/bin/sh
cd $SRCDIR
./autogen.sh
./configure --prefix=/usr
make -j `nproc`
make install
EOF
# chmod +x mkosi.build

And with all that in place we can now build our project into a disk image, simply by typing:

# mkosi

Let’s try it out:

# systemd-nspawn -bi image.raw

Of course, if you do this you’ll notice that building an image like
this can be quite slow. And slow build times are actively hurtful to
your productivity as a developer. Hence let’s make things a bit
faster. First, let’s make use of a package cache shared between runs:

# mkdir mkosi.chache

Building images now should already be substantially faster (and
generate less network traffic) as the packages will now be downloaded
only once and reused. However, you’ll notice that unpacking all those
packages and the rest of the work is still quite slow. But mkosi can
help you with that. Simply use mkosi‘s incremental build feature. In
this mode mkosi will make a copy of the build and final images
immediately before dropping in your build sources or artifacts, so
that building an image becomes a lot quicker: instead of always
starting totally from scratch a build will now reuse everything it can
reuse from a previous run, and immediately begin with building your
sources rather than the build image to build your sources in. To
enable the incremental build feature use -i:

# mkosi -i

Note that if you use this option, the package list is not updated
anymore from your distribution’s servers, as the cached copy is made
after all packages are installed, and hence until you actually delete
the cached copy the distribution’s network servers aren’t contacted
again and no RPMs or DEBs are downloaded. This means the distribution
you use becomes “frozen in time” this way. (Which might be a bad
thing, but also a good thing, as it makes things kinda reproducible.)

Of course, if you run mkosi a couple of times you’ll notice that it
won’t overwrite the generated image when it already exists. You can
either delete the file yourself first (rm image.raw) or let mkosi
do it for you right before building a new image, with mkosi -f. You
can also tell mkosi to not only remove any such pre-existing images,
but also remove any cached copies of the incremental feature, by using
-f twice.

I wrote mkosi originally in order to test systemd, and quickly
generate a disk image of various distributions with the most current
systemd version from git, without all that affecting my host system. I
regularly use mkosi for that today, in incremental mode. The two
commands I use most in that context are:

# mkosi -if && systemd-nspawn -bi image.raw

And sometimes:

# mkosi -iff && systemd-nspawn -bi image.raw

The latter I use only if I want to regenerate everything based on the
very newest set of RPMs provided by Fedora, instead of a cached
snapshot of it.

BTW, the mkosi files for systemd are included in the systemd git
tree:
mkosi.default
and
mkosi.build. This
way, any developer who wants to quickly test something with current
systemd git, or wants to prepare a patch based on it and test it can
check out the systemd repository and simply run mkosi in it and a
few minutes later he has a bootable image he can test in
systemd-nspawn or KVM. casync has similar files:
mkosi.default,
mkosi.build.

Random Interesting Features

1. As mentioned already, mkosi will generate dm-verity enabled
   disk images if you ask for it. For that use the --verity switch on
   the command line or Verity= setting in mkosi.default. Of course,
   dm-verity implies that the root volume is read-only. In this mode
   the top-level dm-verity hash will be placed along-side the output
   disk image in a file named the same way, but with the .roothash
   suffix. If the image is to be created bootable, the root hash is also
   included on the kernel command line in the roothash= parameter,
   which current systemd versions can use to both find and activate the
   root partition in a dm-verity protected way. BTW: it’s a good idea
   to combine this dm-verity mode with the raw_squashfs image mode,
   to generate a genuinely protected, compressed image suitable for
   running in your IoT device.

2. As indicated above, mkosi can automatically create a check-sum
   file SHA256SUMS for you (--checksum) covering all the files it
   outputs (which could be the image file itself, a matching .nspawn
   file using the mkosi.nspawn file mentioned above, as well as the
   .roothash file for the dm-verity root hash.) It can then
   optionally sign this with gpg (--sign). Note that systemd‘s
   machinectl pull-tar and machinectl pull-raw command can download
   these files and the SHA256SUMS file automatically and verify things
   on download. With other words: what mkosi outputs is perfectly
   ready for downloads using these two systemd commands.

3. As mentioned, mkosi is big on supporting UEFI SecureBoot. To
   make use of that, place your X.509 key pair in two files
   mkosi.secureboot.crt and mkosi.secureboot.key, and set
   SecureBoot= or --secure-boot. If so, mkosi will sign the
   kernel/initrd/kernel command line combination during the build. Of
   course, if you use this mode, you should also use
   Verity=/--verity=, otherwise the setup makes only partial
   sense. Note that mkosi will not help you with actually enrolling
   the keys you use in your UEFI BIOS.

4. mkosi has minimal support for GIT checkouts: when it recognizes
   it is run in a git checkout and you use the mkosi.build script
   stuff, the source tree will be copied into the build image, but will
   all files excluded by .gitignore removed.

5. There’s support for encryption in place. Use --encrypt= or
   Encrypt=. Note that the UEFI ESP is never encrypted though, and the
   root partition only if explicitly requested. The /home and /srv
   partitions are unconditionally encrypted if that’s enabled.

6. Images may be built with all documentation removed.

7. The password for the root user and additional kernel command line
   arguments may be configured for the image to generate.

Minimum Requirements

Current mkosi requires Python 3.5, and has a number of dependencies,
listed in the
README. Most
notably you need a somewhat recent systemd version to make use of its
full feature set: systemd 233. Older versions are already packaged for
various distributions, but much of what I describe above is only
available in the most recent release mkosi 3.

The UEFI SecureBoot support requires sbsign which currently isn’t
available in Fedora, but there’s a
COPR.

Future

It is my intention to continue turning mkosi into a tool suitable
for:

1. Testing and debugging projects
2. Building images for secure devices
3. Building portable service images
4. Building images for secure VMs and containers

One of the biggest goals I have for the future is to teach mkosi and
systemd/sd-boot native support for A/B IoT style partition
setups. The idea is that the combination of systemd, casync and
mkosi provides generic building blocks for building secure,
auto-updating devices in a generic way from, even though all pieces
may be used individually, too.

FAQ

1. Why are you reinventing the wheel again? This is exactly like
   $SOMEOTHERPROJECT! — Well, to my knowledge there’s no tool that
   integrates this nicely with your project’s development tree, and can
   do dm-verity and UEFI SecureBoot and all that stuff for you. So
   nope, I don’t think this exactly like $SOMEOTHERPROJECT, thank you
   very much.

2. What about creating MBR/DOS partition images? — That’s really
   out of focus to me. This is an exercise in figuring out how generic
   OSes and devices in the future should be built and an attempt to
   commoditize OS image building. And no, the future doesn’t speak MBR,
   sorry. That said, I’d be quite interested in adding support for
   booting on Raspberry Pi, possibly using a hybrid approach, i.e. using
   a GPT disk label, but arranging things in a way that the Raspberry Pi
   boot protocol (which is built around DOS partition tables), can still
   work.

3. Is this portable? — Well, depends what you mean by
   portable. No, this tool runs on Linux only, and as it uses
   systemd-nspawn during the build process it doesn’t run on
   non-systemd systems either. But then again, you should be able to
   create images for any architecture you like with it, but of course if
   you want the image bootable on bare-metal systems only systems doing
   UEFI are supported (but systemd-nspawn should still work fine on
   them).

4. Where can I get this stuff? — Try
   GitHub. And some distributions
   carry packaged versions, but I think none of them the current v3
   yet.

5. Is this a systemd project? — Yes, it’s hosted under the
   systemd GitHub umbrella. And yes,
   during run-time systemd-nspawn in a current version is required. But
   no, the code-bases are separate otherwise, already because systemd
   is a C project, and mkosi Python.

6. Requiring systemd 233 is a pretty steep requirement, no? —
   Yes, but the feature we need kind of matters (systemd-nspawn‘s
   --overlay= switch), and again, this isn’t supposed to be a tool for
   legacy systems.

7. Can I run the resulting images in LXC or Docker? — Humm, I am
   not an LXC nor Docker guy. If you select directory or subvolume
   as image type, LXC should be able to boot the generated images just
   fine, but I didn’t try. Last time I looked, Docker doesn’t permit
   running proper init systems as PID 1 inside the container, as they
   define their own run-time without intention to emulate a proper
   system. Hence, no I don’t think it will work, at least not with an
   unpatched Docker version. That said, again, don’t ask me questions
   about Docker, it’s not precisely my area of expertise, and quite
   frankly I am not a fan. To my knowledge neither LXC nor Docker are
   able to run containers directly off GPT disk images, hence the
   various raw_xyz image types are definitely not compatible with
   either. That means if you want to generate a single raw disk image
   that can be booted unmodified both in a container and on bare-metal,
   then systemd-nspawn is the container manager to go for
   (specifically, its -i/--image= switch).

Should you care? Is this a tool for you?

Well, that’s up to you really.

If you hack on some complex project and need a quick way to compile
and run your project on a specific current Linux distribution, then
mkosi is an excellent way to do that. Simply drop the mkosi.default
and mkosi.build files in your git tree and everything will be
easy. (And of course, as indicated above: if the project you are
hacking on happens to be called systemd or casync be aware that
those files are already part of the git tree — you can just use them.)

If you hack on some embedded or IoT device, then mkosi is a great
choice too, as it will make it reasonably easy to generate secure
images that are protected against offline modification, by using
dm-verity and UEFI SecureBoot.

If you are an administrator and need a nice way to build images for a
VM or systemd-nspawn container, or a portable service then mkosi
is an excellent choice too.

If you care about legacy computers, old distributions, non-systemd
init systems, old VM managers, Docker, … then no, mkosi is not for
you, but there are plenty of well-established alternatives around that
cover that nicely.

And never forget: mkosi is an Open Source project. We are happy to
accept your patches and other contributions.

Oh, and one unrelated last thing: don’t forget to submit your talk
proposal
and/or buy a ticket for
All Systems Go! 2017 in Berlin — the
conference where things like systemd, casync and mkosi are
discussed, along with a variety of other Linux userspace projects used
for building systems.