Tag Archives: makefile

Create Multiple Builds from the Same Source Using Different AWS CodeBuild Build Specification Files

Post Syndicated from Prakash Palanisamy original https://aws.amazon.com/blogs/devops/create-multiple-builds-from-the-same-source-using-different-aws-codebuild-build-specification-files/

In June 2017, AWS CodeBuild announced you can now specify an alternate build specification file name or location in an AWS CodeBuild project.

In this post, I’ll show you how to use different build specification files in the same repository to create different builds. You’ll find the source code for this post in our GitHub repo.

Requirements

The AWS CLI must be installed and configured.

Solution Overview

I have created a C program (cbsamplelib.c) that will be used to create a shared library and another utility program (cbsampleutil.c) to use that library. I’ll use a Makefile to compile these files.

I need to put this sample application in RPM and DEB packages so end users can easily deploy them. I have created a build specification file for RPM. It will use make to compile this code and the RPM specification file (cbsample.rpmspec) configured in the build specification to create the RPM package. Similarly, I have created a build specification file for DEB. It will create the DEB package based on the control specification file (cbsample.control) configured in this build specification.

RPM Build Project:

The following build specification file (buildspec-rpm.yml) uses build specification version 0.2. As described in the documentation, this version has different syntax for environment variables. This build specification includes multiple phases:

  • As part of the install phase, the required packages is installed using yum.
  • During the pre_build phase, the required directories are created and the required files, including the RPM build specification file, are copied to the appropriate location.
  • During the build phase, the code is compiled, and then the RPM package is created based on the RPM specification.

As defined in the artifact section, the RPM file will be uploaded as a build artifact.

version: 0.2

env:
  variables:
    build_version: "0.1"

phases:
  install:
    commands:
      - yum install rpm-build make gcc glibc -y
  pre_build:
    commands:
      - curr_working_dir=`pwd`
      - mkdir -p ./{RPMS,SRPMS,BUILD,SOURCES,SPECS,tmp}
      - filename="cbsample-$build_version"
      - echo $filename
      - mkdir -p $filename
      - cp ./*.c ./*.h Makefile $filename
      - tar -zcvf /root/$filename.tar.gz $filename
      - cp /root/$filename.tar.gz ./SOURCES/
      - cp cbsample.rpmspec ./SPECS/
  build:
    commands:
      - echo "Triggering RPM build"
      - rpmbuild --define "_topdir `pwd`" -ba SPECS/cbsample.rpmspec
      - cd $curr_working_dir

artifacts:
  files:
    - RPMS/x86_64/cbsample*.rpm
  discard-paths: yes

Using cb-centos-project.json as a reference, create the input JSON file for the CLI command. This project uses an AWS CodeCommit repository named codebuild-multispec and a file named buildspec-rpm.yml as the build specification file. To create the RPM package, we need to specify a custom image name. I’m using the latest CentOS 7 image available in the Docker Hub. I’m using a role named CodeBuildServiceRole. It contains permissions similar to those defined in CodeBuildServiceRole.json. (You need to change the resource fields in the policy, as appropriate.)

{
    "name": "rpm-build-project",
    "description": "Project which will build RPM from the source.",
    "source": {
        "type": "CODECOMMIT",
        "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec",
        "buildspec": "buildspec-rpm.yml"
    },
    "artifacts": {
        "type": "S3",
        "location": "codebuild-demo-artifact-repository"
    },
    "environment": {
        "type": "LINUX_CONTAINER",
        "image": "centos:7",
        "computeType": "BUILD_GENERAL1_SMALL"
    },
    "serviceRole": "arn:aws:iam::012345678912:role/service-role/CodeBuildServiceRole",
    "timeoutInMinutes": 15,
    "encryptionKey": "arn:aws:kms:eu-west-1:012345678912:alias/aws/s3",
    "tags": [
        {
            "key": "Name",
            "value": "RPM Demo Build"
        }
    ]
}

After the cli-input-json file is ready, execute the following command to create the build project.

$ aws codebuild create-project --name CodeBuild-RPM-Demo --cli-input-json file://cb-centos-project.json

{
    "project": {
        "name": "CodeBuild-RPM-Demo", 
        "serviceRole": "arn:aws:iam::012345678912:role/service-role/CodeBuildServiceRole", 
        "tags": [
            {
                "value": "RPM Demo Build", 
                "key": "Name"
            }
        ], 
        "artifacts": {
            "namespaceType": "NONE", 
            "packaging": "NONE", 
            "type": "S3", 
            "location": "codebuild-demo-artifact-repository", 
            "name": "CodeBuild-RPM-Demo"
        }, 
        "lastModified": 1500559811.13, 
        "timeoutInMinutes": 15, 
        "created": 1500559811.13, 
        "environment": {
            "computeType": "BUILD_GENERAL1_SMALL", 
            "privilegedMode": false, 
            "image": "centos:7", 
            "type": "LINUX_CONTAINER", 
            "environmentVariables": []
        }, 
        "source": {
            "buildspec": "buildspec-rpm.yml", 
            "type": "CODECOMMIT", 
            "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec"
        }, 
        "encryptionKey": "arn:aws:kms:eu-west-1:012345678912:alias/aws/s3", 
        "arn": "arn:aws:codebuild:eu-west-1:012345678912:project/CodeBuild-RPM-Demo", 
        "description": "Project which will build RPM from the source."
    }
}

When the project is created, run the following command to start the build. After the build has started, get the build ID. You can use the build ID to get the status of the build.

$ aws codebuild start-build --project-name CodeBuild-RPM-Demo
{
    "build": {
        "buildComplete": false, 
        "initiator": "prakash", 
        "artifacts": {
            "location": "arn:aws:s3:::codebuild-demo-artifact-repository/CodeBuild-RPM-Demo"
        }, 
        "projectName": "CodeBuild-RPM-Demo", 
        "timeoutInMinutes": 15, 
        "buildStatus": "IN_PROGRESS", 
        "environment": {
            "computeType": "BUILD_GENERAL1_SMALL", 
            "privilegedMode": false, 
            "image": "centos:7", 
            "type": "LINUX_CONTAINER", 
            "environmentVariables": []
        }, 
        "source": {
            "buildspec": "buildspec-rpm.yml", 
            "type": "CODECOMMIT", 
            "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec"
        }, 
        "currentPhase": "SUBMITTED", 
        "startTime": 1500560156.761, 
        "id": "CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc", 
        "arn": "arn:aws:codebuild:eu-west-1: 012345678912:build/CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc"
    }
}

$ aws codebuild list-builds-for-project --project-name CodeBuild-RPM-Demo
{
    "ids": [
        "CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc"
    ]
}

$ aws codebuild batch-get-builds --ids CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc
{
    "buildsNotFound": [], 
    "builds": [
        {
            "buildComplete": true, 
            "phases": [
                {
                    "phaseStatus": "SUCCEEDED", 
                    "endTime": 1500560157.164, 
                    "phaseType": "SUBMITTED", 
                    "durationInSeconds": 0, 
                    "startTime": 1500560156.761
                }, 
                {
                    "contexts": [], 
                    "phaseType": "PROVISIONING", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 24, 
                    "startTime": 1500560157.164, 
                    "endTime": 1500560182.066
                }, 
                {
                    "contexts": [], 
                    "phaseType": "DOWNLOAD_SOURCE", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 15, 
                    "startTime": 1500560182.066, 
                    "endTime": 1500560197.906
                }, 
                {
                    "contexts": [], 
                    "phaseType": "INSTALL", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 19, 
                    "startTime": 1500560197.906, 
                    "endTime": 1500560217.515
                }, 
                {
                    "contexts": [], 
                    "phaseType": "PRE_BUILD", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 0, 
                    "startTime": 1500560217.515, 
                    "endTime": 1500560217.662
                }, 
                {
                    "contexts": [], 
                    "phaseType": "BUILD", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 0, 
                    "startTime": 1500560217.662, 
                    "endTime": 1500560217.995
                }, 
                {
                    "contexts": [], 
                    "phaseType": "POST_BUILD", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 0, 
                    "startTime": 1500560217.995, 
                    "endTime": 1500560218.074
                }, 
                {
                    "contexts": [], 
                    "phaseType": "UPLOAD_ARTIFACTS", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 0, 
                    "startTime": 1500560218.074, 
                    "endTime": 1500560218.542
                }, 
                {
                    "contexts": [], 
                    "phaseType": "FINALIZING", 
                    "phaseStatus": "SUCCEEDED", 
                    "durationInSeconds": 4, 
                    "startTime": 1500560218.542, 
                    "endTime": 1500560223.128
                }, 
                {
                    "phaseType": "COMPLETED", 
                    "startTime": 1500560223.128
                }
            ], 
            "logs": {
                "groupName": "/aws/codebuild/CodeBuild-RPM-Demo", 
                "deepLink": "https://console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logEvent:group=/aws/codebuild/CodeBuild-RPM-Demo;stream=57a36755-4d37-4b08-9c11-1468e1682abc", 
                "streamName": "57a36755-4d37-4b08-9c11-1468e1682abc"
            }, 
            "artifacts": {
                "location": "arn:aws:s3:::codebuild-demo-artifact-repository/CodeBuild-RPM-Demo"
            }, 
            "projectName": "CodeBuild-RPM-Demo", 
            "timeoutInMinutes": 15, 
            "initiator": "prakash", 
            "buildStatus": "SUCCEEDED", 
            "environment": {
                "computeType": "BUILD_GENERAL1_SMALL", 
                "privilegedMode": false, 
                "image": "centos:7", 
                "type": "LINUX_CONTAINER", 
                "environmentVariables": []
            }, 
            "source": {
                "buildspec": "buildspec-rpm.yml", 
                "type": "CODECOMMIT", 
                "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec"
            }, 
            "currentPhase": "COMPLETED", 
            "startTime": 1500560156.761, 
            "endTime": 1500560223.128, 
            "id": "CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc", 
            "arn": "arn:aws:codebuild:eu-west-1:012345678912:build/CodeBuild-RPM-Demo:57a36755-4d37-4b08-9c11-1468e1682abc"
        }
    ]
}

DEB Build Project:

In this project, we will use the build specification file named buildspec-deb.yml. Like the RPM build project, this specification includes multiple phases. Here I use a Debian control file to create the package in DEB format. After a successful build, the DEB package will be uploaded as build artifact.

version: 0.2

env:
  variables:
    build_version: "0.1"

phases:
  install:
    commands:
      - apt-get install gcc make -y
  pre_build:
    commands:
      - mkdir -p ./cbsample-$build_version/DEBIAN
      - mkdir -p ./cbsample-$build_version/usr/lib
      - mkdir -p ./cbsample-$build_version/usr/include
      - mkdir -p ./cbsample-$build_version/usr/bin
      - cp -f cbsample.control ./cbsample-$build_version/DEBIAN/control
  build:
    commands:
      - echo "Building the application"
      - make
      - cp libcbsamplelib.so ./cbsample-$build_version/usr/lib
      - cp cbsamplelib.h ./cbsample-$build_version/usr/include
      - cp cbsampleutil ./cbsample-$build_version/usr/bin
      - chmod +x ./cbsample-$build_version/usr/bin/cbsampleutil
      - dpkg-deb --build ./cbsample-$build_version

artifacts:
  files:
    - cbsample-*.deb

Here we use cb-ubuntu-project.json as a reference to create the CLI input JSON file. This project uses the same AWS CodeCommit repository (codebuild-multispec) but a different buildspec file in the same repository (buildspec-deb.yml). We use the default CodeBuild image to create the DEB package. We use the same IAM role (CodeBuildServiceRole).

{
    "name": "deb-build-project",
    "description": "Project which will build DEB from the source.",
    "source": {
        "type": "CODECOMMIT",
        "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec",
        "buildspec": "buildspec-deb.yml"
    },
    "artifacts": {
        "type": "S3",
        "location": "codebuild-demo-artifact-repository"
    },
    "environment": {
        "type": "LINUX_CONTAINER",
        "image": "aws/codebuild/ubuntu-base:14.04",
        "computeType": "BUILD_GENERAL1_SMALL"
    },
    "serviceRole": "arn:aws:iam::012345678912:role/service-role/CodeBuildServiceRole",
    "timeoutInMinutes": 15,
    "encryptionKey": "arn:aws:kms:eu-west-1:012345678912:alias/aws/s3",
    "tags": [
        {
            "key": "Name",
            "value": "Debian Demo Build"
        }
    ]
}

Using the CLI input JSON file, create the project, start the build, and check the status of the project.

$ aws codebuild create-project --name CodeBuild-DEB-Demo --cli-input-json file://cb-ubuntu-project.json

$ aws codebuild list-builds-for-project --project-name CodeBuild-DEB-Demo

$ aws codebuild batch-get-builds --ids CodeBuild-DEB-Demo:e535c4b0-7067-4fbe-8060-9bb9de203789

After successful completion of the RPM and DEB builds, check the S3 bucket configured in the artifacts section for the build packages. Build projects will create a directory in the name of the build project and copy the artifacts inside it.

$ aws s3 ls s3://codebuild-demo-artifact-repository/CodeBuild-RPM-Demo/
2017-07-20 16:16:59       8108 cbsample-0.1-1.el7.centos.x86_64.rpm

$ aws s3 ls s3://codebuild-demo-artifact-repository/CodeBuild-DEB-Demo/
2017-07-20 16:37:22       5420 cbsample-0.1.deb

Override Buildspec During Build Start:

It’s also possible to override the build specification file of an existing project when starting a build. If we want to create the libs RPM package instead of the whole RPM, we will use the build specification file named buildspec-libs-rpm.yml. This build specification file is similar to the earlier RPM build. The only difference is that it uses a different RPM specification file to create libs RPM.

version: 0.2

env:
  variables:
    build_version: "0.1"

phases:
  install:
    commands:
      - yum install rpm-build make gcc glibc -y
  pre_build:
    commands:
      - curr_working_dir=`pwd`
      - mkdir -p ./{RPMS,SRPMS,BUILD,SOURCES,SPECS,tmp}
      - filename="cbsample-libs-$build_version"
      - echo $filename
      - mkdir -p $filename
      - cp ./*.c ./*.h Makefile $filename
      - tar -zcvf /root/$filename.tar.gz $filename
      - cp /root/$filename.tar.gz ./SOURCES/
      - cp cbsample-libs.rpmspec ./SPECS/
  build:
    commands:
      - echo "Triggering RPM build"
      - rpmbuild --define "_topdir `pwd`" -ba SPECS/cbsample-libs.rpmspec
      - cd $curr_working_dir

artifacts:
  files:
    - RPMS/x86_64/cbsample-libs*.rpm
  discard-paths: yes

Using the same RPM build project that we created earlier, start a new build and set the value of the `–buildspec-override` parameter to buildspec-libs-rpm.yml .

$ aws codebuild start-build --project-name CodeBuild-RPM-Demo --buildspec-override buildspec-libs-rpm.yml
{
    "build": {
        "buildComplete": false, 
        "initiator": "prakash", 
        "artifacts": {
            "location": "arn:aws:s3:::codebuild-demo-artifact-repository/CodeBuild-RPM-Demo"
        }, 
        "projectName": "CodeBuild-RPM-Demo", 
        "timeoutInMinutes": 15, 
        "buildStatus": "IN_PROGRESS", 
        "environment": {
            "computeType": "BUILD_GENERAL1_SMALL", 
            "privilegedMode": false, 
            "image": "centos:7", 
            "type": "LINUX_CONTAINER", 
            "environmentVariables": []
        }, 
        "source": {
            "buildspec": "buildspec-libs-rpm.yml", 
            "type": "CODECOMMIT", 
            "location": "https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/codebuild-multispec"
        }, 
        "currentPhase": "SUBMITTED", 
        "startTime": 1500562366.239, 
        "id": "CodeBuild-RPM-Demo:82d05f8a-b161-401c-82f0-83cb41eba567", 
        "arn": "arn:aws:codebuild:eu-west-1:012345678912:build/CodeBuild-RPM-Demo:82d05f8a-b161-401c-82f0-83cb41eba567"
    }
}

After the build is completed successfully, check to see if the package appears in the artifact S3 bucket under the CodeBuild-RPM-Demo build project folder.

$ aws s3 ls s3://codebuild-demo-artifact-repository/CodeBuild-RPM-Demo/
2017-07-20 16:16:59       8108 cbsample-0.1-1.el7.centos.x86_64.rpm
2017-07-20 16:53:54       5320 cbsample-libs-0.1-1.el7.centos.x86_64.rpm

Conclusion

In this post, I have shown you how multiple buildspec files in the same source repository can be used to run multiple AWS CodeBuild build projects. I have also shown you how to provide a different buildspec file when starting the build.

For more information about AWS CodeBuild, see the AWS CodeBuild documentation. You can get started with AWS CodeBuild by using this step by step guide.


About the author

Prakash Palanisamy is a Solutions Architect for Amazon Web Services. When he is not working on Serverless, DevOps or Alexa, he will be solving problems in Project Euler. He also enjoys watching educational documentaries.

mkosi — A Tool for Generating OS Images

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/mkosi-a-tool-for-generating-os-images.html

Introducing mkosi

After blogging about
casync
I realized I never blogged about the
mkosi tool that combines nicely
with it. mkosi has been around for a while already, and its time to
make it a bit better known. mkosi stands for Make Operating System
Image
, and is a tool for precisely that: generating an OS tree or
image that can be booted.

Yes, there are many tools like mkosi, and a number of them are quite
well known and popular. But mkosi has a number of features that I
think make it interesting for a variety of use-cases that other tools
don’t cover that well.

What is mkosi?

What are those use-cases, and what does mkosi precisely set apart?
mkosi is definitely a tool with a focus on developer’s needs for
building OS images, for testing and debugging, but also for generating
production images with cryptographic protection. A typical use-case
would be to add a mkosi.default file to an existing project (for
example, one written in C or Python), and thus making it easy to
generate an OS image for it. mkosi will put together the image with
development headers and tools, compile your code in it, run your test
suite, then throw away the image again, and build a new one, this time
without development headers and tools, and install your build
artifacts in it. This final image is then “production-ready”, and only
contains your built program and the minimal set of packages you
configured otherwise. Such an image could then be deployed with
casync (or any other tool of course) to be delivered to your set of
servers, or IoT devices or whatever you are building.

mkosi is supposed to be legacy-free: the focus is clearly on
today’s technology, not yesteryear’s. Specifically this means that
we’ll generate GPT partition tables, not MBR/DOS ones. When you tell
mkosi to generate a bootable image for you, it will make it bootable
on EFI, not on legacy BIOS. The GPT images generated follow
specifications such as the Discoverable Partitions
Specification
,
so that /etc/fstab can remain unpopulated and tools such as
systemd-nspawn can automatically dissect the image and boot from
them.

So, let’s have a look on the specific images it can generate:

  1. Raw GPT disk image, with ext4 as root
  2. Raw GPT disk image, with btrfs as root
  3. Raw GPT disk image, with a read-only squashfs as root
  4. A plain directory on disk containing the OS tree directly (this is useful for creating generic container images)
  5. A btrfs subvolume on disk, similar to the plain directory
  6. A tarball of a plain directory

When any of the GPT choices above are selected, a couple of additional
options are available:

  1. A swap partition may be added in
  2. The system may be made bootable on EFI systems
  3. Separate partitions for /home and /srv may be added in
  4. The root, /home and /srv partitions may be optionally encrypted with LUKS
  5. The root partition may be protected using dm-verity, thus making offline attacks on the generated system hard
  6. If the image is made bootable, the dm-verity root hash is automatically added to the kernel command line, and the kernel together with its initial RAM disk and the kernel command line is optionally cryptographically signed for UEFI SecureBoot

Note that mkosi is distribution-agnostic. It currently can build
images based on the following Linux distributions:

  1. Fedora
  2. Debian
  3. Ubuntu
  4. ArchLinux
  5. openSUSE

Note though that not all distributions are supported at the same
feature level currently. Also, as mkosi is based on dnf
--installroot
, debootstrap, pacstrap and zypper, and those
packages are not packaged universally on all distributions, you might
not be able to build images for all those distributions on arbitrary
host distributions.

The GPT images are put together in a way that they aren’t just
compatible with UEFI systems, but also with VM and container managers
(that is, at least the smart ones, i.e. VM managers that know UEFI,
and container managers that grok GPT disk images) to a large
degree. In fact, the idea is that you can use mkosi to build a
single GPT image that may be used to:

  1. Boot on bare-metal boxes
  2. Boot in a VM
  3. Boot in a systemd-nspawn container
  4. Directly run a systemd service off, using systemd’s RootImage= unit file setting

Note that in all four cases the dm-verity data is automatically used
if available to ensure the image is not tampered with (yes, you read
that right, systemd-nspawn and systemd’s RootImage= setting
automatically do dm-verity these days if the image has it.)

Mode of Operation

The simplest usage of mkosi is by simply invoking it without
parameters (as root):

# mkosi

Without any configuration this will create a GPT disk image for you,
will call it image.raw and drop it in the current directory. The
distribution used will be the same one as your host runs.

Of course in most cases you want more control about how the image is
put together, i.e. select package sets, select the distribution, size
partitions and so on. Most of that you can actually specify on the
command line, but it is recommended to instead create a couple of
mkosi.$SOMETHING files and directories in some directory. Then,
simply change to that directory and run mkosi without any further
arguments. The tool will then look in the current working directory
for these files and directories and make use of them (similar to how
make looks for a Makefile…). Every single file/directory is
optional, but if they exist they are honored. Here’s a list of the
files/directories mkosi currently looks for:

  1. mkosi.default — This is the main configuration file, here you
    can configure what kind of image you want, which distribution, which
    packages and so on.

  2. mkosi.extra/ — If this directory exists, then mkosi will copy
    everything inside it into the images built. You can place arbitrary
    directory hierarchies in here, and they’ll be copied over whatever is
    already in the image, after it was put together by the distribution’s
    package manager. This is the best way to drop additional static files
    into the image, or override distribution-supplied ones.

  3. mkosi.build — This executable file is supposed to be a build
    script. When it exists, mkosi will build two images, one after the
    other in the mode already mentioned above: the first version is the
    build image, and may include various build-time dependencies such as
    a compiler or development headers. The build script is also copied
    into it, and then run inside it. The script should then build
    whatever shall be built and place the result in $DESTDIR (don’t
    worry, popular build tools such as Automake or Meson all honor
    $DESTDIR anyway, so there’s not much to do here explicitly). It may
    also run a test suite, or anything else you like. After the script
    finished, the build image is removed again, and a second image (the
    final image) is built. This time, no development packages are
    included, and the build script is not copied into the image again —
    however, the build artifacts from the first run (i.e. those placed in
    $DESTDIR) are copied into the image.

  4. mkosi.postinst — If this executable script exists, it is invoked
    inside the image (inside a systemd-nspawn invocation) and can
    adjust the image as it likes at a very late point in the image
    preparation. If mkosi.build exists, i.e. the dual-phased
    development build process used, then this script will be invoked
    twice: once inside the build image and once inside the final
    image. The first parameter passed to the script clarifies which phase
    it is run in.

  5. mkosi.nspawn — If this file exists, it should contain a
    container configuration file for systemd-nspawn (see
    systemd.nspawn(5)
    for details), which shall be shipped along with the final image and
    shall be included in the check-sum calculations (see below).

  6. mkosi.cache/ — If this directory exists, it is used as package
    cache directory for the builds. This directory is effectively bind
    mounted into the image at build time, in order to speed up building
    images. The package installers of the various distributions will
    place their package files here, so that subsequent runs can reuse
    them.

  7. mkosi.passphrase — If this file exists, it should contain a
    pass-phrase to use for the LUKS encryption (if that’s enabled for the
    image built). This file should not be readable to other users.

  8. mkosi.secure-boot.crt and mkosi.secure-boot.key should be an
    X.509 key pair to use for signing the kernel and initrd for UEFI
    SecureBoot, if that’s enabled.

How to use it

So, let’s come back to our most trivial example, without any of the
mkosi.$SOMETHING files around:

# mkosi

As mentioned, this will create a build file image.raw in the current
directory. How do we use it? Of course, we could dd it onto some USB
stick and boot it on a bare-metal device. However, it’s much simpler
to first run it in a container for testing:

# systemd-nspawn -bi image.raw

And there you go: the image should boot up, and just work for you.

Now, let’s make things more interesting. Let’s still not use any of
the mkosi.$SOMETHING files around:

# mkosi -t raw_btrfs --bootable -o foobar.raw
# systemd-nspawn -bi foobar.raw

This is similar as the above, but we made three changes: it’s no
longer GPT + ext4, but GPT + btrfs. Moreover, the system is made
bootable on UEFI systems, and finally, the output is now called
foobar.raw.

Because this system is bootable on UEFI systems, we can run it in KVM:

qemu-kvm -m 512 -smp 2 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -drive format=raw,file=foobar.raw

This will look very similar to the systemd-nspawn invocation, except
that this uses full VM virtualization rather than container
virtualization. (Note that the way to run a UEFI qemu/kvm instance
appears to change all the time and is different on the various
distributions. It’s quite annoying, and I can’t really tell you what
the right qemu command line is to make this work on your system.)

Of course, it’s not all raw GPT disk images with mkosi. Let’s try
a plain directory image:

# mkosi -d fedora -t directory -o quux
# systemd-nspawn -bD quux

Of course, if you generate the image as plain directory you can’t boot
it on bare-metal just like that, nor run it in a VM.

A more complex command line is the following:

# mkosi -d fedora -t raw_squashfs --checksum --xz --package=openssh-clients --package=emacs

In this mode we explicitly pick Fedora as the distribution to use, ask
mkosi to generate a compressed GPT image with a root squashfs,
compress the result with xz, and generate a SHA256SUMS file with
the hashes of the generated artifacts. The package will contain the
SSH client as well as everybody’s favorite editor.

Now, let’s make use of the various mkosi.$SOMETHING files. Let’s
say we are working on some Automake-based project and want to make it
easy to generate a disk image off the development tree with the
version you are hacking on. Create a configuration file:

# cat > mkosi.default <<EOF
[Distribution]
Distribution=fedora
Release=24

[Output]
Format=raw_btrfs
Bootable=yes

[Packages]
# The packages to appear in both the build and the final image
Packages=openssh-clients httpd
# The packages to appear in the build image, but absent from the final image
BuildPackages=make gcc libcurl-devel
EOF

And let’s add a build script:

# cat > mkosi.build <<EOF
#!/bin/sh
./autogen.sh
./configure --prefix=/usr
make -j `nproc`
make install
EOF
# chmod +x mkosi.build

And with all that in place we can now build our project into a disk image, simply by typing:

# mkosi

Let’s try it out:

# systemd-nspawn -bi image.raw

Of course, if you do this you’ll notice that building an image like
this can be quite slow. And slow build times are actively hurtful to
your productivity as a developer. Hence let’s make things a bit
faster. First, let’s make use of a package cache shared between runs:

# mkdir mkosi.cache

Building images now should already be substantially faster (and
generate less network traffic) as the packages will now be downloaded
only once and reused. However, you’ll notice that unpacking all those
packages and the rest of the work is still quite slow. But mkosi can
help you with that. Simply use mkosi‘s incremental build feature. In
this mode mkosi will make a copy of the build and final images
immediately before dropping in your build sources or artifacts, so
that building an image becomes a lot quicker: instead of always
starting totally from scratch a build will now reuse everything it can
reuse from a previous run, and immediately begin with building your
sources rather than the build image to build your sources in. To
enable the incremental build feature use -i:

# mkosi -i

Note that if you use this option, the package list is not updated
anymore from your distribution’s servers, as the cached copy is made
after all packages are installed, and hence until you actually delete
the cached copy the distribution’s network servers aren’t contacted
again and no RPMs or DEBs are downloaded. This means the distribution
you use becomes “frozen in time” this way. (Which might be a bad
thing, but also a good thing, as it makes things kinda reproducible.)

Of course, if you run mkosi a couple of times you’ll notice that it
won’t overwrite the generated image when it already exists. You can
either delete the file yourself first (rm image.raw) or let mkosi
do it for you right before building a new image, with mkosi -f. You
can also tell mkosi to not only remove any such pre-existing images,
but also remove any cached copies of the incremental feature, by using
-f twice.

I wrote mkosi originally in order to test systemd, and quickly
generate a disk image of various distributions with the most current
systemd version from git, without all that affecting my host system. I
regularly use mkosi for that today, in incremental mode. The two
commands I use most in that context are:

# mkosi -if && systemd-nspawn -bi image.raw

And sometimes:

# mkosi -iff && systemd-nspawn -bi image.raw

The latter I use only if I want to regenerate everything based on the
very newest set of RPMs provided by Fedora, instead of a cached
snapshot of it.

BTW, the mkosi files for systemd are included in the systemd git
tree:
mkosi.default
and
mkosi.build. This
way, any developer who wants to quickly test something with current
systemd git, or wants to prepare a patch based on it and test it can
check out the systemd repository and simply run mkosi in it and a
few minutes later he has a bootable image he can test in
systemd-nspawn or KVM. casync has similar files:
mkosi.default,
mkosi.build.

Random Interesting Features

  1. As mentioned already, mkosi will generate dm-verity enabled
    disk images if you ask for it. For that use the --verity switch on
    the command line or Verity= setting in mkosi.default. Of course,
    dm-verity implies that the root volume is read-only. In this mode
    the top-level dm-verity hash will be placed along-side the output
    disk image in a file named the same way, but with the .roothash
    suffix. If the image is to be created bootable, the root hash is also
    included on the kernel command line in the roothash= parameter,
    which current systemd versions can use to both find and activate the
    root partition in a dm-verity protected way. BTW: it’s a good idea
    to combine this dm-verity mode with the raw_squashfs image mode,
    to generate a genuinely protected, compressed image suitable for
    running in your IoT device.

  2. As indicated above, mkosi can automatically create a check-sum
    file SHA256SUMS for you (--checksum) covering all the files it
    outputs (which could be the image file itself, a matching .nspawn
    file using the mkosi.nspawn file mentioned above, as well as the
    .roothash file for the dm-verity root hash.) It can then
    optionally sign this with gpg (--sign). Note that systemd‘s
    machinectl pull-tar and machinectl pull-raw command can download
    these files and the SHA256SUMS file automatically and verify things
    on download. With other words: what mkosi outputs is perfectly
    ready for downloads using these two systemd commands.

  3. As mentioned, mkosi is big on supporting UEFI SecureBoot. To
    make use of that, place your X.509 key pair in two files
    mkosi.secureboot.crt and mkosi.secureboot.key, and set
    SecureBoot= or --secure-boot. If so, mkosi will sign the
    kernel/initrd/kernel command line combination during the build. Of
    course, if you use this mode, you should also use
    Verity=/--verity=, otherwise the setup makes only partial
    sense. Note that mkosi will not help you with actually enrolling
    the keys you use in your UEFI BIOS.

  4. mkosi has minimal support for GIT checkouts: when it recognizes
    it is run in a git checkout and you use the mkosi.build script
    stuff, the source tree will be copied into the build image, but will
    all files excluded by .gitignore removed.

  5. There’s support for encryption in place. Use --encrypt= or
    Encrypt=. Note that the UEFI ESP is never encrypted though, and the
    root partition only if explicitly requested. The /home and /srv
    partitions are unconditionally encrypted if that’s enabled.

  6. Images may be built with all documentation removed.

  7. The password for the root user and additional kernel command line
    arguments may be configured for the image to generate.

Minimum Requirements

Current mkosi requires Python 3.5, and has a number of dependencies,
listed in the
README. Most
notably you need a somewhat recent systemd version to make use of its
full feature set: systemd 233. Older versions are already packaged for
various distributions, but much of what I describe above is only
available in the most recent release mkosi 3.

The UEFI SecureBoot support requires sbsign which currently isn’t
available in Fedora, but there’s a
COPR
.

Future

It is my intention to continue turning mkosi into a tool suitable
for:

  1. Testing and debugging projects
  2. Building images for secure devices
  3. Building portable service images
  4. Building images for secure VMs and containers

One of the biggest goals I have for the future is to teach mkosi and
systemd/sd-boot native support for A/B IoT style partition
setups. The idea is that the combination of systemd, casync and
mkosi provides generic building blocks for building secure,
auto-updating devices in a generic way from, even though all pieces
may be used individually, too.

FAQ

  1. Why are you reinventing the wheel again? This is exactly like
    $SOMEOTHERPROJECT!
    — Well, to my knowledge there’s no tool that
    integrates this nicely with your project’s development tree, and can
    do dm-verity and UEFI SecureBoot and all that stuff for you. So
    nope, I don’t think this exactly like $SOMEOTHERPROJECT, thank you
    very much.

  2. What about creating MBR/DOS partition images? — That’s really
    out of focus to me. This is an exercise in figuring out how generic
    OSes and devices in the future should be built and an attempt to
    commoditize OS image building. And no, the future doesn’t speak MBR,
    sorry. That said, I’d be quite interested in adding support for
    booting on Raspberry Pi, possibly using a hybrid approach, i.e. using
    a GPT disk label, but arranging things in a way that the Raspberry Pi
    boot protocol (which is built around DOS partition tables), can still
    work.

  3. Is this portable? — Well, depends what you mean by
    portable. No, this tool runs on Linux only, and as it uses
    systemd-nspawn during the build process it doesn’t run on
    non-systemd systems either. But then again, you should be able to
    create images for any architecture you like with it, but of course if
    you want the image bootable on bare-metal systems only systems doing
    UEFI are supported (but systemd-nspawn should still work fine on
    them).

  4. Where can I get this stuff? — Try
    GitHub. And some distributions
    carry packaged versions, but I think none of them the current v3
    yet.

  5. Is this a systemd project? — Yes, it’s hosted under the
    systemd GitHub umbrella. And yes,
    during run-time systemd-nspawn in a current version is required. But
    no, the code-bases are separate otherwise, already because systemd
    is a C project, and mkosi Python.

  6. Requiring systemd 233 is a pretty steep requirement, no?
    Yes, but the feature we need kind of matters (systemd-nspawn‘s
    --overlay= switch), and again, this isn’t supposed to be a tool for
    legacy systems.

  7. Can I run the resulting images in LXC or Docker? — Humm, I am
    not an LXC nor Docker guy. If you select directory or subvolume
    as image type, LXC should be able to boot the generated images just
    fine, but I didn’t try. Last time I looked, Docker doesn’t permit
    running proper init systems as PID 1 inside the container, as they
    define their own run-time without intention to emulate a proper
    system. Hence, no I don’t think it will work, at least not with an
    unpatched Docker version. That said, again, don’t ask me questions
    about Docker, it’s not precisely my area of expertise, and quite
    frankly I am not a fan. To my knowledge neither LXC nor Docker are
    able to run containers directly off GPT disk images, hence the
    various raw_xyz image types are definitely not compatible with
    either. That means if you want to generate a single raw disk image
    that can be booted unmodified both in a container and on bare-metal,
    then systemd-nspawn is the container manager to go for
    (specifically, its -i/--image= switch).

Should you care? Is this a tool for you?

Well, that’s up to you really.

If you hack on some complex project and need a quick way to compile
and run your project on a specific current Linux distribution, then
mkosi is an excellent way to do that. Simply drop the mkosi.default
and mkosi.build files in your git tree and everything will be
easy. (And of course, as indicated above: if the project you are
hacking on happens to be called systemd or casync be aware that
those files are already part of the git tree — you can just use them.)

If you hack on some embedded or IoT device, then mkosi is a great
choice too, as it will make it reasonably easy to generate secure
images that are protected against offline modification, by using
dm-verity and UEFI SecureBoot.

If you are an administrator and need a nice way to build images for a
VM or systemd-nspawn container, or a portable service then mkosi
is an excellent choice too.

If you care about legacy computers, old distributions, non-systemd
init systems, old VM managers, Docker, … then no, mkosi is not for
you, but there are plenty of well-established alternatives around that
cover that nicely.

And never forget: mkosi is an Open Source project. We are happy to
accept your patches and other contributions.

Oh, and one unrelated last thing: don’t forget to submit your talk
proposal

and/or buy a ticket for
All Systems Go! 2017 in Berlin — the
conference where things like systemd, casync and mkosi are
discussed, along with a variety of other Linux userspace projects used
for building systems.

mkosi — A Tool for Generating OS Images

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/mkosi-a-tool-for-generating-os-images.html

Introducing mkosi

After blogging about
casync
I realized I never blogged about the
mkosi tool that combines nicely
with it. mkosi has been around for a while already, and its time to
make it a bit better known. mkosi stands for Make Operating System
Image
, and is a tool for precisely that: generating an OS tree or
image that can be booted.

Yes, there are many tools like mkosi, and a number of them are quite
well known and popular. But mkosi has a number of features that I
think make it interesting for a variety of use-cases that other tools
don’t cover that well.

What is mkosi?

What are those use-cases, and what does mkosi precisely set apart?
mkosi is definitely a tool with a focus on developer’s needs for
building OS images, for testing and debugging, but also for generating
production images with cryptographic protection. A typical use-case
would be to add a mkosi.default file to an existing project (for
example, one written in C or Python), and thus making it easy to
generate an OS image for it. mkosi will put together the image with
development headers and tools, compile your code in it, run your test
suite, then throw away the image again, and build a new one, this time
without development headers and tools, and install your build
artifacts in it. This final image is then “production-ready”, and only
contains your built program and the minimal set of packages you
configured otherwise. Such an image could then be deployed with
casync (or any other tool of course) to be delivered to your set of
servers, or IoT devices or whatever you are building.

mkosi is supposed to be legacy-free: the focus is clearly on
today’s technology, not yesteryear’s. Specifically this means that
we’ll generate GPT partition tables, not MBR/DOS ones. When you tell
mkosi to generate a bootable image for you, it will make it bootable
on EFI, not on legacy BIOS. The GPT images generated follow
specifications such as the Discoverable Partitions
Specification
,
so that /etc/fstab can remain unpopulated and tools such as
systemd-nspawn can automatically dissect the image and boot from
them.

So, let’s have a look on the specific images it can generate:

  1. Raw GPT disk image, with ext4 as root
  2. Raw GPT disk image, with btrfs as root
  3. Raw GPT disk image, with a read-only squashfs as root
  4. A plain directory on disk containing the OS tree directly (this is useful for creating generic container images)
  5. A btrfs subvolume on disk, similar to the plain directory
  6. A tarball of a plain directory

When any of the GPT choices above are selected, a couple of additional
options are available:

  1. A swap partition may be added in
  2. The system may be made bootable on EFI systems
  3. Separate partitions for /home and /srv may be added in
  4. The root, /home and /srv partitions may be optionally encrypted with LUKS
  5. The root partition may be protected using dm-verity, thus making offline attacks on the generated system hard
  6. If the image is made bootable, the dm-verity root hash is automatically added to the kernel command line, and the kernel together with its initial RAM disk and the kernel command line is optionally cryptographically signed for UEFI SecureBoot

Note that mkosi is distribution-agnostic. It currently can build
images based on the following Linux distributions:

  1. Fedora
  2. Debian
  3. Ubuntu
  4. ArchLinux
  5. openSUSE

Note though that not all distributions are supported at the same
feature level currently. Also, as mkosi is based on dnf
--installroot
, debootstrap, pacstrap and zypper, and those
packages are not packaged universally on all distributions, you might
not be able to build images for all those distributions on arbitrary
host distributions. For example, Fedora doesn’t package zypper,
hence you cannot build an openSUSE image easily on Fedora, but you can
still build Fedora (obviously…), Debian, Ubuntu and ArchLinux images
on it just fine.

The GPT images are put together in a way that they aren’t just
compatible with UEFI systems, but also with VM and container managers
(that is, at least the smart ones, i.e. VM managers that know UEFI,
and container managers that grok GPT disk images) to a large
degree. In fact, the idea is that you can use mkosi to build a
single GPT image that may be used to:

  1. Boot on bare-metal boxes
  2. Boot in a VM
  3. Boot in a systemd-nspawn container
  4. Directly run a systemd service off, using systemd’s RootImage= unit file setting

Note that in all four cases the dm-verity data is automatically used
if available to ensure the image is not tempered with (yes, you read
that right, systemd-nspawn and systemd’s RootImage= setting
automatically do dm-verity these days if the image has it.)

Mode of Operation

The simplest usage of mkosi is by simply invoking it without
parameters (as root):

# mkosi

Without any configuration this will create a GPT disk image for you,
will call it image.raw and drop it in the current directory. The
distribution used will be the same one as your host runs.

Of course in most cases you want more control about how the image is
put together, i.e. select package sets, select the distribution, size
partitions and so on. Most of that you can actually specify on the
command line, but it is recommended to instead create a couple of
mkosi.$SOMETHING files and directories in some directory. Then,
simply change to that directory and run mkosi without any further
arguments. The tool will then look in the current working directory
for these files and directories and make use of them (similar to how
make looks for a Makefile…). Every single file/directory is
optional, but if they exist they are honored. Here’s a list of the
files/directories mkosi currently looks for:

  1. mkosi.default — This is the main configuration file, here you
    can configure what kind of image you want, which distribution, which
    packages and so on.

  2. mkosi.extra/ — If this directory exists, then mkosi will copy
    everything inside it into the images built. You can place arbitrary
    directory hierarchies in here, and they’ll be copied over whatever is
    already in the image, after it was put together by the distribution’s
    package manager. This is the best way to drop additional static files
    into the image, or override distribution-supplied ones.

  3. mkosi.build — This executable file is supposed to be a build
    script. When it exists, mkosi will build two images, one after the
    other in the mode already mentioned above: the first version is the
    build image, and may include various build-time dependencies such as
    a compiler or development headers. The build script is also copied
    into it, and then run inside it. The script should then build
    whatever shall be built and place the result in $DESTDIR (don’t
    worry, popular build tools such as Automake or Meson all honor
    $DESTDIR anyway, so there’s not much to do here explicitly). It may
    also run a test suite, or anything else you like. After the script
    finished, the build image is removed again, and a second image (the
    final image) is built. This time, no development packages are
    included, and the build script is not copied into the image again —
    however, the build artifacts from the first run (i.e. those placed in
    $DESTDIR) are copied into the image.

  4. mkosi.postinst — If this executable script exists, it is invoked
    inside the image (inside a systemd-nspawn invocation) and can
    adjust the image as it likes at a very late point in the image
    preparation. If mkosi.build exists, i.e. the dual-phased
    development build process used, then this script will be invoked
    twice: once inside the build image and once inside the final
    image. The first parameter passed to the script clarifies which phase
    it is run in.

  5. mkosi.nspawn — If this file exists, it should contain a
    container configuration file for systemd-nspawn (see
    systemd.nspawn(5)
    for details), which shall be shipped along with the final image and
    shall be included in the check-sum calculations (see below).

  6. mkosi.cache/ — If this directory exists, it is used as package
    cache directory for the builds. This directory is effectively bind
    mounted into the image at build time, in order to speed up building
    images. The package installers of the various distributions will
    place their package files here, so that subsequent runs can reuse
    them.

  7. mkosi.passphrase — If this file exists, it should contain a
    pass-phrase to use for the LUKS encryption (if that’s enabled for the
    image built). This file should not be readable to other users.

  8. mkosi.secure-boot.crt and mkosi.secure-boot.key should be an
    X.509 key pair to use for signing the kernel and initrd for UEFI
    SecureBoot, if that’s enabled.

How to use it

So, let’s come back to our most trivial example, without any of the
mkosi.$SOMETHING files around:

# mkosi

As mentioned, this will create a build file image.raw in the current
directory. How do we use it? Of course, we could dd it onto some USB
stick and boot it on a bare-metal device. However, it’s much simpler
to first run it in a container for testing:

# systemd-nspawn -bi image.raw

And there you go: the image should boot up, and just work for you.

Now, let’s make things more interesting. Let’s still not use any of
the mkosi.$SOMETHING files around:

# mkosi -t raw_btrfs --bootable -o foobar.raw
# systemd-nspawn -bi foobar.raw

This is similar as the above, but we made three changes: it’s no
longer GPT + ext4, but GPT + btrfs. Moreover, the system is made
bootable on UEFI systems, and finally, the output is now called
foobar.raw.

Because this system is bootable on UEFI systems, we can run it in KVM:

qemu-kvm -m 512 -smp 2 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd -drive format=raw,file=foobar.raw

This will look very similar to the systemd-nspawn invocation, except
that this uses full VM virtualization rather than container
virtualization. (Note that the way to run a UEFI qemu/kvm instance
appears to change all the time and is different on the various
distributions. It’s quite annoying, and I can’t really tell you what
the right qemu command line is to make this work on your system.)

Of course, it’s not all raw GPT disk images with mkosi. Let’s try
a plain directory image:

# mkosi -d fedora -t directory -o quux
# systemd-nspawn -bD quux

Of course, if you generate the image as plain directory you can’t boot
it on bare-metal just like that, nor run it in a VM.

A more complex command line is the following:

# mkosi -d fedora -t raw_squashfs --checksum --xz --package=openssh-clients --package=emacs

In this mode we explicitly pick Fedora as the distribution to use, ask
mkosi to generate a compressed GPT image with a root squashfs,
compress the result with xz, and generate a SHA256SUMS file with
the hashes of the generated artifacts. The package will contain the
SSH client as well as everybody’s favorite editor.

Now, let’s make use of the various mkosi.$SOMETHING files. Let’s
say we are working on some Automake-based project and want to make it
easy to generate a disk image off the development tree with the
version you are hacking on. Create a configuration file:

# cat > mkosi.default <<EOF
[Distribution]
Distribution=fedora
Release=24

[Output]
Format=raw_btrfs
Bootable=yes

[Packages]
# The packages to appear in both the build and the final image
Packages=openssh-clients httpd
# The packages to appear in the build image, but absent from the final image
BuildPackages=make gcc libcurl-devel
EOF

And let’s add a build script:

# cat > mkosi.build <<EOF
#!/bin/sh
cd $SRCDIR
./autogen.sh
./configure --prefix=/usr
make -j `nproc`
make install
EOF
# chmod +x mkosi.build

And with all that in place we can now build our project into a disk image, simply by typing:

# mkosi

Let’s try it out:

# systemd-nspawn -bi image.raw

Of course, if you do this you’ll notice that building an image like
this can be quite slow. And slow build times are actively hurtful to
your productivity as a developer. Hence let’s make things a bit
faster. First, let’s make use of a package cache shared between runs:

# mkdir mkosi.chache

Building images now should already be substantially faster (and
generate less network traffic) as the packages will now be downloaded
only once and reused. However, you’ll notice that unpacking all those
packages and the rest of the work is still quite slow. But mkosi can
help you with that. Simply use mkosi‘s incremental build feature. In
this mode mkosi will make a copy of the build and final images
immediately before dropping in your build sources or artifacts, so
that building an image becomes a lot quicker: instead of always
starting totally from scratch a build will now reuse everything it can
reuse from a previous run, and immediately begin with building your
sources rather than the build image to build your sources in. To
enable the incremental build feature use -i:

# mkosi -i

Note that if you use this option, the package list is not updated
anymore from your distribution’s servers, as the cached copy is made
after all packages are installed, and hence until you actually delete
the cached copy the distribution’s network servers aren’t contacted
again and no RPMs or DEBs are downloaded. This means the distribution
you use becomes “frozen in time” this way. (Which might be a bad
thing, but also a good thing, as it makes things kinda reproducible.)

Of course, if you run mkosi a couple of times you’ll notice that it
won’t overwrite the generated image when it already exists. You can
either delete the file yourself first (rm image.raw) or let mkosi
do it for you right before building a new image, with mkosi -f. You
can also tell mkosi to not only remove any such pre-existing images,
but also remove any cached copies of the incremental feature, by using
-f twice.

I wrote mkosi originally in order to test systemd, and quickly
generate a disk image of various distributions with the most current
systemd version from git, without all that affecting my host system. I
regularly use mkosi for that today, in incremental mode. The two
commands I use most in that context are:

# mkosi -if && systemd-nspawn -bi image.raw

And sometimes:

# mkosi -iff && systemd-nspawn -bi image.raw

The latter I use only if I want to regenerate everything based on the
very newest set of RPMs provided by Fedora, instead of a cached
snapshot of it.

BTW, the mkosi files for systemd are included in the systemd git
tree:
mkosi.default
and
mkosi.build. This
way, any developer who wants to quickly test something with current
systemd git, or wants to prepare a patch based on it and test it can
check out the systemd repository and simply run mkosi in it and a
few minutes later he has a bootable image he can test in
systemd-nspawn or KVM. casync has similar files:
mkosi.default,
mkosi.build.

Random Interesting Features

  1. As mentioned already, mkosi will generate dm-verity enabled
    disk images if you ask for it. For that use the --verity switch on
    the command line or Verity= setting in mkosi.default. Of course,
    dm-verity implies that the root volume is read-only. In this mode
    the top-level dm-verity hash will be placed along-side the output
    disk image in a file named the same way, but with the .roothash
    suffix. If the image is to be created bootable, the root hash is also
    included on the kernel command line in the roothash= parameter,
    which current systemd versions can use to both find and activate the
    root partition in a dm-verity protected way. BTW: it’s a good idea
    to combine this dm-verity mode with the raw_squashfs image mode,
    to generate a genuinely protected, compressed image suitable for
    running in your IoT device.

  2. As indicated above, mkosi can automatically create a check-sum
    file SHA256SUMS for you (--checksum) covering all the files it
    outputs (which could be the image file itself, a matching .nspawn
    file using the mkosi.nspawn file mentioned above, as well as the
    .roothash file for the dm-verity root hash.) It can then
    optionally sign this with gpg (--sign). Note that systemd‘s
    machinectl pull-tar and machinectl pull-raw command can download
    these files and the SHA256SUMS file automatically and verify things
    on download. With other words: what mkosi outputs is perfectly
    ready for downloads using these two systemd commands.

  3. As mentioned, mkosi is big on supporting UEFI SecureBoot. To
    make use of that, place your X.509 key pair in two files
    mkosi.secureboot.crt and mkosi.secureboot.key, and set
    SecureBoot= or --secure-boot. If so, mkosi will sign the
    kernel/initrd/kernel command line combination during the build. Of
    course, if you use this mode, you should also use
    Verity=/--verity=, otherwise the setup makes only partial
    sense. Note that mkosi will not help you with actually enrolling
    the keys you use in your UEFI BIOS.

  4. mkosi has minimal support for GIT checkouts: when it recognizes
    it is run in a git checkout and you use the mkosi.build script
    stuff, the source tree will be copied into the build image, but will
    all files excluded by .gitignore removed.

  5. There’s support for encryption in place. Use --encrypt= or
    Encrypt=. Note that the UEFI ESP is never encrypted though, and the
    root partition only if explicitly requested. The /home and /srv
    partitions are unconditionally encrypted if that’s enabled.

  6. Images may be built with all documentation removed.

  7. The password for the root user and additional kernel command line
    arguments may be configured for the image to generate.

Minimum Requirements

Current mkosi requires Python 3.5, and has a number of dependencies,
listed in the
README. Most
notably you need a somewhat recent systemd version to make use of its
full feature set: systemd 233. Older versions are already packaged for
various distributions, but much of what I describe above is only
available in the most recent release mkosi 3.

The UEFI SecureBoot support requires sbsign which currently isn’t
available in Fedora, but there’s a
COPR
.

Future

It is my intention to continue turning mkosi into a tool suitable
for:

  1. Testing and debugging projects
  2. Building images for secure devices
  3. Building portable service images
  4. Building images for secure VMs and containers

One of the biggest goals I have for the future is to teach mkosi and
systemd/sd-boot native support for A/B IoT style partition
setups. The idea is that the combination of systemd, casync and
mkosi provides generic building blocks for building secure,
auto-updating devices in a generic way from, even though all pieces
may be used individually, too.

FAQ

  1. Why are you reinventing the wheel again? This is exactly like
    $SOMEOTHERPROJECT!
    — Well, to my knowledge there’s no tool that
    integrates this nicely with your project’s development tree, and can
    do dm-verity and UEFI SecureBoot and all that stuff for you. So
    nope, I don’t think this exactly like $SOMEOTHERPROJECT, thank you
    very much.

  2. What about creating MBR/DOS partition images? — That’s really
    out of focus to me. This is an exercise in figuring out how generic
    OSes and devices in the future should be built and an attempt to
    commoditize OS image building. And no, the future doesn’t speak MBR,
    sorry. That said, I’d be quite interested in adding support for
    booting on Raspberry Pi, possibly using a hybrid approach, i.e. using
    a GPT disk label, but arranging things in a way that the Raspberry Pi
    boot protocol (which is built around DOS partition tables), can still
    work.

  3. Is this portable? — Well, depends what you mean by
    portable. No, this tool runs on Linux only, and as it uses
    systemd-nspawn during the build process it doesn’t run on
    non-systemd systems either. But then again, you should be able to
    create images for any architecture you like with it, but of course if
    you want the image bootable on bare-metal systems only systems doing
    UEFI are supported (but systemd-nspawn should still work fine on
    them).

  4. Where can I get this stuff? — Try
    GitHub. And some distributions
    carry packaged versions, but I think none of them the current v3
    yet.

  5. Is this a systemd project? — Yes, it’s hosted under the
    systemd GitHub umbrella. And yes,
    during run-time systemd-nspawn in a current version is required. But
    no, the code-bases are separate otherwise, already because systemd
    is a C project, and mkosi Python.

  6. Requiring systemd 233 is a pretty steep requirement, no?
    Yes, but the feature we need kind of matters (systemd-nspawn‘s
    --overlay= switch), and again, this isn’t supposed to be a tool for
    legacy systems.

  7. Can I run the resulting images in LXC or Docker? — Humm, I am
    not an LXC nor Docker guy. If you select directory or subvolume
    as image type, LXC should be able to boot the generated images just
    fine, but I didn’t try. Last time I looked, Docker doesn’t permit
    running proper init systems as PID 1 inside the container, as they
    define their own run-time without intention to emulate a proper
    system. Hence, no I don’t think it will work, at least not with an
    unpatched Docker version. That said, again, don’t ask me questions
    about Docker, it’s not precisely my area of expertise, and quite
    frankly I am not a fan. To my knowledge neither LXC nor Docker are
    able to run containers directly off GPT disk images, hence the
    various raw_xyz image types are definitely not compatible with
    either. That means if you want to generate a single raw disk image
    that can be booted unmodified both in a container and on bare-metal,
    then systemd-nspawn is the container manager to go for
    (specifically, its -i/--image= switch).

Should you care? Is this a tool for you?

Well, that’s up to you really.

If you hack on some complex project and need a quick way to compile
and run your project on a specific current Linux distribution, then
mkosi is an excellent way to do that. Simply drop the mkosi.default
and mkosi.build files in your git tree and everything will be
easy. (And of course, as indicated above: if the project you are
hacking on happens to be called systemd or casync be aware that
those files are already part of the git tree — you can just use them.)

If you hack on some embedded or IoT device, then mkosi is a great
choice too, as it will make it reasonably easy to generate secure
images that are protected against offline modification, by using
dm-verity and UEFI SecureBoot.

If you are an administrator and need a nice way to build images for a
VM or systemd-nspawn container, or a portable service then mkosi
is an excellent choice too.

If you care about legacy computers, old distributions, non-systemd
init systems, old VM managers, Docker, … then no, mkosi is not for
you, but there are plenty of well-established alternatives around that
cover that nicely.

And never forget: mkosi is an Open Source project. We are happy to
accept your patches and other contributions.

Oh, and one unrelated last thing: don’t forget to submit your talk
proposal

and/or buy a ticket for
All Systems Go! 2017 in Berlin — the
conference where things like systemd, casync and mkosi are
discussed, along with a variety of other Linux userspace projects used
for building systems.

Bash 4.4 and Readline 7.0 released

Post Syndicated from jake original http://lwn.net/Articles/700982/rss

The GNU Bourne Again SHell
(Bash) project has released version 4.4 of the tool. It comes with a large
number of bug fixes as well as new features:”The most notable new
features are mapfile’s ability to use an arbitrary
record delimiter; a –help option available for nearly all builtins; a
new family of ${[email protected]} expansions that transform the value of
`parameter’; the `local’ builtin’s ability to save and restore the state
of the single-letter shell option flags around function calls; a new
EXECIGNORE variable, which adds the ability to specify names that should
be ignored when searching for commands; and the beginning of an SDK for
loadable builtins, which consists of a set of headers and a Makefile
fragment that can be included in projects wishing to build their own
loadable builtins, augmented by support for a BASH_LOADABLES_PATH variable
that defines a search path for builtins loaded with `enable -f’. The existing
loadable builtin examples are now installed by default with `make
install’.
” In addition, the related Readline
command-line editing library
project has released Readline 7.0.

How to Translate HIPAA Controls to AWS CloudFormation Templates: Part 3 of the Automating HIPAA Compliance Series

Post Syndicated from Chris Crosbie original https://blogs.aws.amazon.com/security/post/Tx2X8A35ONJYE2V/How-to-Translate-HIPAA-Controls-to-AWS-CloudFormation-Templates-Part-3-of-the-Au

In my previous post, I walked through the setup of a DevSecOps environment that gives healthcare developers the ability to launch their own healthcare web server. At the heart of the architecture is AWS CloudFormation, a JSON representation of your architecture that allows security administrators to provision AWS resources according to the compliance standards they define. In today’s post, I will share examples that provide a Top 10 List of CloudFormation code snippets that you can consider when trying to map the requirements of the AWS Business Associates Agreement (BAA) to CloudFormation templates.

The example CloudFormation template I use as an example in today’s post is the same template I used in my previous post to define a healthcare product in AWS Directory Service. The template creates a healthcare web server that follows many of the contractual obligations outlined in the AWS BAA. The template also allows healthcare developers to customize their web server according to the following parameters:

FriendlyName – The name with which you tag your server.

CodeCommitRepo – The cloneUrlHttp field for the Git repository that you would like to release on the web server.

Environment – A choice between PROD and TEST. TEST will create a security group with several secure ports open, including SSH, from within a Classless Inter-Domain Routing (CIDR) block range. Choosing PROD will create a security group with HTTPS that is only accessible from the public Internet. (Exposing production web servers directly to the public Internet is not a best practice and is shown for example purposes only).

PHI – If you need to store protected health information (PHI) on the server. Choosing YES will create an encrypted EBS volume and attach it to the web server.

WebDirectory – This is the name of your website. For example, DNS-NAME/WebDirectory.

InstanceType – This is the Amazon EC2 instance type on which the code will be deployed. Because the AWS BAA requires PHI to be processed on dedicated instances, the choices here are limited to those EC2 instance types that are offered in dedicated tenancy mode.

I will forego CloudFormation tutorials in this post because an abundance of material for learning CloudFormation is easily accessible in AWS documentation. Instead, I will jump right in to share the Top 10 List of CloudFormation code snippets. If you are new to CloudFormation, you might find value in first understanding the capabilities it offers. qwikLABS is a great resource for learning AWS technology and offers multiple ClouldFormation labs to bring you up to speed quickly. The qwikLabs site offers entry-level CloudFormation labs at no cost. 

It’s important to note that the example CloudFormation template from which the following 10 snippets are taken is only an example and does not guarantee HIPAA or AWS BAA compliance. The template is meant as a starting point for developing your own templates that not only help you meet your AWS BAA obligations, but also provide general guidance as you expand beyond a single web server and start to utilize DevSecOps methods for other HIPAA-driven compliance needs.

Without further ado, here is a Top 10 List of CloudFormation compliance snippets that you should consider when building your own CloudFormation templates. In each section, I highlight the code I refer to in the associated description.

1. Set tenancy to dedicated.

To run a web server, you need an EC2 instance on which to install it. This can be accomplished in CloudFormation by adding it as a resource in the template. However, you also want to make sure that the EC2 instance meets your AWS BAA obligations by running in dedicated tenancy mode (in other words, your instance runs on single-tenant hardware).

To enforce this, in the EC2 instance change the tenancy property of the instance to dedicated.

    "EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
          "Tenancy" : "dedicated",
}}

2. Turn on detailed monitoring.

Detailed monitoring provides data about your EC2 instance over 1-minute periods. You can enable this in CloudFormation by adding a parameter to your EC2Instance resource.

When you turn on detailed monitoring, the data is then available for the instance in AWS Management Console graphs or through the API. Because there is an upcharge for detailed monitoring, you might want to turn this on only in your production environments. Having data each minute could be critical to recognizing failures and triggering responses to these failures.

On the other hand, also turning on detailed monitoring in your development environments could help you diagnose issues and prevent you from inadvertently moving such issues to production.

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"Tenancy" : "dedicated",
"Monitoring": "true",
}}

3. Define security group rules based on environment.

CloudFormation allows you to modify the firewall rules on your EC2 instance based on input parameters given to the template when it runs. This is done with AWS security groups and is very useful when you want to enforce certain compliance measures you define, such as disabling SSH access to production web servers or restricting development web servers from being accessed by the public Internet.

To do this, change security group settings based on whether your instance is targeted at test, QA, or production environments. You can do this by using conditions and an intrinsic If function. Intrinsic functions help you modify the security groups between environments according to your compliance standards, but at the same time, maintain consistent infastructure between environments.

"Conditions" : {
"CreatePRODResources" : {"Fn::Equals" : [{"Ref" : "Environment"}, "PROD"]}
},
"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"Tenancy" : "dedicated",
"Monitoring": "true"
}},
"SecurityGroups": [{
"Fn::If": [
"CreateTESTResources",
{"Ref": "InstanceSecurityGroupTEST"},
{"Ref": "InstanceSecurityGroupPROD"}
]
}],
"InstanceSecurityGroupTEST": {
"Type": "AWS::EC2::SecurityGroup",
"Condition" : "CreateTESTResources",
"Properties": {
"GroupDescription": "Enable access only from secure protocols",
"SecurityGroupIngress": [
{ "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "10.0.0.0/24" },
{ "IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "10.0.0.0/24" },
{ "IpProtocol" : "tcp", "FromPort" : "143", "ToPort" : "143", "CidrIp" : "10.0.0.0/24" },
{ "IpProtocol" : "tcp", "FromPort" : "465", "ToPort" : "465", "CidrIp" : "10.0.0.0/24" },
{ "IpProtocol" : "icmp", "FromPort" : "8", "ToPort" : "-1", "CidrIp" : "10.0.0.0/24" }
]
}}

4. Force instance tagging.

EC2 tagging is a common way for auditors and security professionals to understand why EC2 instances were launched and for what purpose. You can require the developer launching the template to enter information that you need for EC2 instance tagging by using CloudFormation parameters.

By using parameters such as AllowedValues and MinLength, you can maintain consistent tagging mechanisms by requiring that the developer enter a tag from a predetermined list of options (AllowedValues), or simply making them enter a text value meeting a certain length (MinLength).

In the following snippet, I use an AllowedValues list of YES and NO to make the developer tag the instance with information about whether or not the EC2 instance will be used to store PHI. I also use the MinLength to make the developer tag the EC2 instance with their email address so that we know who to contact if there is an issue with the instance.

"Parameters": {
"PHI":
{
"Description": "Will this instance need to store protected health information?",
"Default": "YES",
"Type": "String",
"AllowedValues": [
"YES",
"NO"
]
},
"Environment":
{
"Description": "Please specify the target environment",
"Default": "TEST",
"Type": "String",
"AllowedValues": [
"TEST",
"PROD",
“QA”
]
},
},
"InstanceOwnerEmail":
{
"Description": "Please enter the email address of the developer taking responsblity for this server",
"Default": "@mycompany.com",
"Type": "String"
},
"FriendlyName":
{
"Description": "Please enter a friendly name for the server",
"Type": "String",
"MinLength": 3,
"ConstraintDescription": "Must enter a friendy name for the server that is at least three characters long."
},
"Resources": {
"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"Tags":[
{ "Key" : "PHI", "Value" : {"Ref": "PHI"} },
{ "Key" : "Name", "Value" : {"Ref": "FriendlyName"} },
{ "Key" : "Environment", "Value" : {"Ref": "Environment"} },
{ "Key" : "InstanceOwnerEmail", "Value" : {"Ref": "InstanceOwnerEmail"} }
}}

5. Use IAM roles for EC2.

Applications must sign their API requests with AWS credentials. IAM roles are designed so that applications can securely make API requests from your instances, without requiring you to manage the security credentials that the applications use. In this example, I give the EC2 instance permission to perform a Git clone from our AWS CodeCommit repositories and push log data to Amazon CloudWatch.

"Resources": {

"HealthcareWebRole":
{
"Type": "AWS::IAM::Role",
"Properties":
{
"AssumeRolePolicyDocument":
{
"Version" : "2012-10-17",
"Statement":
[ {
"Effect": "Allow",
"Principal":
{
"Service": [ "ec2.amazonaws.com" ]
},
"Action": [ "sts:AssumeRole" ]
} ]
},
"Path": "/",
"ManagedPolicyArns": ["arn:aws:iam::aws:policy/AWSCodeCommitReadOnly", "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"]
}
},
"HealthcareWebInstanceProfile":
{
"Type": "AWS::IAM::InstanceProfile",
"Properties":
{
"Path": "/",
"Roles": [ { "Ref": "HealthcareWebRole" } ]
}
},
"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
IamInstanceProfile": {"Ref": "HealthcareWebInstanceProfile"}
}}

6. Add encrypted storage if you need to store PHI.

Applications that need to store PHI must encrypt the data at rest to meet the AWS BAA requirements. Amazon EBS encryption is one way to do this. The highlighted portion of the following snippet will add an encrypted EBS volume if the developer answers YES to the question, ”Will this instance need to store protected health information?”

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"BlockDeviceMappings" : [
{
"DeviceName" : "/dev/sdm",
"Ebs" : {
"VolumeType" : "io1",
"Iops" : "200",
"DeleteOnTermination" : "false",
"VolumeSize" : "10",
"Encrypted": {
"Fn::If" : [
"ContainsPHI",
"true",
"false"
]
}
}
},
{
"DeviceName" : "/dev/sdk",
"NoDevice" : {}
}
]}

7. Turn on CloudWatch Logs.

On each instance, install the AWS CloudWatch Logs agent, which uses CloudWatch Logs to monitor, store, and access your log files from EC2 instances. You can then retrieve the associated log data from a centralized logging repository that can be segregated from the application development team. 

After turning on the CloudWatch Logs agent, by default logs from the /var/log/messages are sent to CloudWatch. These messages store valuable, nondebug, and noncritical messages. These logs should be considered the general system activity logs, where you can start ensuring that you have the highest level of audit logging. However, you most likely will want to modify the /etc/awslogs/awslogs.conf file to add additional log locations if you choose to use this service in a HIPAA environment.

For example, you may want to add authentication logs (/var/log/auth.log) and set up alerting in CloudWatch to notify an administrator if repeated unauthorized access attempts are made against your server.

The following snippet will start the CloudWatch Logs agent and make sure it gets turned on during each startup.

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData" :
{ "Fn::Base64" :
{ "Fn::Join" :
[
"",
[
"service awslogs startn",
"chkconfig awslogs onn"
]
}}

8. Install Amazon Inspector.

Amazon Inspector is an automated security assessment service (currently offered in preview mode) that can help improve the security and compliance of applications deployed on AWS. Amazon Inspector allows you to run assessments for common best practices, vulnerablities, and exposures, and these findings can then be mapped to your own HIPAA control frameworks. Amazon Inspector makes it easier to validate that applications are adhering to your defined standards, and it helps to manage security issues proactively before a critical event such as a breach occurs.

Amazon Inspector requires an agent-based client to be installed on the EC2 instance. However, this installation can be performed by using a CloudFormation template. In the CloudFormation template used for this blog post, the Amazon Inspector installation is intentionally missing because Amazon Inspector in preview mode is available only in a different region than CodeCommit. However, if you would like to install it while in preview mode, you can use the following snippet.         

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData" :
{ "Fn::Base64" :
{ "Fn::Join" :
[
"",
[
“curl -O https://s3-us-west-2.amazonaws.com/inspector.agent.us-west-2/latest/install”
“sudo bash install”
]
}}

9. Configure SSL for encryption in flight.

As detailed in the AWS BAA, you must encrypt all PHI in flight. For production healthcare applications, open only those ports in the EC2 security group that are used in secure protocols.

The following snippet provides an example of UserData pulling down self-signed certificates from a publicly available Amazon S3 site. Although there may be situations when you have deemed self-signed certificates to be acceptable, a more secure approach would be to store the certificates in a private S3 bucket and give permission to the EC2 role to download the certificates and configurations.

Important: The certificates in the following code snippet are provided for demonstration purposes only and should never be used for any type of security or compliance purpose.

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData" :
{ "Fn::Base64" :
{ "Fn::Join" :
[
"",
[
"wget https://s3.amazonaws.com/awsiammedia/public/sample/hippa-compliance/aws-service-catalog/code-deployments/fakehipaa.crt -P /etc/pki/tls/certsn",

"wget https://s3.amazonaws.com/awsiammedia/public/sample/hippa-compliance/aws-service-catalog/code-deployments/fakehipaa.key -P /etc/pki/tls/private/n",

"wget https://s3.amazonaws.com/awsiammedia/public/sample/hippa-compliance/aws-service-catalog/code-deployments/fakehipaa.csr -P /etc/pki/tls/private/n",

"wget https://s3.amazonaws.com/awsiammedia/public/sample/hippa-compliance/aws-service-catalog/code-deployments/ssl.conf -P /etc/httpd/conf.d/ssl.confn",

"service httpd startn",

"chkconfig httpd onn"
]
}}

10. Clone from AWS CodeCommit.

So far the snippets in this post have focused on getting infrastructure secured in accordance with your compliance standards. However, you also need a process for automated code deployments. A variety of tools and techniques is available for automating code deployments, but in the following snippet, I will demonstrate an automated code deployment using an EC2 role and CodeCommit. This combination of an EC2 role and CodeCommit requires you to set the system-wide Git preferences by modifying the /etc/gitconfig file.

In the following snippet, after the authorized connection to CodeCommit is established, Git clones the repository provided by the developer into the default root folder of an Apache web server. However, this example could easily be extended to look for developer makefiles or to have an extra step that calls shell scripts that are written by the developer but maintained in CodeCommit.

"EC2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"UserData" :
{ "Fn::Base64" :
{ "Fn::Join" :
[

"git config –system credential.https://git-codecommit.us-east-1.amazonaws.com.helper ‘!aws –profile default codecommit credential-helper [email protected]’n",

"git config –system credential.https://git-codecommit.us-east-1.amazonaws.com.UseHttpPath truen",
"aws configure set region us-east-1n",

"cd /var/www/htmln",

"git clone ", {"Ref": "CodeCommitRepo"}, " ", {"Ref": "WebDirectory"}, " n",

[
]
}}

Conclusion

I hope that these 10 code snippets give you a head start to develop your own CloudFormation compliance templates. I encourage you to build on the template I provided to learn more about how CloudFormation works as you take steps to achieve your own DevSecOps architecture.

– Chris

Fscking Visual Studio Code JS Hello World

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/02/fscking-visual-studio-code-js-hello.html

The reason Linux never succeeded on the desktop is the lack of usability testing. Open-source programmers hate users, and created such an ugly baby that only a fanboy could love it. It’s funny watching the same thing happen to “Visual Studio Code”, Microsoft’s answer to the Atom editor. You’d think with Microsoft behind it, that it’d be guided by usability testing. The opposite is true. It spends a lot of time hyping it, but every time I try to use it, I encounter unreasonable hurdles for the simplest of things. It’s the standard open-source paradigm — they only spend effort to make something work in theory without the extra effort to make it usable in practice.The most common thing you’ll want to do is first create a “hello world” program, then debug it. As far as I can tell, there are no resources that’ll explain how to do this. So, for JavaScript on Windows, I thought I’d explain how this works.Firstly, you’ll need to install NodeJS and VS Code. Just choose the defaults, it’s uneventful.Secondly, you need to understand how projects work. This is the first hurdle everyone has with an IDE. You don’t simply run the code or use a makefile. Instead, you manage a lot of other things, such as the debugging environment.VSCode manages projects by putting things in a folder. In other words, you don’t “open a project”, but instead “open a folder”. Any project files (or version control files) that it needs it’ll create as it goes along. When you first tell it to start debugging, it’ll create a config file for the debugger filled in with defaults that’ll work for you.The steps to create/debug a “hello world” program are therefore:mkdir c:tmpfoocode c:tmpfoo[click new file icon in the project, create “app.js”]console.log(“hello world”);[click debug icon, save/close the launch.json file it creates for you][click go icon]By default, execution starts at ‘app.js’. If you’ve named your file something else, like ‘hello.js’, you’ll need to change your launch.json settings. Otherwise, you’ll get a “app.js does not exist” error message.At this point you aren’t out of the woods. When I tried this, I got the error message “cannot launch target (reason: spawn node ENOENT)”Normally, I don’t mind such error messages. When the user has done something so weird that the programmer can’t figure out how broken things are, then cryptic error messages like this are expected.But this isn’t something weird, but the most common problem somebody is likely to have. Specifically, it can’t find the path to “node.exe” to launch NodeJS. That’s because I’d just installed both NodeJS and VSCode, and told the installer to launch VSCode directly. It meant the shell wouldn’t have node.exe in it’s path until I logged out and back in again. Every new user is going to have the same problem. Yet, there is really no help available anywhere that explains what’s going on.After you install both NodeJS and VSCode, make sure that these programs are “added to the PATH variable” (the default in the installer). Then, restart Windows. Or, at this point open a virgin new command-prompt, at which point they’ll be in your path, so the steps I describe above on the command-line should work.Then, I got my “hello world” to work:I’m writing this post with the two error messages above in the hopes that Google can find them. When I googled the error messages, I got nothing helpful.In general, when using VSCode, and you get stuck, just assume that the VSCode programmers are a bunch of fscking retards, and try to reverse engineer what retards would do. In my case, I knew that not having node.exe in the path was the most common problem in the world, but that retards would nonetheless have the most cryptic error message in the world when it encountered the problem.Note that “Visual Studio Code” is a side Microsoft project. It’s nothing like “Visual Studio”, which is a rather awesome environment, though one that sadly doesn’t support JavaScript.

Fscking Visual Studio Code JS Hello World

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/02/fscking-visual-studio-code-js-hello.html

The reason Linux never succeeded on the desktop is the lack of usability testing. Open-source programmers hate users, and created such an ugly baby that only a fanboy could love it. It’s funny watching the same thing happen to “Visual Studio Code”, Microsoft’s answer to the Atom editor. You’d think with Microsoft behind it, that it’d be guided by usability testing. The opposite is true. It spends a lot of time hyping it, but every time I try to use it, I encounter unreasonable hurdles for the simplest of things. It’s the standard open-source paradigm — they only spend effort to make something work in theory without the extra effort to make it usable in practice.The most common thing you’ll want to do is first create a “hello world” program, then debug it. As far as I can tell, there are no resources that’ll explain how to do this. So, for JavaScript on Windows, I thought I’d explain how this works.Firstly, you’ll need to install NodeJS and VS Code. Just choose the defaults, it’s uneventful.Secondly, you need to understand how projects work. This is the first hurdle everyone has with an IDE. You don’t simply run the code or use a makefile. Instead, you manage a lot of other things, such as the debugging environment.VSCode manages projects by putting things in a folder. In other words, you don’t “open a project”, but instead “open a folder”. Any project files (or version control files) that it needs it’ll create as it goes along. When you first tell it to start debugging, it’ll create a config file for the debugger filled in with defaults that’ll work for you.The steps to create/debug a “hello world” program are therefore:mkdir c:tmpfoocode c:tmpfoo[click new file icon in the project, create “app.js”]console.log(“hello world”);[click debug icon, save/close the launch.json file it creates for you][click go icon]By default, execution starts at ‘app.js’. If you’ve named your file something else, like ‘hello.js’, you’ll need to change your launch.json settings. Otherwise, you’ll get a “app.js does not exist” error message.At this point you aren’t out of the woods. When I tried this, I got the error message “cannot launch target (reason: spawn node ENOENT)”Normally, I don’t mind such error messages. When the user has done something so weird that the programmer can’t figure out how broken things are, then cryptic error messages like this are expected.But this isn’t something weird, but the most common problem somebody is likely to have. Specifically, it can’t find the path to “node.exe” to launch NodeJS. That’s because I’d just installed both NodeJS and VSCode, and told the installer to launch VSCode directly. It meant the shell wouldn’t have node.exe in it’s path until I logged out and back in again. Every new user is going to have the same problem. Yet, there is really no help available anywhere that explains what’s going on.After you install both NodeJS and VSCode, make sure that these programs are “added to the PATH variable” (the default in the installer). Then, restart Windows. Or, at this point open a virgin new command-prompt, at which point they’ll be in your path, so the steps I describe above on the command-line should work.Then, I got my “hello world” to work:I’m writing this post with the two error messages above in the hopes that Google can find them. When I googled the error messages, I got nothing helpful.In general, when using VSCode, and you get stuck, just assume that the VSCode programmers are a bunch of fscking retards, and try to reverse engineer what retards would do. In my case, I knew that not having node.exe in the path was the most common problem in the world, but that retards would nonetheless have the most cryptic error message in the world when it encountered the problem.Note that “Visual Studio Code” is a side Microsoft project. It’s nothing like “Visual Studio”, which is a rather awesome environment, though one that sadly doesn’t support JavaScript.

systemd for Developers II

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/socket-activation2.html

It has been way too long since I posted the first
episode
of my systemd for Developers series. Here’s finally the
second part. Make sure you read the first episode of the series before
you start with this part since I’ll assume the reader grokked the wonders
of socket activation.

Socket Activation, Part II

This time we’ll focus on adding socket activation support to real-life
software, more specifically the CUPS printing server. Most current Linux
desktops run CUPS by default these days, since printing is so basic that it’s a
must have, and must just work when the user needs it. However, most desktop
CUPS installations probably don’t actually see more than a handful of print
jobs each month. Even if you are a busy office worker you’ll unlikely to
generate more than a couple of print jobs an hour on your PC. Also, printing is
not time critical. Whether a job takes 50ms or 100ms until it reaches the
printer matters little. As long as it is less than a few seconds the user
probably won’t notice. Finally, printers are usually peripheral hardware: they
aren’t built into your laptop, and you don’t always carry them around plugged
in. That all together makes CUPS a perfect candidate for lazy activation:
instead of starting it unconditionally at boot we just start it on-demand, when
it is needed. That way we can save resources, at boot and at runtime. However,
this kind of activation needs to take place transparently, so that the user
doesn’t notice that the print server was not actually running yet when he tried
to access it. To achieve that we need to make sure that the print server is
started as soon at least one of three conditions hold:

A local client tries to talk to the print server, for example because
a GNOME application opened the printing dialog which tries to enumerate
available printers.

A printer is being plugged in locally, and it should be configured and
enabled and then optionally the user be informed about it.

At boot, when there’s still an unprinted print job lurking in the queue.

Of course, the desktop is not the only place where CUPS is used. CUPS can be
run in small and big print servers, too. In that case the amount of print jobs
is substantially higher, and CUPS should be started right away at boot. That
means that (optionally) we still want to start CUPS unconditionally at boot and
not delay its execution until when it is needed.

Putting this all together we need four kind of activation to make CUPS work
well in all situations at minimal resource usage: socket based activation (to
support condition 1 above), hardware based activation (to support condition 2),
path based activation (for condition 3) and finally boot-time activation (for
the optional server usecase). Let’s focus on these kinds of activation in more
detail, and in particular on socket-based activation.

Socket Activation in Detail

To implement socket-based activation in CUPS we need to make sure that when
sockets are passed from systemd these are used to listen on instead of binding
them directly in the CUPS source code. Fortunately this is relatively easy to
do in the CUPS sources, since it already supports launchd-style socket
activation, as it is used on MacOS X (note that CUPS is nowadays an Apple
project). That means the code already has all the necessary hooks to add
systemd-style socket activation with minimal work.

To begin with our patching session we check out the CUPS sources.
Unfortunately CUPS is still stuck in unhappy Subversion country and not using
git yet. In order to simplify our patching work our first step is to use
git-svn to check it out locally in a way we can access it with the
usual git tools:

git svn clone http://svn.easysw.com/public/cups/trunk/ cups

This will take a while. After the command finished we use the wonderful
git grep to look for all occurences of the word “launchd”, since
that’s probably where we need to add the systemd support too. This reveals scheduler/main.c
as main source file which implements launchd interaction.

Browsing through this file we notice that two functions are primarily
responsible for interfacing with launchd, the appropriately named
launchd_checkin() and launchd_checkout() functions. The
former acquires the sockets from launchd when the daemon starts up, the latter
terminates communication with launchd and is called when the daemon shuts down.
systemd’s socket activation interfaces are much simpler than those of launchd.
Due to that we only need an equivalent of the launchd_checkin() call,
and do not need a checkout function. Our own function
systemd_checkin() can be implemented very similar to
launchd_checkin(): we look at the sockets we got passed and try to map
them to the ones configured in the CUPS configuration. If we got more sockets
passed than configured in CUPS we implicitly add configuration for them. If the
CUPS configuration includes definitions for more listening sockets those will
be bound natively in CUPS. That way we’ll very robustly listen on all ports
that are listed in either systemd or CUPS configuration.

Our function systemd_checkin() uses sd_listen_fds() from
sd-daemon.c to acquire the file descriptors. Then, we use
sd_is_socket() to map the sockets to the right listening configuration
of CUPS, in a loop for each passed socket. The loop corresponds very closely to
the loop from launchd_checkin() however is a lot simpler. Our patch so far looks like this.

Before we can test our patch, we add sd-daemon.c
and sd-daemon.h
as drop-in files to the package, so that sd_listen_fds() and
sd_is_socket() are available for use. After a few minimal changes to
the Makefile we are almost ready to test our socket activated version
of CUPS. The last missing step is creating two unit files for CUPS, one for the
socket (cups.socket), the
other for the service (cups.service). To make things
simple we just drop them in /etc/systemd/system and make sure systemd
knows about them, with systemctl daemon-reload.

Now we are ready to test our little patch: we start the socket with
systemctl start cups.socket. This will bind the socket, but won’t
start CUPS yet. Next, we simply invoke lpq to test whether CUPS is
transparently started, and yupp, this is exactly what happens. We’ll get the
normal output from lpq as if we had started CUPS at boot already, and
if we then check with systemctl status cups.service we see that CUPS
was automatically spawned by our invocation of lpq. Our test
succeeded, socket activation worked!

Hardware Activation in Detail

The next trigger is hardware activation: we want to make sure that CUPS is
automatically started as soon as a local printer is found, regardless whether
that happens as hotplug during runtime or as coldplug during
boot. Hardware activation in systemd is done via udev rules. Any udev device
that is tagged with the systemd tag can pull in units as needed via
the SYSTEMD_WANTS= environment variable. In the case of CUPS we don’t
even have to add our own udev rule to the mix, we can simply hook into what
systemd already does out-of-the-box with rules shipped upstream. More
specifically, it ships a udev rules file with the following lines:

SUBSYSTEM==”printer”, TAG+=”systemd”, ENV{SYSTEMD_WANTS}=”printer.target”
SUBSYSTEM==”usb”, KERNEL==”lp*”, TAG+=”systemd”, ENV{SYSTEMD_WANTS}=”printer.target”
SUBSYSTEM==”usb”, ENV{DEVTYPE}==”usb_device”, ENV{ID_USB_INTERFACES}==”*:0701??:*”, TAG+=”systemd”, ENV{SYSTEMD_WANTS}=”printer.target”

This pulls in the target unit printer.target as soon as at least
one printer is plugged in (supporting all kinds of printer ports). All we now
have to do is make sure that our CUPS service is pulled in by
printer.target and we are done. By placing WantedBy=printer.target
line in the [Install] section of the service file, a
Wants dependency is created from printer.target to
cups.service as soon as the latter is enabled with systemctl
enable. The indirection via printer.target provides us with a
simple way to use systemctl enable and systemctl disable to
manage hardware activation of a service.

Path-based Activation in Detail

To ensure that CUPS is also started when there is a print job still queued
in the printing spool, we write a simple cups.path that
activates CUPS as soon as we find a file in /var/spool/cups.

Boot-based Activation in Detail

Well, starting services on boot is obviously the most boring and well-known
way to spawn a service. This entire excercise was about making this unnecessary,
but we still need to support it for explicit print server machines. Since those
are probably the exception and not the common case we do not enable this kind
of activation by default, but leave it to the administrator to add it in when
he deems it necessary, with a simple command (ln -s
/lib/systemd/system/cups.service
/etc/systemd/system/multi-user.target.wants/ to be precise).

So, now we have covered all four kinds of activation. To finalize our patch
we have a closer look at the [Install] section of cups.service, i.e.
the part of the unit file that controls how systemctl enable
cups.service and systemctl disable cups.service will hook the
service into/unhook the service from the system. Since we don’t want to start
cups at boot we do not place WantedBy=multi-user.target in it like we
would do for those services. Instead we just place an Also= line that
makes sure that cups.path and cups.socket are
automatically also enabled if the user asks to enable cups.service
(they are enabled according to the [Install] sections in those unit
files).

As last step we then integrate our work into the build system. In contrast
to SysV init scripts systemd unit files are supposed to be distribution
independent. Hence it is a good idea to include them in the upstream tarball.
Unfortunately CUPS doesn’t use Automake, but Autoconf with a set of handwritten
Makefiles. This requires a bit more work to get our additions integrated, but
is not too difficult either. And
this is how our final patch looks like
, after we commited our work and ran
git format-patch -1 on it to generate a pretty git patch.

The next step of course is to get this patch integrated into the upstream
and Fedora packages (or whatever other distribution floats your boat). To make
this easy I have prepared a
patch for Tim that makes the necessary packaging changes for Fedora 16
, and
includes the patch intended for upstream linked above. Of course, ideally the
patch is merged upstream, however in the meantime we can already include it in
the Fedora packages.

Note that CUPS was particularly easy to patch since it already supported
launchd-style activation, patching a service that doesn’t support that yet is
only marginally more difficult. (Oh, and we have no plans to offer the complex
launchd API as compatibility kludge on Linux. It simply doesn’t translate very
well, so don’t even ask… ;-))

And that finishes our little blog story. I hope this quick walkthrough how to add
socket activation (and the other forms of activation) to a package were
interesting to you, and will help you doing the same for your own packages. If you
have questions, our IRC channel #systemd on freenode and
our mailing
list
are available, and we are always happy to help!

systemd for Developers II

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/socket-activation2.html

It has been way too long since I posted the first
episode
of my systemd for Developers series. Here’s finally the
second part. Make sure you read the first episode of the series before
you start with this part since I’ll assume the reader grokked the wonders
of socket activation.

Socket Activation, Part II

This time we’ll focus on adding socket activation support to real-life
software, more specifically the CUPS printing server. Most current Linux
desktops run CUPS by default these days, since printing is so basic that it’s a
must have, and must just work when the user needs it. However, most desktop
CUPS installations probably don’t actually see more than a handful of print
jobs each month. Even if you are a busy office worker you’ll unlikely to
generate more than a couple of print jobs an hour on your PC. Also, printing is
not time critical. Whether a job takes 50ms or 100ms until it reaches the
printer matters little. As long as it is less than a few seconds the user
probably won’t notice. Finally, printers are usually peripheral hardware: they
aren’t built into your laptop, and you don’t always carry them around plugged
in. That all together makes CUPS a perfect candidate for lazy activation:
instead of starting it unconditionally at boot we just start it on-demand, when
it is needed. That way we can save resources, at boot and at runtime. However,
this kind of activation needs to take place transparently, so that the user
doesn’t notice that the print server was not actually running yet when he tried
to access it. To achieve that we need to make sure that the print server is
started as soon at least one of three conditions hold:

  1. A local client tries to talk to the print server, for example because
    a GNOME application opened the printing dialog which tries to enumerate
    available printers.
  2. A printer is being plugged in locally, and it should be configured and
    enabled and then optionally the user be informed about it.
  3. At boot, when there’s still an unprinted print job lurking in the queue.

Of course, the desktop is not the only place where CUPS is used. CUPS can be
run in small and big print servers, too. In that case the amount of print jobs
is substantially higher, and CUPS should be started right away at boot. That
means that (optionally) we still want to start CUPS unconditionally at boot and
not delay its execution until when it is needed.

Putting this all together we need four kind of activation to make CUPS work
well in all situations at minimal resource usage: socket based activation (to
support condition 1 above), hardware based activation (to support condition 2),
path based activation (for condition 3) and finally boot-time activation (for
the optional server usecase). Let’s focus on these kinds of activation in more
detail, and in particular on socket-based activation.

Socket Activation in Detail

To implement socket-based activation in CUPS we need to make sure that when
sockets are passed from systemd these are used to listen on instead of binding
them directly in the CUPS source code. Fortunately this is relatively easy to
do in the CUPS sources, since it already supports launchd-style socket
activation, as it is used on MacOS X (note that CUPS is nowadays an Apple
project). That means the code already has all the necessary hooks to add
systemd-style socket activation with minimal work.

To begin with our patching session we check out the CUPS sources.
Unfortunately CUPS is still stuck in unhappy Subversion country and not using
git yet. In order to simplify our patching work our first step is to use
git-svn to check it out locally in a way we can access it with the
usual git tools:

git svn clone http://svn.easysw.com/public/cups/trunk/ cups

This will take a while. After the command finished we use the wonderful
git grep to look for all occurences of the word “launchd”, since
that’s probably where we need to add the systemd support too. This reveals scheduler/main.c
as main source file which implements launchd interaction.

Browsing through this file we notice that two functions are primarily
responsible for interfacing with launchd, the appropriately named
launchd_checkin() and launchd_checkout() functions. The
former acquires the sockets from launchd when the daemon starts up, the latter
terminates communication with launchd and is called when the daemon shuts down.
systemd’s socket activation interfaces are much simpler than those of launchd.
Due to that we only need an equivalent of the launchd_checkin() call,
and do not need a checkout function. Our own function
systemd_checkin() can be implemented very similar to
launchd_checkin(): we look at the sockets we got passed and try to map
them to the ones configured in the CUPS configuration. If we got more sockets
passed than configured in CUPS we implicitly add configuration for them. If the
CUPS configuration includes definitions for more listening sockets those will
be bound natively in CUPS. That way we’ll very robustly listen on all ports
that are listed in either systemd or CUPS configuration.

Our function systemd_checkin() uses sd_listen_fds() from
sd-daemon.c to acquire the file descriptors. Then, we use
sd_is_socket() to map the sockets to the right listening configuration
of CUPS, in a loop for each passed socket. The loop corresponds very closely to
the loop from launchd_checkin() however is a lot simpler. Our patch so far looks like this.

Before we can test our patch, we add sd-daemon.c
and sd-daemon.h
as drop-in files to the package, so that sd_listen_fds() and
sd_is_socket() are available for use. After a few minimal changes to
the Makefile we are almost ready to test our socket activated version
of CUPS. The last missing step is creating two unit files for CUPS, one for the
socket (cups.socket), the
other for the service (cups.service). To make things
simple we just drop them in /etc/systemd/system and make sure systemd
knows about them, with systemctl daemon-reload.

Now we are ready to test our little patch: we start the socket with
systemctl start cups.socket. This will bind the socket, but won’t
start CUPS yet. Next, we simply invoke lpq to test whether CUPS is
transparently started, and yupp, this is exactly what happens. We’ll get the
normal output from lpq as if we had started CUPS at boot already, and
if we then check with systemctl status cups.service we see that CUPS
was automatically spawned by our invocation of lpq. Our test
succeeded, socket activation worked!

Hardware Activation in Detail

The next trigger is hardware activation: we want to make sure that CUPS is
automatically started as soon as a local printer is found, regardless whether
that happens as hotplug during runtime or as coldplug during
boot. Hardware activation in systemd is done via udev rules. Any udev device
that is tagged with the systemd tag can pull in units as needed via
the SYSTEMD_WANTS= environment variable. In the case of CUPS we don’t
even have to add our own udev rule to the mix, we can simply hook into what
systemd already does out-of-the-box with rules shipped upstream. More
specifically, it ships a udev rules file with the following lines:

SUBSYSTEM=="printer", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"
SUBSYSTEM=="usb", KERNEL=="lp*", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ENV{ID_USB_INTERFACES}=="*:0701??:*", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"

This pulls in the target unit printer.target as soon as at least
one printer is plugged in (supporting all kinds of printer ports). All we now
have to do is make sure that our CUPS service is pulled in by
printer.target and we are done. By placing WantedBy=printer.target
line in the [Install] section of the service file, a
Wants dependency is created from printer.target to
cups.service as soon as the latter is enabled with systemctl
enable
. The indirection via printer.target provides us with a
simple way to use systemctl enable and systemctl disable to
manage hardware activation of a service.

Path-based Activation in Detail

To ensure that CUPS is also started when there is a print job still queued
in the printing spool, we write a simple cups.path that
activates CUPS as soon as we find a file in /var/spool/cups.

Boot-based Activation in Detail

Well, starting services on boot is obviously the most boring and well-known
way to spawn a service. This entire excercise was about making this unnecessary,
but we still need to support it for explicit print server machines. Since those
are probably the exception and not the common case we do not enable this kind
of activation by default, but leave it to the administrator to add it in when
he deems it necessary, with a simple command (ln -s
/lib/systemd/system/cups.service
/etc/systemd/system/multi-user.target.wants/
to be precise).

So, now we have covered all four kinds of activation. To finalize our patch
we have a closer look at the [Install] section of cups.service, i.e.
the part of the unit file that controls how systemctl enable
cups.service
and systemctl disable cups.service will hook the
service into/unhook the service from the system. Since we don’t want to start
cups at boot we do not place WantedBy=multi-user.target in it like we
would do for those services. Instead we just place an Also= line that
makes sure that cups.path and cups.socket are
automatically also enabled if the user asks to enable cups.service
(they are enabled according to the [Install] sections in those unit
files).

As last step we then integrate our work into the build system. In contrast
to SysV init scripts systemd unit files are supposed to be distribution
independent. Hence it is a good idea to include them in the upstream tarball.
Unfortunately CUPS doesn’t use Automake, but Autoconf with a set of handwritten
Makefiles. This requires a bit more work to get our additions integrated, but
is not too difficult either. And
this is how our final patch looks like
, after we commited our work and ran
git format-patch -1 on it to generate a pretty git patch.

The next step of course is to get this patch integrated into the upstream
and Fedora packages (or whatever other distribution floats your boat). To make
this easy I have prepared a
patch for Tim that makes the necessary packaging changes for Fedora 16
, and
includes the patch intended for upstream linked above. Of course, ideally the
patch is merged upstream, however in the meantime we can already include it in
the Fedora packages.

Note that CUPS was particularly easy to patch since it already supported
launchd-style activation, patching a service that doesn’t support that yet is
only marginally more difficult. (Oh, and we have no plans to offer the complex
launchd API as compatibility kludge on Linux. It simply doesn’t translate very
well, so don’t even ask… ;-))

And that finishes our little blog story. I hope this quick walkthrough how to add
socket activation (and the other forms of activation) to a package were
interesting to you, and will help you doing the same for your own packages. If you
have questions, our IRC channel #systemd on freenode and
our mailing
list
are available, and we are always happy to help!

systemd for Developers II

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/socket-activation2.html

It has been way too long since I posted the first
episode
of my systemd for Developers series. Here’s finally the
second part. Make sure you read the first episode of the series before
you start with this part since I’ll assume the reader grokked the wonders
of socket activation.

Socket Activation, Part II

This time we’ll focus on adding socket activation support to real-life
software, more specifically the CUPS printing server. Most current Linux
desktops run CUPS by default these days, since printing is so basic that it’s a
must have, and must just work when the user needs it. However, most desktop
CUPS installations probably don’t actually see more than a handful of print
jobs each month. Even if you are a busy office worker you’ll unlikely to
generate more than a couple of print jobs an hour on your PC. Also, printing is
not time critical. Whether a job takes 50ms or 100ms until it reaches the
printer matters little. As long as it is less than a few seconds the user
probably won’t notice. Finally, printers are usually peripheral hardware: they
aren’t built into your laptop, and you don’t always carry them around plugged
in. That all together makes CUPS a perfect candidate for lazy activation:
instead of starting it unconditionally at boot we just start it on-demand, when
it is needed. That way we can save resources, at boot and at runtime. However,
this kind of activation needs to take place transparently, so that the user
doesn’t notice that the print server was not actually running yet when he tried
to access it. To achieve that we need to make sure that the print server is
started as soon at least one of three conditions hold:

  1. A local client tries to talk to the print server, for example because
    a GNOME application opened the printing dialog which tries to enumerate
    available printers.
  2. A printer is being plugged in locally, and it should be configured and
    enabled and then optionally the user be informed about it.
  3. At boot, when there’s still an unprinted print job lurking in the queue.

Of course, the desktop is not the only place where CUPS is used. CUPS can be
run in small and big print servers, too. In that case the amount of print jobs
is substantially higher, and CUPS should be started right away at boot. That
means that (optionally) we still want to start CUPS unconditionally at boot and
not delay its execution until when it is needed.

Putting this all together we need four kind of activation to make CUPS work
well in all situations at minimal resource usage: socket based activation (to
support condition 1 above), hardware based activation (to support condition 2),
path based activation (for condition 3) and finally boot-time activation (for
the optional server usecase). Let’s focus on these kinds of activation in more
detail, and in particular on socket-based activation.

Socket Activation in Detail

To implement socket-based activation in CUPS we need to make sure that when
sockets are passed from systemd these are used to listen on instead of binding
them directly in the CUPS source code. Fortunately this is relatively easy to
do in the CUPS sources, since it already supports launchd-style socket
activation, as it is used on MacOS X (note that CUPS is nowadays an Apple
project). That means the code already has all the necessary hooks to add
systemd-style socket activation with minimal work.

To begin with our patching session we check out the CUPS sources.
Unfortunately CUPS is still stuck in unhappy Subversion country and not using
git yet. In order to simplify our patching work our first step is to use
git-svn to check it out locally in a way we can access it with the
usual git tools:

git svn clone http://svn.easysw.com/public/cups/trunk/ cups

This will take a while. After the command finished we use the wonderful
git grep to look for all occurences of the word “launchd”, since
that’s probably where we need to add the systemd support too. This reveals scheduler/main.c
as main source file which implements launchd interaction.

Browsing through this file we notice that two functions are primarily
responsible for interfacing with launchd, the appropriately named
launchd_checkin() and launchd_checkout() functions. The
former acquires the sockets from launchd when the daemon starts up, the latter
terminates communication with launchd and is called when the daemon shuts down.
systemd’s socket activation interfaces are much simpler than those of launchd.
Due to that we only need an equivalent of the launchd_checkin() call,
and do not need a checkout function. Our own function
systemd_checkin() can be implemented very similar to
launchd_checkin(): we look at the sockets we got passed and try to map
them to the ones configured in the CUPS configuration. If we got more sockets
passed than configured in CUPS we implicitly add configuration for them. If the
CUPS configuration includes definitions for more listening sockets those will
be bound natively in CUPS. That way we’ll very robustly listen on all ports
that are listed in either systemd or CUPS configuration.

Our function systemd_checkin() uses sd_listen_fds() from
sd-daemon.c to acquire the file descriptors. Then, we use
sd_is_socket() to map the sockets to the right listening configuration
of CUPS, in a loop for each passed socket. The loop corresponds very closely to
the loop from launchd_checkin() however is a lot simpler. Our patch so far looks like this.

Before we can test our patch, we add sd-daemon.c
and sd-daemon.h
as drop-in files to the package, so that sd_listen_fds() and
sd_is_socket() are available for use. After a few minimal changes to
the Makefile we are almost ready to test our socket activated version
of CUPS. The last missing step is creating two unit files for CUPS, one for the
socket (cups.socket), the
other for the service (cups.service). To make things
simple we just drop them in /etc/systemd/system and make sure systemd
knows about them, with systemctl daemon-reload.

Now we are ready to test our little patch: we start the socket with
systemctl start cups.socket. This will bind the socket, but won’t
start CUPS yet. Next, we simply invoke lpq to test whether CUPS is
transparently started, and yupp, this is exactly what happens. We’ll get the
normal output from lpq as if we had started CUPS at boot already, and
if we then check with systemctl status cups.service we see that CUPS
was automatically spawned by our invocation of lpq. Our test
succeeded, socket activation worked!

Hardware Activation in Detail

The next trigger is hardware activation: we want to make sure that CUPS is
automatically started as soon as a local printer is found, regardless whether
that happens as hotplug during runtime or as coldplug during
boot. Hardware activation in systemd is done via udev rules. Any udev device
that is tagged with the systemd tag can pull in units as needed via
the SYSTEMD_WANTS= environment variable. In the case of CUPS we don’t
even have to add our own udev rule to the mix, we can simply hook into what
systemd already does out-of-the-box with rules shipped upstream. More
specifically, it ships a udev rules file with the following lines:

SUBSYSTEM=="printer", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"
SUBSYSTEM=="usb", KERNEL=="lp*", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ENV{ID_USB_INTERFACES}=="*:0701??:*", TAG+="systemd", ENV{SYSTEMD_WANTS}="printer.target"

This pulls in the target unit printer.target as soon as at least
one printer is plugged in (supporting all kinds of printer ports). All we now
have to do is make sure that our CUPS service is pulled in by
printer.target and we are done. By placing WantedBy=printer.target
line in the [Install] section of the service file, a
Wants dependency is created from printer.target to
cups.service as soon as the latter is enabled with systemctl
enable
. The indirection via printer.target provides us with a
simple way to use systemctl enable and systemctl disable to
manage hardware activation of a service.

Path-based Activation in Detail

To ensure that CUPS is also started when there is a print job still queued
in the printing spool, we write a simple cups.path that
activates CUPS as soon as we find a file in /var/spool/cups.

Boot-based Activation in Detail

Well, starting services on boot is obviously the most boring and well-known
way to spawn a service. This entire excercise was about making this unnecessary,
but we still need to support it for explicit print server machines. Since those
are probably the exception and not the common case we do not enable this kind
of activation by default, but leave it to the administrator to add it in when
he deems it necessary, with a simple command (ln -s
/lib/systemd/system/cups.service
/etc/systemd/system/multi-user.target.wants/
to be precise).

So, now we have covered all four kinds of activation. To finalize our patch
we have a closer look at the [Install] section of cups.service, i.e.
the part of the unit file that controls how systemctl enable
cups.service
and systemctl disable cups.service will hook the
service into/unhook the service from the system. Since we don’t want to start
cups at boot we do not place WantedBy=multi-user.target in it like we
would do for those services. Instead we just place an Also= line that
makes sure that cups.path and cups.socket are
automatically also enabled if the user asks to enable cups.service
(they are enabled according to the [Install] sections in those unit
files).

As last step we then integrate our work into the build system. In contrast
to SysV init scripts systemd unit files are supposed to be distribution
independent. Hence it is a good idea to include them in the upstream tarball.
Unfortunately CUPS doesn’t use Automake, but Autoconf with a set of handwritten
Makefiles. This requires a bit more work to get our additions integrated, but
is not too difficult either. And
this is how our final patch looks like
, after we commited our work and ran
git format-patch -1 on it to generate a pretty git patch.

The next step of course is to get this patch integrated into the upstream
and Fedora packages (or whatever other distribution floats your boat). To make
this easy I have prepared a
patch for Tim that makes the necessary packaging changes for Fedora 16
, and
includes the patch intended for upstream linked above. Of course, ideally the
patch is merged upstream, however in the meantime we can already include it in
the Fedora packages.

Note that CUPS was particularly easy to patch since it already supported
launchd-style activation, patching a service that doesn’t support that yet is
only marginally more difficult. (Oh, and we have no plans to offer the complex
launchd API as compatibility kludge on Linux. It simply doesn’t translate very
well, so don’t even ask… ;-))

And that finishes our little blog story. I hope this quick walkthrough how to add
socket activation (and the other forms of activation) to a package were
interesting to you, and will help you doing the same for your own packages. If you
have questions, our IRC channel #systemd on freenode and
our mailing
list
are available, and we are always happy to help!

Remember the Verbosity (A Brief Note)

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2007/04/17/linux-verbose-build.html

I don’t remember when it happened, but sometime in the past four years,
the Makefiles for the kernel named Linux changed. I can’t remember
exactly, but I do recall sometime “recently” that the
kernel build output stopped looking like what I remember from 1991,
and started looking like this:

CC arch/i386/kernel/semaphore.o
CC arch/i386/kernel/signal.o

This is a heck of a lot easier to read, but there was something cool
about having make display the whole gcc
command lines, like this:

gcc -m32 -Wp,-MD,arch/i386/kernel/.semaphore.o.d -nostdinc -isystem /usr/lib/gcc/i486-linux-gnu/4.0.3/include -D__KERNEL__ -Iinclude -include include/linux/autoconf.h -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -Os -fomit-frame-pointer -pipe -msoft-float -mpreferred-stack-boundary=2 -march=i686 -mtune=pentium4 -Iinclude/asm-i386/mach-default -Wdeclaration-after-statement -Wno-pointer-sign -D”KBUILD_STR(s)=#s” -D”KBUILD_BASENAME=KBUILD_STR(semaphore)” -D”KBUILD_MODNAME=KBUILD_STR(semaphore)” -c -o arch/i386/kernel/semaphore.o arch/i386/kernel/semaphore.c
gcc -m32 -Wp,-MD,arch/i386/kernel/.signal.o.d -nostdinc -isystem /usr/lib/gcc/i486-linux-gnu/4.0.3/include -D__KERNEL__ -Iinclude -include include/linux/autoconf.h -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -Os -fomit-frame-pointer -pipe -msoft-float -mpreferred-stack-boundary=2 -march=i686 -mtune=pentium4 -Iinclude/asm-i386/mach-default -Wdeclaration-after-statement -Wno-pointer-sign -D”KBUILD_STR(s)=#s” -D”KBUILD_BASENAME=KBUILD_STR(signal)” -D”KBUILD_MODNAME=KBUILD_STR(signal)” -c -o arch/i386/kernel/signal.o arch/i386/kernel/signal.c

I never gave it much thought, since the new form was easier to read. I
figured that those folks who still eat kernel code for breakfast knew
about this change well ahead of time. Of course, they were the only
ones who needed to see the verbose output of the gcc
command lines. I could live with seeing the simpler CC
lines for my purposes, until today.

I was compiling kernel code and for the first time since this change in
the Makefiles, I was using a non-default gcc to build
Linux. I wanted to double-check that I’d given the right options to
make throughout the process. I therefore found myself
looking for a way to see the full output again (and for the first
time). It was easy enough to figure out: giving the variable setting
V=1 to make gives you the verbose version.
For you Debian folks like me, we’re using make-kpkg, so
the line we need looks like: MAKEFLAGS=”V=1″ make-kpkg
kernel_image.

It’s nice sometimes to pretend I’m compiling 0.99pl12 again and not
2.6.20.7. 🙂 No matter which options you give make, it is
still a whole lot easier to bootstrap Linux these days.