Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/systemd-for-admins-2.html

Here’s the second installment of my ongoing series about systemd for administrators.

Which Service Owns Which Processes?

On most Linux systems the number of processes that are running by
default is substantial. Knowing which process does what and where it
belongs to becomes increasingly difficult. Some services even maintain
a couple of worker processes which clutter the “ps” output with
many additional processes that are often not easy to recognize. This is
further complicated if daemons spawn arbitrary 3rd-party processes, as
Apache does with CGI processes, or cron does with user jobs.

A slight remedy for this is often the process inheritance tree, as
shown by “ps xaf“. However this is usually not reliable, as processes
whose parents die get reparented to PID 1, and hence all information
about inheritance gets lost. If a process “double forks” it hence loses
its relationships to the processes that started it. (This actually is
supposed to be a feature and is relied on for the traditional Unix
daemonizing logic.) Furthermore processes can freely change their names
with PR_SETNAME or by patching argv[0], thus making
it harder to recognize them. In fact they can play hide-and-seek with
the administrator pretty nicely this way.

In systemd we place every process that is spawned in a control
group named after its service. Control groups (or cgroups)
at their most basic are simply groups of processes that can be
arranged in a hierarchy and labelled individually. When processes
spawn other processes these children are automatically made members of
the parents cgroup. Leaving a cgroup is not possible for unprivileged
processes. Thus, cgroups can be used as an effective way to label
processes after the service they belong to and be sure that the
service cannot escape from the label, regardless how often it forks or
renames itself. Furthermore this can be used to safely kill a service
and all processes it created, again with no chance of escaping.

In today’s installment I want to introduce you to two commands you
may use to relate systemd services and processes. The first one, is
the well known ps command which has been updated to show
cgroup information along the other process details. And this is how it
looks:

$ ps xawf -eo pid,user,cgroup,args
  PID USER     CGROUP                              COMMAND
    2 root     -                                   [kthreadd]
    3 root     -                                    \_ [ksoftirqd/0]
[...]
 4281 root     -                                    \_ [flush-8:0]
    1 root     name=systemd:/systemd-1             /sbin/init
  455 root     name=systemd:/systemd-1/sysinit.service /sbin/udevd -d
28188 root     name=systemd:/systemd-1/sysinit.service  \_ /sbin/udevd -d
28191 root     name=systemd:/systemd-1/sysinit.service  \_ /sbin/udevd -d
 1096 dbus     name=systemd:/systemd-1/dbus.service /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
 1131 root     name=systemd:/systemd-1/auditd.service auditd
 1133 root     name=systemd:/systemd-1/auditd.service  \_ /sbin/audispd
 1135 root     name=systemd:/systemd-1/auditd.service      \_ /usr/sbin/sedispatch
 1171 root     name=systemd:/systemd-1/NetworkManager.service /usr/sbin/NetworkManager --no-daemon
 4028 root     name=systemd:/systemd-1/NetworkManager.service  \_ /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-wlan0.pid -lf /var/lib/dhclient/dhclient-7d32a784-ede9-4cf6-9ee3-60edc0bce5ff-wlan0.lease -
 1175 avahi    name=systemd:/systemd-1/avahi-daemon.service avahi-daemon: running [epsilon.local]
 1194 avahi    name=systemd:/systemd-1/avahi-daemon.service  \_ avahi-daemon: chroot helper
 1193 root     name=systemd:/systemd-1/rsyslog.service /sbin/rsyslogd -c 4
 1195 root     name=systemd:/systemd-1/cups.service cupsd -C /etc/cups/cupsd.conf
 1207 root     name=systemd:/systemd-1/mdmonitor.service mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
 1210 root     name=systemd:/systemd-1/irqbalance.service irqbalance
 1216 root     name=systemd:/systemd-1/dbus.service /usr/sbin/modem-manager
 1219 root     name=systemd:/systemd-1/dbus.service /usr/libexec/polkit-1/polkitd
 1242 root     name=systemd:/systemd-1/dbus.service /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
 1249 68       name=systemd:/systemd-1/haldaemon.service hald
 1250 root     name=systemd:/systemd-1/haldaemon.service  \_ hald-runner
 1273 root     name=systemd:/systemd-1/haldaemon.service      \_ hald-addon-input: Listening on /dev/input/event3 /dev/input/event9 /dev/input/event1 /dev/input/event7 /dev/input/event2 /dev/input/event0 /dev/input/event8
 1275 root     name=systemd:/systemd-1/haldaemon.service      \_ /usr/libexec/hald-addon-rfkill-killswitch
 1284 root     name=systemd:/systemd-1/haldaemon.service      \_ /usr/libexec/hald-addon-leds
 1285 root     name=systemd:/systemd-1/haldaemon.service      \_ /usr/libexec/hald-addon-generic-backlight
 1287 68       name=systemd:/systemd-1/haldaemon.service      \_ /usr/libexec/hald-addon-acpi
 1317 root     name=systemd:/systemd-1/abrtd.service /usr/sbin/abrtd -d -s
 1332 root     name=systemd:/systemd-1/[email protected]/tty2 /sbin/mingetty tty2
 1339 root     name=systemd:/systemd-1/[email protected]/tty3 /sbin/mingetty tty3
 1342 root     name=systemd:/systemd-1/[email protected]/tty5 /sbin/mingetty tty5
 1343 root     name=systemd:/systemd-1/[email protected]/tty4 /sbin/mingetty tty4
 1344 root     name=systemd:/systemd-1/crond.service crond
 1346 root     name=systemd:/systemd-1/[email protected]/tty6 /sbin/mingetty tty6
 1362 root     name=systemd:/systemd-1/sshd.service /usr/sbin/sshd
 1376 root     name=systemd:/systemd-1/prefdm.service /usr/sbin/gdm-binary -nodaemon
 1391 root     name=systemd:/systemd-1/prefdm.service  \_ /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1 --force-active-vt
 1394 root     name=systemd:/systemd-1/prefdm.service      \_ /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-f2KUOh/database -nolisten tcp vt1
 1495 root     name=systemd:/user/lennart/1             \_ pam: gdm-password
 1521 lennart  name=systemd:/user/lennart/1                 \_ gnome-session
 1621 lennart  name=systemd:/user/lennart/1                     \_ metacity
 1635 lennart  name=systemd:/user/lennart/1                     \_ gnome-panel
 1638 lennart  name=systemd:/user/lennart/1                     \_ nautilus
 1640 lennart  name=systemd:/user/lennart/1                     \_ /usr/libexec/polkit-gnome-authentication-agent-1
 1641 lennart  name=systemd:/user/lennart/1                     \_ /usr/bin/seapplet
 1644 lennart  name=systemd:/user/lennart/1                     \_ gnome-volume-control-applet
 1646 lennart  name=systemd:/user/lennart/1                     \_ /usr/sbin/restorecond -u
 1652 lennart  name=systemd:/user/lennart/1                     \_ /usr/bin/devilspie
 1662 lennart  name=systemd:/user/lennart/1                     \_ nm-applet --sm-disable
 1664 lennart  name=systemd:/user/lennart/1                     \_ gnome-power-manager
 1665 lennart  name=systemd:/user/lennart/1                     \_ /usr/libexec/gdu-notification-daemon
 1670 lennart  name=systemd:/user/lennart/1                     \_ /usr/libexec/evolution/2.32/evolution-alarm-notify
 1672 lennart  name=systemd:/user/lennart/1                     \_ /usr/bin/python /usr/share/system-config-printer/applet.py
 1674 lennart  name=systemd:/user/lennart/1                     \_ /usr/lib64/deja-dup/deja-dup-monitor
 1675 lennart  name=systemd:/user/lennart/1                     \_ abrt-applet
 1677 lennart  name=systemd:/user/lennart/1                     \_ bluetooth-applet
 1678 lennart  name=systemd:/user/lennart/1                     \_ gpk-update-icon
 1408 root     name=systemd:/systemd-1/console-kit-daemon.service /usr/sbin/console-kit-daemon --no-daemon
 1419 gdm      name=systemd:/systemd-1/prefdm.service /usr/bin/dbus-launch --exit-with-session
 1453 root     name=systemd:/systemd-1/dbus.service /usr/libexec/upowerd
 1473 rtkit    name=systemd:/systemd-1/rtkit-daemon.service /usr/libexec/rtkit-daemon
 1496 root     name=systemd:/systemd-1/accounts-daemon.service /usr/libexec/accounts-daemon
 1499 root     name=systemd:/systemd-1/systemd-logger.service /lib/systemd/systemd-logger
 1511 lennart  name=systemd:/systemd-1/prefdm.service /usr/bin/gnome-keyring-daemon --daemonize --login
 1534 lennart  name=systemd:/user/lennart/1        dbus-launch --sh-syntax --exit-with-session
 1535 lennart  name=systemd:/user/lennart/1        /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 1603 lennart  name=systemd:/user/lennart/1        /usr/libexec/gconfd-2
 1612 lennart  name=systemd:/user/lennart/1        /usr/libexec/gnome-settings-daemon
 1615 lennart  name=systemd:/user/lennart/1        /usr/libexec/gvfsd
 1626 lennart  name=systemd:/user/lennart/1        /usr/libexec//gvfs-fuse-daemon /home/lennart/.gvfs
 1634 lennart  name=systemd:/user/lennart/1        /usr/bin/pulseaudio --start --log-target=syslog
 1649 lennart  name=systemd:/user/lennart/1         \_ /usr/libexec/pulse/gconf-helper
 1645 lennart  name=systemd:/user/lennart/1        /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=24
 1668 lennart  name=systemd:/user/lennart/1        /usr/libexec/im-settings-daemon
 1701 lennart  name=systemd:/user/lennart/1        /usr/libexec/gvfs-gdu-volume-monitor
 1707 lennart  name=systemd:/user/lennart/1        /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
 1725 lennart  name=systemd:/user/lennart/1        /usr/libexec/clock-applet
 1727 lennart  name=systemd:/user/lennart/1        /usr/libexec/wnck-applet
 1729 lennart  name=systemd:/user/lennart/1        /usr/libexec/notification-area-applet
 1733 root     name=systemd:/systemd-1/dbus.service /usr/libexec/udisks-daemon
 1747 root     name=systemd:/systemd-1/dbus.service  \_ udisks-daemon: polling /dev/sr0
 1759 lennart  name=systemd:/user/lennart/1        gnome-screensaver
 1780 lennart  name=systemd:/user/lennart/1        /usr/libexec/gvfsd-trash --spawner :1.9 /org/gtk/gvfs/exec_spaw/0
 1864 lennart  name=systemd:/user/lennart/1        /usr/libexec/gvfs-afc-volume-monitor
 1874 lennart  name=systemd:/user/lennart/1        /usr/libexec/gconf-im-settings-daemon
 1903 lennart  name=systemd:/user/lennart/1        /usr/libexec/gvfsd-burn --spawner :1.9 /org/gtk/gvfs/exec_spaw/1
 1909 lennart  name=systemd:/user/lennart/1        gnome-terminal
 1913 lennart  name=systemd:/user/lennart/1         \_ gnome-pty-helper
 1914 lennart  name=systemd:/user/lennart/1         \_ bash
29231 lennart  name=systemd:/user/lennart/1         |   \_ ssh tango
 2221 lennart  name=systemd:/user/lennart/1         \_ bash
 4193 lennart  name=systemd:/user/lennart/1         |   \_ ssh tango
 2461 lennart  name=systemd:/user/lennart/1         \_ bash
29219 lennart  name=systemd:/user/lennart/1         |   \_ emacs systemd-for-admins-1.txt
15113 lennart  name=systemd:/user/lennart/1         \_ bash
27251 lennart  name=systemd:/user/lennart/1             \_ empathy
29504 lennart  name=systemd:/user/lennart/1             \_ ps xawf -eo pid,user,cgroup,args
 1968 lennart  name=systemd:/user/lennart/1        ssh-agent
 1994 lennart  name=systemd:/user/lennart/1        gpg-agent --daemon --write-env-file
18679 lennart  name=systemd:/user/lennart/1        /bin/sh /usr/lib64/firefox-3.6/run-mozilla.sh /usr/lib64/firefox-3.6/firefox
18741 lennart  name=systemd:/user/lennart/1         \_ /usr/lib64/firefox-3.6/firefox
28900 lennart  name=systemd:/user/lennart/1             \_ /usr/lib64/nspluginwrapper/npviewer.bin --plugin /usr/lib64/mozilla/plugins/libflashplayer.so --connection /org/wrapper/NSPlugins/libflashplayer.so/18741-6
 4016 root     name=systemd:/systemd-1/sysinit.service /usr/sbin/bluetoothd --udev
 4094 smmsp    name=systemd:/systemd-1/sendmail.service sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
 4096 root     name=systemd:/systemd-1/sendmail.service sendmail: accepting connections
 4112 ntp      name=systemd:/systemd-1/ntpd.service /usr/sbin/ntpd -n -u ntp:ntp -g
27262 lennart  name=systemd:/user/lennart/1        /usr/libexec/mission-control-5
27265 lennart  name=systemd:/user/lennart/1        /usr/libexec/telepathy-haze
27268 lennart  name=systemd:/user/lennart/1        /usr/libexec/telepathy-logger
27270 lennart  name=systemd:/user/lennart/1        /usr/libexec/dconf-service
27280 lennart  name=systemd:/user/lennart/1        /usr/libexec/notification-daemon
27284 lennart  name=systemd:/user/lennart/1        /usr/libexec/telepathy-gabble
27285 lennart  name=systemd:/user/lennart/1        /usr/libexec/telepathy-salut
27297 lennart  name=systemd:/user/lennart/1        /usr/libexec/geoclue-yahoo

(Note that this output is shortened, I have removed most of the
kernel threads here, since they are not relevant in the context of
this blog story)

In the third column you see the cgroup systemd assigned to each
process. You’ll find that the udev processes are in the
name=systemd:/systemd-1/sysinit.service cgroup, which is
where systemd places all processes started by the
sysinit.service service, which covers early boot.

My personal recommendation is to set the shell alias psc
to the ps command line shown above:

alias psc='ps xawf -eo pid,user,cgroup,args'

With this service information of processes is just four keypresses
away!

A different way to present the same information is the
systemd-cgls tool we ship with systemd. It shows the cgroup
hierarchy in a pretty tree. Its output looks like this:

$ systemd-cgls
+    2 [kthreadd]
[...]
+ 4281 [flush-8:0]
+ user
| \ lennart
|   \ 1
|     +  1495 pam: gdm-password
|     +  1521 gnome-session
|     +  1534 dbus-launch --sh-syntax --exit-with-session
|     +  1535 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
|     +  1603 /usr/libexec/gconfd-2
|     +  1612 /usr/libexec/gnome-settings-daemon
|     +  1615 /ushr/libexec/gvfsd
|     +  1621 metacity
|     +  1626 /usr/libexec//gvfs-fuse-daemon /home/lennart/.gvfs
|     +  1634 /usr/bin/pulseaudio --start --log-target=syslog
|     +  1635 gnome-panel
|     +  1638 nautilus
|     +  1640 /usr/libexec/polkit-gnome-authentication-agent-1
|     +  1641 /usr/bin/seapplet
|     +  1644 gnome-volume-control-applet
|     +  1645 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=24
|     +  1646 /usr/sbin/restorecond -u
|     +  1649 /usr/libexec/pulse/gconf-helper
|     +  1652 /usr/bin/devilspie
|     +  1662 nm-applet --sm-disable
|     +  1664 gnome-power-manager
|     +  1665 /usr/libexec/gdu-notification-daemon
|     +  1668 /usr/libexec/im-settings-daemon
|     +  1670 /usr/libexec/evolution/2.32/evolution-alarm-notify
|     +  1672 /usr/bin/python /usr/share/system-config-printer/applet.py
|     +  1674 /usr/lib64/deja-dup/deja-dup-monitor
|     +  1675 abrt-applet
|     +  1677 bluetooth-applet
|     +  1678 gpk-update-icon
|     +  1701 /usr/libexec/gvfs-gdu-volume-monitor
|     +  1707 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=22
|     +  1725 /usr/libexec/clock-applet
|     +  1727 /usr/libexec/wnck-applet
|     +  1729 /usr/libexec/notification-area-applet
|     +  1759 gnome-screensaver
|     +  1780 /usr/libexec/gvfsd-trash --spawner :1.9 /org/gtk/gvfs/exec_spaw/0
|     +  1864 /usr/libexec/gvfs-afc-volume-monitor
|     +  1874 /usr/libexec/gconf-im-settings-daemon
|     +  1882 /usr/libexec/gvfs-gphoto2-volume-monitor
|     +  1903 /usr/libexec/gvfsd-burn --spawner :1.9 /org/gtk/gvfs/exec_spaw/1
|     +  1909 gnome-terminal
|     +  1913 gnome-pty-helper
|     +  1914 bash
|     +  1968 ssh-agent
|     +  1994 gpg-agent --daemon --write-env-file
|     +  2221 bash
|     +  2461 bash
|     +  4193 ssh tango
|     + 15113 bash
|     + 18679 /bin/sh /usr/lib64/firefox-3.6/run-mozilla.sh /usr/lib64/firefox-3.6/firefox
|     + 18741 /usr/lib64/firefox-3.6/firefox
|     + 27251 empathy
|     + 27262 /usr/libexec/mission-control-5
|     + 27265 /usr/libexec/telepathy-haze
|     + 27268 /usr/libexec/telepathy-logger
|     + 27270 /usr/libexec/dconf-service
|     + 27280 /usr/libexec/notification-daemon
|     + 27284 /usr/libexec/telepathy-gabble
|     + 27285 /usr/libexec/telepathy-salut
|     + 27297 /usr/libexec/geoclue-yahoo
|     + 28900 /usr/lib64/nspluginwrapper/npviewer.bin --plugin /usr/lib64/mozilla/plugins/libflashplayer.so --connection /org/wrapper/NSPlugins/libflashplayer.so/18741-6
|     + 29219 emacs systemd-for-admins-1.txt
|     + 29231 ssh tango
|     \ 29519 systemd-cgls
\ systemd-1
  + 1 /sbin/init
  + ntpd.service
  | \ 4112 /usr/sbin/ntpd -n -u ntp:ntp -g
  + systemd-logger.service
  | \ 1499 /lib/systemd/systemd-logger
  + accounts-daemon.service
  | \ 1496 /usr/libexec/accounts-daemon
  + rtkit-daemon.service
  | \ 1473 /usr/libexec/rtkit-daemon
  + console-kit-daemon.service
  | \ 1408 /usr/sbin/console-kit-daemon --no-daemon
  + prefdm.service
  | + 1376 /usr/sbin/gdm-binary -nodaemon
  | + 1391 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1 --force-active-vt
  | + 1394 /usr/bin/Xorg :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-f2KUOh/database -nolisten tcp vt1
  | + 1419 /usr/bin/dbus-launch --exit-with-session
  | \ 1511 /usr/bin/gnome-keyring-daemon --daemonize --login
  + [email protected]
  | + tty6
  | | \ 1346 /sbin/mingetty tty6
  | + tty4
  | | \ 1343 /sbin/mingetty tty4
  | + tty5
  | | \ 1342 /sbin/mingetty tty5
  | + tty3
  | | \ 1339 /sbin/mingetty tty3
  | \ tty2
  |   \ 1332 /sbin/mingetty tty2
  + abrtd.service
  | \ 1317 /usr/sbin/abrtd -d -s
  + crond.service
  | \ 1344 crond
  + sshd.service
  | \ 1362 /usr/sbin/sshd
  + sendmail.service
  | + 4094 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
  | \ 4096 sendmail: accepting connections
  + haldaemon.service
  | + 1249 hald
  | + 1250 hald-runner
  | + 1273 hald-addon-input: Listening on /dev/input/event3 /dev/input/event9 /dev/input/event1 /dev/input/event7 /dev/input/event2 /dev/input/event0 /dev/input/event8
  | + 1275 /usr/libexec/hald-addon-rfkill-killswitch
  | + 1284 /usr/libexec/hald-addon-leds
  | + 1285 /usr/libexec/hald-addon-generic-backlight
  | \ 1287 /usr/libexec/hald-addon-acpi
  + irqbalance.service
  | \ 1210 irqbalance
  + avahi-daemon.service
  | + 1175 avahi-daemon: running [epsilon.local]
  + NetworkManager.service
  | + 1171 /usr/sbin/NetworkManager --no-daemon
  | \ 4028 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-wlan0.pid -lf /var/lib/dhclient/dhclient-7d32a784-ede9-4cf6-9ee3-60edc0bce5ff-wlan0.lease -cf /var/run/nm-dhclient-wlan0.conf wlan0
  + rsyslog.service
  | \ 1193 /sbin/rsyslogd -c 4
  + mdmonitor.service
  | \ 1207 mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
  + cups.service
  | \ 1195 cupsd -C /etc/cups/cupsd.conf
  + auditd.service
  | + 1131 auditd
  | + 1133 /sbin/audispd
  | \ 1135 /usr/sbin/sedispatch
  + dbus.service
  | +  1096 /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
  | +  1216 /usr/sbin/modem-manager
  | +  1219 /usr/libexec/polkit-1/polkitd
  | +  1242 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid
  | +  1453 /usr/libexec/upowerd
  | +  1733 /usr/libexec/udisks-daemon
  | +  1747 udisks-daemon: polling /dev/sr0
  | \ 29509 /usr/libexec/packagekitd
  + dev-mqueue.mount
  + dev-hugepages.mount
  \ sysinit.service
    +   455 /sbin/udevd -d
    +  4016 /usr/sbin/bluetoothd --udev
    + 28188 /sbin/udevd -d
    \ 28191 /sbin/udevd -d

(This too is shortened, the same way)

As you can see, this command shows the processes by their cgroup
and hence service, as systemd labels the cgroups after the
services. For example, you can easily see that the auditing service
auditd.service spawns three individual processes,
auditd, audisp and sedispatch.

If you look closely you will notice that a number of processes have
been assigned to the cgroup /user/1. At this point let’s
simply leave it at that systemd not only maintains services in cgroups,
but user session processes as well. In a later installment we’ll discuss in
more detail what this about.

So much for now, come back soon for the next installment!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/google-video.html

Google just published a video interview with yours truly. Watch it! Oh, and Vincent, I even put on a red shirt for you!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/systemd-update.html

It has been a while since my original
announcement of systemd. Here’s a little status update, on what
happened since then. For simplicity’s sake I’ll just list here what we
worked on in a bulleted list, with no particular order and without
trying to cover this comprehensively:

• systemd has been accepted as Feature for Fedora 14, and as it
  looks right now everything worked out nicely and we’ll ship F14 with
  systemd as init system.
• We added a number of additional unit types: .timer for
  cron-style timer-based activation of services, .swap exposes
  swap files and partitions the same way we handle mount points, and
  .path can be used to activate units dependending on the
  existance/creation of files or fill status of spool directories.
• We hooked systemd up to SELinux: systemd is now capabale of
  properly labelling directories, sockets and FIFOs it creates according
  to the SELinux policy for the services we maintain.
• We hooked systemd up to the Linux auditing subsystem: as first
  init system at all systemd now generates auditing records for all
  services it starts/stops, including their failure status.
• We hooked systemd up to TCP wrappers, for all socket connections
  it accepts.
• We hooked systemd up to PAM, so that optionally, when systemd runs
  a service as a different user it initializes the usual PAM session
  setup and teardown hooks.
• We hooked systemd up to D-Bus, so that D-Bus passes activation
  requests to systemd and systemd becomes the central point for all
  kinds of activation, thus greatly extending the control of the
  execution environment of bus activated services, and making them
  accessible through the same utilities as SysV services. Also, this
  enables us to do race-free parallelized start-up for D-Bus services
  and their clients, thus speeding up things even further.
• systemd is now able to handle various Debian and OpenSUSE-specific
  extensions to the classic SysV init script formats natively, on top of
  the Fedora extensions we already parse.
• The D-Bus coverage of the systemd interface is now complete,
  allowing both introspection of runtime data and of parsed
  configuration data. It’s fun now to introspect systemd with gdbus
  or d-feet.
• We added a systemd
  PAM module, which assigns the processes of each user session to
  its own cgroup in the systemd cgroup tree. This also enables reliable
  killing of all processes associated with a session when the user logs
  out. This also manages a secure per-user /var/run-style directory
  which is supposed to be used for sockets and similar files that shall
  be cleaned up when the user logs out.
• There’s a new tool systemd-cgls,
  which plots a pretty process tree based on the systemd cgroup
  hierarchy. It’s really pretty. Try it!
• We now have our own cgroup hierarchy beneath
  /cgroup/systemd (though is will move to /sys/fs/
  before the F14 release).
• We have pretty code that automatically spawns a getty on a serial
  port when the kernel console is redirected to a serial TTY.
• systemctl got beefed up substantially (it can even draw
  dependency graphs now, via dot!), and the SysV compatiblity
  tools were extended to more completely and correctly support what was
  historically provided by SysV. For example, we’ll now warn the user
  when systemd service files have changed but systemd was not asked to
  reload its configuration. Also, you can now use systemd’s native
  client tools to reboot or shut-down an Upstart or sysvinit system, to
  facilitate upgrades.
• We provide a reference
  implementation for the socket activation and other APIs for nicer
  interaction with systemd.
• We have a pretty complete set of documentation
  now, some
  of it even extending to areas not directly related to systemd
  itself.
• Quite a number of upstream packages now ship with systemd service
  files out-of-the-box now, that work across all distributions that have
  adopted systemd. It is our intention to unify the boot and service
  management between distributions with systemd, and this shows fruits
  already. Furthermore a number of upstream packages now ship our
  patches for socket-based activation.
• Even more options that control the process execution environment
  or the sockets we create are now supported.
• Earlier today I began my series of blog stories on systemd
  for administrators.
• We reimplemented almost all boot-up and shutdown scripts of the
  standard Fedora install in much smaller, simpler and faster C
  utilities, or in systemd itself. Most of this will not be enabled in
  F14 however, even though it is shipped with systemd upstream. With
  this enabled the entire Linux system gains a completely new feeling as
  the number of shells we spawn approaches zero, and the PID of the
  first user terminal is way < 500 now, and the early boot-up is
  fully parallelized. We looked at the boot scripts of Fedora, OpenSUSE
  and Debian and distilled from this a list of functionality that makes
  up the early boot process and reimplemented this in C, if possible
  following the bahaviour of one of the existing implementations from
  these three distributions. This turned out to be much less effort than
  anticipated, and we are actually quite excited about this. Look
  forward to the fruits of this work in F15, when we might be able to
  present you a shell-less boot at least for standard desktop/laptop
  systems.
• We spent some time reinvestigating the current syslog logic, and
  came up with an elegant and simple scheme to provide /dev/log
  compatible logging right from the time systemd is first initialized
  right until the time the kernel halts the machine. Through the wonders
  of socket based activation we first connect the /dev/log
  socket with a minimal bridge to the kernel log buffer (kmsg)
  and then, as soon as the real syslog is started up as part of the
  later bootup phase, we dynamically replace this minimal bridge by the
  real syslog daemon — without losing a single log message. Since one
  of the first things the real syslog daemon does is flushing the kernel
  log buffer into log files, all logged messages will sooner or later be
  stored on disk, regardless whether they have been generated during
  early boot, late boot or system runtime. On top of that if the syslog
  daemon terminates or is shut down during runtime, the bridge becomes
  active again and log output is written to kmsg again. The same applies
  when the system goes down. This provides a simple an robust way how we
  can ensure that no logs will ever be lost again, and logging is
  available from the beginning of boot-up to the end of
  shut-down. Plymouth will most likely adopt a similar scheme for initrd
  logging, thus ensuring that everything ever logged on the system will
  properly end up in the log files, whether it comes from the kernel,
  from the initrd, from early-boot, from runtime or shutdown. And if
  syslogd is not around, dmesg will provide you with access to
  the log messages. While this bridge is part of systemd upstream, we’ll
  most likely enable this bridge in Fedora only starting with F15. Also
  note that embedded systems that have no interest in shipping a full
  syslogd solution can simply use this syslog bridge during the entire
  runtime, and thus making the kernel log buffer the centralized log
  storage, with all the advantages this offers: zero disk IO at runtime,
  access to serial and netconsole logging, and remote debug access to
  the kernel log buffer.
• We now install autofs units for many “API” kernel virtual file
  systems by default, such as binfmt_misc or
  hugetlbfs. That means that the file system access is readily
  available, client code no longer has to manually load the respective
  kernel modules, as they are autoloaded on first access of the file
  system. This has many advantages: it is not only faster to set up
  during boot, but also simpler for applications, as they can just
  assume the functionality is available. On top of that permission
  problems for the initialization go away, since manual module loading
  requires root privileges.
• Many smaller fixes and enhancements, all across the board, which
  if mentioned here would make this blog story another blog
  novel. Suffice to say, we did a lot of polishing to ready systemd for
  F14.

All in all, systemd is progressing nicely, and the features we have
been working on in the last months are without exception features not
existing in any other of the init systems available on Linux and our
feature set already was far ahead of what the older init
implementations provide. And we have quite a bit planned for the
future. So, stay tuned!

Also note that I’ll speak about systemd at LinuxKongress
2010 in Nuremberg, Germany. Later this year I’ll also be speaking
at the Linux
Plumbers Conference in Boston, MA. Make sure to drop by if you
want to learn about systemd or discuss exiciting new ideas or features
with us.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/systemd-for-admins-1.html

As many of you know, systemd is the new
Fedora init system, starting with F14, and it is also on its way to being adopted in
a number of other distributions as well (for example, OpenSUSE). For administrators
systemd provides a variety of new features and changes and enhances the
administrative process substantially. This blog story is the first part of a
series of articles I plan to post roughly every week for the next months. In
every post I will try to explain one new feature of systemd. Many of these features
are small and simple, so these stories should be interesting to a broader audience.
However, from time to time we’ll dive a little bit deeper into the great new
features systemd provides you with.

Verifying Bootup

Traditionally, when booting up a Linux system, you see a lot of
little messages passing by on your screen. As we work on speeding up
and parallelizing the boot process these messages are becoming visible
for a shorter and shorter time only and be less and less readable —
if they are shown at all, given we use graphical boot splash
technology like Plymouth these days. Nonetheless the information of
the boot screens was and still is very relevant, because it shows you
for each service that is being started as part of bootup, wether it
managed to start up successfully or failed (with those green or red
[ OK ] or [ FAILED ] indicators). To improve the
situation for machines that boot up fast and parallelized and to make
this information more nicely available during runtime, we added a
feature to systemd that tracks and remembers for each service whether
it started up successfully, whether it exited with a non-zero exit
code, whether it timed out, or whether it terminated abnormally (by
segfaulting or similar), both during start-up and runtime. By simply
typing systemctl in your shell you can query the state of all
services, both systemd native and SysV/LSB services:

[root@lambda] ~# systemctl
UNIT                                          LOAD   ACTIVE       SUB          JOB             DESCRIPTION
dev-hugepages.automount                       loaded active       running                      Huge Pages File System Automount Point
dev-mqueue.automount                          loaded active       running                      POSIX Message Queue File System Automount Point
proc-sys-fs-binfmt_misc.automount             loaded active       waiting                      Arbitrary Executable File Formats File System Automount Point
sys-kernel-debug.automount                    loaded active       waiting                      Debug File System Automount Point
sys-kernel-security.automount                 loaded active       waiting                      Security File System Automount Point
sys-devices-pc...0000:02:00.0-net-eth0.device loaded active       plugged                      82573L Gigabit Ethernet Controller
[...]
sys-devices-virtual-tty-tty9.device           loaded active       plugged                      /sys/devices/virtual/tty/tty9
-.mount                                       loaded active       mounted                      /
boot.mount                                    loaded active       mounted                      /boot
dev-hugepages.mount                           loaded active       mounted                      Huge Pages File System
dev-mqueue.mount                              loaded active       mounted                      POSIX Message Queue File System
home.mount                                    loaded active       mounted                      /home
proc-sys-fs-binfmt_misc.mount                 loaded active       mounted                      Arbitrary Executable File Formats File System
abrtd.service                                 loaded active       running                      ABRT Automated Bug Reporting Tool
accounts-daemon.service                       loaded active       running                      Accounts Service
acpid.service                                 loaded active       running                      ACPI Event Daemon
atd.service                                   loaded active       running                      Execution Queue Daemon
auditd.service                                loaded active       running                      Security Auditing Service
avahi-daemon.service                          loaded active       running                      Avahi mDNS/DNS-SD Stack
bluetooth.service                             loaded active       running                      Bluetooth Manager
console-kit-daemon.service                    loaded active       running                      Console Manager
cpuspeed.service                              loaded active       exited                       LSB: processor frequency scaling support
crond.service                                 loaded active       running                      Command Scheduler
cups.service                                  loaded active       running                      CUPS Printing Service
dbus.service                                  loaded active       running                      D-Bus System Message Bus
[email protected]                            loaded active       running                      Getty on tty2
[email protected]                            loaded active       running                      Getty on tty3
[email protected]                            loaded active       running                      Getty on tty4
[email protected]                            loaded active       running                      Getty on tty5
[email protected]                            loaded active       running                      Getty on tty6
haldaemon.service                             loaded active       running                      Hardware Manager
[email protected]                            loaded active       running                      sda shock protection daemon
irqbalance.service                            loaded active       running                      LSB: start and stop irqbalance daemon
iscsi.service                                 loaded active       exited                       LSB: Starts and stops login and scanning of iSCSI devices.
iscsid.service                                loaded active       exited                       LSB: Starts and stops login iSCSI daemon.
livesys-late.service                          loaded active       exited                       LSB: Late init script for live image.
livesys.service                               loaded active       exited                       LSB: Init script for live image.
lvm2-monitor.service                          loaded active       exited                       LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
mdmonitor.service                             loaded active       running                      LSB: Start and stop the MD software RAID monitor
modem-manager.service                         loaded active       running                      Modem Manager
netfs.service                                 loaded active       exited                       LSB: Mount and unmount network filesystems.
NetworkManager.service                        loaded active       running                      Network Manager
ntpd.service                                  loaded maintenance  maintenance                  Network Time Service
polkitd.service                               loaded active       running                      Policy Manager
prefdm.service                                loaded active       running                      Display Manager
rc-local.service                              loaded active       exited                       /etc/rc.local Compatibility
rpcbind.service                               loaded active       running                      RPC Portmapper Service
rsyslog.service                               loaded active       running                      System Logging Service
rtkit-daemon.service                          loaded active       running                      RealtimeKit Scheduling Policy Service
sendmail.service                              loaded active       running                      LSB: start and stop sendmail
[email protected]:22-172.31.0.4:36368.service  loaded active       running                      SSH Per-Connection Server
sysinit.service                               loaded active       running                      System Initialization
systemd-logger.service                        loaded active       running                      systemd Logging Daemon
udev-post.service                             loaded active       exited                       LSB: Moves the generated persistent udev rules to /etc/udev/rules.d
udisks.service                                loaded active       running                      Disk Manager
upowerd.service                               loaded active       running                      Power Manager
wpa_supplicant.service                        loaded active       running                      Wi-Fi Security Service
avahi-daemon.socket                           loaded active       listening                    Avahi mDNS/DNS-SD Stack Activation Socket
cups.socket                                   loaded active       listening                    CUPS Printing Service Sockets
dbus.socket                                   loaded active       running                      dbus.socket
rpcbind.socket                                loaded active       listening                    RPC Portmapper Socket
sshd.socket                                   loaded active       listening                    sshd.socket
systemd-initctl.socket                        loaded active       listening                    systemd /dev/initctl Compatibility Socket
systemd-logger.socket                         loaded active       running                      systemd Logging Socket
systemd-shutdownd.socket                      loaded active       listening                    systemd Delayed Shutdown Socket
dev-disk-by\x1...x1db22a\x1d870f1adf2732.swap loaded active       active                       /dev/disk/by-uuid/fd626ef7-34a4-4958-b22a-870f1adf2732
basic.target                                  loaded active       active                       Basic System
bluetooth.target                              loaded active       active                       Bluetooth
dbus.target                                   loaded active       active                       D-Bus
getty.target                                  loaded active       active                       Login Prompts
graphical.target                              loaded active       active                       Graphical Interface
local-fs.target                               loaded active       active                       Local File Systems
multi-user.target                             loaded active       active                       Multi-User
network.target                                loaded active       active                       Network
remote-fs.target                              loaded active       active                       Remote File Systems
sockets.target                                loaded active       active                       Sockets
swap.target                                   loaded active       active                       Swap
sysinit.target                                loaded active       active                       System Initialization

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

221 units listed. Pass --all to see inactive units, too.
[root@lambda] ~#

(I have shortened the output above a little, and removed a few lines not relevant for this blog post.)

Look at the ACTIVE column, which shows you the high-level state of
a service (or in fact of any kind of unit systemd maintains, which can
be more than just services, but we’ll have a look on this in a later
blog posting), whether it is active (i.e. running),
inactive (i.e. not running) or in any other state. If you look
closely you’ll see one item in the list that is marked maintenance
and highlighted in red. This informs you about a service that failed
to run or otherwise encountered a problem. In this case this is
ntpd. Now, let’s find out what actually
happened to ntpd, with the systemctl status
command:

[root@lambda] ~# systemctl status ntpd.service
ntpd.service - Network Time Service
	  Loaded: loaded (/etc/systemd/system/ntpd.service)
	  Active: maintenance
	    Main: 953 (code=exited, status=255)
	  CGroup: name=systemd:/systemd-1/ntpd.service
[root@lambda] ~#

This shows us that NTP terminated during runtime (when it ran as
PID 953), and tells us exactly the error condition: the process exited
with an exit status of 255.

In a later systemd version, we plan to hook this up to ABRT, as soon as
this enhancement request is fixed. Then, if systemctl
status shows you information about a service that crashed it will
direct you right-away to the appropriate crash dump in ABRT.

Summary: use systemctl and systemctl
status as modern, more complete replacements for the traditional
boot-up status messages of SysV services. systemctl status
not only captures in more detail the error condition but also shows
runtime errors in addition to start-up errors.

That’s it for this week, make sure to come back next week, for the
next posting about systemd for administrators!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/bad-lennart.html

I too forgot to mention that my accommodation at GUADEC was sponsored by the GNOME Foundation. Thanks guys!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/sound-theme-canonical.html

#ignore yes

Today I came
across this blog post of your design team. In context of the recent
criticism you had to endure regarding upstream contributions I am disappointed
that you have not bothered to ping anybody from the upstream freedesktop sound
theme (for example yours truly) about this in advance. No, you went to cook
your own soup. What really disappoints me is that we have asked multiple times
for help and support and contributions for the sound theme, to only very little
success, and I even asked some of the Canonical engineers about this topic and
in particular regarding some clarifications of the licensing of the old Ubuntu
sound theme. I am sorry, but if you had listened, or looked, or asked you would
have been aware that we were looking for somebody to maintain this actively,
upstream — and because we didn’t have the time to maintain this we only
did the absolute minimum work necessary and we only maintain this ourselves
because noone else wanted to.

It should be upstream first, downstream second.

I am sorry if I sound like an always complaining prick to you. But believe
me, I am not saying this because I wouldn’t like you or anything like that.
I am just saying this because I believe you could do things
oh so much better.

Please fix this. We want your contributions. Upstream.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/i-am-more-awesome-than-canonical.html

I guess it’s a bit beating a dead horse, but I had a good laugh today when
I
learned that I alone contributed more to GNOME than the entirety of
Canonical, and only 800 additional commits seperating me from being more awesome than Nokia.

/me is amused

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/i-like-listening-to-myself.html

Here’s a podcast interview
with yours truly where I speak a little about PulseAudio and systemd. Seek
to 64:43 for my lovely impetuous voice. There’s also an interview with Owen
just before mine.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/plumbersconf-2010.html

#nocomments y

The Call
for Papers for the Linux
Plumbers Conference (LPC) in November in Cambridge, Massachusetts is ending
soon, on July 19th 2010 (That’s the upcoming monday!). It’s a conference
about the core infrastructure of Linux systems: the part of the system where
userspace and the kernel interface. It’s the only conference where the focus is
specifically on getting together the kernel people who work on the userspace
interfaces and the userspace people who have to deal with kernel interfaces.
It’s supposed to be a place where all the people doing infrastructure work sit
down and talk, so that both parties understand better what the requirements and
needs of the other are, and where we can work towards fixing the major problems
we currently have with our lower-level infrastructure and APIs.

The two previous LPCs were hugely successful (as reported on LWN on various
occasions), and this time we hope to repeat that.

Like the previous years, I will be running the Audio conference track of
LPC, this time together with Mark Brown. Audio infrastructure on Linux has been
steadily improving the last years all over the place, but there’s still a lot
to do. Join us at the LPC to discuss the next steps and help improving Linux
audio further! If you are doing audio infrastructure work on Linux, make
sure to attend and submit a paper!

Sign up soon!
Send
in your paper quickly! Only three days left to the end of the
CFP!

(I am also planning to do a presentation there about systemd, together
with Kay. Make sure to attend if you are interested in that topic.)

See you in Boston!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/locking2.html

I forgot to mention another central problem in my blog story about file locking
on Linux:

Different machines have access to different features of the same file
system. Here’s an example: let’s say you have two machines in your home LAN.
You want them to share their $HOME directory, so that you (or your family) can
use either machine and have access to all your (or their) data. So you export
/home on one machine via NFS and mount it from the other machine.

So far so good. But what happens to file locking now? Programs on the first
machine see a fully-featured ext3 or ext4 file system, where all kinds of
locking works (even though the API might suck as mentioned in the earlier blog
story). But what about the other machine? If you set up lockd properly
then POSIX locking will work on both. If you didn’t one machine can use POSIX
locking properly, the other cannot. And it gets even worse: as mentioned recent
NFS implementations on Linux transparently convert client-side BSD locking into
POSIX locking on the server side. Now, if the same application uses BSD locking on both
the client and the server side from two instances they will end up with two
orthogonal locks and although both sides think they have properly acquired a
lock (and they actually did) they will overwrite each other’s data, because
those two locks are independent. (And one wonders why the NFS developers
implemented this brokenness nonetheless…).

This basically means that locking cannot be used unless it is verified that
everyone accessing a file system can make use of the same file system feature
set. If you use file locking on a file system you should do so only if you are
sufficiently sure that nobody using a broken or weird NFS implementation might
want to access and lock those files as well. And practically that is
impossible. Even if fpathconf() was improved so that it could inform
the caller whether it can successfully apply a file lock to a file, this would
still not give any hint if the same is true for everybody else accessing the
file. But that is essential when speaking of advisory (i.e. cooperative) file
locking.

And no, this isn’t easy to fix. So again, the recommendation: forget about
file locking on Linux, it’s nothing more than a useless toy.

Also read Jeremy
Allison’s (Samba) take on POSIX file locking. It’s an interesting read.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/locking.html

It’s amazing how far Linux has come without providing for proper file
locking that works and is usable from userspace. A little overview why file
locking is still in a very sad state:

To begin with, there’s a plethora of APIs, and all of them are awful:

• POSIX File locking as available with fcntl(F_SET_LK): the POSIX
  locking API is the most portable one and in theory works across NFS. It can do
  byte-range locking. So much on the good side. On the bad side there’s a lot
  more however: locks are bound to processes, not file descriptors. That means
  that this logic cannot be used in threaded environments unless combined with a
  process-local mutex. This is hard to get right, especially in libraries that do
  not know the environment they are run in, i.e. whether they are used in
  threaded environments or not. The worst part however is that POSIX locks are
  automatically released if a process calls close() on any (!) of
  its open file descriptors for that file. That means that when one part of a
  program locks a file and another by coincidence accesses it too for a short
  time, the first part’s lock will be broken and it won’t be notified about that.
  Modern software tends to load big frameworks (such as Gtk+ or Qt) into memory
  as well as arbitrary modules via mechanisms such as NSS, PAM, gvfs,
  GTK_MODULES, Apache modules, GStreamer modules where one module seldom can
  control what another module in the same process does or accesses. The effect of
  this is that POSIX locks are unusable in any non-trivial program where it
  cannot be ensured that a file that is locked is never accessed by
  any other part of the process at the same time. Example: a user managing
  daemon wants to write /etc/passwd and locks the file for that. At
  the same time in another thread (or from a stack frame further down)
  something calls getpwuid() which internally accesses
  /etc/passwd and causes the lock to be released, the first thread
  (or stack frame) not knowing that. Furthermore should two threads use the
  locking fcntl()s on the same file they will interfere with each other’s locks
  and reset the locking ranges and flags of each other. On top of that locking
  cannot be used on any file that is publicly accessible (i.e. has the R bit set
  for groups/others, i.e. more access bits on than 0600), because that would
  otherwise effectively give arbitrary users a way to indefinitely block
  execution of any process (regardless of the UID it is running under) that wants
  to access and lock the file. This is generally not an acceptable security risk.
  Finally, while POSIX file locks are supposedly NFS-safe they not always really
  are as there are still many NFS implementations around where locking is not properly
  implemented, and NFS tends to be used in heterogenous networks. The biggest
  problem about this is that there is no way to properly detect whether file
  locking works on a specific NFS mount (or any mount) or not.
• The other API for POSIX file locks: lockf() is another API for the
  same mechanism and suffers by the same problems. One wonders why there are two
  APIs for the same messed up interface.
• BSD locking based on flock(). The semantics of this kind of
  locking are much nicer than for POSIX locking: locks are bound to file
  descriptors, not processes. This kind of locking can hence be used safely
  between threads and can even be inherited across fork() and
  exec(). Locks are only automatically broken on the close()
  call for the one file descriptor they were created with (or the last duplicate
  of it). On the other hand this kind of locking does not offer byte-range
  locking and suffers by the same security problems as POSIX locking, and works
  on even less cases on NFS than POSIX locking (i.e. on BSD and Linux < 2.6.12
  they were NOPs returning success). And since BSD locking is not as portable as
  POSIX locking this is sometimes an unsafe choice. Some OSes even find it funny
  to make flock() and fcntl(F_SET_LK) control the same locks.
  Linux treats them independently — except for the cases where it doesn’t: on
  Linux NFS they are transparently converted to POSIX locks, too now. What a chaos!
• Mandatory locking is available too. It’s based on the POSIX locking API but
  not portable in itself. It’s dangerous business and should generally be avoided
  in cleanly written software.
• Traditional lock file based file locking. This is how things where done
  traditionally, based around known atomicity guarantees of certain basic file
  system operations. It’s a cumbersome thing, and requires polling of the file
  system to get notifications when a lock is released. Also, On Linux NFS < 2.6.5
  it doesn’t work properly, since O_EXCL isn’t atomic there. And of course the
  client cannot really know what the server is running, so again this brokeness
  is not detectable.

The Disappointing Summary

File locking on Linux is just broken. The broken semantics of POSIX locking
show that the designers of this API apparently never have tried to actually use
it in real software. It smells a lot like an interface that kernel people
thought makes sense but in reality doesn’t when you try to use it from
userspace.

Here’s a list of places where you shouldn’t use file locking due to the
problems shown above: If you want to lock a file in $HOME, forget about it as
$HOME might be NFS and locks generally are not reliable there. The same applies
to every other file system that might be shared across the network. If the file
you want to lock is accessible to more than your own user (i.e. an access mode
> 0700), forget about locking, it would allow others to block your
application indefinitely. If your program is non-trivial or threaded or uses a
framework such as Gtk+ or Qt or any of the module-based APIs such as NSS, PAM,
… forget about about POSIX locking. If you care about portability, don’t use
file locking.

Or to turn this around, the only case where it is kind of safe to use file locking
is in trivial applications where portability is not key and by using BSD
locking on a file system where you can rely that it is local and on files
inaccessible to others. Of course, that doesn’t leave much, except for private
files in /tmp for trivial user applications.

Or in one sentence: in its current state Linux file locking is unusable.

And that is a shame.

Update: Check out the follow-up story on this topic.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/ids.html

When programming software that cooperates with software running on behalf of
other users, other sessions or other computers it is often necessary to work with
unique identifiers. These can be bound to various hardware and software objects
as well as lifetimes. Often, when people look for such an ID to use they pick
the wrong one because semantics and lifetime or the IDs are not clear. Here’s a
little incomprehensive list of IDs accessible on Linux and how you should or
should not use them.

Hardware IDs

1. /sys/class/dmi/id/product_uuid: The main board product UUID, as
   set by the board manufacturer and encoded in the BIOS DMI information. It may
   be used to identify a mainboard and only the mainboard. It changes when the
   user replaces the main board. Also, often enough BIOS manufacturers write bogus
   serials into it. In addition, it is x86-specific. Access for unprivileged users
   is forbidden. Hence it is of little general use.
2. CPUID/EAX=3 CPU serial number: A CPU UUID, as set by the
   CPU manufacturer and encoded on the CPU chip. It may be used to identify a CPU
   and only a CPU. It changes when the user replaces the CPU. Also, most modern
   CPUs don’t implement this feature anymore, and older computers tend to disable
   this option by default, controllable via a BIOS Setup option. In addition, it
   is x86-specific. Hence this too is of little general use.
3. /sys/class/net/*/address: One or more network MAC addresses, as
   set by the network adapter manufacturer and encoded on some network card
   EEPROM. It changes when the user replaces the network card. Since network cards
   are optional and there may be more than one the availability if this ID is not
   guaranteed and you might have more than one to choose from. On virtual machines
   the MAC addresses tend to be random. This too is hence of little general use.
4. /sys/bus/usb/devices/*/serial: Serial numbers of various USB
   devices, as encoded in the USB device EEPROM. Most devices don’t have a serial
   number set, and if they have it is often bogus. If the user replaces his USB
   hardware or plugs it into another machine these IDs may change or appear in
   other machines. This hence too is of little use.

There are various other hardware IDs available, many of which you may
discover via the ID_SERIAL udev property of various devices, such hard disks
and similar. They all have in common that they are bound to specific
(replacable) hardware, not universally available, often filled with bogus data
and random in virtualized environments. Or in other words: don’t use them, don’t
rely on them for identification, unless you really know what you are doing and
in general they do not guarantee what you might hope they guarantee.

Software IDs

1. /proc/sys/kernel/random/boot_id: A random ID that is regenerated
   on each boot. As such it can be used to identify the local machine’s current
   boot. It’s universally available on any recent Linux kernel. It’s a good and
   safe choice if you need to identify a specific boot on a specific booted
   kernel.
2. gethostname(), /proc/sys/kernel/hostname: A non-random ID
   configured by the administrator to identify a machine in the network. Often
   this is not set at all or is set to some default value such as
   localhost and not even unique in the local network. In addition it
   might change during runtime, for example because it changes based on updated
   DHCP information. As such it is almost entirely useless for anything but
   presentation to the user. It has very weak semantics and relies on correct
   configuration by the administrator. Don’t use this to identify machines in a
   distributed environment. It won’t work unless centrally administered, which
   makes it useless in a globalized, mobile world. It has no place in
   automatically generated filenames that shall be bound to specific hosts. Just
   don’t use it, please. It’s really not what many people think it is.
   gethostname() is standardized in POSIX and hence portable to other
   Unixes.
3. IP Addresses returned by SIOCGIFCONF or the respective Netlink APIs: These
   tend to be dynamically assigned and often enough only valid on local networks
   or even only the local links (i.e. 192.168.x.x style addresses, or even
   169.254.x.x/IPv4LL). Unfortunately they hence have little use outside of
   networking.
4. gethostid(): Returns a supposedly unique 32-bit identifier for the
   current machine. The semantics of this is not clear. On most machines this
   simply returns a value based on a local IPv4 address. On others it is
   administrator controlled via the /etc/hostid file. Since the semantics
   of this ID are not clear and most often is just a value based on the IP address it is
   almost always the wrong choice to use. On top of that 32bit are not
   particularly a lot. On the other hand this is standardized in POSIX and hence
   portable to other Unixes. It’s probably best to ignore this value and if people
   don’t want to ignore it they should probably symlink /etc/hostid to
   /var/lib/dbus/machine-id or something similar.
5. /var/lib/dbus/machine-id: An ID identifying a specific Linux/Unix
   installation. It does not change if hardware is replaced. It is not unreliable
   in virtualized environments. This value has clear semantics and is considered
   part of the D-Bus API. It is supposedly globally unique and portable to all
   systems that have D-Bus. On Linux, it is universally available, given that
   almost all non-embedded and even a fair share of the embedded machines ship
   D-Bus now. This is the recommended way to identify a machine, possibly with a
   fallback to the host name to cover systems that still lack D-Bus. If your
   application links against libdbus, you may access this ID with
   dbus_get_local_machine_id(), if not you can read it directly from the file system.
6. /proc/self/sessionid: An ID identifying a specific Linux login
   session. This ID is maintained by the kernel and part of the auditing logic. It
   is uniquely assigned to each login session during a specific system boot,
   shared by each process of a session, even across su/sudo and cannot be changed
   by userspace. Unfortunately some distributions have so far failed to set things
   up properly for this to work (Hey, you, Ubuntu!), and this ID is always
   (uint32_t) -1 for them. But there’s hope they get this fixed
   eventually. Nonetheless it is a good choice for a unique session identifier on
   the local machine and for the current boot. To make this ID globally unique it
   is best combined with /proc/sys/kernel/random/boot_id.
7. getuid(): An ID identifying a specific Unix/Linux user. This ID is
   usually automatically assigned when a user is created. It is not unique across
   machines and may be reassigned to a different user if the original user was
   deleted. As such it should be used only locally and with the limited validity
   in time in mind. To make this ID globally unique it is not sufficient to
   combine it with /var/lib/dbus/machine-id, because the same ID might be
   used for a different user that is created later with the same UID. Nonetheless
   this combination is often good enough. It is available on all POSIX systems.
8. ID_FS_UUID: an ID that identifies a specific file system in the
   udev tree. It is not always clear how these serials are generated but this
   tends to be available on almost all modern disk file systems. It is not
   available for NFS mounts or virtual file systems. Nonetheless this is often a
   good way to identify a file system, and in the case of the root directory even
   an installation. However due to the weakly defined generation semantics the
   D-Bus machine ID is generally preferrable.

Generating IDs

Linux offers a kernel interface to generate UUIDs on demand, by reading from
/proc/sys/kernel/random/uuid. This is a very simple interface to
generate UUIDs. That said, the logic behind UUIDs is unnecessarily complex and
often it is a better choice to simply read 16 bytes or so from
/dev/urandom.

Summary

And the gist of it all: Use /var/lib/dbus/machine-id! Use
/proc/self/sessionid! Use /proc/sys/kernel/random/boot_id!
Use getuid()! Use /dev/urandom! And forget about the
rest, in particular the host name, or the hardware IDs such as DMI. And keep in
mind that you may combine the aforementioned IDs in various ways to get
different semantics and validity constraints.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/linuxtag-2010-slides.html

On popular request, here are my (terse) slides from LinuxTag on systemd.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/linuxtag2k10.html

The upcoming week I’ll do two talks at LinuxTag 2010 at the Berlin Fair Grounds. One of them was only
added to the schedule today, about
systemd. Systemd has never been presented in a public talk before, so make
sure to attend this historic moment… ;-). Read about what has been written about systemd
so far, so that you can ask the sharpest questions during my
presentation.

My second talk might be about stuff a little less reported in the press, but
still very interesting, about Surround Sound in Gnome.

See you at LinuxTag!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/mango-lassi-is-back.html

Sven Herzberg has recently
been doing a lot of work on Mango Lassi, a
project deserving love but which I as its original author haven’t touched
in 3 years.

His work is already bearing fruits:

Distribution packagers, please go and package his version, Mango Lassi is an
awesome, wonderful tool that needs distributor love.

If you want to use Mango Lassi without waiting for the distribution packagers to catch up, Sven has built some packages for you in the OpenSUSE Build Service.

Sven, KUTGW!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/name-your-threads.html

Stefan Kost recently pointed me to the fact that the Linux system call
prctl(PR_SET_NAME) does not in fact change the process name, but the
task name (comm field) — in contrast to what the
man page suggests.

That makes it very useful for naming threads, since you can read back the
name you set with PR_SET_NAME earlier from the /proc file system
(/proc/$PID/task/$TID/comm on newer kernels,
/proc/$PID/task/$TID/stat‘s second field on older kernels), and hence
distuingish which thread might be responsible for the high CPU load or similar
problems.

So, now go, if you have a project which involves a lot of threads, name them
all individually, and make it easier to debug them. What’s missing now, of
course, is that gdb learns this and shows the comm name when doing info
threads.

I have changed PulseAudio now to name all threads it creates.

Of course, what would be even better than this is full file system extended
attribute support in procfs, so that we could attach arbitrary information to
processes and threads, including references to .desktop files and such.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/systemd-website.html

We now have a web
site, a mailing
list, a bugzilla component and moved our
git repositories to freedesktop.org. Make sure to update your check-outs.

For more details see our new web site.

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/lac-video.html

The great people from the Linux Audio Conference uploaded
the video streams from the event. Among them you can find my
own presentation. Enjoy!

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/when-pa-and-when-not.html

#nocomments yes

One thing became very clear to me during my trip to the Linux Audio Conference 2010
in Utrecht: even many pro audio folks are not sure what Jack does that PulseAudio doesn’t do and what
PulseAudio does that Jack doesn’t do; why they are not competing, why
you cannot replace one by the other, and why merging them (at least in
the short term) might not make immediate sense. In other words, why
millions of phones on this world run PulseAudio and not Jack, and why
a music studio running PulseAudio is crack.

To light this up a bit and for future reference I’ll try to explain in the
following text why there is this seperation between the two systems and why this isn’t
necessarily bad. This is mostly a written up version of (parts of) my slides
from LAC, so if you attended that event you might find little new, but I hope
it is interesting nonetheless.

This is mostly written from my perspective as a hacker working on
consumer audio stuff (more specifically having written most of
PulseAudio), but I am sure most pro audio folks would agree with the
points I raise here, and have more things to add. What I explain below
is in no way comprehensive, just a list of a couple of points I think
are the most important, as they touch the very core of both
systems (and we ignore all the toppings here, i.e. sound effects, yadda, yadda).

First of all let’s clear up the background of the sound server use cases here:

Of course, things are often not as black and white like this, there are uses
that fall in the middle of these two areas.

From the table above a few conclusions may be drawn:

• A consumer sound system must support both low and high latency operation.
  Since low latencies mean high CPU load and hence high power
  consumption^[2] (Heisenberg…), a system should always run with the
  highest latency latency possible, but the lowest latency necessary.
• Since the consumer system is highly dynamic in its use latencies must be
  adjusted dynamically too. That makes a design such as PulseAudio’s timer-based scheduling important.
• A pro audio system’s primary optimization target is low latency. Low
  power usage, dynamic changeble configuration (i.e. a short drop-out while you
  change your pipeline is acceptable) and user-friendliness may be sacrificed for
  that.
• For large buffer sizes a zero-copy design suggests itself: since data
  blocks are large the cache pressure can be considerably reduced by zero-copy
  designs. Only for large buffers the cost of passing pointers around is
  considerable smaller than the cost of passing around the data itself (or the
  other way round: if your audio data has the same size as your pointers, then
  passing pointers around is useless extra work).
• On a resource constrained system the ideal audio pipeline does not touch
  and convert the data passed along it unnecessarily. That makes it important to
  support natively the sample types and interleaving modes of the audio source or
  destination.
• A consumer system needs to simplify the view on the hardware, hide the its
  complexity: hide redundant mixer elements, or merge them while making use of
  the hardware capabilities, and extending it in software so that the same
  functionality is provided on all hardware. A production system should not hide
  or simplify the hardware functionality.
• A consumer system should not drop-out when a client misbehaves or the
  configuration changes (OTOH if it happens in exceptions it is not disastrous
  either). A synchronous pipeline is hence not advisable, clients need to supply
  their data asynchronously.
• In a pro audio system a drop-out during reconfiguration is acceptable,
  during operation unacceptable.
• In consumer audio we need to make compromises on resource usage,
  which pro audio does not have to commit to. Example: a pro audio
  system can issue memlock() with little limitations since the
  hardware is powerful (i.e. a lot of RAM available) and audio is the
  primary purpose. A consumer audio system cannot do that because that
  call practically makes memory unavailable to other applications,
  increasing their swap pressure. And since audio is not the primary
  purpose of the system and resources limited we hence need to find a
  different way.

Jack has been designed for low latencies, where synchronous
operation is advisable, meaning that a misbehaving client call stall
the entire pipeline. Changes of the pipeline or latencies usually
result in drop-outs in one way or the other, since the entire pipeline
is reconfigured, from the hardware to the various clients. Jack only
supports FLOAT32 samples and non-interleaved audio channels (and that
is a good thing). Jack does not employ reference-counted zero-copy
buffers. It does not try to simplify the hardware mixer in any
way.

PulseAudio OTOH can deal with varying latancies, dynamically
adjusting to the lowest latencies any of the connected clients
needs. Client communication is fully asynchronous, a single client
cannot stall the entire pipeline. PulseAudio supports a variety of PCM
formats and channel setups. PulseAudio’s design is heavily based on
reference-counted zero-copy buffers that are passed around, even
between processes, instead of the audio data itself. PulseAudio tries
to simplify the hardware mixer as suggested above.

Now, the two paragraphs above hopefully show how Jack is more
suitable for the pro audio use case and PulseAudio more for the
consumer audio use case. One question asks itself though: can we marry
the two approaches? Yes, we probably can, MacOS has a unified approach
for both uses. However, it is not clear this would be a good
idea. First of all, a system with the complexities introduced by
sample format/channel mapping conversion, as well as dynamically
changing latencies and pipelines, and asynchronous behaviour would
certainly be much less attractive to pro audio developers. In fact,
that Jack limits itself to synchronous, FLOAT32-only,
non-interleaved-only audio streams is one of the big features of its
design. Marrying the two approaches would corrupt that. A merged
solution would probably not have a good stand in the community.

But it goes even further than this: what would the use case for
this be? After all, most of the time, you don’t want your event
sounds, your Youtube, your VoIP and your Rhythmbox mixed into the new
record you are producing. Hence a clear seperation between the two
worlds might even be handy?

Also, let’s not forget that we lack the manpower to even create
such an audio chimera.

So, where to from here? Well, I think we should put the focus on
cooperation instead of amalgamation: teach PulseAudio to go out of the
way as soon as Jack needs access to the device, and optionally make
PulseAudio a normal JACK client while both are running. That way, the
user has the option to use the PulseAudio supplied streams, but
normally does not see them in his pipeline. The first part of this has
already been implemented: Jack2 and PulseAudio do not fight for the
audio device, a friendly handover takes place. Jack takes precedence,
PulseAudio takes the back seat. The second part is still missing: you
still have to manually hookup PulseAudio to Jack if you are interested
in its streams. If both are implemented starting Jack basically has
the effect of replacing PulseAudio’s core with the Jack core, while
still providing full compatibility with PulseAudio clients.

And that I guess is all I have to say on the entire Jack and
PulseAudio story.

Oh, one more thing, while we are at clearing things up: some news
sites claim that PulseAudio’s not necessarily stellar reputation in
some parts of the community comes from Ubuntu and other distributions
having integrated it too early. Well, let me stress here explicitly,
that while they might have made a mistake or two in packaging
PulseAudio and I publicly pointed that out (and probably not in a too
friendly way), I do believe that the point in time they adopted it was
right. Why? Basically, it’s a chicken and egg problem. If it is not
used in the distributions it is not tested, and there is no pressure
to get fixed what then turns out to be broken: in PulseAudio itself,
and in both the layers on top and below of it. Don’t forget that
pushing a new layer into an existing stack will break a lot of
assumptions that the neighboring layers made. Doing this must
break things. Most Free Software projects could probably use more
developers, and that is particularly true for Audio on Linux. And
given that that is how it is, pushing the feature in at that point in
time was the right thing to do. Or in other words, if the features are
right, and things do work correctly as far as the limited test base
the developers control shows, then one day you need to push into the
distributions, even if this might break setups and software that
previously has not been tested, unless you want to stay stuck in your
development indefinitely. So yes, Ubuntu, I think you did well with
adopting PulseAudio when you did.

Footnotes

[1] Side note: yes, consumers tend not to know what dB is, and expect
volume settings in “percentages”, a mostly meaningless unit in
audio. This even spills into projects like VLC or Amarok which expose
linear volume controls (which is a really bad idea).

[2] In case you are wondering why that is the case: if the latency is
low the buffers must be sized smaller. And if the buffers are sized smaller
then the CPU will have to wake up more often to fill them up for the same
playback time. This drives up the CPU load since less actual payload can be
processed for the amount of housekeeping that the CPU has to do during each
buffer iteration. Also, frequent wake-ups make it impossible for the CPU to go
to deeper sleep states. Sleep states are the primary way for modern CPUs
to save power.