Tag Archives: Europe

On journeys

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/03/on-journeys.html

– 1 –

Poland is an ancient country whose history is deeply intertwined with that of the western civilization. In its glory days, the Polish-Lithuanian Commonwealth sprawled across vast expanses of land in central Europe, from Black Sea to Baltic Sea. But over the past two centuries, it suffered a series of military defeats and political partitions at the hands of its closest neighbors: Russia, Austria, Prussia, and – later – Germany.

After more than a hundred years of foreign rule, Poland re-emerged as an independent state in 1918, only to face the armies of Nazi Germany at the onset of World War II. With Poland’s European allies reneging on their earlier military guarantees, the fierce fighting left the country in ruins. Some six million people have died within its borders – more than ten times the death toll in France or in the UK. Warsaw was reduced to a sea of rubble, with perhaps one in ten buildings still standing by the end of the war.

With the collapse of the Third Reich, Franklin D. Roosevelt, Winston Churchill, and Joseph Stalin sat down in Yalta to decide the new order for war-torn Europe. At Stalin’s behest, Poland and its neighboring countries were placed under Soviet political and military control, forming what has become known as the Eastern Bloc.

Over the next several decades, the Soviet satellite states experienced widespread repression and economic decline. But weakened by the expense of the Cold War, the communist chokehold on the region eventually began to wane. In Poland, the introduction of martial law in 1981 could not put an end to sweeping labor unrest. Narrowly dodging the specter of Soviet intervention, the country regained its independence in 1989 and elected its first democratic government; many other Eastern Bloc countries soon followed suit.

Ever since then, Poland has enjoyed a period of unprecedented growth and has emerged as one of the more robust capitalist democracies in the region. In just two decades, it shed many of its backwardly, state-run heavy industries and adopted a modern, service-oriented economy. But the effects of the devastating war and the lost decades under communist rule still linger on – whether you look at the country’s infrastructure, at its socrealist cityscapes, at its political traditions, or at the depressingly low median wage.

When thinking about the American involvement in the Cold War, people around the world may recall Vietnam, Bay of Pigs, or the proxy wars fought in the Middle East. But in Poland and many of its neighboring states, the picture you remember the most is the fall of the Berlin Wall.

– 2 –

I was born in Warsaw in the winter of 1981, at the onset of martial law, with armored vehicles rolling onto Polish streets. My mother, like many of her generation, moved to the capital in the sixties as a part of an effort to rebuild and repopulate the war-torn city. My grandma would tell eerie stories of Germans and Soviets marching through their home village somewhere in the west. I liked listening to the stories; almost every family in Poland had some to tell.

I did not get to know my father. I knew his name; he was a noted cinematographer who worked on big-ticket productions back in the day. He left my mother when I was very young and never showed interest in staying in touch. He had a wife and other children, so it might have been that.

Compared to him, mom hasn’t done well for herself. We ended up in social housing in one of the worst parts of the city, on the right bank of the Vistula river. My early memories from school are that of classmates sniffing glue from crumpled grocery bags. I remember my family waiting in lines for rationed toilet paper and meat. As a kid, you don’t think about it much.

The fall of communism came suddenly. I have a memory of grandma listening to broadcasts from Radio Free Europe, but I did not understand what they were all about. I remember my family cheering one afternoon, transfixed to a black-and-white TV screen. I recall my Russian language class morphing into English; I had my first taste of bananas and grapefruits. There is the image of the monument of Feliks Dzierżyński coming down. I remember being able to go to a better school on the other side of Warsaw – and getting mugged many times on the way.

The transformation brought great wealth to some, but many others have struggled to find their place in the fledgling and sometimes ruthless capitalist economy. Well-educated and well read, my mom ended up in the latter pack, at times barely making ends meet. I think she was in part a victim of circumstance, and in part a slave to way of thinking that did not permit the possibility of taking chances or pursuing happiness.

– 3 –

Mother always frowned upon popular culture, seeing it as unworthy of an educated mind. For a time, she insisted that I only listen to classical music. She angrily shunned video games, comic books, and cartoons. I think she perceived technology as trivia; the only field of science she held in high regard was abstract mathematics, perhaps for its detachment from the mundane world. She hoped that I would learn Latin, a language she could read and write; that I would practice drawing and painting; or that I would read more of the classics of modernist literature.

Of course, I did almost none of that. I hid my grunge rock tapes between Tchaikovsky, listened to the radio under the sheets, and watched the reruns of The A-Team while waiting for her to come back from work. I liked electronics and chemistry a lot more than math. And when I laid my hands on my first computer – an 8-bit relic of British engineering from 1982 – I soon knew that these machines, in their incredible complexity and flexibility, were what I wanted to spend my time on.

I suspected I could be a competent programmer, but never had enough faith in my skill. Yet, in learning about computers, I realized that I had a knack for understanding complex systems and poking holes in how they work. With a couple of friends, we joined the nascent information security community in Europe, comparing notes on mailing lists. Before long, we were taking on serious consulting projects for banks and the government – usually on weekends and after school, but sometimes skipping a class or two. Well, sometimes more than that.

All of the sudden, I was facing an odd choice. I could stop, stay in school and try to get a degree – going back every night to a cramped apartment, my mom sleeping on a folding bed in the kitchen, my personal space limited to a bare futon and a tiny desk. Or, I could seize the moment and try to make it on my own, without hoping that one day, my family would be able to give me a head start.

I moved out, dropped out of school, and took on a full-time job. It paid somewhere around $12,000 a year – a pittance anywhere west of the border, but a solid wage in Poland even today. Not much later, I was making two times as much, about the upper end of what one could hope for in this line of work. I promised myself to keep taking courses after hours, but I wasn’t good at sticking to the plan. I moved in with my girlfriend, and at the age of 19, I felt for the first time that things were going to be all right.

– 4 –

Growing up in Europe, you get used to the barrage of low-brow swipes taken at the United States. Your local news will never pass up the opportunity to snicker about the advances of creationism somewhere in Kentucky. You can stay tuned for a panel of experts telling you about the vastly inferior schools, the medieval justice system, and the striking social inequality on the other side of the pond. You don’t doubt their words – but deep down inside, no matter how smug the critics are, or how seemingly convincing their arguments, the American culture still draws you in.

My moment of truth came in the summer of 2000. A company from Boston asked me if I’d like to talk about a position on their research team; I looked at the five-digit figure and could not believe my luck. Moving to the US was an unreasonable risk for a kid who could barely speak English and had no safety net to fall back to. But that did not matter: I knew I had no prospects of financial independence in Poland – and besides, I simply needed to experience the New World through my own eyes.

Of course, even with a job offer in hand, getting into the United States is not an easy task. An engineering degree and a willing employer opens up a straightforward path; it is simple enough that some companies would abuse the process to source cheap labor for menial, low-level jobs. With a visa tied to the petitioning company, such captive employees could not seek better wages or more rewarding work.

But without a degree, the options shrink drastically. For me, the only route would be a seldom-granted visa reserved for extraordinary skill – meant for the recipients of the Nobel Prize and other folks who truly stand out in their field of expertise. The attorneys looked over my publication record, citations, and the supporting letters from other well-known people in the field. Especially given my age, they thought we had a good shot. A few stressful months later, it turned out that they were right.

On the week of my twentieth birthday, I packed two suitcases and boarded a plane to Boston. My girlfriend joined me, miraculously securing a scholarship at a local university to continue her physics degree; her father helped her with some of the costs. We had no idea what we were doing; we had perhaps few hundred bucks on us, enough to get us through the first couple of days. Four thousand miles away from our place of birth, we were starting a brand new life.

– 5 –

The cultural shock gets you, but not in the sense you imagine. You expect big contrasts, a single eye-opening day to remember for the rest of your life. But driving down a highway in the middle of a New England winter, I couldn’t believe how ordinary the world looked: just trees, boxy buildings, and pavements blanketed with dirty snow.

Instead of a moment of awe, you drown in a sea of small, inconsequential things, draining your energy and making you feel helpless and lost. It’s how you turn on the shower; it’s where you can find a grocery store; it’s what they meant by that incessant “paper or plastic” question at the checkout line. It’s how you get a mailbox key, how you make international calls, it’s how you pay your bills with a check. It’s the rules at the roundabout, it’s your social security number, it’s picking the right toll lane, it’s getting your laundry done. It’s setting up a dial-up account and finding the food you like in the sea of unfamiliar brands. It’s doing all this without Google Maps or a Facebook group to connect with other expats nearby.

The other thing you don’t expect is losing touch with your old friends; you can call or e-mail them every day, but your social frames of reference begin to drift apart, leaving less and less to talk about. The acquaintances you make in the office will probably never replace the folks you grew up with. We managed, but we weren’t prepared for that.

– 6 –

In the summer, we had friends from Poland staying over for a couple of weeks. By the end of their trip, they asked to visit New York City one more time; we liked the Big Apple, so we took them on a familiar ride down I-95. One of them went to see the top of World Trade Center; the rest of us just walked around, grabbing something to eat before we all headed back. A few days later, we were all standing in front of a TV, watching September 11 unfold in real time.

We felt horror and outrage. But when we roamed the unsettlingly quiet streets of Boston, greeted by flags and cardboard signs urging American drivers to honk, we understood that we were strangers a long way from home – and that our future in this country hanged in the balance more than we would have thought.

Permanent residency is a status that gives a foreigner the right to live in the US and do almost anything they please – change jobs, start a business, or live off one’s savings all the same. For many immigrants, the pursuit of this privilege can take a decade or more; for some others, it stays forever out of reach, forcing them to abandon the country in a matter of days as their visas expire or companies fold. With my O-1 visa, I always counted myself among the lucky ones. Sure, it tied me to an employer, but I figured that sorting it out wouldn’t be a big deal.

That proved to be a mistake. In the wake of 9/11, an agency known as Immigration and Naturalization Services was being dismantled and replaced by a division within the Department of Homeland Security. My own seemingly straightforward immigration petition ended up somewhere in the bureaucratic vacuum that formed in between the two administrative bodies. I waited patiently, watching the deepening market slump, and seeing my employer’s prospects get dimmer and dimmer every month. I was ready for the inevitable, with other offers in hand, prepared to make my move perhaps the very first moment I could. But the paperwork just would not come through. With the Boston office finally shutting down, we packed our bags and booked flights. We faced the painful admission that for three years, we chased nothing but a pipe dream. The only thing we had to show for it were two adopted cats, now sitting frightened somewhere in the cargo hold.

The now-worthless approval came through two months later; the lawyers, cheerful as ever, were happy to send me a scan. The hollowed-out remnants of my former employer were eventually bought by Symantec – the very place from where I had my backup offer in hand.

– 7 –

In a way, Europe’s obsession with America’s flaws made it easier to come home without ever explaining how the adventure really played out. When asked, I could just wing it: a mention of the death penalty or permissive gun laws would always get you a knowing nod, allowing the conversation to move on.

Playing to other people’s preconceptions takes little effort; lying to yourself calls for more skill. It doesn’t help that when you come back after three years away from home, you notice all the small annoyances that you used to simply tune out. Back then, Warsaw still had a run-down vibe: the dilapidated road from the airport; the drab buildings on the other side of the river; the uneven pavements littered with dog poop; the dirty walls at my mother’s place, with barely any space to turn. You can live with it, of course – but it’s a reminder that you settled for less, and it’s a sensation that follows you every step of the way.

But more than the sights, I couldn’t forgive myself something else: that I was coming back home with just loose change in my pocket. There are some things that a failed communist state won’t teach you, and personal finance is one of them; I always looked at money just as a reward for work, something you get to spend to brighten your day. The indulgences were never extravagant: perhaps I would take the cab more often, or have take-out every day. But no matter how much I made, I kept living paycheck-to-paycheck – the only way I knew, the way our family always did.

– 8 –

With a three-year stint in the US on your resume, you don’t have a hard time finding a job in Poland. You face the music in a different way. I ended up with a salary around a fourth of what I used to make in Massachusetts, but I simply decided not to think about it much. I wanted to settle down, work on interesting projects, marry my girlfriend, have a child. I started doing consulting work whenever I could, setting almost all the proceeds aside.

After four years with T-Mobile in Poland, I had enough saved to get us through a year or so – and in a way, it changed the way I looked at my work. Being able to take on ambitious challenges and learn new things started to matter more than jumping ships for a modest salary bump. Burned by the folly of pursuing riches in a foreign land, I put a premium on boring professional growth.

Comically, all this introspection made me realize that from where I stood, I had almost nowhere left to go. Sure, Poland had telcos, refineries, banks – but they all consumed the technologies developed elsewhere, shipped here in a shrink-wrapped box; as far as their IT went, you could hardly tell the companies apart. To be a part of the cutting edge, you had to pack your bags, book a flight, and take a jump into the unknown. I sure as heck wasn’t ready for that again.

And then, out of the blue, Google swooped in with an offer to work for them from the comfort of my home, dialing in for a videoconference every now and then. The starting pay was about the same, but I had no second thoughts. I didn’t say it out loud, but deep down inside, I already knew what needed to happen next.

– 9 –

We moved back to the US in 2009, two years after taking the job, already on the hook for a good chunk of Google’s product security and with the comfort of knowing where we stood. In a sense, my motive was petty: you could call it a desire to vindicate a failed adolescent dream. But in many other ways, I have grown fond of the country that shunned us once before; and I wanted our children to grow up without ever having to face the tough choices and the uncertain prospects I had to deal with in my earlier years.

This time, we knew exactly what to do: a quick stop at a grocery store on a way from the airport, followed by e-mail to our immigration folks to get the green card paperwork out the door. A bit more than half a decade later, we were standing in a theater in Campbell, reciting the Oath of Allegiance and clinging on to our new certificates of US citizenship.

The ceremony closed a long and interesting chapter in my life. But more importantly, standing in that hall with people from all over the globe made me realize that my story is not extraordinary; many of them had lived through experiences far more harrowing and captivating than mine. If anything, my tale is hard to tell apart from that of countless other immigrants from the former Eastern Bloc. By some estimates, in the US alone, the Polish diaspora is about 9 million strong.

I know that the Poland of today is not the Poland I grew up in. It’s not not even the Poland I came back to in 2003; the gap to Western Europe is shrinking every single year. But I am grateful to now live in a country that welcomes more immigrants than any other place on Earth – and at the end of their journey, makes many of them them feel at home. It also makes me realize how small and misguided must be the conversations we are having about immigration – not just here, but all over the developed world.

To explore other articles in this short series about Poland, click here. You can also directly proceed to the next entry here.

Revisiting How We Put Together Linux Systems

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html

In a previous blog story I discussed
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems,
I now want to take the opportunity to explain a bit where we want to
take this with
systemd in the
longer run, and what we want to build out of it. This is going to be a
longer story, so better grab a cold bottle of
Club Mate before you start
reading.

Traditional Linux distributions are built around packaging systems
like RPM or dpkg, and an organization model where upstream developers
and downstream packagers are relatively clearly separated: an upstream
developer writes code, and puts it somewhere online, in a tarball. A
packager than grabs it and turns it into RPMs/DEBs. The user then
grabs these RPMs/DEBs and installs them locally on the system. For a
variety of uses this is a fantastic scheme: users have a large
selection of readily packaged software available, in mostly uniform
packaging, from a single source they can trust. In this scheme the
distribution vets all software it packages, and as long as the user
trusts the distribution all should be good. The distribution takes the
responsibility of ensuring the software is not malicious, of timely
fixing security problems and helping the user if something is wrong.

Upstream Projects

However, this scheme also has a number of problems, and doesn’t fit
many use-cases of our software particularly well. Let’s have a look at
the problems of this scheme for many upstreams:

  • Upstream software vendors are fully dependent on downstream
    distributions to package their stuff. It’s the downstream
    distribution that decides on schedules, packaging details, and how
    to handle support. Often upstream vendors want much faster release
    cycles then the downstream distributions follow.

  • Realistic testing is extremely unreliable and next to
    impossible. Since the end-user can run a variety of different
    package versions together, and expects the software he runs to just
    work on any combination, the test matrix explodes. If upstream tests
    its version on distribution X release Y, then there’s no guarantee
    that that’s the precise combination of packages that the end user
    will eventually run. In fact, it is very unlikely that the end user
    will, since most distributions probably updated a number of
    libraries the package relies on by the time the package ends up being
    made available to the user. The fact that each package can be
    individually updated by the user, and each user can combine library
    versions, plug-ins and executables relatively freely, results in a high
    risk of something going wrong.

  • Since there are so many different distributions in so many different
    versions around, if upstream tries to build and test software for
    them it needs to do so for a large number of distributions, which is
    a massive effort.

  • The distributions are actually quite different in many ways. In
    fact, they are different in a lot of the most basic
    functionality. For example, the path where to put x86-64 libraries
    is different on Fedora and Debian derived systems..

  • Developing software for a number of distributions and versions is
    hard: if you want to do it, you need to actually install them, each
    one of them, manually, and then build your software for each.

  • Since most downstream distributions have strict licensing and
    trademark requirements (and rightly so), any kind of closed source
    software (or otherwise non-free) does not fit into this scheme at
    all.

This all together makes it really hard for many upstreams to work
nicely with the current way how Linux works. Often they try to improve
the situation for them, for example by bundling libraries, to make
their test and build matrices smaller.

System Vendors

The toolbox approach of classic Linux distributions is fantastic for
people who want to put together their individual system, nicely
adjusted to exactly what they need. However, this is not really how
many of today’s Linux systems are built, installed or updated. If you
build any kind of embedded device, a server system, or even user
systems, you frequently do your work based on complete system images,
that are linearly versioned. You build these images somewhere, and
then you replicate them atomically to a larger number of systems. On
these systems, you don’t install or remove packages, you get a defined
set of files, and besides installing or updating the system there are
no ways how to change the set of tools you get.

The current Linux distributions are not particularly good at providing
for this major use-case of Linux. Their strict focus on individual
packages as well as package managers as end-user install and update
tool is incompatible with what many system vendors want.

Users

The classic Linux distribution scheme is frequently not what end users
want, either. Many users are used to app markets like Android, Windows
or iOS/Mac have. Markets are a platform that doesn’t package, build or
maintain software like distributions do, but simply allows users to
quickly find and download the software they need, with the app vendor
responsible for keeping the app updated, secured, and all that on the
vendor’s release cycle. Users tend to be impatient. They want their
software quickly, and the fine distinction between trusting a single
distribution or a myriad of app developers individually is usually not
important for them. The companies behind the marketplaces usually try
to improve this trust problem by providing sand-boxing technologies: as
a replacement for the distribution that audits, vets, builds and
packages the software and thus allows users to trust it to a certain
level, these vendors try to find technical solutions to ensure that
the software they offer for download can’t be malicious.

Existing Approaches To Fix These Problems

Now, all the issues pointed out above are not new, and there are
sometimes quite successful attempts to do something about it. Ubuntu
Apps, Docker, Software Collections, ChromeOS, CoreOS all fix part of
this problem set, usually with a strict focus on one facet of Linux
systems. For example, Ubuntu Apps focus strictly on end user (desktop)
applications, and don’t care about how we built/update/install the OS
itself, or containers. Docker OTOH focuses on containers only, and
doesn’t care about end-user apps. Software Collections tries to focus
on the development environments. ChromeOS focuses on the OS itself,
but only for end-user devices. CoreOS also focuses on the OS, but
only for server systems.

The approaches they find are usually good at specific things, and use
a variety of different technologies, on different layers. However,
none of these projects tried to fix this problems in a generic way,
for all uses, right in the core components of the OS itself.

Linux has come to tremendous successes because its kernel is so
generic: you can build supercomputers and tiny embedded devices out of
it. It’s time we come up with a basic, reusable scheme how to solve
the problem set described above, that is equally generic.

What We Want

The systemd cabal (Kay Sievers, Harald Hoyer, Daniel Mack, Tom
Gundersen, David Herrmann, and yours truly) recently met in Berlin
about all these things, and tried to come up with a scheme that is
somewhat simple, but tries to solve the issues generically, for all
use-cases, as part of the systemd project. All that in a way that is
somewhat compatible with the current scheme of distributions, to allow
a slow, gradual adoption. Also, and that’s something one cannot stress
enough: the toolbox scheme of classic Linux distributions is
actually a good one, and for many cases the right one. However, we
need to make sure we make distributions relevant again for all
use-cases, not just those of highly individualized systems.

Anyway, so let’s summarize what we are trying to do:

  • We want an efficient way that allows vendors to package their
    software (regardless if just an app, or the whole OS) directly for
    the end user, and know the precise combination of libraries and
    packages it will operate with.

  • We want to allow end users and administrators to install these
    packages on their systems, regardless which distribution they have
    installed on it.

  • We want a unified solution that ultimately can cover updates for
    full systems, OS containers, end user apps, programming ABIs, and
    more. These updates shall be double-buffered, (at least). This is an
    absolute necessity if we want to prepare the ground for operating
    systems that manage themselves, that can update safely without
    administrator involvement.

  • We want our images to be trustable (i.e. signed). In fact we want a
    fully trustable OS, with images that can be verified by a full
    trust chain from the firmware (EFI SecureBoot!), through the boot loader, through the
    kernel, and initrd. Cryptographically secure verification of the
    code we execute is relevant on the desktop (like ChromeOS does), but
    also for apps, for embedded devices and even on servers (in a post-Snowden
    world, in particular).

What We Propose

So much about the set of problems, and what we are trying to do. So,
now, let’s discuss the technical bits we came up with:

The scheme we propose is built around the variety of concepts of btrfs
and Linux file system name-spacing. btrfs at this point already has a
large number of features that fit neatly in our concept, and the
maintainers are busy working on a couple of others we want to
eventually make use of.

As first part of our proposal we make heavy use of btrfs sub-volumes and
introduce a clear naming scheme for them. We name snapshots like this:

  • usr:<vendorid>:<architecture>:<version> — This refers to a full
    vendor operating system tree. It’s basically a /usr tree (and no
    other directories), in a specific version, with everything you need to boot
    it up inside it. The <vendorid> field is replaced by some vendor
    identifier, maybe a scheme like
    org.fedoraproject.FedoraWorkstation. The <architecture> field
    specifies a CPU architecture the OS is designed for, for example
    x86-64. The <version> field specifies a specific OS version, for
    example 23.4. An example sub-volume name could hence look like this:
    usr:org.fedoraproject.FedoraWorkstation:x86_64:23.4

  • root:<name>:<vendorid>:<architecture> — This refers to an
    instance of an operating system. Its basically a root directory,
    containing primarily /etc and /var (but possibly more). Sub-volumes
    of this type do not contain a populated /usr tree though. The
    <name> field refers to some instance name (maybe the host name of
    the instance). The other fields are defined as above. An example
    sub-volume name is
    root:revolution:org.fedoraproject.FedoraWorkstation:x86_64.

  • runtime:<vendorid>:<architecture>:<version> — This refers to a
    vendor runtime. A runtime here is supposed to be a set of
    libraries and other resources that are needed to run apps (for the
    concept of apps see below), all in a /usr tree. In this regard this
    is very similar to the usr sub-volumes explained above, however,
    while a usr sub-volume is a full OS and contains everything
    necessary to boot, a runtime is really only a set of
    libraries. You cannot boot it, but you can run apps with it. An
    example sub-volume name is: runtime:org.gnome.GNOME3_20:x86_64:3.20.1

  • framework:<vendorid>:<architecture>:<version> — This is very
    similar to a vendor runtime, as described above, it contains just a
    /usr tree, but goes one step further: it additionally contains all
    development headers, compilers and build tools, that allow
    developing against a specific runtime. For each runtime there should
    be a framework. When you develop against a specific framework in a
    specific architecture, then the resulting app will be compatible
    with the runtime of the same vendor ID and architecture. Example:
    framework:org.gnome.GNOME3_20:x86_64:3.20.1

  • app:<vendorid>:<runtime>:<architecture>:<version> — This
    encapsulates an application bundle. It contains a tree that at
    runtime is mounted to /opt/<vendorid>, and contains all the
    application’s resources. The <vendorid> could be a string like
    org.libreoffice.LibreOffice, the <runtime> refers to one the
    vendor id of one specific runtime the application is built for, for
    example org.gnome.GNOME3_20:3.20.1. The <architecture> and
    <version> refer to the architecture the application is built for,
    and of course its version. Example:
    app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133

  • home:<user>:<uid>:<gid> — This sub-volume shall refer to the home
    directory of the specific user. The <user> field contains the user
    name, the <uid> and <gid> fields the numeric Unix UIDs and GIDs
    of the user. The idea here is that in the long run the list of
    sub-volumes is sufficient as a user database (but see
    below). Example: home:lennart:1000:1000.

btrfs partitions that adhere to this naming scheme should be clearly
identifiable. It is our intention to introduce a new GPT partition type
ID for this.

How To Use It

After we introduced this naming scheme let’s see what we can build of
this:

  • When booting up a system we mount the root directory from one of the
    root sub-volumes, and then mount /usr from a matching usr
    sub-volume. Matching here means it carries the same <vendor-id>
    and <architecture>. Of course, by default we should pick the
    matching usr sub-volume with the newest version by default.

  • When we boot up an OS container, we do exactly the same as the when
    we boot up a regular system: we simply combine a usr sub-volume
    with a root sub-volume.

  • When we enumerate the system’s users we simply go through the
    list of home snapshots.

  • When a user authenticates and logs in we mount his home
    directory from his snapshot.

  • When an app is run, we set up a new file system name-space, mount the
    app sub-volume to /opt/<vendorid>/, and the appropriate runtime
    sub-volume the app picked to /usr, as well as the user’s
    /home/$USER to its place.

  • When a developer wants to develop against a specific runtime he
    installs the right framework, and then temporarily transitions into
    a name space where /usris mounted from the framework sub-volume, and
    /home/$USER from his own home directory. In this name space he then
    runs his build commands. He can build in multiple name spaces at the
    same time, if he intends to builds software for multiple runtimes or
    architectures at the same time.

Instantiating a new system or OS container (which is exactly the same
in this scheme) just consists of creating a new appropriately named
root sub-volume. Completely naturally you can share one vendor OS
copy in one specific version with a multitude of container instances.

Everything is double-buffered (or actually, n-fold-buffered), because
usr, runtime, framework, app sub-volumes can exist in multiple
versions. Of course, by default the execution logic should always pick
the newest release of each sub-volume, but it is up to the user keep
multiple versions around, and possibly execute older versions, if he
desires to do so. In fact, like on ChromeOS this could even be handled
automatically: if a system fails to boot with a newer snapshot, the
boot loader can automatically revert back to an older version of the
OS.

An Example

Note that in result this allows installing not only multiple end-user
applications into the same btrfs volume, but also multiple operating
systems, multiple system instances, multiple runtimes, multiple
frameworks. Or to spell this out in an example:

Let’s say Fedora, Mageia and ArchLinux all implement this scheme,
and provide ready-made end-user images. Also, the GNOME, KDE, SDL
projects all define a runtime+framework to develop against. Finally,
both LibreOffice and Firefox provide their stuff according to this
scheme. You can now trivially install of these into the same btrfs
volume:

  • usr:org.fedoraproject.WorkStation:x86_64:24.7
  • usr:org.fedoraproject.WorkStation:x86_64:24.8
  • usr:org.fedoraproject.WorkStation:x86_64:24.9
  • usr:org.fedoraproject.WorkStation:x86_64:25beta
  • usr:org.mageia.Client:i386:39.3
  • usr:org.mageia.Client:i386:39.4
  • usr:org.mageia.Client:i386:39.6
  • usr:org.archlinux.Desktop:x86_64:302.7.8
  • usr:org.archlinux.Desktop:x86_64:302.7.9
  • usr:org.archlinux.Desktop:x86_64:302.7.10
  • root:revolution:org.fedoraproject.WorkStation:x86_64
  • root:testmachine:org.fedoraproject.WorkStation:x86_64
  • root:foo:org.mageia.Client:i386
  • root:bar:org.archlinux.Desktop:x86_64
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.1
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.4
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.5
  • runtime:org.gnome.GNOME3_22:x86_64:3.22.0
  • runtime:org.kde.KDE5_6:x86_64:5.6.0
  • framework:org.gnome.GNOME3_22:x86_64:3.22.0
  • framework:org.kde.KDE5_6:x86_64:5.6.0
  • app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133
  • app:org.libreoffice.LibreOffice:GNOME3_22:x86_64:166
  • app:org.mozilla.Firefox:GNOME3_20:x86_64:39
  • app:org.mozilla.Firefox:GNOME3_20:x86_64:40
  • home:lennart:1000:1000
  • home:hrundivbakshi:1001:1001

In the example above, we have three vendor operating systems
installed. All of them in three versions, and one even in a beta
version. We have four system instances around. Two of them of Fedora,
maybe one of them we usually boot from, the other we run for very
specific purposes in an OS container. We also have the runtimes for
two GNOME releases in multiple versions, plus one for KDE. Then, we
have the development trees for one version of KDE and GNOME around, as
well as two apps, that make use of two releases of the GNOME
runtime. Finally, we have the home directories of two users.

Now, with the name-spacing concepts we introduced above, we can
actually relatively freely mix and match apps and OSes, or develop
against specific frameworks in specific versions on any operating
system. It doesn’t matter if you booted your ArchLinux instance, or
your Fedora one, you can execute both LibreOffice and Firefox just
fine, because at execution time they get matched up with the right
runtime, and all of them are available from all the operating systems
you installed. You get the precise runtime that the upstream vendor of
Firefox/LibreOffice did their testing with. It doesn’t matter anymore
which distribution you run, and which distribution the vendor prefers.

Also, given that the user database is actually encoded in the
sub-volume list, it doesn’t matter which system you boot, the
distribution should be able to find your local users automatically,
without any configuration in /etc/passwd.

Building Blocks

With this naming scheme plus the way how we can combine them on
execution we already came quite far, but how do we actually get these
sub-volumes onto the final machines, and how do we update them? Well,
btrfs has a feature they call “send-and-receive”. It basically allows
you to “diff” two file system versions, and generate a binary
delta. You can generate these deltas on a developer’s machine and then
push them into the user’s system, and he’ll get the exact same
sub-volume too. This is how we envision installation and updating of
operating systems, applications, runtimes, frameworks. At installation
time, we simply deserialize an initial send-and-receive delta into
our btrfs volume, and later, when a new version is released we just
add in the few bits that are new, by dropping in another
send-and-receive delta under a new sub-volume name. And we do it
exactly the same for the OS itself, for a runtime, a framework or an
app. There’s no technical distinction anymore. The underlying
operation for installing apps, runtime, frameworks, vendor OSes, as well
as the operation for updating them is done the exact same way for all.

Of course, keeping multiple full /usr trees around sounds like an
awful lot of waste, after all they will contain a lot of very similar
data, since a lot of resources are shared between distributions,
frameworks and runtimes. However, thankfully btrfs actually is able to
de-duplicate this for us. If we add in a new app snapshot, this simply
adds in the new files that changed. Moreover different runtimes and
operating systems might actually end up sharing the same tree.

Even though the example above focuses primarily on the end-user,
desktop side of things, the concept is also extremely powerful in
server scenarios. For example, it is easy to build your own usr
trees and deliver them to your hosts using this scheme. The usr
sub-volumes are supposed to be something that administrators can put
together. After deserializing them into a couple of hosts, you can
trivially instantiate them as OS containers there, simply by adding a
new root sub-volume for each instance, referencing the usr tree you
just put together. Instantiating OS containers hence becomes as easy
as creating a new btrfs sub-volume. And you can still update the images
nicely, get fully double-buffered updates and everything.

And of course, this scheme also applies great to embedded
use-cases. Regardless if you build a TV, an IVI system or a phone: you
can put together you OS versions as usr trees, and then use
btrfs-send-and-receive facilities to deliver them to the systems, and
update them there.

Many people when they hear the word “btrfs” instantly reply with “is
it ready yet?”. Thankfully, most of the functionality we really need
here is strictly read-only. With the exception of the home
sub-volumes (see below) all snapshots are strictly read-only, and are
delivered as immutable vendor trees onto the devices. They never are
changed. Even if btrfs might still be immature, for this kind of
read-only logic it should be more than good enough.

Note that this scheme also enables doing fat systems: for example,
an installer image could include a Fedora version compiled for x86-64,
one for i386, one for ARM, all in the same btrfs volume. Due to btrfs’
de-duplication they will share as much as possible, and when the image
is booted up the right sub-volume is automatically picked. Something
similar of course applies to the apps too!

This also allows us to implement something that we like to call
Operating-System-As-A-Virus. Installing a new system is little more
than:

  • Creating a new GPT partition table
  • Adding an EFI System Partition (FAT) to it
  • Adding a new btrfs volume to it
  • Deserializing a single usr sub-volume into the btrfs volume
  • Installing a boot loader into the EFI System Partition
  • Rebooting

Now, since the only real vendor data you need is the usr sub-volume,
you can trivially duplicate this onto any block device you want. Let’s
say you are a happy Fedora user, and you want to provide a friend with
his own installation of this awesome system, all on a USB stick. All
you have to do for this is do the steps above, using your installed
usr tree as source to copy. And there you go! And you don’t have to
be afraid that any of your personal data is copied too, as the usr
sub-volume is the exact version your vendor provided you with. Or with
other words: there’s no distinction anymore between installer images
and installed systems. It’s all the same. Installation becomes
replication, not more. Live-CDs and installed systems can be fully
identical.

Note that in this design apps are actually developed against a single,
very specific runtime, that contains all libraries it can link against
(including a specific glibc version!). Any library that is not
included in the runtime the developer picked must be included in the
app itself. This is similar how apps on Android declare one very
specific Android version they are developed against. This greatly
simplifies application installation, as there’s no dependency hell:
each app pulls in one runtime, and the app is actually free to pick
which one, as you can have multiple installed, though only one is used
by each app.

Also note that operating systems built this way will never see
“half-updated” systems, as it is common when a system is updated using
RPM/dpkg. When updating the system the code will either run the old or
the new version, but it will never see part of the old files and part
of the new files. This is the same for apps, runtimes, and frameworks,
too.

Where We Are Now

We are currently working on a lot of the groundwork necessary for
this. This scheme relies on the ability to monopolize the
vendor OS resources in /usr, which is the key of what I described in
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems
a few weeks back. Then, of course, for the full desktop app concept we
need a strong sandbox, that does more than just hiding files from the
file system view. After all with an app concept like the above the
primary interfacing between the executed desktop apps and the rest of the
system is via IPC (which is why we work on kdbus and teach it all
kinds of sand-boxing features), and the kernel itself. Harald Hoyer has
started working on generating the btrfs send-and-receive images based
on Fedora.

Getting to the full scheme will take a while. Currently we have many
of the building blocks ready, but some major items are missing. For
example, we push quite a few problems into btrfs, that other solutions
try to solve in user space. One of them is actually
signing/verification of images. The btrfs maintainers are working on
adding this to the code base, but currently nothing exists. This
functionality is essential though to come to a fully verified system
where a trust chain exists all the way from the firmware to the
apps. Also, to make the home sub-volume scheme fully workable we
actually need encrypted sub-volumes, so that the sub-volume’s
pass-phrase can be used for authenticating users in PAM. This doesn’t
exist either.

Working towards this scheme is a gradual process. Many of the steps we
require for this are useful outside of the grand scheme though, which
means we can slowly work towards the goal, and our users can already
take benefit of what we are working on as we go.

Also, and most importantly, this is not really a departure from
traditional operating systems:

Each app, each OS and each app sees a traditional Unix hierarchy with
/usr, /home, /opt, /var, /etc. It executes in an environment that is
pretty much identical to how it would be run on traditional systems.

There’s no need to fully move to a system that uses only btrfs and
follows strictly this sub-volume scheme. For example, we intend to
provide implicit support for systems that are installed on ext4 or
xfs, or that are put together with traditional packaging tools such as
RPM or dpkg: if the the user tries to install a
runtime/app/framework/os image on a system that doesn’t use btrfs so
far, it can just create a loop-back btrfs image in /var, and push the
data into that. Even us developers will run our stuff like this for a
while, after all this new scheme is not particularly useful for highly
individualized systems, and we developers usually tend to run
systems like that.

Also note that this in no way a departure from packaging systems like
RPM or DEB. Even if the new scheme we propose is used for installing
and updating a specific system, it is RPM/DEB that is used to put
together the vendor OS tree initially. Hence, even in this scheme
RPM/DEB are highly relevant, though not strictly as an end-user tool
anymore, but as a build tool.

So Let’s Summarize Again What We Propose

  • We want a unified scheme, how we can install and update OS images,
    user apps, runtimes and frameworks.

  • We want a unified scheme how you can relatively freely mix OS
    images, apps, runtimes and frameworks on the same system.

  • We want a fully trusted system, where cryptographic verification of
    all executed code can be done, all the way to the firmware, as
    standard feature of the system.

  • We want to allow app vendors to write their programs against very
    specific frameworks, under the knowledge that they will end up being
    executed with the exact same set of libraries chosen.

  • We want to allow parallel installation of multiple OSes and versions
    of them, multiple runtimes in multiple versions, as well as multiple
    frameworks in multiple versions. And of course, multiple apps in
    multiple versions.

  • We want everything double buffered (or actually n-fold buffered), to
    ensure we can reliably update/rollback versions, in particular to
    safely do automatic updates.

  • We want a system where updating a runtime, OS, framework, or OS
    container is as simple as adding in a new snapshot and restarting
    the runtime/OS/framework/OS container.

  • We want a system where we can easily instantiate a number of OS
    instances from a single vendor tree, with zero difference for doing
    this on order to be able to boot it on bare metal/VM or as a
    container.

  • We want to enable Linux to have an open scheme that people can use
    to build app markets and similar schemes, not restricted to a
    specific vendor.

Final Words

I’ll be talking about this at LinuxCon Europe in October. I originally
intended to discuss this at the Linux Plumbers Conference (which I
assumed was the right forum for this kind of major plumbing level
improvement), and at linux.conf.au, but there was no interest in my
session submissions there…

Of course this is all work in progress. These are our current ideas we
are working towards. As we progress we will likely change a number of
things. For example, the precise naming of the sub-volumes might look
very different in the end.

Of course, we are developers of the systemd project. Implementing this
scheme is not just a job for the systemd developers. This is a
reinvention how distributions work, and hence needs great support from
the distributions. We really hope we can trigger some interest by
publishing this proposal now, to get the distributions on board. This
after all is explicitly not supposed to be a solution for one specific
project and one specific vendor product, we care about making this
open, and solving it for the generic case, without cutting corners.

If you have any questions about this, you know how you can reach us
(IRC, mail, G+, …).

The future is going to be awesome!

Revisiting How We Put Together Linux Systems

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html

In a previous blog story I discussed
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems,
I now want to take the opportunity to explain a bit where we want to
take this with
systemd in the
longer run, and what we want to build out of it. This is going to be a
longer story, so better grab a cold bottle of
Club Mate before you start
reading.
Traditional Linux distributions are built around packaging systems
like RPM or dpkg, and an organization model where upstream developers
and downstream packagers are relatively clearly separated: an upstream
developer writes code, and puts it somewhere online, in a tarball. A
packager than grabs it and turns it into RPMs/DEBs. The user then
grabs these RPMs/DEBs and installs them locally on the system. For a
variety of uses this is a fantastic scheme: users have a large
selection of readily packaged software available, in mostly uniform
packaging, from a single source they can trust. In this scheme the
distribution vets all software it packages, and as long as the user
trusts the distribution all should be good. The distribution takes the
responsibility of ensuring the software is not malicious, of timely
fixing security problems and helping the user if something is wrong.
Upstream Projects
However, this scheme also has a number of problems, and doesn’t fit
many use-cases of our software particularly well. Let’s have a look at
the problems of this scheme for many upstreams:

Upstream software vendors are fully dependent on downstream
distributions to package their stuff. It’s the downstream
distribution that decides on schedules, packaging details, and how
to handle support. Often upstream vendors want much faster release
cycles then the downstream distributions follow.

Realistic testing is extremely unreliable and next to
impossible. Since the end-user can run a variety of different
package versions together, and expects the software he runs to just
work on any combination, the test matrix explodes. If upstream tests
its version on distribution X release Y, then there’s no guarantee
that that’s the precise combination of packages that the end user
will eventually run. In fact, it is very unlikely that the end user
will, since most distributions probably updated a number of
libraries the package relies on by the time the package ends up being
made available to the user. The fact that each package can be
individually updated by the user, and each user can combine library
versions, plug-ins and executables relatively freely, results in a high
risk of something going wrong.

Since there are so many different distributions in so many different
versions around, if upstream tries to build and test software for
them it needs to do so for a large number of distributions, which is
a massive effort.

The distributions are actually quite different in many ways. In
fact, they are different in a lot of the most basic
functionality. For example, the path where to put x86-64 libraries
is different on Fedora and Debian derived systems..

Developing software for a number of distributions and versions is
hard: if you want to do it, you need to actually install them, each
one of them, manually, and then build your software for each.

Since most downstream distributions have strict licensing and
trademark requirements (and rightly so), any kind of closed source
software (or otherwise non-free) does not fit into this scheme at
all.

This all together makes it really hard for many upstreams to work
nicely with the current way how Linux works. Often they try to improve
the situation for them, for example by bundling libraries, to make
their test and build matrices smaller.
System Vendors
The toolbox approach of classic Linux distributions is fantastic for
people who want to put together their individual system, nicely
adjusted to exactly what they need. However, this is not really how
many of today’s Linux systems are built, installed or updated. If you
build any kind of embedded device, a server system, or even user
systems, you frequently do your work based on complete system images,
that are linearly versioned. You build these images somewhere, and
then you replicate them atomically to a larger number of systems. On
these systems, you don’t install or remove packages, you get a defined
set of files, and besides installing or updating the system there are
no ways how to change the set of tools you get.
The current Linux distributions are not particularly good at providing
for this major use-case of Linux. Their strict focus on individual
packages as well as package managers as end-user install and update
tool is incompatible with what many system vendors want.
Users
The classic Linux distribution scheme is frequently not what end users
want, either. Many users are used to app markets like Android, Windows
or iOS/Mac have. Markets are a platform that doesn’t package, build or
maintain software like distributions do, but simply allows users to
quickly find and download the software they need, with the app vendor
responsible for keeping the app updated, secured, and all that on the
vendor’s release cycle. Users tend to be impatient. They want their
software quickly, and the fine distinction between trusting a single
distribution or a myriad of app developers individually is usually not
important for them. The companies behind the marketplaces usually try
to improve this trust problem by providing sand-boxing technologies: as
a replacement for the distribution that audits, vets, builds and
packages the software and thus allows users to trust it to a certain
level, these vendors try to find technical solutions to ensure that
the software they offer for download can’t be malicious.
Existing Approaches To Fix These Problems
Now, all the issues pointed out above are not new, and there are
sometimes quite successful attempts to do something about it. Ubuntu
Apps, Docker, Software Collections, ChromeOS, CoreOS all fix part of
this problem set, usually with a strict focus on one facet of Linux
systems. For example, Ubuntu Apps focus strictly on end user (desktop)
applications, and don’t care about how we built/update/install the OS
itself, or containers. Docker OTOH focuses on containers only, and
doesn’t care about end-user apps. Software Collections tries to focus
on the development environments. ChromeOS focuses on the OS itself,
but only for end-user devices. CoreOS also focuses on the OS, but
only for server systems.
The approaches they find are usually good at specific things, and use
a variety of different technologies, on different layers. However,
none of these projects tried to fix this problems in a generic way,
for all uses, right in the core components of the OS itself.
Linux has come to tremendous successes because its kernel is so
generic: you can build supercomputers and tiny embedded devices out of
it. It’s time we come up with a basic, reusable scheme how to solve
the problem set described above, that is equally generic.
What We Want
The systemd cabal (Kay Sievers, Harald Hoyer, Daniel Mack, Tom
Gundersen, David Herrmann, and yours truly) recently met in Berlin
about all these things, and tried to come up with a scheme that is
somewhat simple, but tries to solve the issues generically, for all
use-cases, as part of the systemd project. All that in a way that is
somewhat compatible with the current scheme of distributions, to allow
a slow, gradual adoption. Also, and that’s something one cannot stress
enough: the toolbox scheme of classic Linux distributions is
actually a good one, and for many cases the right one. However, we
need to make sure we make distributions relevant again for all
use-cases, not just those of highly individualized systems.
Anyway, so let’s summarize what we are trying to do:

We want an efficient way that allows vendors to package their
software (regardless if just an app, or the whole OS) directly for
the end user, and know the precise combination of libraries and
packages it will operate with.

We want to allow end users and administrators to install these
packages on their systems, regardless which distribution they have
installed on it.

We want a unified solution that ultimately can cover updates for
full systems, OS containers, end user apps, programming ABIs, and
more. These updates shall be double-buffered, (at least). This is an
absolute necessity if we want to prepare the ground for operating
systems that manage themselves, that can update safely without
administrator involvement.

We want our images to be trustable (i.e. signed). In fact we want a
fully trustable OS, with images that can be verified by a full
trust chain from the firmware (EFI SecureBoot!), through the boot loader, through the
kernel, and initrd. Cryptographically secure verification of the
code we execute is relevant on the desktop (like ChromeOS does), but
also for apps, for embedded devices and even on servers (in a post-Snowden
world, in particular).

What We Propose
So much about the set of problems, and what we are trying to do. So,
now, let’s discuss the technical bits we came up with:
The scheme we propose is built around the variety of concepts of btrfs
and Linux file system name-spacing. btrfs at this point already has a
large number of features that fit neatly in our concept, and the
maintainers are busy working on a couple of others we want to
eventually make use of.
As first part of our proposal we make heavy use of btrfs sub-volumes and
introduce a clear naming scheme for them. We name snapshots like this:

usr:<vendorid>:<architecture>:<version> — This refers to a full
vendor operating system tree. It’s basically a /usr tree (and no
other directories), in a specific version, with everything you need to boot
it up inside it. The <vendorid> field is replaced by some vendor
identifier, maybe a scheme like
org.fedoraproject.FedoraWorkstation. The <architecture> field
specifies a CPU architecture the OS is designed for, for example
x86-64. The <version> field specifies a specific OS version, for
example 23.4. An example sub-volume name could hence look like this:
usr:org.fedoraproject.FedoraWorkstation:x86_64:23.4

root:<name>:<vendorid>:<architecture> — This refers to an
instance of an operating system. Its basically a root directory,
containing primarily /etc and /var (but possibly more). Sub-volumes
of this type do not contain a populated /usr tree though. The
<name> field refers to some instance name (maybe the host name of
the instance). The other fields are defined as above. An example
sub-volume name is
root:revolution:org.fedoraproject.FedoraWorkstation:x86_64.

runtime:<vendorid>:<architecture>:<version> — This refers to a
vendor runtime. A runtime here is supposed to be a set of
libraries and other resources that are needed to run apps (for the
concept of apps see below), all in a /usr tree. In this regard this
is very similar to the usr sub-volumes explained above, however,
while a usr sub-volume is a full OS and contains everything
necessary to boot, a runtime is really only a set of
libraries. You cannot boot it, but you can run apps with it. An
example sub-volume name is: runtime:org.gnome.GNOME3_20:x86_64:3.20.1

framework:<vendorid>:<architecture>:<version> — This is very
similar to a vendor runtime, as described above, it contains just a
/usr tree, but goes one step further: it additionally contains all
development headers, compilers and build tools, that allow
developing against a specific runtime. For each runtime there should
be a framework. When you develop against a specific framework in a
specific architecture, then the resulting app will be compatible
with the runtime of the same vendor ID and architecture. Example:
framework:org.gnome.GNOME3_20:x86_64:3.20.1

app:<vendorid>:<runtime>:<architecture>:<version> — This
encapsulates an application bundle. It contains a tree that at
runtime is mounted to /opt/<vendorid>, and contains all the
application’s resources. The <vendorid> could be a string like
org.libreoffice.LibreOffice, the <runtime> refers to one the
vendor id of one specific runtime the application is built for, for
example org.gnome.GNOME3_20:3.20.1. The <architecture> and
<version> refer to the architecture the application is built for,
and of course its version. Example:
app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133

home:<user>:<uid>:<gid> — This sub-volume shall refer to the home
directory of the specific user. The <user> field contains the user
name, the <uid> and <gid> fields the numeric Unix UIDs and GIDs
of the user. The idea here is that in the long run the list of
sub-volumes is sufficient as a user database (but see
below). Example: home:lennart:1000:1000.

btrfs partitions that adhere to this naming scheme should be clearly
identifiable. It is our intention to introduce a new GPT partition type
ID for this.
How To Use It
After we introduced this naming scheme let’s see what we can build of
this:

When booting up a system we mount the root directory from one of the
root sub-volumes, and then mount /usr from a matching usr
sub-volume. Matching here means it carries the same <vendor-id>
and <architecture>. Of course, by default we should pick the
matching usr sub-volume with the newest version by default.

When we boot up an OS container, we do exactly the same as the when
we boot up a regular system: we simply combine a usr sub-volume
with a root sub-volume.

When we enumerate the system’s users we simply go through the
list of home snapshots.

When a user authenticates and logs in we mount his home
directory from his snapshot.

When an app is run, we set up a new file system name-space, mount the
app sub-volume to /opt/<vendorid>/, and the appropriate runtime
sub-volume the app picked to /usr, as well as the user’s
/home/$USER to its place.

When a developer wants to develop against a specific runtime he
installs the right framework, and then temporarily transitions into
a name space where /usris mounted from the framework sub-volume, and
/home/$USER from his own home directory. In this name space he then
runs his build commands. He can build in multiple name spaces at the
same time, if he intends to builds software for multiple runtimes or
architectures at the same time.

Instantiating a new system or OS container (which is exactly the same
in this scheme) just consists of creating a new appropriately named
root sub-volume. Completely naturally you can share one vendor OS
copy in one specific version with a multitude of container instances.
Everything is double-buffered (or actually, n-fold-buffered), because
usr, runtime, framework, app sub-volumes can exist in multiple
versions. Of course, by default the execution logic should always pick
the newest release of each sub-volume, but it is up to the user keep
multiple versions around, and possibly execute older versions, if he
desires to do so. In fact, like on ChromeOS this could even be handled
automatically: if a system fails to boot with a newer snapshot, the
boot loader can automatically revert back to an older version of the
OS.
An Example
Note that in result this allows installing not only multiple end-user
applications into the same btrfs volume, but also multiple operating
systems, multiple system instances, multiple runtimes, multiple
frameworks. Or to spell this out in an example:
Let’s say Fedora, Mageia and ArchLinux all implement this scheme,
and provide ready-made end-user images. Also, the GNOME, KDE, SDL
projects all define a runtime+framework to develop against. Finally,
both LibreOffice and Firefox provide their stuff according to this
scheme. You can now trivially install of these into the same btrfs
volume:

usr:org.fedoraproject.WorkStation:x86_64:24.7
usr:org.fedoraproject.WorkStation:x86_64:24.8
usr:org.fedoraproject.WorkStation:x86_64:24.9
usr:org.fedoraproject.WorkStation:x86_64:25beta
usr:org.mageia.Client:i386:39.3
usr:org.mageia.Client:i386:39.4
usr:org.mageia.Client:i386:39.6
usr:org.archlinux.Desktop:x86_64:302.7.8
usr:org.archlinux.Desktop:x86_64:302.7.9
usr:org.archlinux.Desktop:x86_64:302.7.10
root:revolution:org.fedoraproject.WorkStation:x86_64
root:testmachine:org.fedoraproject.WorkStation:x86_64
root:foo:org.mageia.Client:i386
root:bar:org.archlinux.Desktop:x86_64
runtime:org.gnome.GNOME3_20:x86_64:3.20.1
runtime:org.gnome.GNOME3_20:x86_64:3.20.4
runtime:org.gnome.GNOME3_20:x86_64:3.20.5
runtime:org.gnome.GNOME3_22:x86_64:3.22.0
runtime:org.kde.KDE5_6:x86_64:5.6.0
framework:org.gnome.GNOME3_22:x86_64:3.22.0
framework:org.kde.KDE5_6:x86_64:5.6.0
app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133
app:org.libreoffice.LibreOffice:GNOME3_22:x86_64:166
app:org.mozilla.Firefox:GNOME3_20:x86_64:39
app:org.mozilla.Firefox:GNOME3_20:x86_64:40
home:lennart:1000:1000
home:hrundivbakshi:1001:1001

In the example above, we have three vendor operating systems
installed. All of them in three versions, and one even in a beta
version. We have four system instances around. Two of them of Fedora,
maybe one of them we usually boot from, the other we run for very
specific purposes in an OS container. We also have the runtimes for
two GNOME releases in multiple versions, plus one for KDE. Then, we
have the development trees for one version of KDE and GNOME around, as
well as two apps, that make use of two releases of the GNOME
runtime. Finally, we have the home directories of two users.
Now, with the name-spacing concepts we introduced above, we can
actually relatively freely mix and match apps and OSes, or develop
against specific frameworks in specific versions on any operating
system. It doesn’t matter if you booted your ArchLinux instance, or
your Fedora one, you can execute both LibreOffice and Firefox just
fine, because at execution time they get matched up with the right
runtime, and all of them are available from all the operating systems
you installed. You get the precise runtime that the upstream vendor of
Firefox/LibreOffice did their testing with. It doesn’t matter anymore
which distribution you run, and which distribution the vendor prefers.
Also, given that the user database is actually encoded in the
sub-volume list, it doesn’t matter which system you boot, the
distribution should be able to find your local users automatically,
without any configuration in /etc/passwd.
Building Blocks
With this naming scheme plus the way how we can combine them on
execution we already came quite far, but how do we actually get these
sub-volumes onto the final machines, and how do we update them? Well,
btrfs has a feature they call “send-and-receive”. It basically allows
you to “diff” two file system versions, and generate a binary
delta. You can generate these deltas on a developer’s machine and then
push them into the user’s system, and he’ll get the exact same
sub-volume too. This is how we envision installation and updating of
operating systems, applications, runtimes, frameworks. At installation
time, we simply deserialize an initial send-and-receive delta into
our btrfs volume, and later, when a new version is released we just
add in the few bits that are new, by dropping in another
send-and-receive delta under a new sub-volume name. And we do it
exactly the same for the OS itself, for a runtime, a framework or an
app. There’s no technical distinction anymore. The underlying
operation for installing apps, runtime, frameworks, vendor OSes, as well
as the operation for updating them is done the exact same way for all.
Of course, keeping multiple full /usr trees around sounds like an
awful lot of waste, after all they will contain a lot of very similar
data, since a lot of resources are shared between distributions,
frameworks and runtimes. However, thankfully btrfs actually is able to
de-duplicate this for us. If we add in a new app snapshot, this simply
adds in the new files that changed. Moreover different runtimes and
operating systems might actually end up sharing the same tree.
Even though the example above focuses primarily on the end-user,
desktop side of things, the concept is also extremely powerful in
server scenarios. For example, it is easy to build your own usr
trees and deliver them to your hosts using this scheme. The usr
sub-volumes are supposed to be something that administrators can put
together. After deserializing them into a couple of hosts, you can
trivially instantiate them as OS containers there, simply by adding a
new root sub-volume for each instance, referencing the usr tree you
just put together. Instantiating OS containers hence becomes as easy
as creating a new btrfs sub-volume. And you can still update the images
nicely, get fully double-buffered updates and everything.
And of course, this scheme also applies great to embedded
use-cases. Regardless if you build a TV, an IVI system or a phone: you
can put together you OS versions as usr trees, and then use
btrfs-send-and-receive facilities to deliver them to the systems, and
update them there.
Many people when they hear the word “btrfs” instantly reply with “is
it ready yet?”. Thankfully, most of the functionality we really need
here is strictly read-only. With the exception of the home
sub-volumes (see below) all snapshots are strictly read-only, and are
delivered as immutable vendor trees onto the devices. They never are
changed. Even if btrfs might still be immature, for this kind of
read-only logic it should be more than good enough.
Note that this scheme also enables doing fat systems: for example,
an installer image could include a Fedora version compiled for x86-64,
one for i386, one for ARM, all in the same btrfs volume. Due to btrfs’
de-duplication they will share as much as possible, and when the image
is booted up the right sub-volume is automatically picked. Something
similar of course applies to the apps too!
This also allows us to implement something that we like to call
Operating-System-As-A-Virus. Installing a new system is little more
than:

Creating a new GPT partition table
Adding an EFI System Partition (FAT) to it
Adding a new btrfs volume to it
Deserializing a single usr sub-volume into the btrfs volume
Installing a boot loader into the EFI System Partition
Rebooting

Now, since the only real vendor data you need is the usr sub-volume,
you can trivially duplicate this onto any block device you want. Let’s
say you are a happy Fedora user, and you want to provide a friend with
his own installation of this awesome system, all on a USB stick. All
you have to do for this is do the steps above, using your installed
usr tree as source to copy. And there you go! And you don’t have to
be afraid that any of your personal data is copied too, as the usr
sub-volume is the exact version your vendor provided you with. Or with
other words: there’s no distinction anymore between installer images
and installed systems. It’s all the same. Installation becomes
replication, not more. Live-CDs and installed systems can be fully
identical.
Note that in this design apps are actually developed against a single,
very specific runtime, that contains all libraries it can link against
(including a specific glibc version!). Any library that is not
included in the runtime the developer picked must be included in the
app itself. This is similar how apps on Android declare one very
specific Android version they are developed against. This greatly
simplifies application installation, as there’s no dependency hell:
each app pulls in one runtime, and the app is actually free to pick
which one, as you can have multiple installed, though only one is used
by each app.
Also note that operating systems built this way will never see
“half-updated” systems, as it is common when a system is updated using
RPM/dpkg. When updating the system the code will either run the old or
the new version, but it will never see part of the old files and part
of the new files. This is the same for apps, runtimes, and frameworks,
too.
Where We Are Now
We are currently working on a lot of the groundwork necessary for
this. This scheme relies on the ability to monopolize the
vendor OS resources in /usr, which is the key of what I described in
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems
a few weeks back. Then, of course, for the full desktop app concept we
need a strong sandbox, that does more than just hiding files from the
file system view. After all with an app concept like the above the
primary interfacing between the executed desktop apps and the rest of the
system is via IPC (which is why we work on kdbus and teach it all
kinds of sand-boxing features), and the kernel itself. Harald Hoyer has
started working on generating the btrfs send-and-receive images based
on Fedora.
Getting to the full scheme will take a while. Currently we have many
of the building blocks ready, but some major items are missing. For
example, we push quite a few problems into btrfs, that other solutions
try to solve in user space. One of them is actually
signing/verification of images. The btrfs maintainers are working on
adding this to the code base, but currently nothing exists. This
functionality is essential though to come to a fully verified system
where a trust chain exists all the way from the firmware to the
apps. Also, to make the home sub-volume scheme fully workable we
actually need encrypted sub-volumes, so that the sub-volume’s
pass-phrase can be used for authenticating users in PAM. This doesn’t
exist either.
Working towards this scheme is a gradual process. Many of the steps we
require for this are useful outside of the grand scheme though, which
means we can slowly work towards the goal, and our users can already
take benefit of what we are working on as we go.
Also, and most importantly, this is not really a departure from
traditional operating systems:
Each app, each OS and each app sees a traditional Unix hierarchy with
/usr, /home, /opt, /var, /etc. It executes in an environment that is
pretty much identical to how it would be run on traditional systems.
There’s no need to fully move to a system that uses only btrfs and
follows strictly this sub-volume scheme. For example, we intend to
provide implicit support for systems that are installed on ext4 or
xfs, or that are put together with traditional packaging tools such as
RPM or dpkg: if the the user tries to install a
runtime/app/framework/os image on a system that doesn’t use btrfs so
far, it can just create a loop-back btrfs image in /var, and push the
data into that. Even us developers will run our stuff like this for a
while, after all this new scheme is not particularly useful for highly
individualized systems, and we developers usually tend to run
systems like that.
Also note that this in no way a departure from packaging systems like
RPM or DEB. Even if the new scheme we propose is used for installing
and updating a specific system, it is RPM/DEB that is used to put
together the vendor OS tree initially. Hence, even in this scheme
RPM/DEB are highly relevant, though not strictly as an end-user tool
anymore, but as a build tool.
So Let’s Summarize Again What We Propose

We want a unified scheme, how we can install and update OS images,
user apps, runtimes and frameworks.

We want a unified scheme how you can relatively freely mix OS
images, apps, runtimes and frameworks on the same system.

We want a fully trusted system, where cryptographic verification of
all executed code can be done, all the way to the firmware, as
standard feature of the system.

We want to allow app vendors to write their programs against very
specific frameworks, under the knowledge that they will end up being
executed with the exact same set of libraries chosen.

We want to allow parallel installation of multiple OSes and versions
of them, multiple runtimes in multiple versions, as well as multiple
frameworks in multiple versions. And of course, multiple apps in
multiple versions.

We want everything double buffered (or actually n-fold buffered), to
ensure we can reliably update/rollback versions, in particular to
safely do automatic updates.

We want a system where updating a runtime, OS, framework, or OS
container is as simple as adding in a new snapshot and restarting
the runtime/OS/framework/OS container.

We want a system where we can easily instantiate a number of OS
instances from a single vendor tree, with zero difference for doing
this on order to be able to boot it on bare metal/VM or as a
container.

We want to enable Linux to have an open scheme that people can use
to build app markets and similar schemes, not restricted to a
specific vendor.

Final Words
I’ll be talking about this at LinuxCon Europe in October. I originally
intended to discuss this at the Linux Plumbers Conference (which I
assumed was the right forum for this kind of major plumbing level
improvement), and at linux.conf.au, but there was no interest in my
session submissions there…
Of course this is all work in progress. These are our current ideas we
are working towards. As we progress we will likely change a number of
things. For example, the precise naming of the sub-volumes might look
very different in the end.
Of course, we are developers of the systemd project. Implementing this
scheme is not just a job for the systemd developers. This is a
reinvention how distributions work, and hence needs great support from
the distributions. We really hope we can trigger some interest by
publishing this proposal now, to get the distributions on board. This
after all is explicitly not supposed to be a solution for one specific
project and one specific vendor product, we care about making this
open, and solving it for the generic case, without cutting corners.
If you have any questions about this, you know how you can reach us
(IRC, mail, G+, …).
The future is going to be awesome!

Revisiting How We Put Together Linux Systems

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html

In a previous blog story I discussed
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems,
I now want to take the opportunity to explain a bit where we want to
take this with
systemd in the
longer run, and what we want to build out of it. This is going to be a
longer story, so better grab a cold bottle of
Club Mate before you start
reading.

Traditional Linux distributions are built around packaging systems
like RPM or dpkg, and an organization model where upstream developers
and downstream packagers are relatively clearly separated: an upstream
developer writes code, and puts it somewhere online, in a tarball. A
packager than grabs it and turns it into RPMs/DEBs. The user then
grabs these RPMs/DEBs and installs them locally on the system. For a
variety of uses this is a fantastic scheme: users have a large
selection of readily packaged software available, in mostly uniform
packaging, from a single source they can trust. In this scheme the
distribution vets all software it packages, and as long as the user
trusts the distribution all should be good. The distribution takes the
responsibility of ensuring the software is not malicious, of timely
fixing security problems and helping the user if something is wrong.

Upstream Projects

However, this scheme also has a number of problems, and doesn’t fit
many use-cases of our software particularly well. Let’s have a look at
the problems of this scheme for many upstreams:

  • Upstream software vendors are fully dependent on downstream
    distributions to package their stuff. It’s the downstream
    distribution that decides on schedules, packaging details, and how
    to handle support. Often upstream vendors want much faster release
    cycles then the downstream distributions follow.

  • Realistic testing is extremely unreliable and next to
    impossible. Since the end-user can run a variety of different
    package versions together, and expects the software he runs to just
    work on any combination, the test matrix explodes. If upstream tests
    its version on distribution X release Y, then there’s no guarantee
    that that’s the precise combination of packages that the end user
    will eventually run. In fact, it is very unlikely that the end user
    will, since most distributions probably updated a number of
    libraries the package relies on by the time the package ends up being
    made available to the user. The fact that each package can be
    individually updated by the user, and each user can combine library
    versions, plug-ins and executables relatively freely, results in a high
    risk of something going wrong.

  • Since there are so many different distributions in so many different
    versions around, if upstream tries to build and test software for
    them it needs to do so for a large number of distributions, which is
    a massive effort.

  • The distributions are actually quite different in many ways. In
    fact, they are different in a lot of the most basic
    functionality. For example, the path where to put x86-64 libraries
    is different on Fedora and Debian derived systems..

  • Developing software for a number of distributions and versions is
    hard: if you want to do it, you need to actually install them, each
    one of them, manually, and then build your software for each.

  • Since most downstream distributions have strict licensing and
    trademark requirements (and rightly so), any kind of closed source
    software (or otherwise non-free) does not fit into this scheme at
    all.

This all together makes it really hard for many upstreams to work
nicely with the current way how Linux works. Often they try to improve
the situation for them, for example by bundling libraries, to make
their test and build matrices smaller.

System Vendors

The toolbox approach of classic Linux distributions is fantastic for
people who want to put together their individual system, nicely
adjusted to exactly what they need. However, this is not really how
many of today’s Linux systems are built, installed or updated. If you
build any kind of embedded device, a server system, or even user
systems, you frequently do your work based on complete system images,
that are linearly versioned. You build these images somewhere, and
then you replicate them atomically to a larger number of systems. On
these systems, you don’t install or remove packages, you get a defined
set of files, and besides installing or updating the system there are
no ways how to change the set of tools you get.

The current Linux distributions are not particularly good at providing
for this major use-case of Linux. Their strict focus on individual
packages as well as package managers as end-user install and update
tool is incompatible with what many system vendors want.

Users

The classic Linux distribution scheme is frequently not what end users
want, either. Many users are used to app markets like Android, Windows
or iOS/Mac have. Markets are a platform that doesn’t package, build or
maintain software like distributions do, but simply allows users to
quickly find and download the software they need, with the app vendor
responsible for keeping the app updated, secured, and all that on the
vendor’s release cycle. Users tend to be impatient. They want their
software quickly, and the fine distinction between trusting a single
distribution or a myriad of app developers individually is usually not
important for them. The companies behind the marketplaces usually try
to improve this trust problem by providing sand-boxing technologies: as
a replacement for the distribution that audits, vets, builds and
packages the software and thus allows users to trust it to a certain
level, these vendors try to find technical solutions to ensure that
the software they offer for download can’t be malicious.

Existing Approaches To Fix These Problems

Now, all the issues pointed out above are not new, and there are
sometimes quite successful attempts to do something about it. Ubuntu
Apps, Docker, Software Collections, ChromeOS, CoreOS all fix part of
this problem set, usually with a strict focus on one facet of Linux
systems. For example, Ubuntu Apps focus strictly on end user (desktop)
applications, and don’t care about how we built/update/install the OS
itself, or containers. Docker OTOH focuses on containers only, and
doesn’t care about end-user apps. Software Collections tries to focus
on the development environments. ChromeOS focuses on the OS itself,
but only for end-user devices. CoreOS also focuses on the OS, but
only for server systems.

The approaches they find are usually good at specific things, and use
a variety of different technologies, on different layers. However,
none of these projects tried to fix this problems in a generic way,
for all uses, right in the core components of the OS itself.

Linux has come to tremendous successes because its kernel is so
generic: you can build supercomputers and tiny embedded devices out of
it. It’s time we come up with a basic, reusable scheme how to solve
the problem set described above, that is equally generic.

What We Want

The systemd cabal (Kay Sievers, Harald Hoyer, Daniel Mack, Tom
Gundersen, David Herrmann, and yours truly) recently met in Berlin
about all these things, and tried to come up with a scheme that is
somewhat simple, but tries to solve the issues generically, for all
use-cases, as part of the systemd project. All that in a way that is
somewhat compatible with the current scheme of distributions, to allow
a slow, gradual adoption. Also, and that’s something one cannot stress
enough: the toolbox scheme of classic Linux distributions is
actually a good one, and for many cases the right one. However, we
need to make sure we make distributions relevant again for all
use-cases, not just those of highly individualized systems.

Anyway, so let’s summarize what we are trying to do:

  • We want an efficient way that allows vendors to package their
    software (regardless if just an app, or the whole OS) directly for
    the end user, and know the precise combination of libraries and
    packages it will operate with.

  • We want to allow end users and administrators to install these
    packages on their systems, regardless which distribution they have
    installed on it.

  • We want a unified solution that ultimately can cover updates for
    full systems, OS containers, end user apps, programming ABIs, and
    more. These updates shall be double-buffered, (at least). This is an
    absolute necessity if we want to prepare the ground for operating
    systems that manage themselves, that can update safely without
    administrator involvement.

  • We want our images to be trustable (i.e. signed). In fact we want a
    fully trustable OS, with images that can be verified by a full
    trust chain from the firmware (EFI SecureBoot!), through the boot loader, through the
    kernel, and initrd. Cryptographically secure verification of the
    code we execute is relevant on the desktop (like ChromeOS does), but
    also for apps, for embedded devices and even on servers (in a post-Snowden
    world, in particular).

What We Propose

So much about the set of problems, and what we are trying to do. So,
now, let’s discuss the technical bits we came up with:

The scheme we propose is built around the variety of concepts of btrfs
and Linux file system name-spacing. btrfs at this point already has a
large number of features that fit neatly in our concept, and the
maintainers are busy working on a couple of others we want to
eventually make use of.

As first part of our proposal we make heavy use of btrfs sub-volumes and
introduce a clear naming scheme for them. We name snapshots like this:

  • usr:<vendorid>:<architecture>:<version> — This refers to a full
    vendor operating system tree. It’s basically a /usr tree (and no
    other directories), in a specific version, with everything you need to boot
    it up inside it. The <vendorid> field is replaced by some vendor
    identifier, maybe a scheme like
    org.fedoraproject.FedoraWorkstation. The <architecture> field
    specifies a CPU architecture the OS is designed for, for example
    x86-64. The <version> field specifies a specific OS version, for
    example 23.4. An example sub-volume name could hence look like this:
    usr:org.fedoraproject.FedoraWorkstation:x86_64:23.4

  • root:<name>:<vendorid>:<architecture> — This refers to an
    instance of an operating system. Its basically a root directory,
    containing primarily /etc and /var (but possibly more). Sub-volumes
    of this type do not contain a populated /usr tree though. The
    <name> field refers to some instance name (maybe the host name of
    the instance). The other fields are defined as above. An example
    sub-volume name is
    root:revolution:org.fedoraproject.FedoraWorkstation:x86_64.

  • runtime:<vendorid>:<architecture>:<version> — This refers to a
    vendor runtime. A runtime here is supposed to be a set of
    libraries and other resources that are needed to run apps (for the
    concept of apps see below), all in a /usr tree. In this regard this
    is very similar to the usr sub-volumes explained above, however,
    while a usr sub-volume is a full OS and contains everything
    necessary to boot, a runtime is really only a set of
    libraries. You cannot boot it, but you can run apps with it. An
    example sub-volume name is: runtime:org.gnome.GNOME3_20:x86_64:3.20.1

  • framework:<vendorid>:<architecture>:<version> — This is very
    similar to a vendor runtime, as described above, it contains just a
    /usr tree, but goes one step further: it additionally contains all
    development headers, compilers and build tools, that allow
    developing against a specific runtime. For each runtime there should
    be a framework. When you develop against a specific framework in a
    specific architecture, then the resulting app will be compatible
    with the runtime of the same vendor ID and architecture. Example:
    framework:org.gnome.GNOME3_20:x86_64:3.20.1

  • app:<vendorid>:<runtime>:<architecture>:<version> — This
    encapsulates an application bundle. It contains a tree that at
    runtime is mounted to /opt/<vendorid>, and contains all the
    application’s resources. The <vendorid> could be a string like
    org.libreoffice.LibreOffice, the <runtime> refers to one the
    vendor id of one specific runtime the application is built for, for
    example org.gnome.GNOME3_20:3.20.1. The <architecture> and
    <version> refer to the architecture the application is built for,
    and of course its version. Example:
    app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133

  • home:<user>:<uid>:<gid> — This sub-volume shall refer to the home
    directory of the specific user. The <user> field contains the user
    name, the <uid> and <gid> fields the numeric Unix UIDs and GIDs
    of the user. The idea here is that in the long run the list of
    sub-volumes is sufficient as a user database (but see
    below). Example: home:lennart:1000:1000.

btrfs partitions that adhere to this naming scheme should be clearly
identifiable. It is our intention to introduce a new GPT partition type
ID for this.

How To Use It

After we introduced this naming scheme let’s see what we can build of
this:

  • When booting up a system we mount the root directory from one of the
    root sub-volumes, and then mount /usr from a matching usr
    sub-volume. Matching here means it carries the same <vendor-id>
    and <architecture>. Of course, by default we should pick the
    matching usr sub-volume with the newest version by default.

  • When we boot up an OS container, we do exactly the same as the when
    we boot up a regular system: we simply combine a usr sub-volume
    with a root sub-volume.

  • When we enumerate the system’s users we simply go through the
    list of home snapshots.

  • When a user authenticates and logs in we mount his home
    directory from his snapshot.

  • When an app is run, we set up a new file system name-space, mount the
    app sub-volume to /opt/<vendorid>/, and the appropriate runtime
    sub-volume the app picked to /usr, as well as the user’s
    /home/$USER to its place.

  • When a developer wants to develop against a specific runtime he
    installs the right framework, and then temporarily transitions into
    a name space where /usris mounted from the framework sub-volume, and
    /home/$USER from his own home directory. In this name space he then
    runs his build commands. He can build in multiple name spaces at the
    same time, if he intends to builds software for multiple runtimes or
    architectures at the same time.

Instantiating a new system or OS container (which is exactly the same
in this scheme) just consists of creating a new appropriately named
root sub-volume. Completely naturally you can share one vendor OS
copy in one specific version with a multitude of container instances.

Everything is double-buffered (or actually, n-fold-buffered), because
usr, runtime, framework, app sub-volumes can exist in multiple
versions. Of course, by default the execution logic should always pick
the newest release of each sub-volume, but it is up to the user keep
multiple versions around, and possibly execute older versions, if he
desires to do so. In fact, like on ChromeOS this could even be handled
automatically: if a system fails to boot with a newer snapshot, the
boot loader can automatically revert back to an older version of the
OS.

An Example

Note that in result this allows installing not only multiple end-user
applications into the same btrfs volume, but also multiple operating
systems, multiple system instances, multiple runtimes, multiple
frameworks. Or to spell this out in an example:

Let’s say Fedora, Mageia and ArchLinux all implement this scheme,
and provide ready-made end-user images. Also, the GNOME, KDE, SDL
projects all define a runtime+framework to develop against. Finally,
both LibreOffice and Firefox provide their stuff according to this
scheme. You can now trivially install of these into the same btrfs
volume:

  • usr:org.fedoraproject.WorkStation:x86_64:24.7
  • usr:org.fedoraproject.WorkStation:x86_64:24.8
  • usr:org.fedoraproject.WorkStation:x86_64:24.9
  • usr:org.fedoraproject.WorkStation:x86_64:25beta
  • usr:org.mageia.Client:i386:39.3
  • usr:org.mageia.Client:i386:39.4
  • usr:org.mageia.Client:i386:39.6
  • usr:org.archlinux.Desktop:x86_64:302.7.8
  • usr:org.archlinux.Desktop:x86_64:302.7.9
  • usr:org.archlinux.Desktop:x86_64:302.7.10
  • root:revolution:org.fedoraproject.WorkStation:x86_64
  • root:testmachine:org.fedoraproject.WorkStation:x86_64
  • root:foo:org.mageia.Client:i386
  • root:bar:org.archlinux.Desktop:x86_64
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.1
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.4
  • runtime:org.gnome.GNOME3_20:x86_64:3.20.5
  • runtime:org.gnome.GNOME3_22:x86_64:3.22.0
  • runtime:org.kde.KDE5_6:x86_64:5.6.0
  • framework:org.gnome.GNOME3_22:x86_64:3.22.0
  • framework:org.kde.KDE5_6:x86_64:5.6.0
  • app:org.libreoffice.LibreOffice:GNOME3_20:x86_64:133
  • app:org.libreoffice.LibreOffice:GNOME3_22:x86_64:166
  • app:org.mozilla.Firefox:GNOME3_20:x86_64:39
  • app:org.mozilla.Firefox:GNOME3_20:x86_64:40
  • home:lennart:1000:1000
  • home:hrundivbakshi:1001:1001

In the example above, we have three vendor operating systems
installed. All of them in three versions, and one even in a beta
version. We have four system instances around. Two of them of Fedora,
maybe one of them we usually boot from, the other we run for very
specific purposes in an OS container. We also have the runtimes for
two GNOME releases in multiple versions, plus one for KDE. Then, we
have the development trees for one version of KDE and GNOME around, as
well as two apps, that make use of two releases of the GNOME
runtime. Finally, we have the home directories of two users.

Now, with the name-spacing concepts we introduced above, we can
actually relatively freely mix and match apps and OSes, or develop
against specific frameworks in specific versions on any operating
system. It doesn’t matter if you booted your ArchLinux instance, or
your Fedora one, you can execute both LibreOffice and Firefox just
fine, because at execution time they get matched up with the right
runtime, and all of them are available from all the operating systems
you installed. You get the precise runtime that the upstream vendor of
Firefox/LibreOffice did their testing with. It doesn’t matter anymore
which distribution you run, and which distribution the vendor prefers.

Also, given that the user database is actually encoded in the
sub-volume list, it doesn’t matter which system you boot, the
distribution should be able to find your local users automatically,
without any configuration in /etc/passwd.

Building Blocks

With this naming scheme plus the way how we can combine them on
execution we already came quite far, but how do we actually get these
sub-volumes onto the final machines, and how do we update them? Well,
btrfs has a feature they call “send-and-receive”. It basically allows
you to “diff” two file system versions, and generate a binary
delta. You can generate these deltas on a developer’s machine and then
push them into the user’s system, and he’ll get the exact same
sub-volume too. This is how we envision installation and updating of
operating systems, applications, runtimes, frameworks. At installation
time, we simply deserialize an initial send-and-receive delta into
our btrfs volume, and later, when a new version is released we just
add in the few bits that are new, by dropping in another
send-and-receive delta under a new sub-volume name. And we do it
exactly the same for the OS itself, for a runtime, a framework or an
app. There’s no technical distinction anymore. The underlying
operation for installing apps, runtime, frameworks, vendor OSes, as well
as the operation for updating them is done the exact same way for all.

Of course, keeping multiple full /usr trees around sounds like an
awful lot of waste, after all they will contain a lot of very similar
data, since a lot of resources are shared between distributions,
frameworks and runtimes. However, thankfully btrfs actually is able to
de-duplicate this for us. If we add in a new app snapshot, this simply
adds in the new files that changed. Moreover different runtimes and
operating systems might actually end up sharing the same tree.

Even though the example above focuses primarily on the end-user,
desktop side of things, the concept is also extremely powerful in
server scenarios. For example, it is easy to build your own usr
trees and deliver them to your hosts using this scheme. The usr
sub-volumes are supposed to be something that administrators can put
together. After deserializing them into a couple of hosts, you can
trivially instantiate them as OS containers there, simply by adding a
new root sub-volume for each instance, referencing the usr tree you
just put together. Instantiating OS containers hence becomes as easy
as creating a new btrfs sub-volume. And you can still update the images
nicely, get fully double-buffered updates and everything.

And of course, this scheme also applies great to embedded
use-cases. Regardless if you build a TV, an IVI system or a phone: you
can put together you OS versions as usr trees, and then use
btrfs-send-and-receive facilities to deliver them to the systems, and
update them there.

Many people when they hear the word “btrfs” instantly reply with “is
it ready yet?”. Thankfully, most of the functionality we really need
here is strictly read-only. With the exception of the home
sub-volumes (see below) all snapshots are strictly read-only, and are
delivered as immutable vendor trees onto the devices. They never are
changed. Even if btrfs might still be immature, for this kind of
read-only logic it should be more than good enough.

Note that this scheme also enables doing fat systems: for example,
an installer image could include a Fedora version compiled for x86-64,
one for i386, one for ARM, all in the same btrfs volume. Due to btrfs’
de-duplication they will share as much as possible, and when the image
is booted up the right sub-volume is automatically picked. Something
similar of course applies to the apps too!

This also allows us to implement something that we like to call
Operating-System-As-A-Virus. Installing a new system is little more
than:

  • Creating a new GPT partition table
  • Adding an EFI System Partition (FAT) to it
  • Adding a new btrfs volume to it
  • Deserializing a single usr sub-volume into the btrfs volume
  • Installing a boot loader into the EFI System Partition
  • Rebooting

Now, since the only real vendor data you need is the usr sub-volume,
you can trivially duplicate this onto any block device you want. Let’s
say you are a happy Fedora user, and you want to provide a friend with
his own installation of this awesome system, all on a USB stick. All
you have to do for this is do the steps above, using your installed
usr tree as source to copy. And there you go! And you don’t have to
be afraid that any of your personal data is copied too, as the usr
sub-volume is the exact version your vendor provided you with. Or with
other words: there’s no distinction anymore between installer images
and installed systems. It’s all the same. Installation becomes
replication, not more. Live-CDs and installed systems can be fully
identical.

Note that in this design apps are actually developed against a single,
very specific runtime, that contains all libraries it can link against
(including a specific glibc version!). Any library that is not
included in the runtime the developer picked must be included in the
app itself. This is similar how apps on Android declare one very
specific Android version they are developed against. This greatly
simplifies application installation, as there’s no dependency hell:
each app pulls in one runtime, and the app is actually free to pick
which one, as you can have multiple installed, though only one is used
by each app.

Also note that operating systems built this way will never see
“half-updated” systems, as it is common when a system is updated using
RPM/dpkg. When updating the system the code will either run the old or
the new version, but it will never see part of the old files and part
of the new files. This is the same for apps, runtimes, and frameworks,
too.

Where We Are Now

We are currently working on a lot of the groundwork necessary for
this. This scheme relies on the ability to monopolize the
vendor OS resources in /usr, which is the key of what I described in
Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems
a few weeks back. Then, of course, for the full desktop app concept we
need a strong sandbox, that does more than just hiding files from the
file system view. After all with an app concept like the above the
primary interfacing between the executed desktop apps and the rest of the
system is via IPC (which is why we work on kdbus and teach it all
kinds of sand-boxing features), and the kernel itself. Harald Hoyer has
started working on generating the btrfs send-and-receive images based
on Fedora.

Getting to the full scheme will take a while. Currently we have many
of the building blocks ready, but some major items are missing. For
example, we push quite a few problems into btrfs, that other solutions
try to solve in user space. One of them is actually
signing/verification of images. The btrfs maintainers are working on
adding this to the code base, but currently nothing exists. This
functionality is essential though to come to a fully verified system
where a trust chain exists all the way from the firmware to the
apps. Also, to make the home sub-volume scheme fully workable we
actually need encrypted sub-volumes, so that the sub-volume’s
pass-phrase can be used for authenticating users in PAM. This doesn’t
exist either.

Working towards this scheme is a gradual process. Many of the steps we
require for this are useful outside of the grand scheme though, which
means we can slowly work towards the goal, and our users can already
take benefit of what we are working on as we go.

Also, and most importantly, this is not really a departure from
traditional operating systems:

Each app, each OS and each app sees a traditional Unix hierarchy with
/usr, /home, /opt, /var, /etc. It executes in an environment that is
pretty much identical to how it would be run on traditional systems.

There’s no need to fully move to a system that uses only btrfs and
follows strictly this sub-volume scheme. For example, we intend to
provide implicit support for systems that are installed on ext4 or
xfs, or that are put together with traditional packaging tools such as
RPM or dpkg: if the the user tries to install a
runtime/app/framework/os image on a system that doesn’t use btrfs so
far, it can just create a loop-back btrfs image in /var, and push the
data into that. Even us developers will run our stuff like this for a
while, after all this new scheme is not particularly useful for highly
individualized systems, and we developers usually tend to run
systems like that.

Also note that this in no way a departure from packaging systems like
RPM or DEB. Even if the new scheme we propose is used for installing
and updating a specific system, it is RPM/DEB that is used to put
together the vendor OS tree initially. Hence, even in this scheme
RPM/DEB are highly relevant, though not strictly as an end-user tool
anymore, but as a build tool.

So Let’s Summarize Again What We Propose

  • We want a unified scheme, how we can install and update OS images,
    user apps, runtimes and frameworks.

  • We want a unified scheme how you can relatively freely mix OS
    images, apps, runtimes and frameworks on the same system.

  • We want a fully trusted system, where cryptographic verification of
    all executed code can be done, all the way to the firmware, as
    standard feature of the system.

  • We want to allow app vendors to write their programs against very
    specific frameworks, under the knowledge that they will end up being
    executed with the exact same set of libraries chosen.

  • We want to allow parallel installation of multiple OSes and versions
    of them, multiple runtimes in multiple versions, as well as multiple
    frameworks in multiple versions. And of course, multiple apps in
    multiple versions.

  • We want everything double buffered (or actually n-fold buffered), to
    ensure we can reliably update/rollback versions, in particular to
    safely do automatic updates.

  • We want a system where updating a runtime, OS, framework, or OS
    container is as simple as adding in a new snapshot and restarting
    the runtime/OS/framework/OS container.

  • We want a system where we can easily instantiate a number of OS
    instances from a single vendor tree, with zero difference for doing
    this on order to be able to boot it on bare metal/VM or as a
    container.

  • We want to enable Linux to have an open scheme that people can use
    to build app markets and similar schemes, not restricted to a
    specific vendor.

Final Words

I’ll be talking about this at LinuxCon Europe in October. I originally
intended to discuss this at the Linux Plumbers Conference (which I
assumed was the right forum for this kind of major plumbing level
improvement), and at linux.conf.au, but there was no interest in my
session submissions there…

Of course this is all work in progress. These are our current ideas we
are working towards. As we progress we will likely change a number of
things. For example, the precise naming of the sub-volumes might look
very different in the end.

Of course, we are developers of the systemd project. Implementing this
scheme is not just a job for the systemd developers. This is a
reinvention how distributions work, and hence needs great support from
the distributions. We really hope we can trigger some interest by
publishing this proposal now, to get the distributions on board. This
after all is explicitly not supposed to be a solution for one specific
project and one specific vendor product, we care about making this
open, and solving it for the generic case, without cutting corners.

If you have any questions about this, you know how you can reach us
(IRC, mail, G+, …).

The future is going to be awesome!

Announcing Amazon SES Endpoint in Europe!

Post Syndicated from Mahae Koh original http://sesblog.amazon.com/post/Tx3PU3POPUV0CC8/Announcing-Amazon-SES-Endpoint-in-Europe

The Amazon Simple Email Service team is very excited to announce that SES has expanded into the EU (Ireland) AWS Region. SES has always been a global service, but until today, it was hosted only in the US East (N. Virginia) Region. SES’s expansion into Europe represents the first step in a series of planned region expansions that we hope will provide an improved email sending experience.

Why does this matter to you? Email sending applications based in Europe can achieve reduced network latency by using SES’s EU endpoint instead of making cross-region SES calls to US East.

Here’s what you need to know about using SES in this new region:

Production access: You will need to request production access for each region you intend to use. Your existing production access status will remain the same in the US East Region.

Quotas: Your sending quota will apply to each region separately. This means that, while your existing quota will remain the same in the US East Region, you will need to separately apply for quota increases in the EU Region as necessary.

Domain/email address verification: Your existing domain and email address verification statuses will only be valid for the US East Region. You will have to verify your domains and email addresses separately for the EU Region. This also means that you will need to separately set up DKIM and feedback notifications for your sending identities in the EU Region.

Suppression list: If you need to remove an address from the suppression list, you will only need to remove it once and the removal will apply to all Regions.

SMTP credentials: Your SMTP credentials are global. You may use the same credentials for all Regions.

For more information, see the SES Developer Guide.

Happy sending, and thanks for using SES!

Announcing Amazon SES Endpoint in Europe!

Post Syndicated from Mahae Koh original http://sesblog.amazon.com/post/Tx3PU3POPUV0CC8/Announcing-Amazon-SES-Endpoint-in-Europe

The Amazon Simple Email Service team is very excited to announce that SES has expanded into the EU (Ireland) AWS Region. SES has always been a global service, but until today, it was hosted only in the US East (N. Virginia) Region. SES’s expansion into Europe represents the first step in a series of planned region expansions that we hope will provide an improved email sending experience.

Why does this matter to you? Email sending applications based in Europe can achieve reduced network latency by using SES’s EU endpoint instead of making cross-region SES calls to US East.

Here’s what you need to know about using SES in this new region:

Production access: You will need to request production access for each region you intend to use. Your existing production access status will remain the same in the US East Region.

Quotas: Your sending quota will apply to each region separately. This means that, while your existing quota will remain the same in the US East Region, you will need to separately apply for quota increases in the EU Region as necessary.

Domain/email address verification: Your existing domain and email address verification statuses will only be valid for the US East Region. You will have to verify your domains and email addresses separately for the EU Region. This also means that you will need to separately set up DKIM and feedback notifications for your sending identities in the EU Region.

Suppression list: If you need to remove an address from the suppression list, you will only need to remove it once and the removal will apply to all Regions.

SMTP credentials: Your SMTP credentials are global. You may use the same credentials for all Regions.

For more information, see the SES Developer Guide.

Happy sending, and thanks for using SES!

FaiFCast Release, and Submit to FOSDEM Legal & Policy Issues DevRoom

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2011/12/16/faif-fosdem.html

Today Karen Sandler and I
released Episode 0x1E of
the Free as in Freedom oggcast
(available
in ogg
and mp3
formats). There are two important things discussed on that oggcast that
I want to draw your attention to:

Submit a proposal for the Legal & Policy Issues DevRoom
CFP

Tom
Marble
, Richard
Fontana
, Karen Sandler, and I are coordinating
the Legal
and Policy Issues DevRoom

at FOSDEM 2012.
The Call
for Participation for the DevRoom is now available
. I’d like to
ask anyone reading this blog post who has an interest in policy and/or
legal issues related to software freedom to submit a talk by Friday 30
December 2011, by
emailing <[email protected]>.

We only have about six slots for speakers (it’s a one-day DevRoom), so
we won’t be able to accept all proposals. I just wanted to let everyone
know that so you don’t flame me if you submit and get rejected.
Meanwhile, note that our goal is to avoid the “this is what
copyrights, trademarks and patents are” introductory talks. Our
focus is on complex issues for those already informed about the basics.
We really felt that the level of discourse about legal and policy issues
at software freedom conferences needs to rise.

There are, of course, plenty of secret membership
clubs 0, even some with their own
private conferences, where these sorts of important issues are discussed.
I personally seek to move high-level policy discussion and debate out of
the secret “old-boys” club backrooms and into a public space
where the entire software freedom community can discuss openly important
legal and policy questions in the community. I hope this DevRoom is a
first step in that direction!

Issues & Questions List for the Software Freedom Non-Profits Debate

I’ve made
reference
recently
to debates about the value of non-profit organizations for software
freedom projects.
In FaiFCast 0x1E,
Karen and I discuss the debate in depth. As part of that, as you’ll see
in the show notes, I’ve made a list of issues that I think were fully
conflated during the recent debates. I can’t spare the time to opine in
detail on them right now (although Karen and I do a bit of that in the
oggcast itself), but I did want to copy the list over here in my blog,
mainly to list them out as issues worth thinking about in a software
freedom non-profit:

Should a non-profit home decide what technical infrastructure is
used for a software freedom project? And if so, what should it be?

If the non-profit doesn’t provide technological services, should
non-profits allow their projects to rely on for-profits for
technological or other services?

Should a non-profit home set political and social positions that
must be followed by the projects? If so, how strictly should they be
enforced?

Should copyrights be held by the non-profit home of the project, or
with the developers, or a mix of the two?

Should the non-profit dictate licensing requirements on the
project? If so, how many licenses and which licenses are
acceptable?

Should a non-profit dictate strict copyright provenance
requirements on their projects? If not, should the non-profit at least
provide guidelines and recommendations?

This list of questions is far from exhaustive, but I
think it’s a pretty good start.

0 Admittedly, I’ve got a
proverbial axe to grind about these secretive membership-only groups,
since, for nearly all of them, I’m persona non grata. My frustration
level in this reached a crescendo when, during a session at LinuxCon
Europe recently, I asked for the criteria to join one such private legal
issues discussions group, and I was told the criteria themselves were
secret. I pointed out to the coordinators of the forum that this wasn’t a
particularly Free Software friendly way to run a discussion group, and
they simply changed the subject. My hope is that this FOSDEM DevRoom can
be a catalyst to start a new discussion forum for legal and policy issues
related to software freedom that doesn’t have this problem.

BTW, just to clarify: I’m not talking
about FLOSS Foundations as
one of these secretive, members-only clubs. While the FLOSS Foundations
main mailing list is indeed invite-only, it’s very easy to join and the
only requirement is: “if you repost emails from this list
publicly, you’ll probably be taken off the mailing list”. There
is
no “Chatham
House Rule”
or other silly, unenforceable, and
spend-inordinate-amount-of-times-remembering-how-to-follow rules in
place for FLOSS Foundations, but such silly rulesets are now common with
these other secretive legal issues meeting groups.

Finally, I know I haven’t named publicly the members-only clubs I’m
talking about here, and that’s by design. This is the first time I’ve
mentioned them at all in my blog, and my hope is that they’ll change
their behaviors soon. I don’t want to publicly shame them by name until
I give them a bit more time to change their behaviors. Also, I don’t
want to inadvertently promote these fora either, since IMO their very
structure is flawed and community-unfriendly.

Update: Some
have claimed incorrectly
that the text in the footnote above somehow indicates my unwillingness to
follow the Chatham House Rule (CHR).
I refuted that
on identi.ca, noting that the text above doesn’t say that, and those who
think it does have simply misunderstood. My primary point (which I’ll now
state even more explicitly) is that CHR is difficult to follow,
particularly when it is mis-applied to a mailing list. CHR is designed
for meetings, which have a clear start time and a finish time. Mailing
lists aren’t meetings, so the behavior of CHR when applied to a mailing
list is often undefined.

I should furthermore note that people who have lived under CHR for a
series of meetings also have similar concerns as mine. For
example, Allison
Randal, who worked under CHR
on Project
Harmony
noted:

The group decided to adopt Chatham House Rule for our
discussions. … At first glance it seems
quite sensible: encourage open participation by being careful about what
you share publicly. But, after almost a year of working under it, I have
to say I’m not a big fan. It’s really quite awkward sometimes figuring out
what you can and can’t say publicly. I’m trying to follow it in this post,
but I’ve probably missed in spots. The simple rule is tricky to apply.

I agree with Allison.

Last Four FaiF Episodes

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2011/11/11/more-faif.html

Those of you that follow my blog have probably wondered we’re I’ve
been. Quite frankly, there is just so much work going on at Conservancy
that I have almost had no time to do anything but Conservancy work, eat
and sleep. My output on this blog and
on identi.ca surely shows that.

The one thing that I’ve kept up with
is the oggcast, Free as in
Freedom
that I co-host
with Karen Sandler, and which is
produced by Dan Lynch.

Since I last made a blog post here, Karen, Dan and I released four
oggcasts. I’ll discuss them here in reverse chronological order:

In Episode 0x1C, which was released today, we published Karen’s
interview with Adam Dingle
of Yorba. IMO (which is undoubtedly
biased), this episode is an important one since it relates to the
issues of non-profit organizations in our community who waiting in the
501(c)(3) application queue. This is a detailed and specific follow-up
to the issues that Karen and I discussed
on FaiF’s Episode
0x13
.

In Episode 0x1B, Karen and I discuss in some detail about the work that
we’ve been up to. Both Karen and I are full-time Executive Directors,
and the amount of work that job takes always seems insurmountable.
Although, after we recorded the episode, I somewhat embarrassingly
remembered
the Bush/Kerry
debate where George W. Bush kept saying his job as president is hard
work
. It’s certainly annoying when a chief executive goes on
and on about how hard his job is, so I apologize if I did a little too
much of that in Episode 0x1B.

In Episode 0x1A,
Karen and I discussed in detail Steve Jobs’ death and the various news
coverage about it. The subject is a bit old news now that I write this,
but I’m glad we did that episode, since it gave me an opportunity to say
everything I wanted to stay about Steve Jobs’ life and death.

In Episode 0x19, we
played Karen’s interview
with Jos Poortvliet,
discussed the identi.ca upgrade, and Karen discussed GNOME 3.2.

My plan is to at least keep the FaiF oggcast going, and I’m
even bugging
Fontana that he and I should start an oggcast too
. Beyond that, I
can’t necessarily commit to any other activities outside of that (and my
job at Conservancy and volunteer duties at FSF). BTW, I recently
attended a few conferences (both LinxCon Europe and the Summer of Code
Mentor Summit). At both of them, multiple folks asked me why I haven’t
been blogging more. I appreciate people’s interest in what I’m writing,
but at the moment, my day-job at Conservancy and volunteer work at FSF
has had to take absolute priority.

Based on the ebb and flow (yes, that’s the first time I’ve actually
used that phrase on my ebb.org blog 🙂 of the Free Software community
that I’ve gotten used to over the last decade and a half, I usually find
that things slow down in mid-December until mid-January. Since
Conservancy’s work is based on the needs of its Free Software projects,
I’ll likely be able to return a “normal” 50 hour work week
(instead of the 60-70 I’ve been doing lately) in December. Thus, I’ll
probably try to write some queued blog posts then to slowly push out
over the few months that follow.

Finally, I want to mention
that Conservancy
has an donation appeal
up on its website. I hope you’ll give
generously to support Conservancy’s work. On that, I’ll just briefly
mention my “hard work” again, to assure you
that donors to
Conservancy
definitely get their money’s worth when I’m on the job.
Since I’m on the topic of that, I also thank everyone who
has donated to FSF
and Conservancy over the
years. I’ve been fortunate to have worked full-time at both
organizations, and I appreciate the community that has supported all
that work over the years.

Kernel Hackers Panel

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/linuxcon-kernel-panel.html

At LinuxCon Europe/ELCE I had the chance to moderate the kernel hackers
panel with Linus Torvalds, Alan Cox, Paul McKenney and Thomas Gleixner on
stage
. I like to believe it went quite well, but check it out for yourself, as
a video recording is now available online:

For me personally I think the most notable topic covered was Control Groups,
and the clarification that they are something that is needed even though their
implementation right now is in many ways less than perfect. But in the end there is no
reasonable way around it, and much like SMP, technology that complicates things
substantially but is ultimately unavoidable.

Other videos from ELCE are online now, too.

Kernel Hackers Panel

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/linuxcon-kernel-panel.html

At LinuxCon Europe/ELCE I had the chance to moderate the kernel hackers
panel with Linus Torvalds, Alan Cox, Paul McKenney and Thomas Gleixner on
stage
. I like to believe it went quite well, but check it out for yourself, as
a video recording is now available online:

For me personally I think the most notable topic covered was Control Groups,
and the clarification that they are something that is needed even though their
implementation right now is in many ways less than perfect. But in the end there is no
reasonable way around it, and much like SMP, technology that complicates things
substantially but is ultimately unavoidable.

Other videos from ELCE are online now, too.

Kernel Hackers Panel

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/linuxcon-kernel-panel.html

At LinuxCon Europe/ELCE I had the chance to moderate the kernel hackers
panel with Linus Torvalds, Alan Cox, Paul McKenney and Thomas Gleixner on
stage
. I like to believe it went quite well, but check it out for yourself, as
a video recording is now available online:

For me personally I think the most notable topic covered was Control Groups,
and the clarification that they are something that is needed even though their
implementation right now is in many ways less than perfect. But in the end there is no
reasonable way around it, and much like SMP, technology that complicates things
substantially but is ultimately unavoidable.

Other videos from ELCE are online now, too.

Desktop Summit 2011

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2011/08/21/desktop-summit.html

I realize nearly ten days after the end of a conference is a bit late
to blog about it. However, I needed some time to recover my usual
workflow, having attended two conferences almost
back-to-back, OSCON 2011
and Desktop Summit. (The
strain of the back-to-back conferences, BTW, made it impossible for me
to
attend Linux
Con North America
2011, although I’ll be
at Linux
Con Europe
. I hope next year’s summer conference schedule is not so
tight.)

This was my first Desktop Summit, as I was unable to attend
the first one in
Grand Canaria two years ago
. I must admit, while it might be a bit
controversial to say so, that I felt the conference was still like two
co-located conferences rather than one conference. I got a chance to
speak to my KDE colleagues about various things, but I ended up mostly
attending GNOME talks and therefore felt more like I was at GUADEC than
at a Desktop Summit for most of the time.

The big exception to that, however, was in fact the primary reason I
was at Desktop Summit this year: to participate in a panel discussion
with Mark Shuttleworth and Michael Meeks
(who
gave the panel a quick one-sentence summary on his blog
). That was
plenary session and the room was filled with KDE and GNOME developers
alike, all of whom seemed very interested in the issue.

Photo of The CAA/CLA panel discussion at Desktop Summit 2011.

The panel format was slightly frustrating — primarily due to
Mark’s insistence that we all make very long open statements —
although Karen Sandler
nevertheless did a good job moderating it and framing the
discussion.

I get the impression most of the audience was already pretty well
informed about all of our positions, although I think I shocked some by
finally saying clearly in a public forum (other than identi.ca) that I
have been lobbying FSF to make copyright assignment for FSF-assigned
projects optional rather than mandatory. Nevertheless, we were cast
well into our three roles: Mark, who wants broad licensing control over
projects his company sponsors so he can control the assets (and possibly
sell them); Michael, who has faced so many troubles in the
OpenOffice.org/LibreOffice debacle that he believes inbound=outbound can
be The Only Way; and me, who believes that copyright assignment is
useful for non-profits willing to promise to do the public good to
enforce the GPL, but otherwise is a Bad Thing.

Lydia
tells me that the videos will be available eventually from Desktop
Summit
, and I’ll update this blog post when they are so folks can
watch the panel. I encourage everyone concerned about the issue of
rights transfers from individual developers to entities (be they via
copyright assignment or other broad CLA means) to watch the video once
it’s available. For the
moment, Jake Edge’s LWN
article about the panel
is a pretty good summary.

My favorite moment of the panel, though, was when Shuttleworth claimed
he was but a distant observer of Project Harmony. Karen, as moderator,
quickly pointed out that he was billed as Project Harmony’s originator
in the panel materials. It’s disturbing that Shuttleworth thinks he can
get away with such a claim: it’s a matter of public record,
that Amanda
Brock
(Canonical, Ltd.’s General Counsel) initiated Project Harmony,
led it for most of its early drafts, and then Canonical Ltd. paid Mark
Radcliffe (a
lawyer who
represents companies that violate the GPL
) to finish the drafting.
I suppose Shuttleworth’s claim is narrowly true (if misleading) since
his personal involvement as an individual was only
tangential, but his money and his staff were clearly central: even now,
it’s led by his employee, Allison Randal. If you run the company that
runs a project, it’s your project: after all, doesn’t that fit clearly
with Shuttleworth’s suppositions about why he should be entitled to be
the recipient of copyright assignments and broad CLAs in the first
place?

The rest of my time at Desktop Summit was more as an attendee than a
speaker. Since I’m not desktop or GUI developer by any means, I mostly
went to talks and learned what others had to teach. I was delighted,
however, that no less than six people came up to me and said they really
liked this blog. It’s always good to be told that something you put a
lot of volunteer work into is valuable to at least a few people, and
fortunately everyone on the Internet is famous to at least six
people. 🙂

Sponsored by the GNOME Foundation!

Meanwhile, I want to thank the GNOME Foundation for sponsoring my trip to
Desktop Summit 2011, as
they did last
year for GUADEC 2010
. Given my own work and background, I’m very
appreciative of a non-profit with limited resources providing travel
funding for conferences. It’s a big expense, and I’m thankful that the
GNOME Foundation has funded my trips to their annual conference.

BTW, while we await the videos from Desktop Summit, there’s some
“proof” you can see that I attended Desktop Summit, as
I appear
in the group photo
, although you’ll need
to view
the hi-res version and scroll to the lower right of the image, and find
me
. I’m in the second/third (depending on how you count) row back,
2-3 from the right, and two to the left
from Lydia Pintscher.

Finally, I did my best
to live dent from the
Desktop Summit 2011
. That might be of interest to some as well, for
example, if you want to dig back and see what folks said in some of the
talks I attended. There was also
a two threads
after the panel that may be of interest.

Post-Bilski Steps for Anti-Software-Patent Advocates

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2010/06/30/bilski.html

Lots of people are opining about
the USA
Supreme Court’s ruling in the Bilski case
. Yesterday, I participated
in
a oggcast
with the folks at SFLC
. In that oggcast, Dan Ravicher explained most
of the legal details of Bilski; I could never cover them as well as he
did, and I wouldn’t even try.

Anyway, as a non-lawyer worried about the policy questions, I’m pretty
much only concerned about those forward-looking policy questions.
However, to briefly look back at how our community responded to this
Bilski situation over the last 18 months: it seems similar to what
happened
while the Eldred
case
was working its way to the Supreme Court. In the months
preceding both Eldred and Bilski, there seemed to be a mass hypnosis that
the Supreme Court would actually change copyright law (Eldred) or patent
law (Bilski) to make it better for freedom of computer users.

In both cases, that didn’t happen. There was admittedly less of that
giddy optimism before Bilski as there was before Eldred, but the ultimate
outcome for computer users is roughly no different in both cases: as we
were with Eldred, we’re left back with the same policy situation we had
before Bilski ever started making its way through the various courts. As
near as I can tell from what I’ve learned, the entire “Bilski
thing” appears to be a no-op. In short, as before, the Patent
Office sometimes can and will deny applications that it determines are
only abstract ideas, and the Supreme Court has now confirmed that the
Patent Office can reject such an application if the Patent Office knows
an abstract idea when it sees it. Nothing has changed regarding most
patents that are granted every day, including those that read on software.
Those of us that oppose software patents continue to believe that software
algorithms are indeed merely abstract ideas and pure mathematics and
shouldn’t be patentable subject matter. The governmental powers still
seems to disagree with us, or, at least, just won’t comment on that
question.

Looking forward, my largest concern, from a policy
perspective, is that the “patent reform” crowd,
who claim to be the allies of the anti-software-patent folks,
will use this decision to declare that the system works.
Bilski’s patent was ultimately denied, but on grounds that leave us no
closer to abolishing software patents. Patent reformists will
say: Well, invalid patents get denied, leaving space for the valid
ones. Those valid ones, they will say, do and should include
lots of patents that read on software. But only the really good
ideas should be patented, they will insist.

We must not yield to the patent reformists, particularly at a time like
this. (BTW, be sure to read
RMS’ classic and still relevant essay,
Patent
Reform Is Not Enough
, if you haven’t already.)

Since Bilski has given us no new tools for abolishing software patents,
we must redouble efforts with tools we already have to mitigate the
threat patents pose to software freedom. Here are a few suggestions,
which I think are actually all implementable by the average developer,
to will keep up the fight against software patents, or at least,
mitigate their impact:

License your software using the
AGPLv3,
GPLv3,
LGPLv3,
or Apache-2.0.
Among the copyleft
licenses, AGPLv3
and GPLv3 offer the
best patent
protections; LGPLv3
offers the best among the weak copyleft
licenses; Apache
License 2.0
offers the best patent protections among the permissive
licenses. These are the licenses we should gravitate toward,
particularly since multiple companies with software patents are
regularly attacking Free Software. At least when such companies
contribute code to projects under these licenses, we know those
particular codebases will be safe from that particular company’s
patents.

Demand real patent licenses from companies, not mere
promises. Patent promises are not
enough0. The Free Software
community deserves to know it has real patent licenses from companies
that hold patents. At the very least, we should demand unilateral
patent licenses for all their patents perpetually for all
possible copylefted code (i.e., companies should grant, ahead of
time, the exact same license that the community would get if the
company had contributed to a yet-to-exist GPLv3’d
codebase)1. Note
further that some companies, that claim to be part of the
FLOSS community, haven’t even given the
(inadequate-but-better-than-nothing) patent promises.
For example,
BlackDuck holds a
patent related to FLOSS
, but
despite saying
they would consider at least a patent promise
, have failed to do
even that minimal effort.

Support organizations/efforts that work to oppose and end
software patents. In particular, be sure that the efforts
you support are not merely “patent reform” efforts hidden
behind anti-software patent rhetoric. Here are a few initiatives that
I’ve recently seen doing work regarding complete abolition of software
patents. I suggest you support them (with your time or dollars):

End Software
Patents
(a project of
FSF),
FFII
(European-specific)
, and
APRIL
(France-specific)
.

Write your legislators. This never hurts. In the
USA, it’s unlikely we can convince Congress to change patent law,
because there are just too many lobbying dollars from those big
patent-holding companies (e.g., the same ones that wrote
those nasty
amicus
briefs
in Bilski). But, writing your Senators and Congresspeople once a year
to remind them of your opposition patents that read on software simply
can’t hurt, and may theoretically help a tiny bit. Now would be a good
time to do it, since you can mention how the Bilski decision convinced
you there’s a need for legislative abolition of software patents.
Meanwhile, remember, it’s even better if you show up at political
debates during election season and ask these candidates to oppose
software patents!

Explain to your colleagues why software patents should be
abolished, particularly if you work in computing. Software
patent abolition is actually a broad spectrum issue across the
computing industry. Only big and powerful companies benefit from
software patents. The little guy — even the little guy
proprietary developer — is hurt by software patents.
Even if you can’t convince your colleagues who write proprietary
software that they should switch to writing Free Software,
you can instead convince them that software patents
are bad for them personally and for their chances to succeed in
software. Share the film,
Patent
Absurdity
, with them and then discuss the issue with them
after they’ve viewed it. Blog, tweet, dent, and the like about the
issue regularly.

(added 2010-07-01 on tmarble‘s
suggestion) Avoid products from pro-software-patent
companies. This is tough to do, and it’s why I didn’t call
for an all-out boycott. Most companies that make computers are
pro-software-patent, so it’s actually tough to buy a computer (or even
components for one) without buying from a pro-software-patent company.
However, avoiding the companies who are most aggressive with patent
aggression is easy: starting with avoiding Apple products is a good
first step (there are plenty of other reasons to avoid Apple anyway).
Microsoft would be next on the list, since they specifically use
software patents to attack FLOSS projects. Those are likely the big
two to avoid, but always remember that all large companies with
proprietary software products actively enforce patents, even if they
don’t file lawsuits. In other words, go with the little guy if you
can; it’s more likely to be a patent-free zone.

If you have a good idea, publish it and make sure the great
idea is well described in code comments and documentation, and that
everything is well archived by date. I put this one last on
my list, because it’s more of a help for the software patent
reformists than it is for the software patent abolitionists.
Nevertheless, sometimes, patents will get in the way of Free Software,
and it will be good if there is strong prior art showing that the idea
was already thought of, implemented, and put out into the world before
the patent was filed. But, fact is,
the “valid”
software patents with no prior art are a bigger threat to software
freedom
. The stronger the patent, the worst the threat, because
it’s more likely to be innovative, new technology that we want to
implement in Free Software.

I sat and thought of what else I could add to this list that
individuals can do to help abolish software patents. I was sad that
these were the only five six things that
I could collect, but that’s all the more reason to do
these five six
things in earnest. The battle for software freedom for all users is not
one we’ll win in our lifetimes. It’s also possible abolition of
software patents will take a generation as well. Those of us that seek
this outcome must be prepared for patience and lifelong, diligent work
so that the right outcome happens, eventually.

0 Update: I
was asked
for a longer write up on software patent licenses as
compared to mere “promises”.
Unfortunately, I
don’t have one, so the best I was able to offer

was the interview I did
on Linux Outlaws, Episode 102, about Microsoft’s patent
promise
. I’ve also added a TODO to write something up more completely
on this particular issue.

1 I am not
leaving my permissively-license-preferring friends out of this issue
without careful consideration. Specifically, I just don’t think it’s
practical or even fair to ask companies to license their patents for
all permissively-licensed code, since that would be the same as
licensing to everyone, including their proprietary software
competitors. An ahead-of-time perpetual license to practice the
teachings of all the company’s patents under AGPLv3 basically makes
sure that code that’s eternally Free Software will also eternally be
patent-licensed from that company, even if the company never
contributes to the AGPLv3’d codebase. Anyone trying to make
proprietary code that infringed the patent wouldn’t have benefit of
the license; only Free Software users, distributors and modifiers
would have the benefit. If a company supports copyleft generally,
then there is no legitimate reason for the company to refuse such a
broad license for copyleft distributions and deployments.

Das Leben der Anderen

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/leben-der-anderen.html

German movies are usually not my thing – I don’t like the topics, I don’t
like the scripting, I don’t like the acting, I don’t like the actors, I don’t
like the drama and I don’t like the humor. (Ok, they usually lack humor entirely, so there’s not much not to like of the humor.)

However, there’s now a notable exception: Das Leben der Anderen (The Lives of
Others) is a very good film, one that I really like. It’s an absorbing drama, the scripting is good and the acting is fine. There’s a good reason that
it has won the European Movie Award (Best Film) and is one of the top
contenders for next years’ Oscar (at least the foreign language one).

If you get the chance to see this movie, do it! It’s worth it.

Das Leben der Anderen

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/leben-der-anderen.html

German movies are usually not my thing – I don’t like the topics, I don’t
like the scripting, I don’t like the acting, I don’t like the actors, I don’t
like the drama and I don’t like the humor. (Ok, they usually lack humor entirely, so there’s not much not to like of the humor.)

However, there’s now a notable exception: Das Leben der Anderen (The Lives of
Others
) is a very good film, one that I really like. It’s an absorbing drama, the scripting is good and the acting is fine. There’s a good reason that
it has won the European Movie Award (Best Film) and is one of the top
contenders for next years’ Oscar (at least the foreign language one).

If you get the chance to see this movie, do it! It’s worth it.

Unique Eyebrows

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/photos/unique-eyebrows.html

Dear American People,

I guess you’ll find businesses selling unique eyebrow
designs only in god’s own country:

Unique Eyebrows

And what does “unique” mean? Do their customers get two
different designs for their two eyebrows? – What a bargain!

Groucho Marx’ greasepaint eyebrows are unique, in a way. Maybe that’s what they are selling?

Confused,
     Lennart (a worried European)

Unique Eyebrows

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/photos/unique-eyebrows.html

Dear American People,

I guess you’ll find businesses selling unique eyebrow
designs
only in god’s own country:

Unique Eyebrows

And what does “unique” mean? Do their customers get two
different designs for their two eyebrows? – What a bargain!

Groucho Marx’ greasepaint eyebrows are unique, in a way. Maybe that’s what they are selling?

Confused,
     Lennart (a worried European)