When your apt-mirror is always downloading

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2008/01/24/apt-mirror-2.html

When I started building our apt-mirror, I ran into a problem: the
machine was throttled against ubuntu.com’s servers, but I had completed
much of the download (which took weeks to get multiple distributions).
I really wanted to roll out the solution quickly, particularly because
the service from the remote servers was worse than ever due to the
throttling that the mirroring created. But, with the mirror incomplete,
I couldn’t so easily make available incomplete repositories.

The solution was to simply let apache redirect users on to the real
servers if the mirror doesn’t have the file. The first order of
business for that is to rewrite and redirect URLs when files aren’t
found. This is a straightforward Apache configuration:

RewriteEngine on
RewriteLogLevel 0
RewriteCond %{REQUEST_FILENAME} !^/cgi/
RewriteCond /var/spool/apt-mirror/mirror/archive.ubuntu.com%{REQUEST_FILENAME} !-F
RewriteCond /var/spool/apt-mirror/mirror/archive.ubuntu.com%{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !(Packages|Sources).bz2$
RewriteCond %{REQUEST_URI} !/index.[^/]*$ [NC]
RewriteRule ^(http://%{HTTP_HOST})?/(.*) http://91.189.88.45/$2 [P]

Note a few things there:

I have to hard-code an IP number, because as I mentioned in
the last
post on this subject
, I’ve faked out DNS
for archive.ubuntu.com and other sites I’m mirroring. (Note:
this has the unfortunate side-effect that I can’t easily take advantage
of round-robin DNS on the other side.)

I avoid taking Packages.bz2 from the other site, because
apt-mirror actually doesn’t mirror the bz2 files (although I’ve
submitted a patch to it so it will eventually).

I make sure that index files get built by my Apache and not
redirected.

I am using Apache proxying, which gives me Yet Another type of
cache temporarily while I’m still downloading the other packages. (I
should actually work out a way to have these caches used by apt-mirror
itself in case a user has already requested a new package while waiting
for apt-mirror to get it.)

Once I do a rewrite like this for each of the hosts I’m replacing with
a mirror, I’m almost done. The problem is that if for any reason my
site needs to give a 403 to the clients, I would actually like to
double-check to be sure that the URL doesn’t happen to work at the place
I’m mirroring from.

My hope was that I could write a RewriteRule based on what the
HTTP return code would be when the request completed. This was
really hard to do, it seemed, and perhaps undoable. The quickest
solution I found was to write a CGI script to do the redirect. So, in
the Apache config I have:

ErrorDocument 403 /cgi/redirect-forbidden.cgi

And, the CGI script looks like this:

#!/usr/bin/perl

use strict;
use CGI qw(:standard);

my $val = $ENV{REDIRECT_SCRIPT_URI};

$val =~ s%^http://(S+).sflc.info(/.*)$%$2%;
if ($1 eq “ubuntu-security”) {
$val = “http://91.189.88.37$val”;
} else {
$val = “http://91.189.88.45$val”;
}

print redirect($val);

With these changes, the user will be redirected to the original when
the files aren’t available on the mirror, and as the mirror gets more
accurate, they’ll get more files from the mirror.

I still have problems if for any reason the user gets a Packages or
Sources file from the original site before the mirror is synchronized,
but this rarely happens since apt-mirror is pretty careful. The only
time it might happen is if the user did an apt-get update when
not connected to our VPN and only a short time later did one while
connected.

apt-mirror and Other Caching for Debian/Ubuntu Repositories

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2008/01/16/apt-mirror-1.html

Working for a small non-profit, everyone has to wear lots of hats, and
one that I have to wear from time to time (since no one else here can)
is “sysadmin”. One of the perennial rules of system
administration is: you can never give users enough bandwidth. The
problem is, they eventually learn how fast your connection to the
outside is, and then complain any time a download doesn’t run at that
speed. Of course, if you have a T1 or better, it’s usually the other
side that’s the problem. So, I look to use our extra bandwidth during
off hours to cache large pools of data that are often downloaded. With
a organization full of Ubuntu machines, the Ubuntu repositories are an
important target for caching.

apt-mirror is a
program that mirrors large Debian-based repositories, including the
Ubuntu ones. There
are already
tutorials
available on how to set it up. What I’m writing about here is a way to
“force” users to use that repository.

The obvious way, of course, is to make
everyone’s /etc/apt/sources.list point at the mirrored
repository. This often isn’t a good option. Save the servers, the user
base here is all laptops, which means that they will often be on
networks that may actually be closer to another package repository and
perhaps I want to avoid interfering with that. (Although given that I
can usually give almost any IP number in the world better than the
30kbs/sec that ubuntu.com’s servers seem to quickly throttle to, that
probably doesn’t matter so much).

The bigger problem is that I don’t want to be married to the idea that
the apt-mirror is part of our essential 24/7 infrastructure. I don’t
want an angry late-night call from a user because they can’t install a
package, and I want the complete freedom to discontinue the server at
any time, if I find it to be unreliable. I can’t do this easily if
sources.list files on traveling machines are hard-coded with
the apt-mirror server’s name or address, especially when I don’t know
when exactly they’ll connect back to our VPN.

The easier solution is to fake out the DNS lookups via the DNS server
used by the VPN and the internal network. This way, user only get the
mirror when they are connected to the VPN or in the office; otherwise,
the get the normal Ubuntu servers. I had actually forgotten you could
fake out DNS on a per host basis, but asking my friend Paul reminded me
quickly. In /etc/bin/named.conf.local (on Debian/Ubuntu), I
just add:

zone “archive.ubuntu.com” {
type master;
file “/etc/bind/db.archive.ubuntu-fake”;
};

And in /etc/bind/db.archive.ubuntu-fake:

$TTL 604800
@ IN SOA archive.ubuntu.com. root.vpn. (
2008011001 ; serial number
10800 3600 604800 3600)
IN NS my-dns-server.vpn.

;
; Begin name records
;
archive.ubuntu.com. IN A MY.EXTERNAL.FACING.IP

And there I have it; I just do one of those for each address I want to
replace (e.g., security.ubuntu.com). Now, when client machines
lookup archive.ubuntu.com (et al), they’ll
get MY.EXTERNAL.FACING.IP, but only
when my-dns-server.vpn is first in their resolv.conf.

Next time, I’ll talk about some other ideas on how I make the
apt-mirror even better.

Postfix Trick to Force Secondary MX to Deliver Locally

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2008/01/09/postfix-secondary-mx-local-deliver.html

Suppose you have a domain name, example.org, that has a primary MX host
(mail.example.org) that does most of the delivery. However,
one of the users, who works at example.com, actually gets delivery of
<[email protected]> at work (from the primary MX for example.com,
mail.example.com). Of course, a simple .forward
or /etc/aliases entry would work, but this would pointlessly
push email back and forth between the two mail servers — in some
cases, up to three pointless passes before the final destination!
That’s particularly an issue in today’s SPAM-laden world. Here’s how to
solve this waste of bandwidth using Postfix.

This tutorial here assumes you have a some reasonable background
knowledge of Postfix MTA administration. If you don’t, this might go a
bit fast for you.

To begin, first note that this setup assumes that you have something
like this with regard to your MX setup:

$ host -t mx example.org
example.org mail is handled by 10 mail.example.org.
example.org mail is handled by 20 mail.example.com.
$ host -t mx example.com
example.com mail is handled by 10 mail.example.com.

Our first task is to avoid
example.org SPAM
backscatter
on mail.example.com. To do that, we make a file with
all the valid accounts for example.org and put it
in mail.example.com:/etc/postfix/relay_recipients. (For more
information, read
the Postfix
docs

or various
tutorials
about this.) After that, we
have something like this in mail.example.com:/etc/postfix/main.cf:

relay_domains = example.org
relay_recipient_maps = hash:/etc/postfix/relay_recipients

And this in /etc/postfix/transport:

example.org smtp:[mail.example.org]

This will give proper delivery for our friend <[email protected]>
(assuming mail.example.org is forwarding that address properly to
<[email protected]>), but mail will push mail back and forth
unnecessarily when mail.example.com gets a message for
<[email protected]>. What we actually want is to wise up
mail.example.com so it “knows” that mail for
<[email protected]> is ultimately going to be delivered locally on
that server.

To do this, we add <[email protected]> to
the virtual_alias_maps, with an entry like:

[email protected] user

so that the key [email protected] resolves to the local
username user. Fortunately, Postfix is smart enough to look at
the virtual table first before performing a relay.

Now, what about aliases like <[email protected]>, that
actually forwards to <[email protected]>? That will have the same
pointless forwarding from server-to-server unless we address it
specifically. To do so, we use the transport file. of course, we
should already have that catch-all entry there to do the relaying:

example.org smtp:[mail.example.org]

But, we can also add email address specific entries for certain
addresses in the example.org domain. Fortunately, email address matches
in the transport table take precedence over whole domain match entries
(see the transport man
page for details
.). Therefore, we simply add entries to
that transport file like this for each of user’s
aliases:

[email protected] local:user

(Note: that assumes you have a delivery method in master.cf
called local. Use whatever transport you typically use to
force local delivery.)

And there you have it! If you have (those albeit rare) friendly and
appreciative users, user will thank you for the slightly
quicker mail delivery, and you’ll be glad that you aren’t pointlessly
shipping SPAM back and forth between MX’s unnecessarily.

GIT Mirrors of my SVN Repositories

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/svn-git.html

To whom it may concern: as a first step to move away from SVN and towards
GIT for all my code, I have now configured live GIT mirrors for all my SVN repositories. The plan is to move fully
to GIT, but not as long as the GIT integration into Trac is as painful as it is
right now. The scripts I used to initialize and update the mirrors are svn-live-init and svn-live-update, for those
interested. They are based on scripts CJ van den Berg supplied me with.

It would be great to have the mirror to be both ways. Lazyweb, do you know how to do that?

GIT Mirrors of my SVN Repositories

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/svn-git.html

To whom it may concern: as a first step to move away from SVN and towards
GIT for all my code, I have now configured live GIT mirrors for all my SVN repositories. The plan is to move fully
to GIT, but not as long as the GIT integration into Trac is as painful as it is
right now. The scripts I used to initialize and update the mirrors are svn-live-init and svn-live-update, for those
interested. They are based on scripts CJ van den Berg supplied me with.

It would be great to have the mirror to be both ways. Lazyweb, do you know how to do that?

Apache 2.0 -> 2.2 LDAP Changes on Ubuntu

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2008/01/01/apache-2-2-ldap.html

I thought the following might be of use to those of you who are still
using Apache 2.0 with LDAP and wish to upgrade to 2.2. I found this
basic information around online, but I had to search pretty hard for it.
Perhaps presenting this in a more straightforward way might help the
next searcher to find an answer more quickly. It’s probably only of
interest if you are using LDAP as your authentication system with an
older Apache (e.g., 2.0) and have upgraded to 2.2 on an Ubuntu or Debian
system (such as upgrading from dapper to gutsy.)

When running dapper on my intranet web server with Apache
2.0.55-4ubuntu2.2, I had something like this:

<Directory /var/www/intranet>
Order allow,deny
Allow from 192.168.1.0/24

Satisfy All
AuthLDAPEnabled on
AuthType Basic
AuthName “Example.Org Intranet”
AuthLDAPAuthoritative on
AuthLDAPBindDN uid=apache,ou=roles,dc=example,dc=org
AuthLDAPBindPassword APACHE_BIND_ACCT_PW
AuthLDAPURL ldap://127.0.0.1/ou=staff,ou=people,dc=example,dc=org?cn
AuthLDAPGroupAttributeIsDN off
AuthLDAPGroupAttribute memberUid

require valid-user
</Directory>

I upgraded that server to gutsy (via dapper → edgy → feisty
→ gutsy in succession, just because it’s safer), and it now has
Apache 2.2.4-3build1. The methods to do LDAP authentication is a bit
more straightforward now, but it does require this change:

<Directory /var/www/intranet>
Order allow,deny
Allow from 192.168.1.0/24

AuthType Basic
AuthName “Example.Org Intranet”
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthLDAPBindDN uid=apache,ou=roles,dc=example,dc=org
AuthLDAPBindPassword APACHE_BIND_ACCT_PW
AuthLDAPURL ldap://127.0.0.1/ou=staff,ou=people,dc=example,dc=org

require valid-user
Satisfy all
</Directory>

However, this wasn’t enough. When I set this up, I got rather strange
error messages such as:

[error] [client MYIP] GROUP: USERNAME not in required group(s).

I found somewhere online (I’ve now lost the link!) that you couldn’t
have standard pam auth competing with the LDAP authentication. This
seemed strange to me, since I’ve told it I want the authentication
provided by LDAP, but anyway, doing the following on the system:

a2dismod auth_pam
a2dismod auth_sys_group

solved the problem. I decided to move on rather than dig deeper into the
true reasons. Sometimes, administration life is actually better with a
mystery about.

Avahi/Zeroconf patch for distcc updated

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/avahi-distcc.html

I finally found them time to sit down and update my venerable Avahi/Zeroconf patch for
distcc
. A patched distcc
automatically discovers suitable compiler servers on the local network, without
the need to manually configure them. (Announcement).

Here’s a quick HOWTO for using a patched distcc like this:

Make sure to start distccd (the server) with the new
–zeroconf switch, This will make it announce its services on the
network.

Edit your $HOME/.distcc/hosts and add +zeroconf. This
magic string will enable Zeroconf support in the client, i.e. will be expanded
to the list of available suitable distcc servers on your LAN.

Now set $CC to distcc gcc globally for your login
sessions. This will tell all well-behaving build systems to use distcc
for compilation (this doesn’t work for the kernel, as one notable exception).
Even better than setting $CC to distcc gcc is setting it to
ccache distcc gcc which will enable ccache in addition to distcc. i.e. stick something like this in your ~/.bash_profile: export CC=”ccache distcc gcc”

And finally use make -j `distcc -j` instead of plain make
to enable parallel building with the right number of concurrent processes.
Setting $MAKEFLAGS properly is an alternative option, however is suboptimal if
the evalutation is only done once at login time.

If this doesn’t work for you than it is a good idea to run distcc
–show-hosts to get a list of discovered distcc servers. If this list
isn’t complete then this is most likely due to mismatching GCC versions or
architectures. To check if that’s the case use avahi-browse -r
_distcc._tcp and compare the values of the cc_machine and
cc_version fields. Please note that different Linux distributions use
different GCC machine strings. Which is expected since GCC is usually patched quite
a bit on the different distributions. This means that a Fedora distcc
(the client) will not find a Debian distccd (the server) and vice
versa. But again: that’s a feature, not a bug.

The new -j and –show-hosts options for distcc are useful for non-zeroconf setups, too.

The patch will automatically discover the number of CPUs on remote machines
and make use of that information to better distribute jobs.

In short: Zeroconf support in distcc is totally hot, everyone should have it!

For more information have a look on the announcement of my original
patch from 2004
(at that time for the historic HOWL Zeroconf daemon), or read the new
announcement linked above.

Distribution packagers! Please merge this new patch into your packages! It
would be a pity to withhold Zeroconf support in distcc from your users any
longer!

Unfortunately, Fedora doesn’t include any distcc packages. Someone should be
changing that (who’s not me ;-)).

You like this patch? Then give me a kudo on ohloh.net. Now that I earned a golden 10 (after kicking Larry Ewing from position 64. Ha, take that Mr. Ewing!), I need to make sure I don’t fall into silver oblivion again. 😉

Avahi/Zeroconf patch for distcc updated

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/avahi-distcc.html

I finally found them time to sit down and update my venerable Avahi/Zeroconf patch for
distcc
. A patched distcc
automatically discovers suitable compiler servers on the local network, without
the need to manually configure them. (Announcement).

Here’s a quick HOWTO for using a patched distcc like this:

  • Make sure to start distccd (the server) with the new
    --zeroconf switch, This will make it announce its services on the
    network.
  • Edit your $HOME/.distcc/hosts and add +zeroconf. This
    magic string will enable Zeroconf support in the client, i.e. will be expanded
    to the list of available suitable distcc servers on your LAN.
  • Now set $CC to distcc gcc globally for your login
    sessions. This will tell all well-behaving build systems to use distcc
    for compilation (this doesn’t work for the kernel, as one notable exception).
    Even better than setting $CC to distcc gcc is setting it to
    ccache distcc gcc which will enable ccache in addition to distcc. i.e. stick something like this in your ~/.bash_profile: export CC="ccache distcc gcc"
  • And finally use make -j `distcc -j` instead of plain make
    to enable parallel building with the right number of concurrent processes.
    Setting $MAKEFLAGS properly is an alternative option, however is suboptimal if
    the evalutation is only done once at login time.

If this doesn’t work for you than it is a good idea to run distcc
--show-hosts
to get a list of discovered distcc servers. If this list
isn’t complete then this is most likely due to mismatching GCC versions or
architectures. To check if that’s the case use avahi-browse -r
_distcc._tcp
and compare the values of the cc_machine and
cc_version fields. Please note that different Linux distributions use
different GCC machine strings. Which is expected since GCC is usually patched quite
a bit on the different distributions. This means that a Fedora distcc
(the client) will not find a Debian distccd (the server) and vice
versa. But again: that’s a feature, not a bug.

The new -j and --show-hosts options for distcc are useful for non-zeroconf setups, too.

The patch will automatically discover the number of CPUs on remote machines
and make use of that information to better distribute jobs.

In short: Zeroconf support in distcc is totally hot, everyone should have it!

For more information have a look on the announcement of my original
patch from 2004
(at that time for the historic HOWL Zeroconf daemon), or read the new
announcement linked above.

Distribution packagers! Please merge this new patch into your packages! It
would be a pity to withhold Zeroconf support in distcc from your users any
longer!

Unfortunately, Fedora doesn’t include any distcc packages. Someone should be
changing that (who’s not me ;-)).

You like this patch? Then give me a kudo on ohloh.net. Now that I earned a golden 10 (after kicking Larry Ewing from position 64. Ha, take that Mr. Ewing!), I need to make sure I don’t fall into silver oblivion again. 😉

Avahi 0.6.22

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/avahi-0.6.22.html

A couple of minutes ago I released Avahi
0.6.22
into the wild, the newest iteration of everyone’s favourite zero
configuration networking suite.

Avahi Logo

You ask why this is something to blog about?

Firstly, new in this version is Sjoerd Simons’ avahi-gobject
library, a GObject wrapper around the Avahi API. It allows full GObject-style
object oriented programming of Zeroconf applications, with signals and
everything. To all you GNOME/Gtk+ hackers out there: now it is even more fun to
hack your own Zeroconf applications for GNOME/Gtk+!

Secondly, this is the first release to ship i18n support. For those who
prefer to run their systems with non-english locales[1] this should
be good news. I’ve always been a little afraid of adding i18n support, since
this either meant that I would have contstantly had to commit i18n patches, or that I
would have needed to move my code to GNOME SVN. However, we now have Fedora’s Transifex,
which allows me to open up my SVN for translators without much organizational
work on my side. Translations are handled centrally, and commited back to my
repository when needed. It’s a bit like Canonical’s Rosetta, but with a focus
on commiting i18n changes upstream, and without being closed-source crap.

You like this release? Then give me a kudo on ohloh.net. My
ego still thirsts for gold, and I am still (or again) 25 positions away from
that. 😉

Footnotes

[1] Personally, I run my desktop with $LC_MESSAGES=C, but
LANG=de_DE, which are the settings I can recommend to everyone who is from Germany and wants to stay
sane. Unfortunately it is a PITA to configure this on
GNOME, though.

Avahi 0.6.22

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/avahi-0.6.22.html

A couple of minutes ago I released Avahi
0.6.22
into the wild, the newest iteration of everyone’s favourite zero
configuration networking suite.

Avahi Logo

You ask why this is something to blog about?

Firstly, new in this version is Sjoerd Simons’ avahi-gobject
library, a GObject wrapper around the Avahi API. It allows full GObject-style
object oriented programming of Zeroconf applications, with signals and
everything. To all you GNOME/Gtk+ hackers out there: now it is even more fun to
hack your own Zeroconf applications for GNOME/Gtk+!

Secondly, this is the first release to ship i18n support. For those who
prefer to run their systems with non-english locales[1] this should
be good news. I’ve always been a little afraid of adding i18n support, since
this either meant that I would have contstantly had to commit i18n patches, or that I
would have needed to move my code to GNOME SVN. However, we now have Fedora’s Transifex,
which allows me to open up my SVN for translators without much organizational
work on my side. Translations are handled centrally, and commited back to my
repository when needed. It’s a bit like Canonical’s Rosetta, but with a focus
on commiting i18n changes upstream, and without being closed-source crap.

You like this release? Then give me a kudo on ohloh.net. My
ego still thirsts for gold, and I am still (or again) 25 positions away from
that. 😉

Footnotes

[1] Personally, I run my desktop with $LC_MESSAGES=C, but
LANG=de_DE, which are the settings I can recommend to everyone who is from Germany and wants to stay
sane. Unfortunately it is a PITA to configure this on
GNOME, though.

Back from India

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/photos/india.html

FOSS.in was one of the best conferences I have ever
been to, and a lot of fun. The organization was flawless and I can
only heartily recommend everyone to send in a presentation proposal for next year’s
iteration. I certainly hope the commitee is going to accept my proposals next year again. Especially the food was gorgeous.

I will spare you the usual conference photos, you can find a lot of those on flickr. However, what I will not spare you are a couple of photos I shot in Bangalore, Srirangapatna and Mysore.

India  
India  
India  
India  

India  
India  
India  
India  
India  
India  
India  

India  
India  
India  
India  
India  
India  
India

Panorama

Lazyweb: POSIX Process Groups and Sessions

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/pgrp-vs-session.html

Dear Lazyweb,

I have trouble understanding what exactly POSIX process groups and sessions
are good for. The POSIX
docs
are very vague on this. What exactly is the effect of being in a process
group with some other process, and what does being in the same session with it
add on top? And what is the benefit of being a group/session leader in contrast of just being a normal random process in the group/session?

The only thing I understood is that kill(2) with a negative first
parameter can be used to “multicast” signals to entire process groups, and that
SIGINT on C-c is delivered that way. But, is that all? The POSIX docs say
“… for the purpose of signaling, placement in foreground or background,
and other job control actions”, which is very vague. What are those
“other job control actions?”. What does job control persist of besides
multicasting signals? And what is “placement in foreground or background” other
than delivering signals?

And I totally don’t get POSIX sessions and how they differ from POSIX process groups. Please enlighten me!

Puzzled,
    Lennart

Lazyweb: POSIX Process Groups and Sessions

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/pgrp-vs-session.html

Dear Lazyweb,

I have trouble understanding what exactly POSIX process groups and sessions
are good for. The POSIX
docs
are very vague on this. What exactly is the effect of being in a process
group with some other process, and what does being in the same session with it
add on top? And what is the benefit of being a group/session leader in contrast of just being a normal random process in the group/session?

The only thing I understood is that kill(2) with a negative first
parameter can be used to “multicast” signals to entire process groups, and that
SIGINT on C-c is delivered that way. But, is that all? The POSIX docs say
… for the purpose of signaling, placement in foreground or background,
and other job control actions
“, which is very vague. What are those
other job control actions?“. What does job control persist of besides
multicasting signals? And what is “placement in foreground or background” other
than delivering signals?

And I totally don’t get POSIX sessions and how they differ from POSIX process groups. Please enlighten me!

Puzzled,
    Lennart

stet and AGPLv3

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2007/11/21/stet-and-agplv3.html

Many people don’t realize that the GPLv3 process actually began long
before the November 2005 announcement. For me and a few others, the GPLv3
process started much earlier. Also, in my view, it didn’t actually end
until this week, the FSF released the AGPLv3. Today, I’m particularly
proud that stet was the first software released covered by the terms of
that license.

The GPLv3 process focused on the idea of community, and a community is
built from bringing together many individual experiences. I am grateful
for all my personal experiences throughout this process. Indeed, I
would guess that other GPL fans like myself remember, as I do, the first
time the heard the phrase “GPLv3”. For me, it was a bit
early — on Tuesday 8 January 2002 in a conference room at MIT. On
that day, Richard Stallman, Eben Moglen and I sat down to have an
all-day meeting that included discussions regarding updating GPL. A key
issue that we sought to address was (in those days) called the
“Application Service Provider (ASP) problem” — now
called “Software as a Service (SaaS)”.

A few days later, on the telephone with Moglen2 one morning, as I stood in my
kitchen making oatmeal, we discussed this problem. I pointed out the
oft-forgotten section 2(c) of the GPL [version 2]. I argued that contrary
to popular belief, it does have restrictions on some minor
modifications. Namely, you have to maintain those print statements for
copyright and warranty disclaimer information. It’s reasonable, in other
words, to restrict some minor modifications to defend freedom.

We also talked about that old Computer Science problem of having a
program print its own source code. I proposed that maybe we needed a
section 2(d) that required that if a program prints its own source to
the user, that you can’t remove that feature, and that the feature must
always print the complete and corresponding source.

Within two months, Affero
GPLv1 was published
— an authorized fork of the GPL to test
the idea. From then until AGPLv3, that “Affero clause”
has had many changes, iterations and improvements, and I’m grateful
for all the excellent feedback, input and improvements that have gone
into it. The
result, the
Affero GPLv3 (AGPLv3) released on Monday
, is an excellent step
forward for software freedom licensing. While the community process
indicated that the preference was for the Affero clause to be part of
a separate license, I’m nevertheless elated that the clause continues
to live on and be part of the licensing infrastructure defending
software freedom.

Other than coining the Affero clause, my other notable personal
contribution to the GPLv3 was management of a software development
project to create the online public commenting system. To do the
programming, we contracted with Orion Montoya, who has extensive
experience doing semantic markup of source texts from an academic
perspective. Orion gave me my first introduction to the whole
“Web 2.0” thing, and I was amazed how useful the result was;
it helped the leaders of the process easily grok the public response.
For example, the intensity highlighting — which shows the hot
spots in the text that received the most comments — gives a very
quick picture of sections that are really of concern to the public. In
reviewing the drafts today, I was reminded that the big red area in
section 1 about “encryption and authorization codes”
is
substantially
changed and less intensely highlighted by draft 4
. That quick-look
gives a clear picture of how the community process operated to get a
better license for everyone.

Orion, a Classics scholar as an undergrad, named the
software stet for its original Latin definition: “let it
stand as it is”. It was his hope that stet (the software) would
help along the GPLv3 process so that our whole community, after filing
comments on each successive draft, could look at the final draft and
simply say: Stet!

Stet has a special place in software history, I believe, even if it’s
just a purely geeky one. It is the first software system in history to
be meta-licensed. Namely, it was software whose output was its own
license. It’s with that exciting hacker concept that I put up today
a Trac instance
for stet, licensed under the terms of the AGPLv3 [ which is now on
Gitorious ]
1.

Stet is by no means ready for drop-in production. Like most software
projects, we didn’t estimate perfectly how much work would be needed.
We got lazy about organization early on, which means it still requires a
by-hand install, and new texts must be carefully marked up by hand.
We’ve moved on to other projects, but hopefully SFLC will host the Trac
instance indefinitely so that other developers can make it better.
That’s what copylefted FOSS is all about — even when it’s
SaaS.

1Actually, it’s
under AGPLv3 plus an exception to allow for combining with the
GPLv2-only Request Tracker, with which parts of stet combine.

2Update
2016-01-06:After writing this blog post, I found
evidence in my email archives from early 2002, wherein Henry Poole (who
originally suggested the need for Affero GPL to FSF), began cc’ing me anew
on an existing thread. In that thread, Poole quoted text from Moglen
proposing the original AGPLv1 idea to Poole. Moglen’s quoted text in
Poole’s email proposed the idea as if it were solely Moglen’s own. Based
on the timeline of the emails I have, Moglen seems to have written to Poole
within 36-48 hours of my original formulation of the idea.

While I do not accuse Moglen of plagiarism, I believe he does at least
misremember my idea as his own, which is particularly surprising, as Moglen
(at that time, in 2002) seemed unfamiliar with the Computer Science concept
of a quine; I had to explain that concept as part of my presentation of my
idea. Furthermore, Moglen and I discussed this matter in a personal
conversation in 2007 (around the time I made this blog post originally) and
Moglen said to me: “you certainly should take credit for the Affero
GPL”. Thus, I thought the matter was thus fully settled back in
2007, and thus Moglen’s post-2007 claims of credit that write me out of
Affero GPL’s history are simply baffling. To clear up the confusion his
ongoing claims create, I added this footnote to communicate unequivocally
that my memory of that phone call is solid, because it was the first time I
ever came up with a particularly interesting licensing idea, so the memory
became extremely precious to me immediately. I am therefore completely
sure I was the first to propose the original idea of mandating preservation
of a quine-like feature in AGPLv1§2(d) (as a fork/expansion of
GPLv2§2(c)) on the telephone to Moglen, as described above. Moglen
has never produced evidence to dispute my recollection, and even agreed
with the events as I told them back in 2007.

Nevertheless, unlike Moglen, I do admit that creation of the final text of
AGPLv1 was a collaborative process, which included contributions from
Moglen, Poole, RMS, and a lawyer (whose name I don’t recall) whom Poole
hired. AGPLv3§13’s drafting was similarly collaborative, and included
input from Richard Fontana, David Turner, and Brett Smith, too.

Finally, I note my surprise at this outcome. In my primary community
— the Free Software community — people are generally extremely
good at giving proper credit. Unlike the Free Software community, legal
communities apparently are cutthroat on the credit issue, so I’ve
learned.

Emulated atomic operations and real-time scheduling

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/atomic-rt.html

Unfortunately not all CPU architectures have native support for atomic
operations
, or only support a very limited subset. Most
prominently ARMv5 (and
older) hasn’t any support besides the most basic atomic swap
operation[1]. Now, more and more free code is starting to
use atomic operations and lock-free
algorithms
, one being my own project, PulseAudio. If you have ever done real-time
programming
you probably know that you cannot really do it without
support for atomic operations. One question remains however: what to
do on CPUs which support only the most basic atomic operations
natively?

On the kernel side atomic ops are very easy to emulate: just disable
interrupts temporarily, then do your operation non-atomically, and afterwards
enable them again. That’s relatively cheap and works fine (unless you are on SMP — which
fortunately you usually are not for those CPUs). The Linux
kernel does it this way and it is good. But what to do in user-space, where you cannot just go and disable interrupts?

Let’s see how the different userspace libraries/frameworks do it
for ARMv5, a very relevant architecture that only knows an atomic swap (exchange)
but no CAS
or even atomic arithmetics. Let’s start with an excerpt from glibc’s
atomic operations implementation for ARM
:

/* Atomic compare and exchange. These sequences are not actually atomic;
there is a race if *MEM != OLDVAL and we are preempted between the two
swaps. However, they are very close to atomic, and are the best that a
pre-ARMv6 implementation can do without operating system support.
LinuxThreads has been using these sequences for many years. */

This comment says it all. Not good. The more you make use of atomic
operations the more likely you’re going to to hit this race. Let’s
hope glibc is not a heavy user of atomic operations. PulseAudio however is, and
PulseAudio happens to be my focus.

Let’s have a look on how Qt4
does it:

extern Q_CORE_EXPORT char q_atomic_lock;

inline char q_atomic_swp(volatile char *ptr, char newval)
{
register int ret;
asm volatile(“swpb %0,%1,[%2]”
: “=&r”(ret)
: “r”(newval), “r”(ptr)
: “cc”, “memory”);
return ret;
}

inline int q_atomic_test_and_set_int(volatile int *ptr, int expected, int newval)
{
int ret = 0;
while (q_atomic_swp(&q_atomic_lock, ~0) != 0);
if (*ptr == expected) {
*ptr = newval;
ret = 1;
}
q_atomic_swp(&q_atomic_lock, 0);
return ret;
}

So, what do we have here? A slightly better version. In standard
situations it actually works. But it sucks big time, too. Why? It
contains a spin lock: the variable q_atomic_lock is used for
locking the atomic operation. The code tries to set it to non-zero,
and if that fails it tries again, until it succeeds, in the hope that
the other thread — which currently holds the lock — gives it up. The
big problem here is: it might take a while until that happens, up to
1/HZ time on Linux. Usually you want to use atomic operations to
minimize the need for mutexes and thus speed things up. Now, here you
got a lock, and it’s the worst kind: the spinning lock. Not
good. Also, if used from a real-time thread the machine simply locks
up when we enter the loop in contended state, because preemption is
disabled for RT threads and thus the loop will spin forever. Evil. And
then, there’s another problem: it’s a big bottleneck, because all
atomic operations are synchronized via a single variable which is
q_atomic_lock. Not good either. And let’s not forget that
only code that has access to q_atomic_lock actually can
execute this code safely. If you want to use it for
lock-free IPC via shared memory this is going to break. And let’s not
forget that it is unusable from signal handlers (which probably
doesn’t matter much, though). So, in summary: this code sucks,
too.

Next try, let’s have a look on how glib
does it:

static volatile int atomic_spin = 0;

static int atomic_spin_trylock (void)
{
int result;

asm volatile (
“swp %0, %1, [%2]n”
: “=&r,&r” (result)
: “r,0” (1), “r,r” (&atomic_spin)
: “memory”);
if (result == 0)
return 0;
else
return -1;
}

static void atomic_spin_lock (void)
{
while (atomic_spin_trylock())
sched_yield();
}

static void atomic_spin_unlock (void)
{
atomic_spin = 0;
}

gint
g_atomic_int_exchange_and_add (volatile gint *atomic,
gint val)
{
gint result;

atomic_spin_lock();
result = *atomic;
*atomic += val;
atomic_spin_unlock();

return result;
}

Once again, a spin loop. However, this implementation makes use of
sched_yield() for asking the OS to reschedule. It’s a bit
better than the Qt version, since it doesn’t spin just burning CPU,
but instead tells the kernel to execute something else, increasing the
chance that the thread currently holding the lock is scheduled. It’s a
bit friendlier, but it’s not great either because this might still delay
execution quite a bit. It’s better then the Qt version. And probably
one of the very few ligitimate occasions where using
sched_yield() is OK. It still doesn’t work for RT — because
sched_yield() in most cases is a NOP on for RT threads, so
you still get a machine lockup. And it still has the
one-lock-to-rule-them-all bottleneck. And it still is not compatible
with shared memory.

Then, there’s libatomic_ops. It’s
the most complex code, so I’ll spare you to paste it here. Basically
it uses the same spin loop. With three differences however:

16 lock variables instead of a single one are used. The variable
that is used is picked via simple hashing of the pointer to the atomic variable
that shall be modified. This removes the one-lock-to-rule-them-all
bottleneck.

Instead of pthread_yield() it uses select() with
a small timeval parameter to give the current holder of the lock some
time to give it up. To make sure that the select() is not
optimized away by the kernel and the thread thus never is preempted
the sleep time is increased on every loop iteration.

It explicitly disables signals before doing the atomic operation.

It’s certainly the best implementation of the ones discussed here:
It doesn’t suffer by the one-lock-to-rule-them-all bottleneck. It’s
(supposedly) signal handler safe (which however comes at the cost of
doing two syscalls on every atomic operation — probably a very high
price). It actually works on RT, due to sleeping for an explicit
time. However it still doesn’t deal with priority
inversion
problems — which is a big issue for real-time
programming. Also, the time slept in the select() call might
be relatively long, since at least on Linux the time passed to
select() is rounded up to 1/HZ — not good for RT either. And
then, it still doesn’t work for shared memory IPC.

So, what do we learn from this? At least one thing: better don’t do
real-time programming with ARMv5[2]. But more practically, how
could a good emulation for atomic ops, solely based on atomic swap
look like? Here are a few ideas:

Use an implementation inspired by libatomic_ops. Right
now it’s the best available. It’s probably a good idea, though, to
replace select() by a nanosleep(), since on recent
kernels the latter doesn’t round up to 1/HZ anymore, at least when you
have high-resolution timers[3] Then, if you can live
without signal handler safety, drop the signal mask changing.

If you use something based on libatomic_ops and want to
use it for shared memory IPC, then you have the option to move the
lock variables into shared memory too. Note however, that this allows
evil applications to lock up your process by taking the locks and
never giving them up. (Which however is always a problem if not all
atomic operations you need are available in hardware) So if you do
this, make sure that only trusted processes can attach to your memory
segment.

Alternatively, spend some time and investigate if it is possible
to use futexes to sleep on the lock variables. This is not trivial
though, since futexes right now expect the availability of an atomic
increment operation. But it might be possible to emulate this good
enough with the swap operation. There’s now even a FUTEX_LOCK_PI
operation which would allow priority inheritance.

Alternatively, find a a way to allow user space disabling
interrupts cheaply (requires kernel patching). Since enabling
RT scheduling is a priviliged operation already (since you may easily
lock up your machine with it), it might not be too problematic to
extend the ability to disable interrupts to user space: it’s just yet
another way to lock up your machine.

For the libatomic_ops based algorithm: if you’re lucky
and defined a struct type for your atomic integer types, like the
kernel does, or like I do in PulseAudio with pa_atomic_t,
then you can stick the lock variable directly into your
structure. This makes shared memory support transparent, and removes
the one-lock-to-rule-them-all bottleneck completely. Of course, OTOH it
increases the memory consumption a bit and increases cache pressure
(though I’d assume that this is neglible).

For the libatomic_ops based algorithm: start sleeping for
the time returned by clock_getres()
(cache the result!). You cannot sleep shorter than that anyway.

Yepp, that’s as good as it gets. Unfortunately I cannot serve you
the optimal solution on a silver platter. I never actually did
development for ARMv5, this blog story just sums up my thoughts on all
the code I saw which emulates atomic ops on ARMv5. But maybe someone
who actually cares about atomic operations on ARM finds this
interesting and maybe invests some time to prepare patches for Qt,
glib, glibc — and PulseAudio.

Update: I added two more ideas to the list above.

Update 2: Andrew Haley just posted something like the optimal solution for the problem. It would be great if people would start using this.

Footnotes

[1] The Nokia 770 has an ARMv5 chip, N800 has ARMv6. The OpenMoko phone apparently uses ARMv5.

[2] And let’s not even think about CPUs which don’t even have an atomic swap!

[3] Which however you probably won’t, given that they’re only available on x86 on stable Linux kernels for now — but still, it’s cleaner.

Emulated atomic operations and real-time scheduling

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/atomic-rt.html

Unfortunately not all CPU architectures have native support for atomic
operations
, or only support a very limited subset. Most
prominently ARMv5 (and
older) hasn’t any support besides the most basic atomic swap
operation[1]. Now, more and more free code is starting to
use atomic operations and lock-free
algorithms
, one being my own project, PulseAudio. If you have ever done real-time
programming
you probably know that you cannot really do it without
support for atomic operations. One question remains however: what to
do on CPUs which support only the most basic atomic operations
natively?

On the kernel side atomic ops are very easy to emulate: just disable
interrupts temporarily, then do your operation non-atomically, and afterwards
enable them again. That’s relatively cheap and works fine (unless you are on SMP — which
fortunately you usually are not for those CPUs). The Linux
kernel does it this way and it is good. But what to do in user-space, where you cannot just go and disable interrupts?

Let’s see how the different userspace libraries/frameworks do it
for ARMv5, a very relevant architecture that only knows an atomic swap (exchange)
but no CAS
or even atomic arithmetics. Let’s start with an excerpt from glibc’s
atomic operations implementation for ARM
:

/* Atomic compare and exchange.  These sequences are not actually atomic;
   there is a race if *MEM != OLDVAL and we are preempted between the two
   swaps.  However, they are very close to atomic, and are the best that a
   pre-ARMv6 implementation can do without operating system support.
   LinuxThreads has been using these sequences for many years.  */

This comment says it all. Not good. The more you make use of atomic
operations the more likely you’re going to to hit this race. Let’s
hope glibc is not a heavy user of atomic operations. PulseAudio however is, and
PulseAudio happens to be my focus.

Let’s have a look on how Qt4
does it:

extern Q_CORE_EXPORT char q_atomic_lock;

inline char q_atomic_swp(volatile char *ptr, char newval)
{
    register int ret;
    asm volatile("swpb %0,%1,[%2]"
                 : "=&r"(ret)
                 : "r"(newval), "r"(ptr)
                 : "cc", "memory");
    return ret;
}

inline int q_atomic_test_and_set_int(volatile int *ptr, int expected, int newval)
{
    int ret = 0;
    while (q_atomic_swp(&q_atomic_lock, ~0) != 0);
    if (*ptr == expected) {
	*ptr = newval;
	ret = 1;
    }
    q_atomic_swp(&q_atomic_lock, 0);
    return ret;
}

So, what do we have here? A slightly better version. In standard
situations it actually works. But it sucks big time, too. Why? It
contains a spin lock: the variable q_atomic_lock is used for
locking the atomic operation. The code tries to set it to non-zero,
and if that fails it tries again, until it succeeds, in the hope that
the other thread — which currently holds the lock — gives it up. The
big problem here is: it might take a while until that happens, up to
1/HZ time on Linux. Usually you want to use atomic operations to
minimize the need for mutexes and thus speed things up. Now, here you
got a lock, and it’s the worst kind: the spinning lock. Not
good. Also, if used from a real-time thread the machine simply locks
up when we enter the loop in contended state, because preemption is
disabled for RT threads and thus the loop will spin forever. Evil. And
then, there’s another problem: it’s a big bottleneck, because all
atomic operations are synchronized via a single variable which is
q_atomic_lock. Not good either. And let’s not forget that
only code that has access to q_atomic_lock actually can
execute this code safely. If you want to use it for
lock-free IPC via shared memory this is going to break. And let’s not
forget that it is unusable from signal handlers (which probably
doesn’t matter much, though). So, in summary: this code sucks,
too.

Next try, let’s have a look on how glib
does it:

static volatile int atomic_spin = 0;

static int atomic_spin_trylock (void)
{
  int result;

  asm volatile (
    "swp %0, %1, [%2]\n"
    : "=&r,&r" (result)
    : "r,0" (1), "r,r" (&atomic_spin)
    : "memory");
  if (result == 0)
    return 0;
  else
    return -1;
}

static void atomic_spin_lock (void)
{
  while (atomic_spin_trylock())
    sched_yield();
}

static void atomic_spin_unlock (void)
{
  atomic_spin = 0;
}

gint
g_atomic_int_exchange_and_add (volatile gint *atomic,
			       gint           val)
{
  gint result;

  atomic_spin_lock();
  result = *atomic;
  *atomic += val;
  atomic_spin_unlock();

  return result;
}

Once again, a spin loop. However, this implementation makes use of
sched_yield() for asking the OS to reschedule. It’s a bit
better than the Qt version, since it doesn’t spin just burning CPU,
but instead tells the kernel to execute something else, increasing the
chance that the thread currently holding the lock is scheduled. It’s a
bit friendlier, but it’s not great either because this might still delay
execution quite a bit. It’s better then the Qt version. And probably
one of the very few ligitimate occasions where using
sched_yield() is OK. It still doesn’t work for RT — because
sched_yield() in most cases is a NOP on for RT threads, so
you still get a machine lockup. And it still has the
one-lock-to-rule-them-all bottleneck. And it still is not compatible
with shared memory.

Then, there’s libatomic_ops. It’s
the most complex code, so I’ll spare you to paste it here. Basically
it uses the same spin loop. With three differences however:

  1. 16 lock variables instead of a single one are used. The variable
    that is used is picked via simple hashing of the pointer to the atomic variable
    that shall be modified. This removes the one-lock-to-rule-them-all
    bottleneck.
  2. Instead of pthread_yield() it uses select() with
    a small timeval parameter to give the current holder of the lock some
    time to give it up. To make sure that the select() is not
    optimized away by the kernel and the thread thus never is preempted
    the sleep time is increased on every loop iteration.
  3. It explicitly disables signals before doing the atomic operation.

It’s certainly the best implementation of the ones discussed here:
It doesn’t suffer by the one-lock-to-rule-them-all bottleneck. It’s
(supposedly) signal handler safe (which however comes at the cost of
doing two syscalls on every atomic operation — probably a very high
price). It actually works on RT, due to sleeping for an explicit
time. However it still doesn’t deal with priority
inversion
problems — which is a big issue for real-time
programming. Also, the time slept in the select() call might
be relatively long, since at least on Linux the time passed to
select() is rounded up to 1/HZ — not good for RT either. And
then, it still doesn’t work for shared memory IPC.

So, what do we learn from this? At least one thing: better don’t do
real-time programming with ARMv5[2]. But more practically, how
could a good emulation for atomic ops, solely based on atomic swap
look like? Here are a few ideas:

  • Use an implementation inspired by libatomic_ops. Right
    now it’s the best available. It’s probably a good idea, though, to
    replace select() by a nanosleep(), since on recent
    kernels the latter doesn’t round up to 1/HZ anymore, at least when you
    have high-resolution timers[3] Then, if you can live
    without signal handler safety, drop the signal mask changing.
  • If you use something based on libatomic_ops and want to
    use it for shared memory IPC, then you have the option to move the
    lock variables into shared memory too. Note however, that this allows
    evil applications to lock up your process by taking the locks and
    never giving them up. (Which however is always a problem if not all
    atomic operations you need are available in hardware) So if you do
    this, make sure that only trusted processes can attach to your memory
    segment.
  • Alternatively, spend some time and investigate if it is possible
    to use futexes to sleep on the lock variables. This is not trivial
    though, since futexes right now expect the availability of an atomic
    increment operation. But it might be possible to emulate this good
    enough with the swap operation. There’s now even a FUTEX_LOCK_PI
    operation which would allow priority inheritance.
  • Alternatively, find a a way to allow user space disabling
    interrupts cheaply (requires kernel patching). Since enabling
    RT scheduling is a priviliged operation already (since you may easily
    lock up your machine with it), it might not be too problematic to
    extend the ability to disable interrupts to user space: it’s just yet
    another way to lock up your machine.
  • For the libatomic_ops based algorithm: if you’re lucky
    and defined a struct type for your atomic integer types, like the
    kernel does, or like I do in PulseAudio with pa_atomic_t,
    then you can stick the lock variable directly into your
    structure. This makes shared memory support transparent, and removes
    the one-lock-to-rule-them-all bottleneck completely. Of course, OTOH it
    increases the memory consumption a bit and increases cache pressure
    (though I’d assume that this is neglible).
  • For the libatomic_ops based algorithm: start sleeping for
    the time returned by clock_getres()
    (cache the result!). You cannot sleep shorter than that anyway.

Yepp, that’s as good as it gets. Unfortunately I cannot serve you
the optimal solution on a silver platter. I never actually did
development for ARMv5, this blog story just sums up my thoughts on all
the code I saw which emulates atomic ops on ARMv5. But maybe someone
who actually cares about atomic operations on ARM finds this
interesting and maybe invests some time to prepare patches for Qt,
glib, glibc — and PulseAudio.

Update: I added two more ideas to the list above.

Update 2: Andrew Haley just posted something like the optimal solution for the problem. It would be great if people would start using this.

Footnotes

[1] The Nokia 770 has an ARMv5 chip, N800 has ARMv6. The OpenMoko phone apparently uses ARMv5.

[2] And let’s not even think about CPUs which don’t even have an atomic swap!

[3] Which however you probably won’t, given that they’re only available on x86 on stable Linux kernels for now — but still, it’s cleaner.

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close