Adblock

Post Syndicated from RealEnder original https://alex.stanev.org/blog/?p=335

Скоро не се бях занимавал активно с българските списъци за блокиране на нежелано съдържание. След като хората зад Adblock Plus ореваха, че насоките на страницата са доста out-of-date, преди малко направих доста обновления:

  • Премахната поддръжката за urlfilter.ini за Opera – това беше доста крив вариант за филтриране, който отдавна е deprecated;
  • Добавена информация за Adblock Plus за Android;
  • Коригирани статистиките за операционните системи – имаше проблем с визуализацията на Mac OS;
  • Обновен списъка с филтрите – след голямото коледно прочистване са добавени доста нови, подадени от потребители филтри.

Ако все още не сте изпробвали технологията, открийте как на Adblock за българи. Промените могат да се следят и в Twitter.

MariaDB living in interesting times

Post Syndicated from Monty original http://monty-says.blogspot.com/2013/02/mariadb-living-in-interesting-times.html

An old Chinese curse says “May you live in interesting times“.
Fortunately this can be taken both as a curse or as a blessing.

For MariaDB this has definitely been a blessing the last few months.

Since we announced the MariaDB foundation in December 2012 a lot has happened:

We will soon be able to announce that also some of the other very big Internet companies have moved to MariaDB. (stay tuned 🙂

The above means that the installation base of MariaDB will increase from about one million to 5-10 million within this year. You can find here a full updated lists of all distributions that supports MariaDB.

We are really living in interesting times!

Happy Chinese New Year to you all!

The Biggest Myths

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/the-biggest-myths.html

Since we first proposed systemd
for inclusion in the distributions it has been frequently discussed in
many forums, mailing lists and conferences. In these discussions one
can often hear certain myths about systemd, that are repeated over and
over again, but certainly don’t gain any truth by constant
repetition. Let’s take the time to debunk a few of them:

  1. Myth: systemd is monolithic.

    If you build systemd with all configuration options enabled you
    will build 69 individual binaries. These binaries all serve different
    tasks, and are neatly separated for a number of reasons. For example,
    we designed systemd with security in mind, hence most daemons run at
    minimal privileges (using kernel capabilities, for example) and are
    responsible for very specific tasks only, to minimize their security
    surface and impact. Also, systemd parallelizes the boot more than any
    prior solution. This parallization happens by running more processes
    in parallel. Thus it is essential that systemd is nicely split up into
    many binaries and thus processes. In fact, many of these
    binaries[1] are separated out so nicely, that they are very
    useful outside of systemd, too.

    A package involving 69 individual binaries can hardly be called
    monolithic. What is different from prior solutions however,
    is that we ship more components in a single tarball, and maintain them
    upstream in a single repository with a unified release cycle.

  2. Myth: systemd is about speed.

    Yes, systemd is fast (A
    pretty complete userspace boot-up in ~900ms, anyone?
    ), but that’s
    primarily just a side-effect of doing things right. In fact, we
    never really sat down and optimized the last tiny bit of performance
    out of systemd. Instead, we actually frequently knowingly picked the
    slightly slower code paths in order to keep the code more
    readable. This doesn’t mean being fast was irrelevant for us, but
    reducing systemd to its speed is certainly quite a misconception,
    since that is certainly not anywhere near the top of our list of
    goals.

  3. Myth: systemd’s fast boot-up is irrelevant for
    servers.

    That is just completely not true. Many administrators actually are
    keen on reduced downtimes during maintenance windows. In High
    Availability setups it’s kinda nice if the failed machine comes back
    up really fast. In cloud setups with a large number of VMs or
    containers the price of slow boots multiplies with the number of
    instances. Spending minutes of CPU and IO on really slow boots of
    hundreds of VMs or containers reduces your system’s density
    drastically, heck, it even costs you more energy. Slow boots can be
    quite financially expensive. Then, fast booting of containers allows
    you to implement a logic such as socket
    activated containers
    , allowing you to drastically increase the
    density of your cloud system.

    Of course, in many server setups boot-up is indeed irrelevant, but
    systemd is supposed to cover the whole range. And yes, I am aware
    that often it is the server firmware that costs the most time at
    boot-up, and the OS anyways fast compared to that, but well, systemd
    is still supposed to cover the whole range (see above…), and no,
    not all servers have such bad firmware, and certainly not VMs and
    containers, which are servers of a kind, too.[2]

  4. Myth: systemd is incompatible with shell scripts.

    This is entirely bogus. We just don’t use them for the boot
    process, because we believe they aren’t the best tool for that
    specific purpose, but that doesn’t mean systemd was incompatible with
    them. You can easily run shell scripts as systemd services, heck, you
    can run scripts written in any language as systemd services,
    systemd doesn’t care the slightest bit what’s inside your
    executable. Moreover, we heavily use shell scripts for our own
    purposes, for installing, building, testing systemd. And you can stick
    your scripts in the early boot process, use them for normal services,
    you can run them at latest shutdown, there are practically no
    limits.

  5. Myth: systemd is difficult.

    This also is entire non-sense. A systemd platform is actually much
    simpler than traditional Linuxes because it unifies
    system objects and their dependencies as systemd units. The
    configuration file language is very simple, and redundant
    configuration files we got rid of. We provide uniform tools for much
    of the configuration of the system. The system is much less
    conglomerate than traditional Linuxes are. We also have pretty
    comprehensive documentation (all linked
    from the homepage
    ) about pretty much every detail of systemd, and
    this not only covers admin/user-facing interfaces, but also developer
    APIs.

    systemd certainly comes with a learning curve. Everything
    does. However, we like to believe that it is actually simpler to
    understand systemd than a Shell-based boot for most people. Surprised
    we say that? Well, as it turns out, Shell is not a pretty language to
    learn, it’s syntax is arcane and complex. systemd unit files are
    substantially easier to understand, they do not expose a programming
    language, but are simple and declarative by nature. That all said, if
    you are experienced in shell, then yes, adopting systemd will take a
    bit of learning.

    To make learning easy we tried hard to provide the maximum
    compatibility to previous solutions. But not only that, on many
    distributions you’ll find that some of the traditional tools will now
    even tell you — while executing what you are asking for — how you
    could do it with the newer tools instead, in a possibly nicer way.

    Anyway, the take-away is probably that systemd is probably as
    simple as such a system can be, and that we try hard to make it easy
    to learn. But yes, if you know sysvinit then adopting systemd will
    require a bit learning, but quite frankly if you mastered sysvinit,
    then systemd should be easy for you.

  6. Myth: systemd is not modular.

    Not true at all. At compile time you have a number of
    configure switches to select what you want to build, and what
    not. And we
    document
    how you can select in even more detail what you need,
    going beyond our configure switches.

    This modularity is not totally unlike the one of the Linux kernel,
    where you can select many features individually at compile time. If the
    kernel is modular enough for you then systemd should be pretty close,
    too.

  7. Myth: systemd is only for desktops.

    That is certainly not true. With systemd we try to cover pretty
    much the same range as Linux itself does. While we care for desktop
    uses, we also care pretty much the same way for server uses, and
    embedded uses as well. You can bet that Red Hat wouldn’t make it a
    core piece of RHEL7 if it wasn’t the best option for managing services
    on servers.

    People from numerous companies work on systemd. Car manufactureres
    build it into cars, Red Hat uses it for a server operating system, and
    GNOME uses many of its interfaces for improving the desktop. You find
    it in toys, in space telescopes, and in wind turbines.

    Most features I most recently worked on are probably relevant
    primarily on servers, such as container
    support
    , resource
    management
    or the security
    features
    . We cover desktop systems pretty well already, and there
    are number of companies doing systemd development for embedded, some
    even offer consulting services in it.

  8. Myth: systemd was created as result of the NIH syndrome.

    This is not true. Before we began working on systemd we were
    pushing for Canonical’s Upstart to be widely adopted (and Fedora/RHEL
    used it too for a while). However, we eventually came to the
    conclusion that its design was inherently flawed at its core (at least
    in our eyes: most fundamentally, it leaves dependency management to
    the admin/developer, instead of solving this hard problem in code),
    and if something’s wrong in the core you better replace it, rather
    than fix it. This was hardly the only reason though, other things that
    came into play, such as the licensing/contribution agreement mess
    around it. NIH wasn’t one of the reasons, though…[3]

  9. Myth: systemd is a freedesktop.org project.

    Well, systemd is certainly hosted at fdo, but freedesktop.org is
    little else but a repository for code and documentation. Pretty much
    any coder can request a repository there and dump his stuff there (as
    long as it’s somewhat relevant for the infrastructure of free
    systems). There’s no cabal involved, no “standardization” scheme, no
    project vetting, nothing. It’s just a nice, free, reliable place to
    have your repository. In that regard it’s a bit like SourceForge,
    github, kernel.org, just not commercial and without over-the-top
    requirements, and hence a good place to keep our stuff.

    So yes, we host our stuff at fdo, but the implied assumption of
    this myth in that there was a group of people who meet and then agree
    on how the future free systems look like, is entirely bogus.

  10. Myth: systemd is not UNIX.

    There’s certainly some truth in that. systemd’s sources do not
    contain a single line of code originating from original UNIX. However,
    we derive inspiration from UNIX, and thus there’s a ton of UNIX in
    systemd. For example, the UNIX idea of “everything is a file” finds
    reflection in that in systemd all services are exposed at runtime in a
    kernel file system, the cgroupfs. Then, one of the original
    features of UNIX was multi-seat support, based on built-in terminal
    support. Text terminals are hardly the state of the art how you
    interface with your computer these days however. With systemd we
    brought native multi-seat
    support back, but this time with full support for today’s hardware,
    covering graphics, mice, audio, webcams and more, and all that fully
    automatic, hotplug-capable and without configuration. In fact the
    design of systemd as a suite of integrated tools that each have their
    individual purposes but when used together are more than just the sum
    of the parts, that’s pretty much at the core of UNIX philosophy. Then,
    the way our project is handled (i.e. maintaining much of the core OS
    in a single git repository) is much closer to the BSD model (which is
    a true UNIX, unlike Linux) of doing things (where most of the core OS
    is kept in a single CVS/SVN repository) than things on Linux ever
    were.

    Ultimately, UNIX is something different for everybody. For us
    systemd maintainers it is something we derive inspiration from. For
    others it is a religion, and much like the other world religions there
    are different readings and understandings of it. Some define UNIX
    based on specific pieces of code heritage, others see it just as a set
    of ideas, others as a set of commands or APIs, and even others as a
    definition of behaviours. Of course, it is impossible to ever make all
    these people happy.

    Ultimately the question whether something is UNIX or not matters
    very little. Being technically excellent is hardly exclusive to
    UNIX. For us, UNIX is a major influence (heck, the biggest one), but
    we also have other influences. Hence in some areas systemd will be
    very UNIXy, and in others a little bit less.

  11. Myth: systemd is complex.

    There’s certainly some truth in that. Modern computers are complex
    beasts, and the OS running on it will hence have to be complex
    too. However, systemd is certainly not more complex than prior
    implementations of the same components. Much rather, it’s simpler, and
    has less redundancy (see above). Moreover, building a simple OS based
    on systemd will involve much fewer packages than a traditional Linux
    did. Fewer packages makes it easier to build your system, gets rid of
    interdependencies and of much of the different behaviour of every
    component involved.

  12. Myth: systemd is bloated.

    Well, bloated certainly has many different definitions. But in
    most definitions systemd is probably the opposite of bloat. Since
    systemd components share a common code base, they tend to share much
    more code for common code paths. Here’s an example: in a traditional
    Linux setup, sysvinit, start-stop-daemon, inetd, cron, dbus, all
    implemented a scheme to execute processes with various configuration
    options in a certain, hopefully clean environment. On systemd the code
    paths for all of this, for the configuration parsing, as well as the
    actual execution is shared. This means less code, less place for
    mistakes, less memory and cache pressure, and is thus a very good
    thing. And as a side-effect you actually get a ton more functionality
    for it…

    As mentioned above, systemd is also pretty modular. You can choose
    at build time which components you need, and which you don’t
    need. People can hence specifically choose the level of “bloat” they
    want.

    When you build systemd, it only requires three dependencies: glibc,
    libcap and dbus. That’s it. It can make use of more dependencies, but
    these are entirely optional.

    So, yeah, whichever way you look at it, it’s really not
    bloated.

  13. Myth: systemd being Linux-only is not nice to the BSDs.

    Completely wrong. The BSD folks are pretty much uninterested in
    systemd. If systemd was portable, this would change nothing, they
    still wouldn’t adopt it. And the same is true for the other Unixes in
    the world. Solaris has SMF, BSD has their own “rc” system, and they
    always maintained it separately from Linux. The init system is very
    close to the core of the entire OS. And these other operating systems
    hence define themselves among other things by their core
    userspace. The assumption that they’d adopt our core userspace if we
    just made it portable, is completely without any foundation.

  14. Myth: systemd being Linux-only makes it impossible for Debian to adopt it as default.

    Debian supports non-Linux kernels in their distribution. systemd
    won’t run on those. Is that a problem though, and should that hinder
    them to adopt system as default? Not really. The folks who ported
    Debian to these other kernels were willing to invest time in a massive
    porting effort, they set up test and build systems, and patched and
    built numerous packages for their goal. The maintainance of both a
    systemd unit file and a classic init script for the packaged services
    is a negligable amount of work compared to that, especially since
    those scripts more often than not exist already.

  15. Myth: systemd could be ported to other kernels if its maintainers just wanted to.

    That is simply not true. Porting systemd to other kernel is not
    feasible. We just use too many Linux-specific interfaces. For a few
    one might find replacements on other kernels, some features one might
    want to turn off, but for most this is nor really possible. Here’s a
    small, very incomprehensive list: cgroups, fanotify, umount2(),
    /proc/self/mountinfo
    (including notification), /dev/swaps (same),
    udev, netlink,
    the structure of /sys, /proc/$PID/comm,
    /proc/$PID/cmdline, /proc/$PID/loginuid, /proc/$PID/stat,
    /proc/$PID/session, /proc/$PID/exe, /proc/$PID/fd, tmpfs, devtmpfs,
    capabilities, namespaces of all kinds, various prctl()s, numerous
    ioctls,
    the mount() system call and its semantics, selinux, audit,
    inotify, statfs, O_DIRECTORY, O_NOATIME, /proc/$PID/root, waitid(),
    SCM_CREDENTIALS, SCM_RIGHTS, mkostemp(), /dev/input, ...

    And no, if you look at this list and pick out the few where you can
    think of obvious counterparts on other kernels, then think again, and
    look at the others you didn’t pick, and the complexity of replacing
    them.

  16. Myth: systemd is not portable for no reason.

    Non-sense! We use the Linux-specific functionality because we need
    it to implement what we want. Linux has so many features that
    UNIX/POSIX didn’t have, and we want to empower the user with
    them. These features are incredibly useful, but only if they are
    actually exposed in a friendly way to the user, and that’s what we do
    with systemd.

  17. Myth: systemd uses binary configuration files.

    No idea who came up with this crazy myth, but it’s absolutely not
    true. systemd is configured pretty much exclusively via simple text
    files. A few settings you can also alter with the kernel command line
    and via environment variables. There’s nothing binary in its
    configuration (not even XML). Just plain, simple, easy-to-read text
    files.

  18. Myth: systemd is a feature creep.

    Well, systemd certainly covers more ground that it used to. It’s
    not just an init system anymore, but the basic userspace building
    block to build an OS from, but we carefully make sure to keep most of
    the features optional. You can turn a lot off at compile time, and
    even more at runtime. Thus you can choose freely how much feature
    creeping you want.

  19. Myth: systemd forces you to do something.

    systemd is not the mafia. It’s Free Software, you can do with it
    whatever you want, and that includes not using it. That’s pretty much
    the opposite of “forcing”.

  20. Myth: systemd makes it impossible to run syslog.

    Not true, we carefully made sure when we introduced
    the journal
    that all data is also passed on to any syslog daemon
    running. In fact, if something changed, then only that syslog gets
    more complete data now than it got before, since we now cover early
    boot stuff as well as STDOUT/STDERR of any system service.

  21. Myth: systemd is incompatible.

    We try very hard to provide the best possible compatibility with
    sysvinit. In fact, the vast majority of init scripts should work just
    fine on systemd, unmodified. However, there actually are indeed a few
    incompatibilities, but we try to document
    these
    and explain what to do about them. Ultimately every system
    that is not actually sysvinit itself will have a certain amount of
    incompatibilities with it since it will not share the exect same code
    paths.

    It is our goal to ensure that differences between the various
    distributions are kept at a minimum. That means unit files usually
    work just fine on a different distribution than you wrote it on, which
    is a big improvement over classic init scripts which are very hard to
    write in a way that they run on multiple Linux distributions, due to
    numerous incompatibilities between them.

  22. Myth: systemd is not scriptable, because of its D-Bus use.

    Not true. Pretty much every single D-Bus interface systemd provides
    is also available in a command line tool, for example in systemctl,
    loginctl,
    timedatectl,
    hostnamectl,
    localectl
    and suchlike. You can easily call these tools from shell scripts, they
    open up pretty much the entire API from the command line with
    easy-to-use commands.

    That said, D-Bus actually has bindings for almost any scripting
    language this world knows. Even from the shell you can invoke
    arbitrary D-Bus methods with dbus-send
    or gdbus. If
    anything, this improves scriptability due to the good support of D-Bus
    in the various scripting languages.

  23. Myth: systemd requires you to use some arcane configuration
    tools instead of allowing you to edit your configuration files
    directly.

    Not true at all. We offer some configuration tools, and using them
    gets you a bit of additional functionality (for example, command line
    completion for all settings!), but there’s no need at all to use
    them. You can always edit the files in question directly if you wish,
    and that’s fully supported. Of course sometimes you need to explicitly
    reload configuration of some daemon after editing the configuration,
    but that’s pretty much true for most UNIX services.

  24. Myth: systemd is unstable and buggy.

    Certainly not according to our data. We have been monitoring the
    Fedora bug tracker (and some others) closely for a long long time. The
    number of bugs is very low for such a central component of the OS,
    especially if you discount the numerous RFE bugs we track for the
    project. We are pretty good in keeping systemd out of the list of
    blocker bugs of the distribution. We have a relatively fast
    development cycle with mostly incremental changes to keep quality and
    stability high.

  25. Myth: systemd is not debuggable.

    False. Some people try to imply that the shell was a good
    debugger. Well, it isn’t really. In systemd we provide you with actual
    debugging features instead. For example: interactive debugging,
    verbose tracing, the ability to mask any component during boot, and
    more. Also, we provide documentation
    for it
    .

    It’s certainly well debuggable, we needed that for our own
    development work, after all. But we’ll grant you one thing: it uses
    different debugging tools, we believe more appropriate ones for the
    purpose, though.

  26. Myth: systemd makes changes for the changes’ sake.

    Very much untrue. We pretty much exclusively have technical
    reasons for the changes we make, and we explain them in the various
    pieces of documentation, wiki pages, blog articles, mailing list
    announcements. We try hard to avoid making incompatible changes, and
    if we do we try to document the why and how in detail. And if you
    wonder about something, just ask us!

  27. Myth: systemd is a Red-Hat-only project, is private property
    of some smart-ass developers, who use it to push their views to the
    world.

    Not true. Currently, there are 16 hackers with commit powers to the
    systemd git tree. Of these 16 only six are employed by Red Hat. The 10
    others are folks from ArchLinux, from Debian, from Intel, even from
    Canonical, Mandriva, Pantheon and a number of community folks with
    full commit rights. And they frequently commit big stuff, major
    changes. Then, there are 374 individuals with patches in our tree, and
    they too came from a number of different companies and backgrounds,
    and many of those have way more than one patch in the tree. The
    discussions about where we want to take systemd are done in the open,
    on our IRC channel (#systemd on freenode, you are always
    weclome), on our mailing
    list
    , and on public hackfests (such
    as our next one in Brno
    , you are invited). We regularly attend
    various conferences, to collect feedback, to explain what we are doing
    and why, like few others do. We maintain blogs, engage in social
    networks (we actually
    have some pretty interesting content on Google+
    , and our Google+
    Community is pretty alive, too
    .), and try really hard to explain
    the why and the how how we do things, and to listen to feedback and
    figure out where the current issues are (for example, from that
    feedback we compiled this lists of often heard myths about
    systemd…).

    What most systemd contributors probably share is a rough idea how a
    good OS should look like, and the desire to make it happen. However,
    by the very nature of the project being Open Source, and rooted in the
    community systemd is just what people want it to be, and if it’s not
    what they want then they can drive the direction with patches and
    code, and if that’s not feasible, then there are numerous other
    options to use, too, systemd is never exclusive.

    One goal of systemd is to unify the dispersed Linux landscape a
    bit. We try to get rid of many of the more pointless differences of
    the various distributions in various areas of the core OS. As part of
    that we sometimes adopt schemes that were previously used by only one
    of the distributions and push it to a level where it’s the default of
    systemd, trying to gently push everybody towards the same set of basic
    configuration. This is never exclusive though, distributions can
    continue to deviate from that if they wish, however, if they end-up
    using the well-supported default their work becomes much easier and
    they might gain a feature or two. Now, as it turns out, more
    frequently than not we actually adopted schemes that where Debianisms,
    rather than Fedoraisms/Redhatisms as best supported scheme by
    systemd. For example, systems running systemd now generally store
    their hostname in /etc/hostname, something that used to be
    specific to Debian and now is used across distributions.

    One thing we’ll grant you though, we sometimes can be
    smart-asses. We try to be prepared whenever we open our mouth, in
    order to be able to back-up with facts what we claim. That might make
    us appear as smart-asses.

    But in general, yes, some of the more influental contributors of
    systemd work for Red Hat, but they are in the minority, and systemd is
    a healthy, open community with different interests, different
    backgrounds, just unified by a few rough ideas where the trip should
    go, a community where code and its design counts, and certainly not
    company affiliation.

  28. Myth: systemd doesn’t support /usr split from the root directory.

    Non-sense. Since its beginnings systemd supports the
    --with-rootprefix= option to its configure script
    which allows you to tell systemd to neatly split up the stuff needed
    for early boot and the stuff needed for later on. All this logic is
    fully present and we keep it up-to-date right there in systemd’s build
    system.

    Of course, we still don’t think that actually
    booting with /usr unavailable is a good idea
    , but we
    support this just fine in our build system. This won’t fix the
    inherent problems of the scheme that you’ll encounter all across the
    board, but you can’t blame that on systemd, because in systemd we
    support this just fine.

  29. Myth: systemd doesn’t allow your to replace its components.

    Not true, you can turn off and replace pretty much any part of
    systemd, with very few exceptions. And those exceptions (such as
    journald) generally allow you to run an alternative side by side to
    it, while cooperating nicely with it.

  30. Myth: systemd’s use of D-Bus instead of sockets makes it intransparent.

    This claim is already contradictory in itself: D-Bus uses sockets
    as transport, too. Hence whenever D-Bus is used to send something
    around, a socket is used for that too. D-Bus is mostly a standardized
    serialization of messages to send over these sockets. If anything this
    makes it more transparent, since this serialization is well
    documented, understood and there are numerous tracing tools and
    language bindings for it. This is very much unlike the usual
    homegrown protocols the various classic UNIX daemons use to
    communicate locally.

Hmm, did I write I just wanted to debunk a “few” myths? Maybe these
were more than just a few… Anyway, I hope I managed to clear up a
couple of misconceptions. Thanks for your time.

Footnotes

[1] For example, systemd-detect-virt,
systemd-tmpfiles,
systemd-udevd are.

[2] Also, we are trying to do our little part on maybe
making this better. By exposing boot-time performance of the firmware
more prominently in systemd’s boot output we hope to shame the
firmware writers to clean up their stuff.

[3] And anyways, guess which project includes a library “libnih” — Upstart or systemd?[4]

[4] Hint: it’s not systemd!

Пайнерът и европроектите

Post Syndicated from Longanlon original http://kaka-cuuka.com/3254

По повод придобилия небивала известност европейски проект по програма “Конкурентноспособност” на фирма Пайнер ООД на стойност към 2 млн., си припомних колко напред е Митьо Пайнера в развиването на бизнеса си и печеленето на пари, както и колко много аз харесвам чалгата (не музикално, а идеологически).

Но по-важно, припомних си какво написах преди време в списание Forbes относно европейските проекти като цяло и техния смисъл (или по-скоро липсата на такъв). Не, проект за развитие на конкурентсноспособността на Пайнер ООД не е неморален – напротив – спечелили са хората, да си го изпълняват. Неморална е самата система на европейските проекти, която раздава пари за абсолютни глупости. Статията ми в сайта на Forbes, отразяваща мнението ми, формирано след почти десетгодишна работа в сферата на европейските проекти.

(Чети още…) (0 думи)

systemd for Administrators, Part XX

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/socket-activated-containers.html

This is no time for procrastination,
here
is
already the twentieth
installment
of

my ongoing series
on
systemd
for
Administrators:

Socket Activated Internet Services and OS Containers

Socket
Activation
is an important feature of systemd. When
we first
announced
systemd we already tried to make the point how great
socket activation is for increasing parallelization and robustness of
socket services, but also for simplifying the dependency logic of the
boot. In this episode I’d like to explain why socket activation is an
important tool for drastically improving how many services and even
containers you can run on a single system with the same resource
usage. Or in other words, how you can drive up the density of customer
sites on a system while spending less on new hardware.

Socket Activated Internet Services

First, let’s take a step back. What was socket activation again? —
Basically, socket activation simply means that systemd sets up
listening sockets (IP or otherwise) on behalf of your services
(without these running yet), and then starts (activates) the
services as soon as the first connection comes in. Depending on the
technology the services might idle for a while after having processed
the connection and possible follow-up connections before they exit on
their own, so that systemd will again listen on the sockets and
activate the services again the next time they are connected to. For
the client it is not visible whether the service it is interested in
is currently running or not. The service’s IP socket stays continously
connectable, no connection attempt ever fails, and all connects will
be processed promptly.

A setup like this lowers resource usage: as services are only
running when needed they only consume resources when required. Many
internet sites and services can benefit from that. For example, web
site hosters will have noticed that of the multitude of web sites that
are on the Internet only a tiny fraction gets a continous stream of
requests: the huge majority of web sites still needs to be available
all the time but gets requests only very unfrequently. With a scheme
like socket activation you take benefit of this. By hosting many of
these sites on a single system like this and only activating their
services as necessary allows a large degree of over-commit: you can
run more sites on your system than the available resources actually
allow. Of course, one shouldn’t over-commit too much to avoid
contention during peak times.

Socket activation like this is easy to use in systemd. Many modern
Internet daemons already support socket activation out of the box (and
for those which don’t yet it’s not
hard
to add). Together with systemd’s instantiated
units support
it is easy to write a pair of service and socket
templates that then may be instantiated multiple times, once for each
site. Then, (optionally) make use of some of the security
features
of systemd to nicely isolate the customer’s site’s
services from each other (think: each customer’s service should only
see the home directory of the customer, everybody else’s directories
should be invisible), and there you go: you now have a highly scalable
and reliable server system, that serves a maximum of securely
sandboxed services at a minimum of resources, and all nicely done with
built-in technology of your OS.

This kind of setup is already in production use in a number of
companies. For example, the great folks at Pantheon are running their
scalable instant Drupal system on a setup that is similar to this. (In
fact, Pantheon’s David Strauss pioneered this scheme. David, you
rock!)

Socket Activated OS Containers

All of the above can already be done with older versions of
systemd. If you use a distribution that is based on systemd, you can
right-away set up a system like the one explained above. But let’s
take this one step further. With systemd 197 (to be included in Fedora
19), we added support for socket activating not only individual
services, but entire OS containers. And I really have to say it
at this point: this is stuff I am really excited
about. 😉

Basically, with socket activated OS containers, the host’s systemd
instance will listen on a number of ports on behalf of a container,
for example one for SSH, one for web and one for the database, and as
soon as the first connection comes in, it will spawn the container
this is intended for, and pass to it all three sockets. Inside of the
container, another systemd is running and will accept the sockets and
then distribute them further, to the services running inside the
container using normal socket activation. The SSH, web and database
services will only see the inside of the container, even though they
have been activated by sockets that were originally created on the
host! Again, to the clients this all is not visible. That an entire OS
container is spawned, triggered by simple network connection is entirely
transparent to the client side.[1]

The OS containers may contain (as the name suggests) a full
operating system, that might even be a different distribution than is
running on the host. For example, you could run your host on Fedora,
but run a number of Debian containers inside of it. The OS containers
will have their own systemd init system, their own SSH instances,
their own process tree, and so on, but will share a number of other
facilities (such as memory management) with the host.

For now, only systemd’s own trivial container manager, systemd-nspawn
has been updated to support this kind of socket activation. We hope
that libvirt-lxc will
soon gain similar functionality. At this point, let’s see in more
detail how such a setup is configured in systemd using nspawn:

First, please use a tool such as debootstrap or yum’s
--installroot to set up a container OS
tree[2]. The details of that are a bit out-of-focus
for this story, there’s plenty of documentation around how to do
this. Of course, make sure you have systemd v197 installed inside
the container. For accessing the container from the command line,
consider using systemd-nspawn
itself. After you configured everything properly, try to boot it up
from the command line with systemd-nspawn’s -b switch.

Assuming you now have a working container that boots up fine, let’s
write a service file for it, to turn the container into a systemd
service on the host you can start and stop. Let’s create
/etc/systemd/system/mycontainer.service on the host:

[Unit]
Description=My little container

[Service]
ExecStart=/usr/bin/systemd-nspawn -jbD /srv/mycontainer 3
KillMode=process

This service can already be started and stopped via systemctl
start
and systemctl stop. However, there’s no nice way
to actually get a shell prompt inside the container. So let’s add SSH
to it, and even more: let’s configure SSH so that a connection to the
container’s SSH port will socket-activate the entire container. First,
let’s begin with telling the host that it shall now listen on the SSH
port of the container. Let’s create
/etc/systemd/system/mycontainer.socket on the host:

[Unit]
Description=The SSH socket of my little container

[Socket]
ListenStream=23

If we start this unit with systemctl start on the host
then it will listen on port 23, and as soon as a connection comes in
it will activate our container service we defined above. We pick port
23 here, instead of the usual 22, as our host’s SSH is already
listening on that. nspawn virtualizes the process list and the file
system tree, but does not actually virtualize the network stack, hence
we just pick different ports for the host and the various containers
here.

Of course, the system inside the container doesn’t yet know what to
do with the socket it gets passed due to socket activation. If you’d
now try to connect to the port, the container would start-up but the
incoming connection would be immediately closed since the container
can’t handle it yet. Let’s fix that!

All that’s necessary for that is teach SSH inside the container
socket activation. For that let’s simply write a pair of socket and
service units for SSH. Let’s create
/etc/systemd/system/sshd.socket in the container:

[Unit]
Description=SSH Socket for Per-Connection Servers

[Socket]
ListenStream=23
Accept=yes

Then, let’s add the matching SSH service file
/etc/systemd/system/[email protected] in the container:

[Unit]
Description=SSH Per-Connection Server for %I

[Service]
ExecStart=-/usr/sbin/sshd -i
StandardInput=socket

Then, make sure to hook sshd.socket into the
sockets.target so that unit is started automatically when the
container boots up:

ln -s /etc/systemd/system/sshd.socket /etc/systemd/system/sockets.target.wants/

And that’s it. If we now activate mycontainer.socket on
the host, the host’s systemd will bind the socket and we can connect
to it. If we do this, the host’s systemd will activate the container,
and pass the socket in to it. The container’s systemd will then take
the socket, match it up with sshd.socket inside the
container. As there’s still our incoming connection queued on it, it
will then immediately trigger an instance of [email protected],
and we’ll have our login.

And that’s already everything there is to it. You can easily add
additional sockets to listen on to
mycontainer.socket. Everything listed therein will be passed
to the container on activation, and will be matched up as good as
possible with all socket units configured inside the
container. Sockets that cannot be matched up will be closed, and
sockets that aren’t passed in but are configured for listening will be
bound be the container’s systemd instance.

So, let’s take a step back again. What did we gain through all of
this? Well, basically, we can now offer a number of full OS containers
on a single host, and the containers can offer their services without
running continously. The density of OS containers on the host can
hence be increased drastically.

Of course, this only works for kernel-based virtualization, not for
hardware virtualization. i.e. something like this can only be
implemented on systems such as libvirt-lxc or nspawn, but not in
qemu/kvm.

If you have a number of containers set up like this, here’s one
cool thing the journal allows you to do. If you pass -m to
journalctl on the host, it will automatically discover the
journals of all local containers and interleave them on
display. Nifty, eh?

With systemd 197 you have everything to set up your own socket
activated OS containers on-board. However, there are a couple of
improvements we’re likely to add soon: for example, right now even if
all services inside the container exit on idle, the container still
will stay around, and we really should make it exit on idle too, if
all its services exited and no logins are around. As it turns out we
already have much of the infrastructure for this around: we can reuse
the auto-suspend functionality we added for laptops: detecting when a
laptop is idle and suspending it then is a very similar problem to
detecting when a container is idle and shutting it down then.

Anyway, this blog story is already way too long. I hope I haven’t
lost you half-way already with all this talk of virtualization,
sockets, services, different OSes and stuff. I hope this blog story is
a good starting point for setting up powerful highly scalable server
systems. If you want to know more, consult the documentation and drop
by our IRC channel. Thank you!

Footnotes

[1] And BTW, this
is another reason
why fast boot times the way systemd offers them
are actually a really good thing on servers, too.

[2] To make it easy: you need a command line such as yum
--releasever=19 --nogpg --installroot=/srv/mycontainer/ --disablerepo='*'
--enablerepo=fedora install systemd passwd yum fedora-release vim-minimal

to install Fedora, and debootstrap --arch=amd64 unstable
/srv/mycontainer/
to install Debian. Also see the bottom of systemd-nspawn(1).
Also note that auditing is currently broken for containers, and if enabled in
the kernel will cause all kinds of errors in the container. Use
audit=0 on the host’s kernel command line to turn it off.

systemd for Administrators, Part XIX

Post Syndicated from Lennart Poettering original https://0pointer.net/blog/projects/detect-virt.html

Happy new year
2013! Here
is
now the nineteenth
installment
of

my ongoing series
on
systemd
for
Administrators:

Detecting Virtualization

When we started working on systemd
we had a closer look on what the various existing init scripts used on
Linux where actually doing. Among other things we noticed that a
number of them where checking explicitly whether they were running in
a virtualized environment (i.e. in a kvm, VMWare, LXC guest or
suchlike) or not. Some init scripts disabled themselves in such
cases[1], others enabled themselves only in such
cases[2]. Frequently, it would probably have been a better
idea to check for other conditions rather than explicitly checking for
virtualization, but after looking at this from all sides we came to
the conclusion that in many cases explicitly conditionalizing services
based on detected virtualization is a valid thing to do. As a result
we added a new configuration option to systemd that can be used to
conditionalize services this way: ConditionVirtualization;
we also added a small tool that can be used in shell scripts to detect
virtualization: systemd-detect-virt(1);
and finally, we added a minimal bus interface to query this from other
applications.

Detecting whether your code is run inside a virtualized environment
is
actually not that hard
. Depending on what precisely you want to
detect it’s little more than running the CPUID instruction and maybe
checking a few files in /sys and /proc. The
complexity is mostly about knowing the strings to look for, and
keeping this list up-to-date. Currently, the the virtualization
detection code in systemd can detect the following virtualization
systems:

  • Hardware virtualization (i.e. VMs):

    • qemu
    • kvm
    • vmware
    • microsoft
    • oracle
    • xen
    • bochs
  • Same-kernel virtualization (i.e. containers):

Let’s have a look how one may make use if this functionality.

Conditionalizing Units

Adding ConditionVirtualization
to the [Unit] section of a unit file is enough to
conditionalize it depending on which virtualization is used or whether
one is used at all. Here’s an example:

[Unit]
Name=My Foobar Service (runs only only on guests)
ConditionVirtualization=yes

[Service]
ExecStart=/usr/bin/foobard

Instead of specifiying “yes” or “no” it is possible
to specify the ID of a specific virtualization solution (Example:
“kvm“, “vmware“, …), or either
“container” or “vm” to check whether the kernel is
virtualized or the hardware. Also, checks can be prefixed with an exclamation mark (“!”) to invert a check. For further details see the manual page.

In Shell Scripts

In shell scripts it is easy to check for virtualized systems with
the systemd-detect-virt(1)
tool. Here’s an example:

if systemd-detect-virt -q ; then
        echo "Virtualization is used:" `systemd-detect-virt`
else
        echo "No virtualization is used."
fi

If this tool is run it will return with an exit code of zero
(success) if a virtualization solution has been found, non-zero
otherwise. It will also print a short identifier of the used
virtualization solution, which can be suppressed with
-q. Also, with the -c and -v parameters it is
possible to detect only kernel or only hardware virtualization
environments. For further details see the manual
page
.

In Programs

Whether virtualization is available is also exported on the system bus:

$ gdbus call --system --dest org.freedesktop.systemd1 --object-path /org/freedesktop/systemd1 --method org.freedesktop.DBus.Properties.Get org.freedesktop.systemd1.Manager Virtualization
(<'systemd-nspawn'>,)

This property contains the empty string if no virtualization is
detected. Note that some container environments cannot be detected
directly from unprivileged code. That’s why we expose this property on
the bus rather than providing a library — the bus implicitly solves
the privilege problem quite nicely.

Note that all of this will only ever detect and return information
about the “inner-most” virtualization solution. If you stack
virtualization (“We must go deeper!”) then these interfaces will
expose the one the code is most directly interfacing
with. Specifically that means that if a container solution is used
inside of a VM, then only the container is generally detected and
returned.

Footonotes

[1] For example: running certain device management service in a
container environment that has no access to any physical hardware makes little sense.

[2] For example: some VM solutions work best if certain
vendor-specific userspace components are running that connect the
guest with the host in some way.

Perl is Free Software’s COBOL, and That’s Ok!

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2012/12/18/perl-cobol.html

In 1991, I’d just gotten my first real programming job for two reasons:
nepotism, and a willingness to write code for $12/hour. I was working
as a contractor to a blood testing laboratory, where the main
development job was writing custom software to handle, process, and do
statistical calculations on blood testing results, primarily for
paternity testing.

My father had been a software developer since the early 1970s, and worked
as a contractor at this blood lab since the late 1970s. As the calendar
had marched toward the early 1990s, technology cruft had collected. The
old TI mainframe, once the primary computer, now only had one job left:
statistical calculation for paternity testing, written in TI’s Pascal.
Slowly but surely, the other software had been rewritten and moved to an
AT&T 3B2/600 running Unix System VR3.2.3. That latter machine was the
first access I had to a real computer, and certainly the first time I had
access to Usenet. This changed my life.

Ironically, even on that 3B2, the accounting system software was written
in COBOL. This seemed like “more cruft” to me, but fortunately
there was a third-party vendor who handled that software, so I didn’t have
to program in COBOL.

I had the good fortune, actually, to help with the interesting problems,
which included grokking data from a blood testing machine that dumped a
bunch of data in some weird reporting format onto its RS-232 port at the
end of every testing cycle. We had to pull the data of that RS-232
interface and load the data in the database. Perl, since it treated
regular expressions as first-class citizens, and had all the Unix block
device fundamentals baked in as native (for the RS-232 I/O), was the
obvious choice.

After that project, I was intrigued by this programming language that had
made the job so easy. My father gave me a copy of the Camel book —
which was, at that point, almost hot off the presses. I read it over a
weekend and I decided that I didn’t really want to program in any other
language again. Perl was just 4 years old then; it was a young language
— Perl 4 had just been released. I started trying to embed Perl into
our database system, but it wasn’t designed for embedding into other
systems as a scripting language. So, I ended up using Tcl instead for the big
project of rewriting the statical calculation software to replace the TI
mainframe. After a year or two writing tens of thousands of lines of Tcl,
I was even more convinced that I’d rather be writing in Perl. When
Perl 5 was released, I switched back to Perl and never really looked
back.

Perl ultimately became my first Free Software community. I lurked on
perl5-porters for years, almost always a bit too timid to post, or ever
send in a patch. But, as I finished my college degree and went to graduate
school, I focused my
thesis work on Perl and virtual machines
. I went to the Perl
conference every year. I was even in the room for the perl5-porters
meeting the day
after Jon
Orwant’s staged tantrum
, which was the catalyst for the Perl 6
effort. I wrote more than a few RFC’s during
the Perl 6 specification
process
. And, to this day, even though I’ve since
done plenty of Python
development
, too, when I need to program to do something, I open an Emacs
buffer and start typing #!/usr/bin/perl.

Meanwhile, I never did learn COBOL. But, I was amazed to hear that
multiple folks who graduated with me eventually got jobs at a health
insurance company. The company trained them in COBOL, so that they could
maintain COBOL systems all day. Everyone once in a while, I idly search a
job site for COBOL. Today, that search is returning 2,338 open jobs. Most
developers never hear about it, of course. It’s far from the exciting new
technology, but it’s there, it’s needed and it’s obviously useful to
someone. Indeed, the COBOL standard was just updated 10 years ago, in
2002!

I notice these days, though, that when I mentioned having done a lot of
Perl development in my life, the average Javascript, Python, or Haskell
developer looks at me like I looked at my dad when he told me that
accounting system was written in COBOL. I’d bet they’d have my same
sigh of relief when told that “someone else” maintains that
code and they won’t have to bother with it.

Yet, I still know people heavily immersed in the Perl community. Indeed,
there is a very active Perl community out there, just like there’s an
active COBOL community. I’m not active in Perl like I once was, but it’s a
community of people, who write new code and maintain old code in Perl, and
that has value. More importantly, though, (and unlike COBOL), Perl was
born on Usenet, and was released as Free Software from the day of its first
release, twenty-five years ago today. Perl was born as part of Free
Software culture, and it lives on.

So, I get it now. I once scoffed at the idea that anyone would write
in COBOL anymore, as if the average COBOL programmer was some sort of
second-class technology citizen. COBOL programmers in 1991, and even
today, are surely good programmers — doing useful things for their
jobs. The same is true of Perl these days: maybe Perl is finally
getting a bit old fashioned — but there are good developers, still
doing useful things with Perl. Perl is becoming Free Software’s COBOL:
an aging language that still has value.

Perl turns 25
years old
today. COBOL was 25 years old in 1984, right at the time
when I first started programming. To those young people who start
programming today: I hope you’ll learn from my mistake. Don’t scoff at the
Perl programmers. 25 years from now, you may regret scoffing at them as
much as I regret scoffing at the COBOL developers. Programmers are
programmers; don’t judge them because you don’t like their favorite
language.

Update (2013-04-12):
I posted
a comment on Allison Randal’s blog about similar issues of Perl’s
popularity
.

Perl is Free Software’s COBOL, and That’s Ok!

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2012/12/18/perl-cobol.html

In 1991, I’d just gotten my first real programming job for two reasons:
nepotism, and a willingness to write code for $12/hour. I was working
as a contractor to a blood testing laboratory, where the main
development job was writing custom software to handle, process, and do
statistical calculations on blood testing results, primarily for
paternity testing.

My father had been a software developer since the early 1970s, and worked
as a contractor at this blood lab since the late 1970s. As the calendar
had marched toward the early 1990s, technology cruft had collected. The
old TI mainframe, once the primary computer, now only had one job left:
statistical calculation for paternity testing, written in TI’s Pascal.
Slowly but surely, the other software had been rewritten and moved to an
AT&T 3B2/600 running Unix System VR3.2.3. That latter machine was the
first access I had to a real computer, and certainly the first time I had
access to Usenet. This changed my life.

Ironically, even on that 3B2, the accounting system software was written
in COBOL. This seemed like “more cruft” to me, but fortunately
there was a third-party vendor who handled that software, so I didn’t have
to program in COBOL.

I had the good fortune, actually, to help with the interesting problems,
which included grokking data from a blood testing machine that dumped a
bunch of data in some weird reporting format onto its RS-232 port at the
end of every testing cycle. We had to pull the data of that RS-232
interface and load the data in the database. Perl, since it treated
regular expressions as first-class citizens, and had all the Unix block
device fundamentals baked in as native (for the RS-232 I/O), was the
obvious choice.

After that project, I was intrigued by this programming language that had
made the job so easy. My father gave me a copy of the Camel book —
which was, at that point, almost hot off the presses. I read it over a
weekend and I decided that I didn’t really want to program in any other
language again. Perl was just 4 years old then; it was a young language
— Perl 4 had just been released. I started trying to embed Perl into
our database system, but it wasn’t designed for embedding into other
systems as a scripting language. So, I ended up using Tcl instead for the big
project of rewriting the statical calculation software to replace the TI
mainframe. After a year or two writing tens of thousands of lines of Tcl,
I was even more convinced that I’d rather be writing in Perl. When
Perl 5 was released, I switched back to Perl and never really looked
back.

Perl ultimately became my first Free Software community. I lurked on
perl5-porters for years, almost always a bit too timid to post, or ever
send in a patch. But, as I finished my college degree and went to graduate
school, I focused my
thesis work on Perl and virtual machines
. I went to the Perl
conference every year. I was even in the room for the perl5-porters
meeting the day
after Jon
Orwant’s staged tantrum
, which was the catalyst for the Perl 6
effort. I wrote more than a few RFC’s during
the Perl 6 specification
process
. And, to this day, even though I’ve since
done plenty of Python
development
, too, when I need to program to do something, I open an Emacs
buffer and start typing #!/usr/bin/perl.

Meanwhile, I never did learn COBOL. But, I was amazed to hear that
multiple folks who graduated with me eventually got jobs at a health
insurance company. The company trained them in COBOL, so that they could
maintain COBOL systems all day. Everyone once in a while, I idly search a
job site for COBOL. Today, that search is returning 2,338 open jobs. Most
developers never hear about it, of course. It’s far from the exciting new
technology, but it’s there, it’s needed and it’s obviously useful to
someone. Indeed, the COBOL standard was just updated 10 years ago, in
2002!

I notice these days, though, that when I mentioned having done a lot of
Perl development in my life, the average Javascript, Python, or Haskell
developer looks at me like I looked at my dad when he told me that
accounting system was written in COBOL. I’d bet they’d have my same
sigh of relief when told that “someone else” maintains that
code and they won’t have to bother with it.

Yet, I still know people heavily immersed in the Perl community. Indeed,
there is a very active Perl community out there, just like there’s an
active COBOL community. I’m not active in Perl like I once was, but it’s a
community of people, who write new code and maintain old code in Perl, and
that has value. More importantly, though, (and unlike COBOL), Perl was
born on Usenet, and was released as Free Software from the day of its first
release, twenty-five years ago today. Perl was born as part of Free
Software culture, and it lives on.

So, I get it now. I once scoffed at the idea that anyone would write
in COBOL anymore, as if the average COBOL programmer was some sort of
second-class technology citizen. COBOL programmers in 1991, and even
today, are surely good programmers — doing useful things for their
jobs. The same is true of Perl these days: maybe Perl is finally
getting a bit old fashioned — but there are good developers, still
doing useful things with Perl. Perl is becoming Free Software’s COBOL:
an aging language that still has value.

Perl turns 25
years old
today. COBOL was 25 years old in 1984, right at the time
when I first started programming. To those young people who start
programming today: I hope you’ll learn from my mistake. Don’t scoff at the
Perl programmers. 25 years from now, you may regret scoffing at them as
much as I regret scoffing at the COBOL developers. Programmers are
programmers; don’t judge them because you don’t like their favorite
language.

Update (2013-04-12):
I posted
a comment on Allison Randal’s blog about similar issues of Perl’s
popularity
.

Perl is Free Software’s COBOL, and That’s Ok!

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2012/12/18/perl-cobol.html

In 1991, I’d just gotten my first real programming job for two reasons:
nepotism, and a willingness to write code for $12/hour. I was working
as a contractor to a blood testing laboratory, where the main
development job was writing custom software to handle, process, and do
statistical calculations on blood testing results, primarily for
paternity testing.

My father had been a software developer since the early 1970s, and worked
as a contractor at this blood lab since the late 1970s. As the calendar
had marched toward the early 1990s, technology cruft had collected. The
old TI mainframe, once the primary computer, now only had one job left:
statistical calculation for paternity testing, written in TI’s Pascal.
Slowly but surely, the other software had been rewritten and moved to an
AT&T 3B2/600 running Unix System VR3.2.3. That latter machine was the
first access I had to a real computer, and certainly the first time I had
access to Usenet. This changed my life.

Ironically, even on that 3B2, the accounting system software was written
in COBOL. This seemed like “more cruft” to me, but fortunately
there was a third-party vendor who handled that software, so I didn’t have
to program in COBOL.

I had the good fortune, actually, to help with the interesting problems,
which included grokking data from a blood testing machine that dumped a
bunch of data in some weird reporting format onto its RS-232 port at the
end of every testing cycle. We had to pull the data of that RS-232
interface and load the data in the database. Perl, since it treated
regular expressions as first-class citizens, and had all the Unix block
device fundamentals baked in as native (for the RS-232 I/O), was the
obvious choice.

After that project, I was intrigued by this programming language that had
made the job so easy. My father gave me a copy of the Camel book —
which was, at that point, almost hot off the presses. I read it over a
weekend and I decided that I didn’t really want to program in any other
language again. Perl was just 4 years old then; it was a young language
— Perl 4 had just been released. I started trying to embed Perl into
our database system, but it wasn’t designed for embedding into other
systems as a scripting language. So, I ended up using Tcl instead for the big
project of rewriting the statical calculation software to replace the TI
mainframe. After a year or two writing tens of thousands of lines of Tcl,
I was even more convinced that I’d rather be writing in Perl. When
Perl 5 was released, I switched back to Perl and never really looked
back.

Perl ultimately became my first Free Software community. I lurked on
perl5-porters for years, almost always a bit too timid to post, or ever
send in a patch. But, as I finished my college degree and went to graduate
school, I focused my
thesis work on Perl and virtual machines
. I went to the Perl
conference every year. I was even in the room for the perl5-porters
meeting the day
after Jon
Orwant’s staged tantrum
, which was the catalyst for the Perl 6
effort. I wrote more than a few RFC’s during
the Perl 6 specification
process
. And, to this day, even though I’ve since
done plenty of Python
development
, too, when I need to program to do something, I open an Emacs
buffer and start typing #!/usr/bin/perl.

Meanwhile, I never did learn COBOL. But, I was amazed to hear that
multiple folks who graduated with me eventually got jobs at a health
insurance company. The company trained them in COBOL, so that they could
maintain COBOL systems all day. Everyone once in a while, I idly search a
job site for COBOL. Today, that search is returning 2,338 open jobs. Most
developers never hear about it, of course. It’s far from the exciting new
technology, but it’s there, it’s needed and it’s obviously useful to
someone. Indeed, the COBOL standard was just updated 10 years ago, in
2002!

I notice these days, though, that when I mentioned having done a lot of
Perl development in my life, the average Javascript, Python, or Haskell
developer looks at me like I looked at my dad when he told me that
accounting system was written in COBOL. I’d bet they’d have my same
sigh of relief when told that “someone else” maintains that
code and they won’t have to bother with it.

Yet, I still know people heavily immersed in the Perl community. Indeed,
there is a very active Perl community out there, just like there’s an
active COBOL community. I’m not active in Perl like I once was, but it’s a
community of people, who write new code and maintain old code in Perl, and
that has value. More importantly, though, (and unlike COBOL), Perl was
born on Usenet, and was released as Free Software from the day of its first
release, twenty-five years ago today. Perl was born as part of Free
Software culture, and it lives on.

So, I get it now. I once scoffed at the idea that anyone would write
in COBOL anymore, as if the average COBOL programmer was some sort of
second-class technology citizen. COBOL programmers in 1991, and even
today, are surely good programmers — doing useful things for their
jobs. The same is true of Perl these days: maybe Perl is finally
getting a bit old fashioned — but there are good developers, still
doing useful things with Perl. Perl is becoming Free Software’s COBOL:
an aging language that still has value.

Perl turns 25
years old
today. COBOL was 25 years old in 1984, right at the time
when I first started programming. To those young people who start
programming today: I hope you’ll learn from my mistake. Don’t scoff at the
Perl programmers. 25 years from now, you may regret scoffing at them as
much as I regret scoffing at the COBOL developers. Programmers are
programmers; don’t judge them because you don’t like their favorite
language.

Update (2013-04-12):
I posted
a comment on Allison Randal’s blog about similar issues of Perl’s
popularity
.

Perl is Free Software’s COBOL, and That’s Ok!

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2012/12/18/perl-cobol.html

In 1991, I’d just gotten my first real programming job for two reasons:
nepotism, and a willingness to write code for $12/hour. I was working
as a contractor to a blood testing laboratory, where the main
development job was writing custom software to handle, process, and do
statistical calculations on blood testing results, primarily for
paternity testing.

My father had been a software developer since the early 1970s, and worked
as a contractor at this blood lab since the late 1970s. As the calendar
had marched toward the early 1990s, technology cruft had collected. The
old TI mainframe, once the primary computer, now only had one job left:
statistical calculation for paternity testing, written in TI’s Pascal.
Slowly but surely, the other software had been rewritten and moved to an
AT&T 3B2/600 running Unix System VR3.2.3. That latter machine was the
first access I had to a real computer, and certainly the first time I had
access to Usenet. This changed my life.

Ironically, even on that 3B2, the accounting system software was written
in COBOL. This seemed like “more cruft” to me, but fortunately
there was a third-party vendor who handled that software, so I didn’t have
to program in COBOL.

I had the good fortune, actually, to help with the interesting problems,
which included grokking data from a blood testing machine that dumped a
bunch of data in some weird reporting format onto its RS-232 port at the
end of every testing cycle. We had to pull the data of that RS-232
interface and load the data in the database. Perl, since it treated
regular expressions as first-class citizens, and had all the Unix block
device fundamentals baked in as native (for the RS-232 I/O), was the
obvious choice.

After that project, I was intrigued by this programming language that had
made the job so easy. My father gave me a copy of the Camel book —
which was, at that point, almost hot off the presses. I read it over a
weekend and I decided that I didn’t really want to program in any other
language again. Perl was just 4 years old then; it was a young language
— Perl 4 had just been released. I started trying to embed Perl into
our database system, but it wasn’t designed for embedding into other
systems as a scripting language. So, I ended up using Tcl instead for the big
project of rewriting the statical calculation software to replace the TI
mainframe. After a year or two writing tens of thousands of lines of Tcl,
I was even more convinced that I’d rather be writing in Perl. When
Perl 5 was released, I switched back to Perl and never really looked
back.

Perl ultimately became my first Free Software community. I lurked on
perl5-porters for years, almost always a bit too timid to post, or ever
send in a patch. But, as I finished my college degree and went to graduate
school, I focused my
thesis work on Perl and virtual machines
. I went to the Perl
conference every year. I was even in the room for the perl5-porters
meeting the day
after Jon
Orwant’s staged tantrum
, which was the catalyst for the Perl 6
effort. I wrote more than a few RFC’s during
the Perl 6 specification
process
. And, to this day, even though I’ve since
done plenty of Python
development
, too, when I need to program to do something, I open an Emacs
buffer and start typing #!/usr/bin/perl.

Meanwhile, I never did learn COBOL. But, I was amazed to hear that
multiple folks who graduated with me eventually got jobs at a health
insurance company. The company trained them in COBOL, so that they could
maintain COBOL systems all day. Everyone once in a while, I idly search a
job site for COBOL. Today, that search is returning 2,338 open jobs. Most
developers never hear about it, of course. It’s far from the exciting new
technology, but it’s there, it’s needed and it’s obviously useful to
someone. Indeed, the COBOL standard was just updated 10 years ago, in
2002!

I notice these days, though, that when I mentioned having done a lot of
Perl development in my life, the average Javascript, Python, or Haskell
developer looks at me like I looked at my dad when he told me that
accounting system was written in COBOL. I’d bet they’d have my same
sigh of relief when told that “someone else” maintains that
code and they won’t have to bother with it.

Yet, I still know people heavily immersed in the Perl community. Indeed,
there is a very active Perl community out there, just like there’s an
active COBOL community. I’m not active in Perl like I once was, but it’s a
community of people, who write new code and maintain old code in Perl, and
that has value. More importantly, though, (and unlike COBOL), Perl was
born on Usenet, and was released as Free Software from the day of its first
release, twenty-five years ago today. Perl was born as part of Free
Software culture, and it lives on.

So, I get it now. I once scoffed at the idea that anyone would write
in COBOL anymore, as if the average COBOL programmer was some sort of
second-class technology citizen. COBOL programmers in 1991, and even
today, are surely good programmers — doing useful things for their
jobs. The same is true of Perl these days: maybe Perl is finally
getting a bit old fashioned — but there are good developers, still
doing useful things with Perl. Perl is becoming Free Software’s COBOL:
an aging language that still has value.

Perl turns 25
years old
today. COBOL was 25 years old in 1984, right at the time
when I first started programming. To those young people who start
programming today: I hope you’ll learn from my mistake. Don’t scoff at the
Perl programmers. 25 years from now, you may regret scoffing at them as
much as I regret scoffing at the COBOL developers. Programmers are
programmers; don’t judge them because you don’t like their favorite
language.

Update (2013-04-12):
I posted
a comment on Allison Randal’s blog about similar issues of Perl’s
popularity
.

Perl is Free Software’s COBOL, and That’s Ok!

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2012/12/18/perl-cobol.html

In 1991, I’d just gotten my first real programming job for two reasons:
nepotism, and a willingness to write code for $12/hour. I was working
as a contractor to a blood testing laboratory, where the main
development job was writing custom software to handle, process, and do
statistical calculations on blood testing results, primarily for
paternity testing.

My father had been a software developer since the early 1970s, and worked
as a contractor at this blood lab since the late 1970s. As the calendar
had marched toward the early 1990s, technology cruft had collected. The
old TI mainframe, once the primary computer, now only had one job left:
statistical calculation for paternity testing, written in TI’s Pascal.
Slowly but surely, the other software had been rewritten and moved to an
AT&T 3B2/600 running Unix System VR3.2.3. That latter machine was the
first access I had to a real computer, and certainly the first time I had
access to Usenet. This changed my life.

Ironically, even on that 3B2, the accounting system software was written
in COBOL. This seemed like “more cruft” to me, but fortunately
there was a third-party vendor who handled that software, so I didn’t have
to program in COBOL.

I had the good fortune, actually, to help with the interesting problems,
which included grokking data from a blood testing machine that dumped a
bunch of data in some weird reporting format onto its RS-232 port at the
end of every testing cycle. We had to pull the data of that RS-232
interface and load the data in the database. Perl, since it treated
regular expressions as first-class citizens, and had all the Unix block
device fundamentals baked in as native (for the RS-232 I/O), was the
obvious choice.

After that project, I was intrigued by this programming language that had
made the job so easy. My father gave me a copy of the Camel book —
which was, at that point, almost hot off the presses. I read it over a
weekend and I decided that I didn’t really want to program in any other
language again. Perl was just 4 years old then; it was a young language
— Perl 4 had just been released. I started trying to embed Perl into
our database system, but it wasn’t designed for embedding into other
systems as a scripting language. So, I ended up using Tcl instead for the big
project of rewriting the statical calculation software to replace the TI
mainframe. After a year or two writing tens of thousands of lines of Tcl,
I was even more convinced that I’d rather be writing in Perl. When
Perl 5 was released, I switched back to Perl and never really looked
back.

Perl ultimately became my first Free Software community. I lurked on
perl5-porters for years, almost always a bit too timid to post, or ever
send in a patch. But, as I finished my college degree and went to graduate
school, I focused my
thesis work on Perl and virtual machines
. I went to the Perl
conference every year. I was even in the room for the perl5-porters
meeting the day
after Jon
Orwant’s staged tantrum
, which was the catalyst for the Perl 6
effort. I wrote more than a few RFC’s during
the Perl 6 specification
process
. And, to this day, even though I’ve since
done plenty of Python
development
, too, when I need to program to do something, I open an Emacs
buffer and start typing #!/usr/bin/perl.

Meanwhile, I never did learn COBOL. But, I was amazed to hear that
multiple folks who graduated with me eventually got jobs at a health
insurance company. The company trained them in COBOL, so that they could
maintain COBOL systems all day. Everyone once in a while, I idly search a
job site for COBOL. Today, that search is returning 2,338 open jobs. Most
developers never hear about it, of course. It’s far from the exciting new
technology, but it’s there, it’s needed and it’s obviously useful to
someone. Indeed, the COBOL standard was just updated 10 years ago, in
2002!

I notice these days, though, that when I mentioned having done a lot of
Perl development in my life, the average Javascript, Python, or Haskell
developer looks at me like I looked at my dad when he told me that
accounting system was written in COBOL. I’d bet they’d have my same
sigh of relief when told that “someone else” maintains that
code and they won’t have to bother with it.

Yet, I still know people heavily immersed in the Perl community. Indeed,
there is a very active Perl community out there, just like there’s an
active COBOL community. I’m not active in Perl like I once was, but it’s a
community of people, who write new code and maintain old code in Perl, and
that has value. More importantly, though, (and unlike COBOL), Perl was
born on Usenet, and was released as Free Software from the day of its first
release, twenty-five years ago today. Perl was born as part of Free
Software culture, and it lives on.

So, I get it now. I once scoffed at the idea that anyone would write
in COBOL anymore, as if the average COBOL programmer was some sort of
second-class technology citizen. COBOL programmers in 1991, and even
today, are surely good programmers — doing useful things for their
jobs. The same is true of Perl these days: maybe Perl is finally
getting a bit old fashioned — but there are good developers, still
doing useful things with Perl. Perl is becoming Free Software’s COBOL:
an aging language that still has value.

Perl turns 25
years old
today. COBOL was 25 years old in 1984, right at the time
when I first started programming. To those young people who start
programming today: I hope you’ll learn from my mistake. Don’t scoff at the
Perl programmers. 25 years from now, you may regret scoffing at them as
much as I regret scoffing at the COBOL developers. Programmers are
programmers; don’t judge them because you don’t like their favorite
language.

Update (2013-04-12):
I posted
a comment on Allison Randal’s blog about similar issues of Perl’s
popularity
.

The collective thoughts of the interwebz