Tag Archives: religion

За религиозността на няколко хиляди в социалките и защо повечето такива изследвания са счупени

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2017/religia-i-schupeni-anketi/

Постоянно сме облъчвани от анкети и социологически проучвания. Особено когато става дума за избори, политика и мнения по актуални за деня въпроси. През май излезе едно такова изследване на Pew Research засягащо отношението на религията в източна Европа. То, както и няколко често споделяни мемета в социалните мрежи, ме накараха да направя свой експеримент преди месец. Пуснах анкета за „религиозността на българите“.

Преди да задълбая в резултатите, има няколко важни детайла, които трябва да обсъдим. Първият е, че анкетата ми в никакъв случай не претендира за представителност. Не съм започвал анкетата с тази идея и нямам илюзии, че може да е. Причините за това е важно да се разберат, ако искаме да разпознаваме лошата „статистика“ и ще ги обсъдя подробно по-долу.

Въпреки това, интересно и до голяма степен неочаквано е, че беше попълнена от 3120 души, 2400 от които в първите 24 часа. Това само по себе си показва, че темата за религията занимава много хора от всички страни на дебата и определено е нещо, за което трябва да говорим повече. В този смисъл, самата дискусия и активност около анкетата може да е дори по-показателна от самите данни.

Също така, някои обърнаха внимание на някои странни въпроси като този за църковния данък. Целта му, както и подборът на отговорите на повечето въпроси, беше да илюстрирам същността на критиката ми към изследването на Pew Research. Тезата ми не е, че умишлено са подправили данните, а че липсата на разбиране за културните различния ги обезсмисля.

Не на последно място, дори взимайки предвид всички условности, резултатите от тези няколко хиляди отговора си остават интересни и с доста уговорки може да ни покажат позициите и самоопределянето ако не на друго, то поне на балона от познати и техни приятели, които има около мен в социалките.

Самоопределяне, политика и изучаване на религия

След като отметнахме всичко това, мисля да започна отзад напред. Не заради друго, а просто защото отговорите са интересни.

В следващите няколко графики съм обединил всички самоопределящи се към някоя религия в една група и всички, които не се самоопределят – в друга. Знам, че това ще раздразни някои, но за по-кратко ще ги наричам „религиозни“ и „нерелигиозни“. Разпределението впрочем е съответно 51.4% „религиозни“ срещу 44.7% с още 3.9% отговорили „друго“. Разбивка на отговорите им ще видите вдясно, а вляво е средното за всички анкетирани. Скрил съм колебаещите се. В червено са отворите „вярно“, а в лилаво – „невярно“.

При първия въпрос няма изненади – 91% от „нерелигиозните“ са отговорили, че религията не е важна за тях. Интересно е, че 20% от „религиозните“ са отговорили по същия начин. Значително повече хора и от двете групи отговарят обаче, че религията е (предимно) важна част от българската идентичност – 71% и 26%. 50% от всички отговорили са дали този отговор срещу 30% отговорили с „не“. Това съвпада с изводите на Pew.

Разминаване идва обаче при следващите въпроси. Едва 17% от отговорилите са казали, че религията влияе на религиозните им убеждения, а 19% – че трябва да играе роля в политическия живот. Дори сред отговорилите, че религията е определено важна за тях, едва половината казват, че тя им влияе политически. В България някак този въпрос е естествен – „вържи попа … да ти е мирно селото“, както се казваше в приказката. В доста държави обаче не е така и щатите определено са сред тях. В анализите на Pew се опитват да направят връзка между религиозността на източна Европа и мнението за Русия. Внушават също, че увеличаване на религиозността би имала някакъв осезаем ефект във външнополитическите ни взаимоотношенията с тях. Макар за отделни маргинални групи това да е вярно, то определено не е сериозен фактор в политиката ни, най-малкото заради мястото на официалната църква.

Следващият въпрос също не изненадва – почти половината от самоопределящите се с някоя религия смятат, че религиозният канон трябва да се изучава в училище, докато 90% от „нерелигиозните“ не са съгласни. Интересно е обаче единодушието по последния въпрос – да се изучава историята на всички религии в час. Над 60% от анкетираните отговарят така и само 24% са против.

Религиозни, религиозни, колкото да сме религиозни

Така, нека да обясня. Чест проблем при съставянето на анкетите е приемането, че всички ще възприемат концепциите по един и същ начин. Мона Чалаби от The Guardian дава един интересен пример с нашумяло изследване за мнението на мюсюлманите във Великобритания за джихада. Мнозинството са отговорили, че го подкрепят и точно тези няколко думи обикалят световните медии. Не става ясно обаче, че повечето анкетирани също отговарят, че възприемат дефиницията за „джихад“ като „мирна вътрешна борба за лично израстване“.

Аналогично, въпросът за самоопределянето с дадена религия може да има доста различен смисъл. Именно на база това направихме разбивка на данните до сега. В следващите въпроси се вижда, че дори „религиозните“ се възпротивяват срещу въвеждането на църковен данък. Едва 15% от „религиозните“ смятат, че трябва да се въведе, а 20% биха го плащали, ако се наложи. 74% от „религиозните“, които смятат, че трябва да се въведе, биха се регистрирали и плащали.

Изборът на точно тази сума не е случайна. Взимам за пример Германия, където такъв църковен данък има отдавна и е процент от заплатата. Този процент варира, но съпоставен със средните доходи там, би се равнявал на около 25 лв. месечно у нас. В Германия този данък се плаща единствено, ако се регистрираш като принадлежащ към някоя от църквите. Доста емигрирали българи правят грешката да се пишат просто „християни“ и още с първата си заплата осъзнават, че вече спонсорират деноминацията на църквата, към която конкретният данъчен служител принадлежи. Лесно е да се отрегистрираш и немалко немци го правят като станат на 18 години. Тогава обаче нямат право на църковна сватба, кръщене или погребение с поп. Общо взето или си в клуба и плащаш, или не си. Затова много немци плащат църковен данък и годишно той генерира около 17 млрд. евро разпределени най-вече между католическата и евангелистката църква.

Този принцип е толкова залегнал в разбирането им за религиозност, че ако зададеш въпрос на немец дали е религиозен, почти сигурно е да отговори с „не“, ако не плаща данъка. Онези, които плащат данъци пък са особено религиозни. Всъщност Германия въпреки либералния си облик е доста религиозна държава. Едва наскоро научих, че в петъка преди Великден всички клубове са затворени и партита забранени. Стига се до викане на полиция и глоби. Това идва изцяло от християнския канон. Все още общините са разделени по линиите католици/евангелисти и гледат да се придържат към общностите си. За празничните паради да не обяснявам. Не мислете и че думата „християн-“ в името на партията на Меркел е просто така – религията има сериозна роля в политиките и кампаниите им.

В същото време, поне съдейки по тези данни, ако в България бъде въведен такъв данък, 4/5 от идентифициращите се с някоя религия и 2/3 от отговорилите, че религията е важна за тях ще „окапят“. Причината да посочвам това, че различни мемета в мрежата показват колко по-религиозни сме от тях. Едно такова нарежда Германия сред атеистичните държави, които са и мирни и прави връзка между тези неща. Някои посочват като източник изследвания като това на Галъп, където наистина се твърди, че Германия има драстично повече атеисти. Ако и в България се налага да плащаш колкото месечна винетка за отговора „религиозен“, съвсем лесно ще ги задминем в атеизма.

В графиката горе ще забележите и че с 50% повече „нерелигиозни“ смятат, че трябва да се въведе данъка, в сравнение с „религиозните“. Причината може би е, че държавата отдели 4.5 млн. лв от бюджета за 2016-та за субсидии на вероизповеданията. През годините има тенденция на увеличаване и повечето пари отиват за ремонти. Няколко стотин хиляди са за подпомагане на български църкви зад граница. Ако се въведе данък и наистина онези 7.7% отговорили с твърдо „да“ се регистрират, то субсидията ще набъбне на поне 60 млн. Това би разтоварило останалите данъкоплатци с поне няколко милиона, с което си обяснявам отговорите на „нерелигиозните“. Разбира се, всичко това е условно, защото анкетата далеч не е представителна за всички работещи.

Социо-стъкмистика

Въпросът за представителността на анкетата беше засегнат още от самото ѝ пускане. Навсякъде коментирах, че целта въобще не е представителност и че нямам намерение да правя обобщения за „религиозността на нацията“. Изводите, които описах до тук, идват с много условности и уговорки. Похвално е, че толкова много хора ги забелязаха от рано и това ми дава надежда, че все пак критичното мислене не е залязло толкова, колкото си мислим.

Най-очевиднияt проблем с анкетата ми е, че се провежда в мрежата. Това само по себе си силно ограничава качеството, защото не покрива представителна извадка от населението. Отделно, анкетата беше разпространяване в социалките от човек на човек, което още повече предполага затвореност на кръга на хора, които я попълват. Най-ясно изразеният белег за това се вижда в активността и групирането на отговарящите през времето. Тази графика показва точно това – в синьо са попълнилите на час, а в червено – дялът самоопределящи се с някоя религия. Вижда се ясно, че в 9 до 11 сутринта е имало много попълващи „нерелигиозни“, докато тенденцията се обръща около 14 часа и вечерта. Това съвпада като време със споделянията на групи на тема атеизъм и религия.

Представителността е най-ключовото понятие, за което трябва да внимаваме. Белезите за „счупена“ представителност са много – анкети в интернет, само пред стационарен телефон или друго ограничено средство за комуникация, концентриране само върху градовете или дори центъра им и прочие. Най-ясен пример за такива дефекти се виждат в социологическите проучвания около изборите. Почти без изключение те са безсмислени като данни, но за сметка на това имат огромно влияние върху общественото мнение. Няма по-известен пример от провала на прогнозите за изборите през 2016-та в щатите. Сайтът FiveThirtyEight има доста добро обяснение защо се случи това. Сред основните причини е именно пропускането на цели демографски групи в допитванията.

Друг фактор в мащабът. Тук всъщност той говори в моя полза, тъй като дори големите социологически агенции боравят с няколко хиляди анкетирани. Ако се вгледате в доклада на Pew Research, техните изводи за България се основават на само 1619 души. Разбира се, броят няма значение, когато в извадката няма представителност. Те твърдят, че са използвали методи осигуряващи такава. Това обаче трябва да се докаже. Обикновено сред въпросите се поставя такъв, резултатите от които може да се сравнят с обективна статистика. Например – възрастовото разпределение на населението, броят женени или брой деца в домакинството. Ако средните стойности по възрастови групи или местоживеене не отговарят на средното за страната, то извадката не е представителна. Именно такъв сериозен дефект илюстрирах с едно друго нашумяло изследване разглеждащо отлагането на деца и брака сред младите в България. Pew още не са публикували данните си, за да сравним. 1600 души обаче не са много. Вчера Инфограф предупреди, че привидното повишаване на дяла бедни българи може да се дължи на увеличената извадка в анкетата на НСИ – от 7300 на 8600 домакинства. Това може да е индикация, че старата извадка е била твърде малка или недостатъчно представителна. Разликата между анкетираните от НСИ и Pew e 4.5 пъти.

Дефект на всяка една подобна анкета е, че разчита на субективни фактори. В повечето случаи, като етническото и религиозното самоопределяне, това е неизбежно. Ако не сме внимателни обаче, това може да доведе до сериозни проблеми. Чалаби посочва, че онази анкета за мюсюлманите и джихада също е била през интернет и отговорилите са били доста малко предвид голямото мюсюлманско население на Великобритания. По-важното обаче е, че е нямало абсолютно никакъв начин да се потвърди, че някой не е попълвал анкетата без да е мюсюлманин просто така – за да изкриви резултата. Това съвсем не е за пренебрегване предвид медийният отзвук, който такива „изследвания“ привличат. Бихте се учудили колко от сензационните новини, които четем, всъщност се осланят именно на изследвания като цитираното от Чалаби и това, което аз направих. Разликата е, че там не говорят за условностите, а дори сериозни медии скачат на бомбастични изводи без да направят критична проверка.

Не по-малко важни са и предложените отговори. Вече обсъдихме как въпросите може да се разбират различно в различните държави и култури. Изборът на отговори също може да изкриви значително резултата. Нарочно добавих отговор „други религии“ и „други“ при въпроса за самоопределянето. Идеята какво е религия доста се разминава между хората. Аз, например се самоопределям като православен християнин, но в никакъв случай не бих се самоопределил с БПЦ. Също така вярванията ми са много по-близки до агностиците, отколкото до религиозните. Това разминаване между възприятие за религия, традиции, празници и спиритуалност трудно се вписва в която и да е от кутийките, които сам дадох като отговори. Съвсем естествено липсата на „атеист“ и различни степени от спектъра на вярващите раздразни някои хора. Отново обаче повтарям – целта ми не беше да изследвам религиозността, което е невъзможно предвид вложените дефекти описани тук. Основната ми цел е да илюстрирам тези дефекти.

Другите въпроси

В анкетата имаше още няколко точки, които добавих просто, защото бяха интересни и присъстваха в оригиналното изследване на Pew. Малко над 22% от отговорилите и около 41% от религиозните казват, че ходят поне веднъж месечно в храм. 57% от всички или посещават църкви, джамии и синагоги като туристи, или не ходят въобще. Интересно е, че едва 25% посещаващите храм всяка седмица, казват, че са съгласни с въвеждането на църковен данък. Ако се въведе, по-малко от половината от тях (45%) казват, че биха го плащали. Струва ми се, че това показва друг интересен аспект на връзката между религиозност и църква поне сред балона ни от познати в мрежата.

В следващите отговори се забелязва относително единодушие. Разделени сме дали възрастните са повече или по-малко религиозни от соца насам. За религиозността на младите има известно разминаване между двата лагера, но мнозинството не смята, че се е увеличила. Това се разминава и с резултатите на Pew. Над 60% от всички групи смятат, че религията има място само в семейни традиции и празници. Това пък потвърждава голямото мнозинство отговорили, че религията няма място в политиката. Все пак, също толкова смятат, че политиците се опитват и навярно успяват да я използват в кампаниите си.

Тук виждате разбивката по възрасти. Не се различава драстично от разпределението на населението на страната, но това не го прави приложимо към населението. Също така, 75% от попълнилите живеят в град на 100 хиляди души, а 10% са в чужбина. Това говори повече за социалния ми балон, отколкото за религиозността на българите като цяло.

Тук съм показал религиозността по възрасти. Вижда се ясна пропорционална корелация между възрастта и самоопределянието с някоя религия. Отрязал съм графиката на 50 години, защото попълнилите посочили над тази възраст са по 10-20, което не позволява да се правят изводи.

Всъщност, именно затова не направих сравнение на отговорите по възрастови групи. Тъй като анкетата е проведена в интернет, се вижда ясно практическата липса на отговорили над 65 години. Така не мога да сравнявам „млади“ и „стари“. Това не означава, че доста „изследвания“ го правят – взима се извадка от 1000 души, 10 от тях са в пенсионна възраст и на тяхна база се съди за всички 1.4 млн. българи в тази група.

Данни и препратки

Статиите на Pew Research заедно с въпросниците и целият доклад за религията в източна Европа ще намерите на страницата им. Данните от моята анкета ще може да свалите като Excel файл. Може да го използвате както намерите за добре, но ще се радвам, ако сложите обратна връзка.

Други подобни анализи са разяснението ми за смъртността и как не разбираме тази метрика, раждаемостта и как е пълно с фалшиви новини, броя на работещите българи в чужбина и как тази статистика е просто измислена и не на последно място – защо не разбираме данните за бедността.

Running R on Amazon Athena

Post Syndicated from Gopal Wunnava original https://aws.amazon.com/blogs/big-data/running-r-on-amazon-athena/

Data scientists are often concerned about managing the infrastructure behind big data platforms while running SQL on R. Amazon Athena is an interactive query service that works directly with data stored in S3 and makes it easy to analyze data using standard SQL without the need to manage infrastructure. Integrating R with Amazon Athena gives data scientists a powerful platform for building interactive analytical solutions.

In this blog post, you’ll connect R/RStudio running on an Amazon EC2 instance with Athena.

Prerequisites

Before you get started, complete the following steps.

    1. Have your AWS account administrator give your AWS account the required permissions to access Athena via Amazon’s Identity and Access Management (IAM) console. This can be done by attaching the associated Athena policies to your data scientist user group in IAM.

 

RAthena_1

  1. Provide a staging directory in the form of an Amazon S3 bucket. Athena will use this to query datasets and store results. We’ll call this staging bucket s3://athenauser-athena-r in the instructions that follow.

NOTE: In this blog post, I create all AWS resources in the US-East region. Use the Region Table to check the availability of Athena in other regions.

Set up R and RStudio on EC2 

  1. Follow the instructions in the blog post “Running R on AWS” to set up R on an EC2 instance (t2.medium or greater) running Amazon Linux . Read the step below before you begin.
  1. In that blog post under “Advanced Details,” when you reach step 3 use the following bash script to install the latest version of RStudio. Modify the password for RStudio as needed.
#!/bin/bash
#install R
yum install -y R
#install RStudio-Server
wget https://download2.rstudio.org/rstudio-server-rhel-1.0.136-x86_64.rpm
yum install -y --nogpgcheck rstudio-server-rhel-1.0.136-x86_64.rpm
#add user(s)
useradd rstudio
echo rstudio:rstudio | chpasswd

Install Java 8 

  1. SSH into this EC2 instance.
  2. Remove older versions of Java.
  3. Install Java 8. This is required to work with Athena.
  4. Run the following commands on the command line.
#install Java 8, select ‘y’ from options presented to proceed with installation
sudo yum install java-1.8.0-openjdk-devel
#remove version 7 of Java, select ‘y’ from options to proceed with removal
sudo yum remove java-1.7.0-openjdk
#configure java, choose 1 as your selection option for java 8 configuration
sudo /usr/sbin/alternatives --config java
#run command below to add Java support to R
sudo R CMD javareconf

#following libraries are required for the interactive application we build later
sudo yum install -y libpng-devel
sudo yum install -y libjpeg-turbo-devel

Set up .Renviron

You need to configure the R environment variable .Renviron with the required Athena credentials.

  1. Get the required credentials from your AWS Administrator in the form of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  1. Type the following command from the Linux command prompt and bring up the vi editor.
sudo vim /home/rstudio/.Renviron

Provide your Athena credentials in the following form into the editor:
ATHENA_USER=< AWS_ACCESS_KEY_ID >
ATHENA_PASSWORD=< AWS_SECRET_ACCESS_KEY>
  1. Save this file and exit from the editor. 

Log in to RStudio

Next, you’ll log in to RStudio on your EC2 instance.

  1. Get the public IP address of your instance from the EC2 dashboard and paste it followed by :8787 (port number for RStudio) into your browser window.
  1. Confirm that your IP address has been whitelisted for inbound access to port 8787 as part of the configuration for the security group associated with your EC2 instance.
  1. Log in to RStudio with the username and password you provided previously.

Install R packages

Next, you’ll install and load the required R packages.

#--following R packages are required for connecting R with Athena
install.packages("rJava")
install.packages("RJDBC")
library(rJava)
library(RJDBC)

#--following R packages are required for the interactive application we build later
#--steps below might take several minutes to complete
install.packages(c("plyr","dplyr","png","RgoogleMaps","ggmap"))
library(plyr)
library(dplyr)
library(png)
library(RgoogleMaps)
library(ggmap)

Connect to Athena

The following steps in R download the Athena driver and set up the required connection. Use the JDBC URL associated with your region.

#verify Athena credentials by inspecting results from command below
Sys.getenv()
#set up URL to download Athena JDBC driver
URL <- 'https://s3.amazonaws.com/athena-downloads/drivers/AthenaJDBC41-1.0.0.jar'
fil <- basename(URL)
#download the file into current working directory
if (!file.exists(fil)) download.file(URL, fil)
#verify that the file has been downloaded successfully
fil
#set up driver connection to JDBC
drv <- JDBC(driverClass="com.amazonaws.athena.jdbc.AthenaDriver", fil, identifier.quote="'")
#connect to Athena using the driver, S3 working directory and credentials for Athena 
#replace ‘athenauser’ below with prefix you have set up for your S3 bucket
con <- jdbcConnection <- dbConnect(drv, 'jdbc:awsathena://athena.us-east-1.amazonaws.com:443/',
s3_staging_dir="s3://athenauser-athena-r",
user=Sys.getenv("ATHENA_USER"),
password=Sys.getenv("ATHENA_PASSWORD"))
#in case of error or warning from step above ensure rJava and RJDBC packages have #been loaded 
#also ensure you have Java 8 running and configured for R as outlined earlier

Now you’re ready to start querying Athena from RStudio. 

Sample Queries to test

# get a list of all tables currently in Athena 
dbListTables(con)
# run a sample query
dfelb=dbGetQuery(con, "SELECT * FROM sampledb.elb_logs limit 10")
head(dfelb,2)

RAthena_2

Interactive Use Case

Next, you’ll practice interactively querying Athena from R for analytics and visualization. For this purpose, you’ll use GDELT, a publicly available dataset hosted on S3.

Create a table in Athena from R using the GDELT dataset. This step can also be performed from the AWS management console as illustrated in the blog post “Amazon Athena – Interactive SQL Queries for Data in Amazon S3.”

#---sql  create table statement in Athena
dbSendQuery(con, 
"
CREATE EXTERNAL TABLE IF NOT EXISTS sampledb.gdeltmaster (
GLOBALEVENTID BIGINT,
SQLDATE INT,
MonthYear INT,
Year INT,
FractionDate DOUBLE,
Actor1Code STRING,
Actor1Name STRING,
Actor1CountryCode STRING,
Actor1KnownGroupCode STRING,
Actor1EthnicCode STRING,
Actor1Religion1Code STRING,
Actor1Religion2Code STRING,
Actor1Type1Code STRING,
Actor1Type2Code STRING,
Actor1Type3Code STRING,
Actor2Code STRING,
Actor2Name STRING,
Actor2CountryCode STRING,
Actor2KnownGroupCode STRING,
Actor2EthnicCode STRING,
Actor2Religion1Code STRING,
Actor2Religion2Code STRING,
Actor2Type1Code STRING,
Actor2Type2Code STRING,
Actor2Type3Code STRING,
IsRootEvent INT,
EventCode STRING,
EventBaseCode STRING,
EventRootCode STRING,
QuadClass INT,
GoldsteinScale DOUBLE,
NumMentions INT,
NumSources INT,
NumArticles INT,
AvgTone DOUBLE,
Actor1Geo_Type INT,
Actor1Geo_FullName STRING,
Actor1Geo_CountryCode STRING,
Actor1Geo_ADM1Code STRING,
Actor1Geo_Lat FLOAT,
Actor1Geo_Long FLOAT,
Actor1Geo_FeatureID INT,
Actor2Geo_Type INT,
Actor2Geo_FullName STRING,
Actor2Geo_CountryCode STRING,
Actor2Geo_ADM1Code STRING,
Actor2Geo_Lat FLOAT,
Actor2Geo_Long FLOAT,
Actor2Geo_FeatureID INT,
ActionGeo_Type INT,
ActionGeo_FullName STRING,
ActionGeo_CountryCode STRING,
ActionGeo_ADM1Code STRING,
ActionGeo_Lat FLOAT,
ActionGeo_Long FLOAT,
ActionGeo_FeatureID INT,
DATEADDED INT,
SOURCEURL STRING )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION 's3://support.elasticmapreduce/training/datasets/gdelt'
;
"
)



dbListTables(con)

You should see this newly created table named ‘gdeltmaster’ appear in your RStudio console after executing the statement above.

RAthena_3

Query this Athena table to get a count of all CAMEO events that took place in the US in 2015.

#--get count of all CAMEO events that took place in US in year 2015 
#--save results in R dataframe
dfg<-dbGetQuery(con,"SELECT eventcode,count(*) as count
FROM sampledb.gdeltmaster
where year = 2015 and ActionGeo_CountryCode IN ('US')
group by eventcode
order by eventcode desc"
)
str(dfg)
head(dfg,2)

RAthena_4

#--get list of top 5 most frequently occurring events in US in 2015
dfs=head(arrange(dfg,desc(count)),5)
dfs

RAthena_5

From the R output shown above, you can see that CAMEO event “042” has the highest count. From the CAMEO manual, you can determine that this event has the description “Travel to another location for a meeting or other event.”

Next, you’ll use the knowledge gained from this analysis to get a list of all geo-coordinates associated with this specific event from the Athena table.

#--get a list of latitude and longitude associated with event “042” 
#--save results in R dataframe
dfgeo<-dbGetQuery(con,"SELECT actiongeo_lat,actiongeo_long
FROM sampledb.gdeltmaster
where year = 2015 and ActionGeo_CountryCode IN ('US')
and eventcode = '042'
"
)
#--duration of above query will depend on factors like size of chosen EC2 instance
#--now rename columns in dataframe for brevity
names(dfgeo)[names(dfgeo)=="actiongeo_lat"]="lat"
names(dfgeo)[names(dfgeo)=="actiongeo_long"]="long"
names(dfgeo)
#let us inspect this R dataframe
str(dfgeo)
head(dfgeo,5)

RAthena_6
Next, generate a map for the United States. 

#--generate map for the US using the ggmap package
map=qmap('USA',zoom=3)
map

RAthena_7

Now you’ll plot the geodata obtained from your Athena table onto this map. This will help you visualize all US locations where these events had occurred in 2015. 

#--plot our geo-coordinates on the US map
map + geom_point(data = dfgeo, aes(x = dfgeo$long, y = dfgeo$lat), color="blue", size=0.5, alpha=0.5)

RAthena_8

By visually inspecting the results, you can determine that this specific event was heavily concentrated in the Northeastern part of the US.

Conclusion

You’ve learned how to build a simple interactive application with Athena and R. Athena can be used to store and query the underlying data for your big data applications using standard SQL, while R can be used to interactively query Athena and generate analytical insights using the powerful set of libraries that R provides.

If you have questions or suggestions, please leave your feedback in the comments.

 


About the Author

gopalGopal Wunnava is a Partner Solution Architect with the AWS GSI Team. He works with partners and customers on big data engagements, and is passionate about building analytical solutions that drive business capabilities and decision making. In his spare time, he loves all things sports and movies related and is fond of old classics like Asterix, Obelix comics and Hitchcock movies.

 

 


Related

Derive Insights from IoT in Minutes using AWS IoT, Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight

o_IoT_Minutes_1

 

Utopia

Post Syndicated from Eevee original https://eev.ee/blog/2017/03/08/utopia/

It’s been a while, but someone’s back on the Patreon blog topic tier! IndustrialRobot asks:

What does your personal utopia look like? Do you think we (as mankind) can achieve it? Why/why not?

Hm.

I spent the month up to my eyeballs in a jam game, but this question was in the back of my mind a lot. I could use it as a springboard to opine about anything, especially in the current climate: politics, religion, nationalism, war, economics, etc., etc. But all of that has been done to death by people who actually know what they’re talking about.

The question does say “personal”. So in a less abstract sense… what do I want the world to look like?

Mostly, I want everyone to have the freedom to make things.

I’ve been having a surprisingly hard time writing the rest of this without veering directly into the ravines of “basic income is good” and “maybe capitalism is suboptimal”. Those are true, but not really the tone I want here, and anyway they’ve been done to death by better writers than I. I’ve talked this out with Mel a few times, and it sounds much better aloud, so I’m going to try to drop my Blog Voice and just… talk.

*ahem*

Art versus business

So, art. Art is good.

I’m construing “art” very broadly here. More broadly than “media”, too. I’m including shitty robots, weird Twitter almost-bots, weird Twitter non-bots, even a great deal of open source software. Anything that even remotely resembles creative work — driven perhaps by curiosity, perhaps by practicality, but always by a soul bursting with ideas and a palpable need to get them out.

Western culture thrives on art. Most culture thrives on art. I’m not remotely qualified to defend this, but I suspect you could define culture in terms of art. It’s pretty important.

You’d think this would be reflected in how we discuss art, but often… it’s not. Tell me how often you’ve heard some of these gems.

  • I could do that.”
  • My eight-year-old kid could do that.”
  • Jokes about the worthlessness of liberal arts degrees.
  • Jokes about people trying to write novels in their spare time, the subtext being that only dreamy losers try to write novels, or something.
  • The caricature of a hippie working on a screenplay at Starbucks.

Oh, and then there was the guy who made a bot to scrape tons of art from artists who were using Patreon as a paywall — and a primary source of income. The justification was that artists shouldn’t expect to make a living off of, er, doing art, and should instead get “real jobs”.

I do wonder. How many of the people repeating these sentiments listen to music, or go to movies, or bought an iPhone because it’s prettier? Are those things not art that took real work to create? Is creating those things not a “real job”?

Perhaps a “real job” has to be one that’s not enjoyable, not a passion? And yet I can’t recall ever hearing anyone say that Taylor Swift should get a “real job”. Or that, say, pro football players should get “real jobs”. What do pro football players even do? They play a game a few times a year, and somehow this drives the flow of unimaginable amounts of money. We dress it up in the more serious-sounding “sport”, but it’s a game in the same general genre as hopscotch. There’s nothing wrong with that, but somehow it gets virtually none of the scorn that art does.

Another possible explanation is America’s partly-Christian, partly-capitalist attitude that you deserve exactly whatever you happen to have at the moment. (Whereas I deserve much more and will be getting it any day now.) Rich people are rich because they earned it, and we don’t question that further. Poor people are poor because they failed to earn it, and we don’t question that further, either. To do so would suggest that the system is somehow unfair, and hard work does not perfectly correlate with any particular measure of success.

I’m sure that factors in, but it’s not quite satisfying: I’ve also seen a good deal of spite aimed at people who are making a fairly decent chunk through Patreon or similar. Something is missing.

I thought, at first, that the key might be the American worship of work. Work is an inherent virtue. Politicians run entire campaigns based on how many jobs they’re going to create. Notably, no one seems too bothered about whether the work is useful, as long as someone decided to pay you for it.

Finally I stumbled upon the key. America doesn’t actually worship work. America worships business. Business means a company is deciding to pay you. Business means legitimacy. Business is what separates a hobby from a career.

And this presents a problem for art.

If you want to provide a service or sell a product, that’ll be hard, but America will at least try to look like it supports you. People are impressed that you’re an entrepreneur, a small business owner. Politicians will brag about policies made in your favor, whether or not they’re stabbing you in the back.

Small businesses have a particular structure they can develop into. You can divide work up. You can have someone in sales, someone in accounting. You can provide specifications and pay a factory to make your product. You can defer all of the non-creative work to someone else, whether that means experts in a particular field or unskilled labor.

But if your work is inherently creative, you can’t do that. The very thing you’re making is your idea in your style, driven by your experience. This is not work that’s readily parallelizable. Even if you sell physical merchandise and register as an LLC and have a dedicated workspace and do various other formal business-y things, the basic structure will still look the same: a single person doing the thing they enjoy. A hobbyist.

Consider the bulleted list from above. Those are all individual painters or artists or authors or screenwriters. The kinds of artists who earn respect without question are generally those managed by a business, those with branding: musical artists signed to labels, actors working for a studio. Even football players are part of a tangle of business.

(This doesn’t mean that business automatically confers respect, of course; tech in particular is full of anecdotes about nerds’ disdain for people whose jobs are design or UI or documentation or whathaveyou. But a businessy look seems to be a significant advantage.)

It seems that although art is a large part of what informs culture, we have a culture that defines “serious” endeavors in such a way that independent art cannot possibly be “serious”.

Art versus money

Which wouldn’t really matter at all, except that we also have a culture that expects you to pay for food and whatnot.

The reasoning isn’t too outlandish. Food is produced from a combination of work and resources. In exchange for getting the food, you should give back some of your own work and resources.

Obviously this is riddled with subtle flaws, but let’s roll with it for now and look at a case study. Like, uh, me!

Mel and I built and released two games together in the six weeks between mid-January and the end of February. Together, those games have made $1,000 in sales. The sales trail off fairly quickly within a few days of release, so we’ll call that the total gross for our effort.

I, dumb, having never actually sold anything before, thought this was phenomenal. Then I had the misfortune of doing some math.

Itch takes at least 10%, so we’re down to $900 net. Divided over six weeks, that’s $150 per week, before taxes — or $3.75 per hour if we’d been working full time.

Ah, but wait! There are two of us. And we hadn’t been working full time — we’d been working nearly every waking hour, which is at least twice “full time” hours. So we really made less than a dollar an hour. Even less than that, if you assume overtime pay.

From the perspective of capitalism, what is our incentive to do this? Between us, we easily have over thirty years of experience doing the things we do, and we spent weeks in crunch mode working on something, all to earn a small fraction of minimum wage. Did we not contribute back our own work and resources? Was our work worth so much less than waiting tables?

Waiting tables is a perfectly respectable way to earn a living, mind you. Ah, but wait! I’ve accidentally done something clever here. It is generally expected that you tip your waiter, because waiters are underpaid by the business, because the business assumes they’ll be tipped. Not tipping is actually, almost impressively, one of the rudest things you can do. And yet it’s not expected that you tip an artist whose work you enjoy, even though many such artists aren’t being paid at all.

Now, to be perfectly fair, both games were released for free. Even a dollar an hour is infinitely more than the zero dollars I was expecting — and I’m amazed and thankful we got as much as we did! Thank you so much. I bring it up not as a complaint, but as an armchair analysis of our systems of incentives.

People can take art for granted and whatever, yes, but there are several other factors at play here that hamper the ability for art to make money.

For one, I don’t want to sell my work. I suspect a great deal of independent artists and writers and open source developers (!) feel the same way. I create things because I want to, because I have to, because I feel so compelled to create that having a non-creative full-time job was making me miserable. I create things for the sake of expressing an idea. Attaching a price tag to something reduces the number of people who’ll experience it. In other words, selling my work would make it less valuable in my eyes, in much the same way that adding banner ads to my writing would make it less valuable.

And yet, I’m forced to sell something in some way, or else I’ll have to find someone who wants me to do bland mechanical work on their ideas in exchange for money… at the cost of producing sharply less work of my own. Thank goodness for Patreon, at least.

There’s also the reverse problem, in that people often don’t want to buy creative work. Everyone does sometimes, but only sometimes. It’s kind of a weird situation, and the internet has exacerbated it considerably.

Consider that if I write a book and print it on paper, that costs something. I have to pay for the paper and the ink and the use of someone else’s printer. If I want one more book, I have to pay a little more. I can cut those costs pretty considerable by printing a lot of books at once, but each copy still has a price, a marginal cost. If I then gave those books away, I would be actively losing money. So I can pretty well justify charging for a book.

Along comes the internet. Suddenly, copying costs nothing. Not only does it cost nothing, but it’s the fundamental operation. When you download a file or receive an email or visit a web site, you’re really getting a copy! Even the process which ultimately shows it on your screen involves a number of copies. This is so natural that we don’t even call it copying, don’t even think of it as copying.

True, bandwidth does cost something, but the rate is virtually nothing until you start looking at very big numbers indeed. I pay $60/mo for hosting this blog and a half dozen other sites — even that’s way more than I need, honestly, but downgrading would be a hassle — and I get 6TB of bandwidth. Even the longest of my posts haven’t exceeded 100KB. A post could be read by 64 million people before I’d start having a problem. If that were the population of a country, it’d be the 23rd largest in the world, between Italy and the UK.

How, then, do I justify charging for my writing? (Yes, I realize the irony in using my blog as an example in a post I’m being paid $88 to write.)

Well, I do pour effort and expertise and a fraction of my finite lifetime into it. But it doesn’t cost me anything tangible — I already had this hosting for something else! — and it’s easier all around to just put it online.

The same idea applies to a vast bulk of what’s online, and now suddenly we have a bit of a problem. Not only are we used to getting everything for free online, but we never bothered to build any sensible payment infrastructure. You still have to pay for everything by typing in a cryptic sequence of numbers from a little physical plastic card, which will then give you a small loan and charge the seller 30¢ plus 2.9% for the “convenience”.

If a website could say “pay 5¢ to read this” and you clicked a button in your browser and that was that, we might be onto something. But with our current setup, it costs far more than 5¢ to transfer 5¢, even though it’s just a number in a computer somewhere. The only people with the power and resources to fix this don’t want to fix it — they’d rather be the ones charging you the 30¢ plus 2.9%.

That leads to another factor of platforms and publishers, which are more than happy to eat a chunk of your earnings even when you do sell stuff. Google Play, the App Store, Steam, and anecdotally many other big-name comparative platforms all take 30% of your sales. A third! And that’s good! It seems common among book publishers to take 85% to 90%. For ebook sales — i.e., ones that don’t actually cost anything — they may generously lower that to a mere 75% to 85%.

Bless Patreon for only taking 5%. Itch.io is even better: it defaults to 10%, but gives you a slider, which you can set to anything from 0% to 100%.

I’ve mentioned all this before, so here’s a more novel thought: finite disposable income. Your audience only has so much money to spend on media right now. You can try to be more compelling to encourage them to spend more of it, rather than saving it, but ultimately everyone has a limit before they just plain run out of money.

Now, popularity is heavily influenced by social and network effects, so it tends to create a power law distribution: a few things are ridiculously hyperpopular, and then there’s a steep drop to a long tail of more modestly popular things.

If a new hyperpopular thing comes out, everyone is likely to want to buy it… but then that eats away a significant chunk of that finite pool of money that could’ve gone to less popular things.

This isn’t bad, and buying a popular thing doesn’t make you a bad person; it’s just what happens. I don’t think there’s any satisfying alternative that doesn’t involve radically changing the way we think about our economy.

Taylor Swift, who I’m only picking on because her infosec account follows me on Twitter, has sold tens of millions of albums and is worth something like a quarter of a billion dollars. Does she need more? If not, should she make all her albums free from now on?

Maybe she does, and maybe she shouldn’t. The alternative is for someone to somehow prevent her from making more money, which doesn’t sit well. Yet it feels almost heretical to even ask if someone “needs” more money, because we take for granted that she’s earned it — in part by being invested in by a record label and heavily advertised. The virtue is work, right? Don’t a lot of people work just as hard? (“But you have to be talented too!” Then please explain how wildly incompetent CEOs still make millions, and leave burning businesses only to be immediately hired by new ones? Anyway, are we really willing to bet there is no one equally talented but not as popular by sheer happenstance?)

It’s kind of a moot question anyway, since she’s probably under contract with billionaires and it’s not up to her.

Where the hell was I going with this.


Right, so. Money. Everyone needs some. But making it off art can be tricky, unless you’re one of the lucky handful who strike gold.

And I’m still pretty goddamn lucky to be able to even try this! I doubt I would’ve even gotten into game development by now if I were still working for an SF tech company — it just drained so much of my creative energy, and it’s enough of an uphill battle for me to get stuff done in the first place.

How many people do I know who are bursting with ideas, but have to work a tedious job to keep the lights on, and are too tired at the end of the day to get those ideas out? Make no mistake, making stuff takes work — a lot of it. And that’s if you’re already pretty good at the artform. If you want to learn to draw or paint or write or code, you have to do just as much work first, with much more frustration, and not as much to show for it.

Utopia

So there’s my utopia. I want to see a world where people have the breathing room to create the things they dream about and share them with the rest of us.

Can it happen? Maybe. I think the cultural issues are a fairly big blocker; we’d be much better off if we treated independent art with the same reverence as, say, people who play with a ball for twelve hours a year. Or if we treated liberal arts degrees as just as good as computer science degrees. (“But STEM can change the world!” Okay. How many people with computer science degrees would you estimate are changing the world, and how many are making a website 1% faster or keeping a lumbering COBOL beast running or trying to trick 1% more people into clicking on ads?)

I don’t really mean stuff like piracy, either. Piracy is a thing, but it’s… complicated. In my experience it’s not even artists who care the most about piracy; it’s massive publishers, the sort who see artists as a sponge to squeeze money out of. You know, the same people who make everything difficult to actually buy, infest it with DRM so it doesn’t work on half the stuff you own, and don’t even sell it in half the world.

I mean treating art as a free-floating commodity, detached from anyone who created it. I mean neo-Nazis adopting a comic book character as their mascot, against the creator’s wishes. I mean politicians and even media conglomerates using someone else’s music in well-funded videos and ads without even asking. I mean assuming Google Image Search, wonder that it is, is some kind of magical free art machine. I mean the snotty Reddit post I found while looking up Patreon’s fee structure, where some doofus was insisting that Patreon couldn’t possibly pay for a full-time YouTuber’s time, because not having a job meant they had lots of time to spare.

Maybe I should go one step further: everyone should create at least once or twice. Everyone should know what it’s like to have crafted something out of nothing, to be a fucking god within the microcosm of a computer screen or a sewing machine or a pottery table. Everyone should know that spark of inspiration that we don’t seem to know how to teach in math or science classes, even though it’s the entire basis of those as well. Everyone should know that there’s a good goddamn reason I listed open source software as a kind of art at the beginning of this post.

Basic income and more arts funding for public schools. If Uber can get billions of dollars for putting little car icons on top of Google Maps and not actually doing any of their own goddamn service themselves, I think we can afford to pump more cash into webcomics and indie games and, yes, even underwater basket weaving.

The false-false-balance problem

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/11/the-false-false-balance-problem.html

Until recently, journalism in America prided itself on objectivity — to report the truth, without taking sides. That’s because big debates are always complexed and nuanced, and that both sides are equally reasonable. Therefore, when writing an article, reporters attempt to achieve balance by quoting people/experts/proponents on both sides of an issue.

But what about those times when one side is clearly unreasonable? You’d never try to achieve balance by citing those who believe in aliens and big-foot, for example.Thus, journalists have come up with the theory of false-balance to justify being partisan and one-sided on certain issues.
Typical examples where journalists cite false-balance is reporting on anti-vaxxers, climate-change denialists, and Creationists. More recently, false-balance has become an issue in the 2016 Trump election.
But this concept of false-balance is wrong. It’s not that anti-vaxxers, denialists, Creationists, and white supremacists are reasonable. Instead, the issue is that the left-wing has reframed the debate. They’ve simplified it into something black-and-white, removing nuance, in a way that shows their opponents as being unreasonable. The media then adopts the reframed debate.
Let’s talk anti-vaxxers. One of the policy debates is whether the government has the power to force vaccinations on people (or on people’s children). Reasonable people say the government doesn’t have this power. Many (if not most) people hold this opinion while agreeing that vaccines are both safe and effective (that they don’t cause autism).
Consider this February 2015 interview with Chris Christy. He’s one of the few politicians who have taken the position that government can override personal choice, such as in the case of an outbreak. Yet, when he said “parents need to have some measure of choice in things as well, so that’s the balance that the government has to decide“, he was broadly reviled as an anti-vaxxer throughout the media. The press reviled other Republican candidates the same way, even while ignoring almost identical statements made at the same time by the Obama administration. They also ignored clearly anti-vax comments from both Hillary and Obama during the 2008 election.
Yes, we can all agree that anti-vaxxers are a bunch of crazy nutjobs. In calling for objectivity, we aren’t saying that you should take them seriously. Instead, we are pointing out the obvious bias in the way the media attacked Republican candidates as being anti-vaxxers, and then hiding behind “false-balance”.
Now let’s talk evolution. The issue is this: Darwinism has been set up as some sort of competing religion against belief in God(s). High-schools teach children to believe in Darwinism, but not to understand Darwinism. Few kids graduate understanding Darwinism, which is why it’s invariably misrepresented in mass-media (X-Men, Planet of the Apes, Waterworld, Godzilla, Jurassic Park, etc.). The only movie I can recall getting evolution correct is Idiocracy.
Also, evolution has holes in it. This isn’t a bad thing in science, every scientific theory has holes. Science isn’t a religion. We don’t care about the holes. That some things remain unexplained by a theory doesn’t bother us. Science has no problem with gaps in knowledge, where we admit “I don’t know”. It’s religion that has “God of the gaps”, where ignorance isn’t tolerated, and everything unexplained is explained by a deity.
The hole in evolution is how the cell evolved. The fossil record teaches us a lot about multi-cellular organisms over the last 400-million years, but not much about how the cell evolved in the 4-billion years on planet Earth before that. I can point to radio isotope dating and fossil finds to prove dinosaurs existed 250,000 million to 60 million years ago, thus disproving your crazy theory of a 10,000 year-old Earth. But I can’t point to anything that disagrees with your view that a deity created the original cellular organisms. I don’t agree with that theory, but I can’t disprove it, either.
The point is that Christians have a good point that Darwinism is taught as a competing religion. You see this in the way books that deny holes in knowledge, insisting that Darwinism explains even how cells evolved, and that doubting Darwin is blasphemy. 
The Creationist solution is wrong, we can’t teach religion in schools. But they have a reasonable concern about religious Darwinism. The solution there is to do a better job teaching it as a science. If kids want to believe that one of the deities created the first cells, then that’s okay, as long as they understand the fossil record and radioisotope dating.
Now let’s talk Climate Change. This is a tough one, because you people have lost your collective minds. The debate is over how much change? how much danger? how much costs?. The debate is not over Is it true?. We all agree it’s true, even most Republicans. By keeping the debate between the black-and-white “Is global warming true?”, the left-wing can avoid the debate “How much warming?”.
Consider this exchange from one of the primary debates:
Moderator: …about climate change…
RUBIO: Because we’re not going to destroy our economy …
Moderator: Governor Christie, … what do you make of skeptics of climate change such as Senator Rubio?
CHRISTIE: I don’t think Senator Rubio is a skeptic of climate change.
RUBIO: I’m not a denier/skeptic of climate change.
The media (in this case CNN) is so convinced that Republican deny climate change that they can’t hear any other statement. Rubio clearly didn’t deny Climate Change, but the moderator was convinced that he did. Every statement is seen as outright denial, or code words for denial. Thus, convinced of the falseness of false-balance, the media never sees the fact that most Republicans are reasonable.
Similar proof of Republican non-denial is this page full of denialism quotes. If you actually look at the quotes, you’ll see that when taken in context, virtually none of the statements deny climate change. For example, when Senator Dan Sulliven says “no concrete scientific consensus on the extent to which humans contribute to climate change“, he is absolutely right. There is 97% consensus that mankind contributes to climate change, but there is widespread disagreement on how much.
That “97% consensus” is incredibly misleading. Whenever it’s quoted, the speaker immediately moves the bar, claiming that scientists also agree with whatever crazy thing the speaker wants, like hurricanes getting worse (they haven’t — at least, not yet).
There’s no inherent reason why Republicans would disagree with addressing Climate Change. For example, Washington State recently voted on a bill to impose a revenue neutral carbon tax. The important part is “revenue neutral”: Republicans hate expanding government, but they don’t oppose policies that keep government the same side. Democrats opposed this bill, precisely because it didn’t expand the size of government. That proves that Democrats are less concerned with a bipartisan approach to addressing climate change, but instead simply use it as a wedge issue to promote their agenda of increased regulation and increased spending. 
If you are serious about address Climate Change, then agree that Republicans aren’t deniers, and then look for bipartisan solutions.
Conclusion

The point here is not to try to convince you of any political opinion. The point here is to describe how the press has lost objectivity by adopting the left-wing’s reframing of the debate. Instead of seeing balanced debate between two reasonable sides, they see a warped debate between a reasonable (left-wing) side and an unreasonable (right-wing) side. That the opposing side is unreasonable is so incredible seductive they can never give it up.
That Christie had to correct the moderator in the debate should teach you that something is rotten in journalism. Christie understood Rubio’s remarks, but the debate moderator could not. Journalists cannot even see the climate debate because they are wedded to the left-wing’s corrupt view of the debate.
The issue of false-balance is wrong. In debates that evenly divide the population, the issues are complex and nuanced, both sides are reasonable. That’s the law. It doesn’t matter what the debate is. If you see the debate simplified to the point where one side is obviously unreasonable, then it’s you who has a problem.

Dinner with Rajneeshees

One evening I answered the doorbell to find a burgundy clad couple on the doorstep. They were followers of the Bagwan Shree Rajneesh, whose cult had recently purchased a large ranch in the eastern part of the state. No, they weren’t there to convert us. They had come for dinner. My father had invited them.
My father was a journalist, who had been covering the controversies with the cult’s neighbors. Yes, they were a crazy cult which later would breakup after committing acts of domestic terrorism.  But this couple was a pair of young professionals (lawyers) who, except for their clothing, looked and behaved like normal people. They would go on to live normal lives after the cult.
Growing up, I lived in two worlds. One was the normal world, which encourages you to demonize those who disagree with you. On the political issues that concern you most, you divide the world into the righteous and the villains. It’s not enough to believe the other side wrong, you most also believe them to be evil.
The other world was that of my father, teaching me to see the other side of the argument. I guess I grew up with my own Atticus Finch (from To Kill a Mockingbird), who set an ideal. In much the same way that Atticus told his children that they couldn’t hate even Hitler, I was told I couldn’t hate even the crazy Rajneeshees.

On Trump

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2016/11/on-trump.html

I dislike commenting on politics. I think it’s difficult to contribute any novel thought – and in today’s hyper-polarized world, stating an unpopular or half-baked opinion is a recipe for losing friends or worse. Still, with many of my colleagues expressing horror and disbelief over what happened on Tuesday night, I reluctantly decided to jot down my thoughts.

I think that in trying to explain away the meteoric rise of Mr. Trump, many of the mainstream commentators have focused on two phenomena. Firstly, they singled out the emergence of “filter bubbles” – a mechanism that allows people to reinforce their own biases and shields them from opposing views. Secondly, they implicated the dark undercurrents of racism, misogynism, or xenophobia that still permeate some corners of our society. From that ugly place, the connection to Mr. Trump’s foul-mouthed populism was not hard to make; his despicable bragging about women aside, to his foes, even an accidental hand gesture or an inane 4chan frog meme was proof enough. Once we crossed this line, the election was no longer about economic policy, the environment, or the like; it was an existential battle for equality and inclusiveness against the forces of evil that lurk in our midst. Not a day went by without a comparison between Mr. Trump and Adolf Hitler in the press. As for the moderate voters, the pundits had an explanation, too: the right-wing filter bubble must have clouded their judgment and created a false sense of equivalency between a horrid, conspiracy-peddling madman and our cozy, liberal status quo.

Now, before I offer my take, let me be clear that I do not wish to dismiss the legitimate concerns about the overtones of Mr. Trump’s campaign. Nor do I desire to downplay the scale of discrimination and hatred that the societies around the world are still grappling with, or the potential that the new administration could make it worse. But I found the aforementioned explanation of Mr. Trump’s unexpected victory to be unsatisfying in many ways. Ultimately, we all live in bubbles and we all have biases; in that regard, not much sets CNN apart from Fox News, Vox from National Review, or The Huffington Post from Breitbart. The reason why most of us would trust one and despise the other is that we instinctively recognize our own biases as more benign. After all, in the progressive world, we are fighting for an inclusive society that gives all people a fair chance to succeed. As for the other side? They seem like a bizarre, cartoonishly evil coalition of dimwits, racists, homophobes, and the ultra-rich. We even have serious scientific studies to back that up; their authors breathlessly proclaim that the conservative brain is inferior to the progressive brain. Unlike the conservatives, we believe in science, so we hit the “like” button and retweet the news.

But here’s the thing: I know quite a few conservatives, many of whom have probably voted for Mr. Trump – and they are about as smart, as informed, and as compassionate as my progressive friends. I think that the disconnect between the worldviews stems from something else: if you are a well-off person in a coastal city, you know people who are immigrants or who belong to other minorities, making you acutely attuned to their plight; but you may lack the same, deeply personal connection to – say – the situation of the lower middle class in the Midwest. You might have seen surprising charts or read a touching story in Mother Jones few years back, but it’s hard to think of them as individuals; they are more of a socioeconomic obstacle, a problem to be solved. The same goes for our understanding of immigration or globalization: these phenomena make our high-tech hubs more prosperous and more open; the externalities of our policies, if any, are just an abstract price that somebody else ought to bear for doing what’s morally right. And so, when Mr. Trump promises to temporarily ban travel from Muslim countries linked to terrorism or anti-American sentiments, we (rightly) gasp in disbelief; but when Mr. Obama paints an insulting caricature of rural voters as simpletons who “cling to guns or religion or antipathy to people who aren’t like them”, we smile and praise him for his wit, not understanding how the other side could be so offended by the truth. Similarly, when Mrs. Clinton chuckles while saying “we are going to put a lot of coal miners out of business” to a cheering crowd, the scene does not strike us as a thoughtless, offensive, or in poor taste. Maybe we will read a story about the miners in Mother Jones some day?

Of course, liberals take pride in caring for the common folk, but I suspect that their leaders’ attempts to reach out to the underprivileged workers in the “flyover states” often come across as ham-fisted and insincere. The establishment schools the voters about the inevitability of globalization, as if it were some cosmic imperative; they are told that to reject the premise would not just be wrong – but that it’d be a product of a diseased, nativist mind. They hear that the factories simply had to go to China or Mexico, and the goods just have to come back duty-free – all so that our complex, interconnected world can be a happier place. The workers are promised entitlements, but it stands to reason that they want dignity and hope for their children, not a lifetime on food stamps. The idle, academic debates about automation, post-scarcity societies, and Universal Basic Income probably come across as far-fetched and self-congratulatory, too.

The discourse is poisoned by cognitive biases in many other ways. The liberal media keeps writing about the unaccountable right-wing oligarchs who bankroll the conservative movement and supposedly poison people’s minds – but they offer nothing but praise when progressive causes are being bankrolled by Mr. Soros or Mr. Bloomberg. They claim that the conservatives represent “post-truth” politics – but their fact-checkers shoot down conservative claims over fairly inconsequential mistakes, while giving their favored politicians a pass on half-true platitudes about immigration, gun control, crime, or the sources of inequality. Mr. Obama sneers at the conservative bias of Fox News, but has no concern with the striking tilt to the left in the academia or in the mainstream press. The Economist finds it appropriate to refer to Trump supporters as “trumpkins” in print – but it would be unthinkable for them to refer to the fans of Mrs. Clinton using any sort of a mocking term. The pundits ponder the bold artistic statement made by the nude statues of the Republican nominee – but they would be disgusted if a conservative sculptor portrayed the Democratic counterpart in a similarly unflattering light. The commentators on MSNBC read into every violent incident at Trump rallies – but when a a random group of BLM protesters starts chanting about killing police officers, we all agree it would not be fair to cast the entire movement in a negative light.

Most progressives are either oblivious to these biases, or dismiss them as a harmless casualty of fighting the good fight. Perhaps so – and it is not my intent to imply equivalency between the causes of the left and of the right. But in the end, I suspect that the liberal echo chamber contributed to the election of Mr. Trump far more than anything that ever transpired on the right. It marginalized and excluded legitimate but alien socioeconomic concerns from the mainstream political discourse, binning them with truly bigoted and unintelligent speech – and leaving the “flyover underclass” no option other than to revolt. And it wasn’t just a revolt of the awful fringes. On the right, we had Mr. Trump – a clumsy outsider who eschews many of the core tenets of the conservative platform, and who does not convincingly represent neither the neoconservative establishment of the Bush era, nor the Bible-thumping religious right of the Tea Party. On the left, we had Mr. Sanders – an unaccomplished Senator who offered simplistic but moving slogans, who painted the accumulation of wealth as the source of our ills, and who promised to mold the United States into an idyllic version of the social democracies of Europe – supposedly governed by the workers, and not by the exploitative elites.

I think that people rallied behind Mr. Sanders and Mr. Trump not because they particularly loved the candidates or took all their promises seriously – but because they had no other credible herald for their cause. When the mainstream media derided their rebellion and the left simply laughed it off, it only served as a battle cry. When tens of millions of Trump supporters were labeled as xenophobic and sexist deplorables who deserved no place in politics, it only pushed more moderates toward the fringe. Suddenly, rational people could see themselves voting for a politically inexperienced and brash billionaire – a guy who talks about cutting taxes for the rich, who wants to cozy up to Russia, and whose VP pick previously wasn’t so sure about LGBT rights. I think it all happened not because of Mr. Trump’s character traits or thoughtful political positions, and not because half of the country hates women and minorities. He won because he was the only one to promise to “drain the swamp” – and to promise hope, not handouts, to the lower middle class.

There is a risk that this election will prove to be a step back for civil rights, or that Mr. Trump’s bold but completely untested economic policies will leave the world worse off; while not certain, it pains me to even contemplate this possibility. When we see injustice, we should fight tooth and nail. But for now, I am not swayed by the preemptively apocalyptic narrative on the left. Perhaps naively, I have faith in the benevolence of our compatriots and the strength of the institutions of – as cheesy as it sounds – one of the great nations of the world.

Exploring Geospatial Intelligence using SparkR on Amazon EMR

Post Syndicated from Gopal Wunnava original https://blogs.aws.amazon.com/bigdata/post/Tx1MECZ47VAV84F/Exploring-Geospatial-Intelligence-using-SparkR-on-Amazon-EMR

Gopal Wunnava is a Senior Consultant with AWS Professional Services

The number of data sources that use location, such as smartphones and sensory devices used in IoT (Internet of things), is expanding rapidly. This explosion has increased demand for analyzing spatial data.

Geospatial intelligence (GEOINT) allows you to analyze data that has geographical or spatial dimensions and present it based on its location. GEOINT can be applied to many industries, including social and environmental sciences, law enforcement, defense, and weather and disaster management. In this blog post, I show you how to build a simple GEOINT application using SparkR that will allow you to appreciate GEOINT capabilities.

R has been a popular platform for GEOINT due to its wide range of functions and packages related to spatial analysis. SparkR provides a great solution for overcoming known limitations in R because it lets you run geospatial applications in a distributed environment provided by the Spark engine. By implementing your SparkR geospatial application on Amazon EMR, you combine  these benefits with the flexibility and ease-of-use provided by EMR.

Overview of a GEOINT application

You’ll use the GDELT project for building this GEOINT application. The GDELT dataset you will use is available on Amazon S3  in the form of a tab-delimited text file with an approximate size of 13 GB.

Your GEOINT application will generate images like the one below.

Building your GEOINT application

Create an EMR cluster (preferably with the latest AMI version) in your region of choice, specifying Spark, Hive, and Ganglia.  To learn how to create a cluster, see Getting Started: Analyzing Big Data with Amazon EMR.

I suggest the r3 instance family for this application (1 master and 8 core nodes, all r38xlarge in this case), as it is suitable for the type of SparkR application you will create here. While I have chosen this cluster size for better performance, a smaller cluster size could work as well.

After your cluster is ready and you SSH into the master node, run the following command to install the files required for this application to run on EMR:

sudo yum install libjpeg-turbo-devel

For this GEOINT application, you identify and display locations where certain events of interest related to the economy are taking place in the US. For a more detailed description of these events, see the CAMEO manual available from the GDELT website.

You can use either the SparkR shell (type sparkR on the command line) or RStudio to develop this GEOINT application. To learn how to configure RStudio on EMR, see the Crunching Statistics at Scale with SparkR on Amazon EMR blog post.

You need to install the required R packages for this application onto the cluster. This can be done by executing the following R statement:

install.packages(c("plyr","dplyr","mapproj","RgoogleMaps","ggmap"))

Note: The above step can take up to thirty minutes because a number of dependent packages must be installed onto your EMR cluster.

After the required packages are installed the next step is to load these packages into your R environment:

library(plyr)
library(dplyr)
library(mapproj)
library(RgoogleMaps)
library(ggmap)

You can save the images generated by this application as a PDF document. Unless you use the setwd() function to set your desired path for this file, it defaults to your current working directory.

setwd("/home/hadoop")
pdf("SparkRGEOINTEMR.pdf")

If you are using RStudio, the plots appear in the lower-right corner of your workspace.

Now, create the Hive context that is required to access the external table from within the Spark environment:

#set up Hive context
hiveContext <- sparkRHive.init(sc)

Note: If you are using the SparkR shell in EMR, the spark context ‘sc’ is created automatically for you.  If you are using RStudio, follow the instructions in the Crunching Statistics at Scale with SparkR on Amazon EMR blog post to create the Spark context.

Next, create an external table that points to your source GDELT dataset on S3.

sql(hiveContext,
"
CREATE EXTERNAL TABLE IF NOT EXISTS gdelt (
GLOBALEVENTID BIGINT,
SQLDATE INT,
MonthYear INT,
Year INT,
FractionDate DOUBLE,
Actor1Code STRING,
Actor1Name STRING,
Actor1CountryCode STRING,
Actor1KnownGroupCode STRING,
Actor1EthnicCode STRING,
Actor1Religion1Code STRING,
Actor1Religion2Code STRING,
Actor1Type1Code STRING,
Actor1Type2Code STRING,
Actor1Type3Code STRING,
Actor2Code STRING,
Actor2Name STRING,
Actor2CountryCode STRING,
Actor2KnownGroupCode STRING,
Actor2EthnicCode STRING,
Actor2Religion1Code STRING,
Actor2Religion2Code STRING,
Actor2Type1Code STRING,
Actor2Type2Code STRING,
Actor2Type3Code STRING,
IsRootEvent INT,
EventCode STRING,
EventBaseCode STRING,
EventRootCode STRING,
QuadClass INT,
GoldsteinScale DOUBLE,
NumMentions INT,
NumSources INT,
NumArticles INT,
AvgTone DOUBLE,
Actor1Geo_Type INT,
Actor1Geo_FullName STRING,
Actor1Geo_CountryCode STRING,
Actor1Geo_ADM1Code STRING,
Actor1Geo_Lat FLOAT,
Actor1Geo_Long FLOAT,
Actor1Geo_FeatureID INT,
Actor2Geo_Type INT,
Actor2Geo_FullName STRING,
Actor2Geo_CountryCode STRING,
Actor2Geo_ADM1Code STRING,
Actor2Geo_Lat FLOAT,
Actor2Geo_Long FLOAT,
Actor2Geo_FeatureID INT,
ActionGeo_Type INT,
ActionGeo_FullName STRING,
ActionGeo_CountryCode STRING,
ActionGeo_ADM1Code STRING,
ActionGeo_Lat FLOAT,
ActionGeo_Long FLOAT,
ActionGeo_FeatureID INT,
DATEADDED INT,
SOURCEURL STRING )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION 's3://support.elasticmapreduce/training/datasets/gdelt'
");

Note: You might encounter an error in the above statement, with an error message that specifies an unused argument. This can be due to an overwritten Spark context. If this error appears, restart your SparkR or RStudio session.

Next, apply your filters to extract desired events in the last two years for the country of interest and store the results in a SparkR dataframe  named ‘gdelt’. In this post, I focus on spatial analysis of the data for the US only. For code samples that illustrate how this can be done for other countries, such as India, see the Exploring GDELT – Geospatial Analysis using SparkR on EMR GitHub site for the Big Data Blog.

gdelt<-sql(hiveContext,"SELECT * FROM gdelt WHERE ActionGeo_CountryCode IN ('US') AND Year >= 2014")

Register and cache this table in-memory:

registerTempTable(gdelt, "gdelt")
cacheTable(hiveContext, "gdelt")

Rename the columns for readability:

names(gdelt)[names(gdelt)=="actiongeo_countrycode"]="cn"
names(gdelt)[names(gdelt)=="actiongeo_lat"]="lat"
names(gdelt)[names(gdelt)=="actiongeo_long"]="long"
names(gdelt)

Next, extract a subset of columns from your original SparkR dataframe (‘gdelt’).  Of particular interest is the “lat” and “long” columns, as these attributes provide you with the locations where these events have taken place. Register and cache this result table into another SparkR dataframe  named ‘gdt’:

gdt=gdelt[,
		 c("sqldate",
		 "eventcode",
		 "globaleventid", 
		 "cn",
	         "year",
	         "lat",
	         "long")
             ]
registerTempTable(gdt, "gdt")
cacheTable(hiveContext, "gdt")

Now, filter down to specific events of interest. For this blog post, I have chosen certain event codes that relate to economic aid and co-operation.  While the CAMEO manual provides more details on what these specific event codes represent, I have provided a table below for quick reference. Refer to the manual to choose events from other news categories that may be of particular interest to you.

Store your chosen event codes into an R vector object:

ecocodes <- c("0211","0231","0311","0331","061","071")	

Next, apply a filter operation and store the results of this operation into another SparkR dataset named ‘gdeltusinf’. Follow the same approach of registering and caching this table.

gdeltusinf <- filter(gdt,gdt$eventcode %in% ecocodes)
registerTempTable(gdeltusinf, "gdeltusinf")
cacheTable(hiveContext, "gdeltusinf")

Now that you have a smaller dataset that is a subset of the original, collect this SparkR dataframe into a local R dataframe. By doing this, you can leverage the spatial libraries installed previously into your R environment.

dflocale1=collect(select(gdeltusinf,"*"))	

Save the R dataframe to a local file system in case you need to quit your SparkR session and want to reuse the file at a later point. You can also share this saved file with other R users and sessions.  

save(dflocale1, file = "gdeltlocale1.Rdata")

Now, create separate dataframes by country as this allows you to plot the corresponding event locations on separate maps:

dflocalus1=subset(dflocale1,dflocale1$cn=='US')

Next, provide a suitable title for the maps and prepare to plot them:

plot.new()
title("GDELT Analysis for Economy related Events in 2014-2015")
map=qmap('USA',zoom=3)

The first plot identifies locations where all events related to economic aid and co-operation have taken place in the US within the last two years (2014-2015). For this example, these locations are marked in red, but you can choose another color.

map + geom_point(data = dflocalus1, aes(x = dflocalus1$long, y = dflocalus1$lat), color="red", size=0.5, alpha=0.5)
title("All GDELT Event Locations in USA related to Economy in 2014-2015")

It may take several seconds for the image to be displayed. From the above image, you can infer that the five chosen events related to economy took place in locations all over the US in 2014-2015. While this map provides you with the insight that these five events were fairly widespread in the US, you might want to drill down further and identify locations where each of these five specific events took place.

For this purpose, display only certain chosen events. Start by displaying locations where events related to ‘0211’ (Economic Co-op for Appeals) have taken place in the US, using the color blue:

dflocalus0211=subset(dflocalus1,dflocalus1$eventcode=='0211')
x0211=geom_point(data = dflocalus0211, aes(x = dflocalus0211$long, y = dflocalus0211$lat), color="blue", size=2, alpha=0.5)
map+x0211
title("GDELT Event Locations in USA: Economic Co-op(appeals)-Code 0211")

From the image above, you can see that the event ‘0211’ (Economic Co-op for Appeals) was fairly widespread as well, but there is more of a concentration within the Eastern region of the US, specifically the Northeast.

Next, follow the same process, but this time for a different event –‘0231’ (Economic Aid for Appeals). Notice the use of the color yellow for this purpose.

dflocalus0231=subset(dflocalus1,dflocalus1$eventcode=='0231')
x0231=geom_point(data = dflocalus0231, aes(x = dflocalus0231$long, y = dflocalus0231$lat), color="yellow", size=2, alpha=0.5)
map+x0231
title("GDELT Event Locations in USA:Economic Aid(appeals)-Code 0231")

From the above image, you can see that there is a heavy concentration of this event type in the Midwest, Eastern, and Western parts of the US while the North-Central region is sparser.

You can follow a similar approach to prepare separate R dataframes for each event of interest. Choosing a different color for each event allows you to identify each event type and locations much more easily.

dflocalus0311=subset(dflocalus1,dflocalus1$eventcode=='0311')
dflocalus0331=subset(dflocalus1,dflocalus1$eventcode=='0331')
dflocalus061=subset(dflocalus1,dflocalus1$eventcode=='061')
dflocalus071=subset(dflocalus1,dflocalus1$eventcode=='071')

x0211=geom_point(data = dflocalus0211, aes(x = dflocalus0211$long, y = dflocalus0211$lat), color="blue", size=3, alpha=0.5)
x0231=geom_point(data = dflocalus0231, aes(x = dflocalus0231$long, y = dflocalus0231$lat), color="yellow", size=1, alpha=0.5)
x0311=geom_point(data = dflocalus0311, aes(x = dflocalus0311$long, y = dflocalus0311$lat), color="red", size=1, alpha=0.5)
x0331=geom_point(data = dflocalus0331, aes(x = dflocalus0331$long, y = dflocalus0331$lat), color="green", size=1, alpha=0.5)
x061=geom_point(data = dflocalus061, aes(x = dflocalus061$long, y = dflocalus061$lat), color="orange", size=1, alpha=0.5)
x071=geom_point(data = dflocalus071, aes(x = dflocalus071$long, y = dflocalus071$lat), color="violet", size=1, alpha=0.5)

Using this approach allows you to overlay locations of different events on the same map, where each event is represented by a specific color. To illustrate this with an example, use the following code to display locations of three specific events from the steps above:

map+x0211+x0231+x0311

legend(‘bottomleft’,c("0211:Appeal for Economic Co-op","0231:Appeal for Economic Aid","0311:Intent for Economic Co-op"),col=c("blue","yellow","red"),pch=16) title("GDELT Locations In USA: Economy related Events in 2014-2015")

Note: If you are using RStudio, you might have to hit the refresh button on the Plots tab to display the map in the lower-right portion of your workspace.

From the above image, you can see that the Northeast region has the heaviest concentration of events, while certain areas such as North Central are sparser.

Conclusion

I’ve shown you how to build a simple yet powerful geospatial application using SparkR on EMR.  Though R helps with geospatial analysis, native R limitations prevent GEOINT applications from scaling to the extent required by large-scale GEOINT applications. You can overcome these limitations by mixing SparkR with native R workloads, a method referred to as big data / small learning.  This approach lets you take your GEOINT application performance to the next level while still running the R analytics you know and love.

You can find the code samples for this GEOINT application, along with other use cases for this application, on the Exploring GDELT – Geospatial Analysis using SparkR on EMR GitHub site.  You can also find code samples that make use of the concept of pipelines and the pipeR package to implement this functionality. Along similar lines, you can make use of the magrittr package to implement the functionality represented in this application.

If you have any questions or suggestions, please leave a comment below.

————————————

Related

Crunching Statistics at Scale with SparkR on Amazon EMR

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

Subjective explainer: gun debate in the US

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/10/subjective-explainer-gun-debate-in-us.html

In the wake of the tragic events in Roseburg, I decided to briefly return to the topic of looking at the US culture from the perspective of a person born in Europe. In particular, I wanted to circle back to the topic of firearms.

Contrary to popular beliefs, the United States has witnessed a dramatic decline in violence over the past 20 years. In fact, when it comes to most types of violent crime – say, robbery, assault, or rape – the country now compares favorably to the UK and many other OECD nations. But as I explored in my earlier posts, one particular statistic – homicide – is still registering about three times as high as in many other places within the EU.

The homicide epidemic in the United States has a complex nature and overwhelmingly affects ethnic minorities and other disadvantaged social groups; perhaps because of this, the phenomenon sees very little honest, public scrutiny. It is propelled into the limelight only in the wake of spree shootings and other sickening, seemingly random acts of terror; such incidents, although statistically insignificant, take a profound mental toll on the American society. At the same time, the effects of high-profile violence seem strangely short-lived: they trigger a series of impassioned political speeches, invariably focusing on the connection between violence and guns – but the nation soon goes back to business as usual, knowing full well that another massacre will happen soon, perhaps the very same year.

On the face of it, this pattern defies all reason – angering my friends in Europe and upsetting many brilliant and well-educated progressives in the US. They utter frustrated remarks about the all-powerful gun lobby and the spineless politicians, blaming the partisan gridlock for the failure to pass even the most reasonable and toothless gun control laws. I used to be in the same camp; today, I think the reality is more complex than that.

To get to the bottom of this mystery, it helps to look at the spirit of radical individualism and classical liberalism that remains the national ethos of the United States – and in fact, is enjoying a degree of resurgence unseen for many decades prior. In Europe, it has long been settled that many individual liberties – be it the freedom of speech or the natural right to self-defense – can be constrained to advance even some fairly far-fetched communal goals. On the old continent, such sacrifices sometimes paid off, and sometimes led to atrocities; but the basic premise of European collectivism is not up for serious debate. In America, the same notion certainly cannot be taken for granted today.

When it comes to firearm ownership in particular, the country is facing a fundamental choice between two possible realities:

A largely disarmed society that depends on the state to protect it from almost all harm, and where citizens are generally not permitted to own guns without presenting a compelling cause. In this model, adopted by many European countries, firearms tend to be less available to common criminals – simply by the virtue of limited supply and comparatively high prices in black market trade. At the same time, it can be argued that any nation subscribing to this doctrine becomes more vulnerable to foreign invasion or domestic terror, should its government ever fail to provide adequate protection to all citizens. Disarmament can also limit civilian recourse against illegitimate, totalitarian governments – a seemingly outlandish concern, but also a very fresh memory for many European countries subjugated not long ago under the auspices of the Soviet Bloc.

A well-armed society where firearms are available to almost all competent adults, and where the natural right to self-defense is subject to few constraints. This is the model currently employed in the United States, where it arises from the straightfoward, originalist interpretation of the Second Amendment – as recognized by roughly 75% of all Americans and affirmed by the Supreme Court. When following such a doctrine, a country will likely witness greater resiliency in the face of calamities or totalitarian regimes. At the same time, its citizens might have to accept some inherent, non-trivial increase in violent crime due to the prospect of firearms more easily falling into the wrong hands.

It seems doubtful that a viable middle-ground approach can exist in the United States. With more than 300 million civilian firearms in circulation, most of them in unknown hands, the premise of reducing crime through gun control would inevitably and critically depend on some form of confiscation; without such drastic steps, the supply of firearms to the criminal underground or to unfit individuals would not be disrupted in any meaningful way. Because of this, intellectual integrity requires us to look at many of the legislative proposals not only through the prism of their immediate utility, but also to give consideration to the societal model they are likely to advance.

And herein lies the problem: many of the current “common-sense” gun control proposals have very little merit when considered in isolation. There is scant evidence that reinstating the ban on military-looking semi-automatic rifles (“assault weapons”), or rolling out the prohibition on private sales at gun shows, would deliver measurable results. There is also no compelling reason to believe that ammo taxes, firearm owner liability insurance, mandatory gun store cameras, firearm-free school zones, bans on open carry, or federal gun registration can have any impact on violent crime. And so, the debate often plays out like this:

At the same time, by the virtue of making weapons more difficult, expensive, and burdensome to own, many of the legislative proposals floated by progressives would probably gradually erode the US gun culture; intentionally or not, their long-term outcome would be a society less passionate about firearms and more willing to follow in the footsteps of Australia or the UK. Only as we cross that line and confiscate hundreds of millions of guns, it’s fathomable – yet still far from certain – that we would see a sharp drop in homicides.

This method of inquiry helps explain the visceral response from gun rights advocates: given the legislation’s dubious benefits and its predicted long-term consequences, many pro-gun folks are genuinely worried that making concessions would eventually mean giving up one of their cherished civil liberties – and on some level, they are right.

Some feel that this argument is a fallacy, a tell tale invented by a sinister corporate “gun lobby” to derail the political debate for personal gain. But the evidence of such a conspiracy is hard to find; in fact, it seems that the progressives themselves often fan the flames. In the wake of Roseburg, both Barack Obama and Hillary Clinton came out praising the confiscation-based gun control regimes employed in Australia and the UK – and said that they would like the US to follow suit. Depending on where you stand on the issue, it was either an accidental display of political naivete, or the final reveal of their sinister plan. For the latter camp, the ultimate proof of a progressive agenda came a bit later: in response to the terrorist attack in San Bernardino, several eminent Democratic-leaning newspapers published scathing editorials demanding civilian disarmament while downplaying the attackers’ connection to Islamic State.

Another factor that poisons the debate is that despite being highly educated and eloquent, the progressive proponents of gun control measures are often hopelessly unfamiliar with the very devices they are trying to outlaw:

I’m reminded of the widespread contempt faced by Senator Ted Stevens following his attempt to compare the Internet to a “series of tubes” as he was arguing against net neutrality. His analogy wasn’t very wrong – it just struck a nerve as simplistic and out-of-date. My progressive friends did not react the same way when Representative Carolyn McCarthy – one of the key proponents of the ban on assault weapons – showed no understanding of the supposedly lethal firearm features she was trying to eradicate. Such bloopers are not rare, too; not long ago, Mr. Bloomberg, one of the leading progressive voices on gun control in America, argued against semi-automatic rifles without understanding how they differ from the already-illegal machine guns:

Yet another example comes Representative Diana DeGette, the lead sponsor of a “common-sense” bill that sought to prohibit the manufacture of magazines with capacity over 15 rounds. She defended the merits of her legislation while clearly not understanding how a magazine differs from ammunition – or that the former can be reused:

“I will tell you these are ammunition, they’re bullets, so the people who have those know they’re going to shoot them, so if you ban them in the future, the number of these high capacity magazines is going to decrease dramatically over time because the bullets will have been shot and there won’t be any more available.”

Treating gun ownership with almost comical condescension has become vogue among a good number of progressive liberals. On a campaign stop in San Francisco, Mr. Obama sketched a caricature of bitter, rural voters who “cling to guns or religion or antipathy to people who aren’t like them”. Not much later, one Pulitzer Prize-winning columnist for The Washington Post spoke of the Second Amendment as “the refuge of bumpkins and yeehaws who like to think they are protecting their homes against imagined swarthy marauders desperate to steal their flea-bitten sofas from their rotting front porches”. Many of the newspaper’s readers probably had a good laugh – and then wondered why it has gotten so difficult to seek sensible compromise.

There are countless dubious and polarizing claims made by the supporters of gun rights, too; examples include a recent NRA-backed tirade by Dana Loesch denouncing the “godless left”, or the constant onslaught of conspiracy theories spewed by Alex Jones and Glenn Beck. But when introducing new legislation, the burden of making educated and thoughtful arguments should rest on its proponents, not other citizens. When folks such as Bloomberg prescribe sweeping changes to the American society while demonstrating striking ignorance about the topics they want to regulate, they come across as elitist and flippant – and deservedly so.

Given how controversial the topic is, I think it’s wise to start an open, national conversation about the European model of gun control and the risks and benefits of living in an unarmed society. But it’s also likely that such a debate wouldn’t last very long. Progressive politicians like to say that the dialogue is impossible because of the undue influence of the National Rifle Association – but as I discussed in my earlier blog posts, the organization’s financial resources and power are often overstated: it does not even make it onto the list of top 100 lobbyists in Washington, and its support comes mostly from member dues, not from shadowy business interests or wealthy oligarchs. In reality, disarmament just happens to be a very unpopular policy in America today: the support for gun ownership is very strong and has been growing over the past 20 years – even though hunting is on the decline.

Perhaps it would serve the progressive movement better to embrace the gun culture – and then think of ways to curb its unwanted costs. Addressing inner-city violence, especially among the disadvantaged youth, would quickly bring the US homicide rate much closer to the rest of the highly developed world. But admitting the staggering scale of this social problem can be an uncomfortable and politically charged position to hold. For Democrats, it would be tantamount to singling out minorities. For Republicans, it would be just another expansion of the nanny state.

PS. If you are interested in a more systematic evaluation of the scale, the impact, and the politics of gun ownership in the United States, you may enjoy an earlier entry on this blog. Or, if you prefer to read my entire series comparing the life in Europe and in the US, try this link.

Subjective explainer: gun debate in the US

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/10/subjective-explainer-gun-debate-in-us.html

In the wake of the tragic events in Roseburg, I decided to briefly return to the topic of looking at the US culture from the perspective of a person born in Europe. In particular, I wanted to circle back to the topic of firearms.

Contrary to popular beliefs, the United States has witnessed a dramatic decline in violence over the past 20 years. In fact, when it comes to most types of violent crime – say, robbery, assault, or rape – the country now compares favorably to the UK and many other OECD nations. But as I explored in my earlier posts, one particular statistic – homicide – is still registering about three times as high as in many other places within the EU.

The homicide epidemic in the United States has a complex nature and overwhelmingly affects ethnic minorities and other disadvantaged social groups; perhaps because of this, the phenomenon sees very little honest, public scrutiny. It is propelled into the limelight only in the wake of spree shootings and other sickening, seemingly random acts of terror; such incidents, although statistically insignificant, take a profound mental toll on the American society. At the same time, the effects of high-profile violence seem strangely short-lived: they trigger a series of impassioned political speeches, invariably focusing on the connection between violence and guns – but the nation soon goes back to business as usual, knowing full well that another massacre will happen soon, perhaps the very same year.

On the face of it, this pattern defies all reason – angering my friends in Europe and upsetting many brilliant and well-educated progressives in the US. They utter frustrated remarks about the all-powerful gun lobby and the spineless politicians, blaming the partisan gridlock for the failure to pass even the most reasonable and toothless gun control laws. I used to be in the same camp; today, I think the reality is more complex than that.

To get to the bottom of this mystery, it helps to look at the spirit of radical individualism and classical liberalism that remains the national ethos of the United States – and in fact, is enjoying a degree of resurgence unseen for many decades prior. In Europe, it has long been settled that many individual liberties – be it the freedom of speech or the natural right to self-defense – can be constrained to advance even some fairly far-fetched communal goals. On the old continent, such sacrifices sometimes paid off, and sometimes led to atrocities; but the basic premise of European collectivism is not up for serious debate. In America, the same notion certainly cannot be taken for granted today.

When it comes to firearm ownership in particular, the country is facing a fundamental choice between two possible realities:

A largely disarmed society that depends on the state to protect it from almost all harm, and where citizens are generally not permitted to own guns without presenting a compelling cause. In this model, adopted by many European countries, firearms tend to be less available to common criminals – simply by the virtue of limited supply and comparatively high prices in black market trade. At the same time, it can be argued that any nation subscribing to this doctrine becomes more vulnerable to foreign invasion or domestic terror, should its government ever fail to provide adequate protection to all citizens. Disarmament can also limit civilian recourse against illegitimate, totalitarian governments – a seemingly outlandish concern, but also a very fresh memory for many European countries subjugated not long ago under the auspices of the Soviet Bloc.

A well-armed society where firearms are available to almost all competent adults, and where the natural right to self-defense is subject to few constraints. This is the model currently employed in the United States, where it arises from the straightfoward, originalist interpretation of the Second Amendment – as recognized by roughly 75% of all Americans and affirmed by the Supreme Court. When following such a doctrine, a country will likely witness greater resiliency in the face of calamities or totalitarian regimes. At the same time, its citizens might have to accept some inherent, non-trivial increase in violent crime due to the prospect of firearms more easily falling into the wrong hands.

It seems doubtful that a viable middle-ground approach can exist in the United States. With more than 300 million civilian firearms in circulation, most of them in unknown hands, the premise of reducing crime through gun control would inevitably and critically depend on some form of confiscation; without such drastic steps, the supply of firearms to the criminal underground or to unfit individuals would not be disrupted in any meaningful way. Because of this, intellectual integrity requires us to look at many of the legislative proposals not only through the prism of their immediate utility, but also to give consideration to the societal model they are likely to advance.

And herein lies the problem: many of the current “common-sense” gun control proposals have very little merit when considered in isolation. There is scant evidence that reinstating the ban on military-looking semi-automatic rifles (“assault weapons”), or rolling out the prohibition on private sales at gun shows, would deliver measurable results. There is also no compelling reason to believe that ammo taxes, firearm owner liability insurance, mandatory gun store cameras, firearm-free school zones, bans on open carry, or federal gun registration can have any impact on violent crime. And so, the debate often plays out like this:

At the same time, by the virtue of making weapons more difficult, expensive, and burdensome to own, many of the legislative proposals floated by progressives would probably gradually erode the US gun culture; intentionally or not, their long-term outcome would be a society less passionate about firearms and more willing to follow in the footsteps of Australia or the UK. Only as we cross that line and confiscate hundreds of millions of guns, it’s fathomable – yet still far from certain – that we would see a sharp drop in homicides.

This method of inquiry helps explain the visceral response from gun rights advocates: given the legislation’s dubious benefits and its predicted long-term consequences, many pro-gun folks are genuinely worried that making concessions would eventually mean giving up one of their cherished civil liberties – and on some level, they are right.

Some feel that this argument is a fallacy, a tell tale invented by a sinister corporate “gun lobby” to derail the political debate for personal gain. But the evidence of such a conspiracy is hard to find; in fact, it seems that the progressives themselves often fan the flames. In the wake of Roseburg, both Barack Obama and Hillary Clinton came out praising the confiscation-based gun control regimes employed in Australia and the UK – and said that they would like the US to follow suit. Depending on where you stand on the issue, it was either an accidental display of political naivete, or the final reveal of their sinister plan. For the latter camp, the ultimate proof of a progressive agenda came a bit later: in response to the terrorist attack in San Bernardino, several eminent Democratic-leaning newspapers published scathing editorials demanding civilian disarmament while downplaying the attackers’ connection to Islamic State.

Another factor that poisons the debate is that despite being highly educated and eloquent, the progressive proponents of gun control measures are often hopelessly unfamiliar with the very devices they are trying to outlaw:

I’m reminded of the widespread contempt faced by Senator Ted Stevens following his attempt to compare the Internet to a “series of tubes” as he was arguing against net neutrality. His analogy wasn’t very wrong – it just struck a nerve as simplistic and out-of-date. My progressive friends did not react the same way when Representative Carolyn McCarthy – one of the key proponents of the ban on assault weapons – showed no understanding of the supposedly lethal firearm features she was trying to eradicate. Such bloopers are not rare, too; not long ago, Mr. Bloomberg, one of the leading progressive voices on gun control in America, argued against semi-automatic rifles without understanding how they differ from the already-illegal machine guns:

Yet another example comes Representative Diana DeGette, the lead sponsor of a “common-sense” bill that sought to prohibit the manufacture of magazines with capacity over 15 rounds. She defended the merits of her legislation while clearly not understanding how a magazine differs from ammunition – or that the former can be reused:

“I will tell you these are ammunition, they’re bullets, so the people who have those know they’re going to shoot them, so if you ban them in the future, the number of these high capacity magazines is going to decrease dramatically over time because the bullets will have been shot and there won’t be any more available.”

Treating gun ownership with almost comical condescension has become vogue among a good number of progressive liberals. On a campaign stop in San Francisco, Mr. Obama sketched a caricature of bitter, rural voters who “cling to guns or religion or antipathy to people who aren’t like them”. Not much later, one Pulitzer Prize-winning columnist for The Washington Post spoke of the Second Amendment as “the refuge of bumpkins and yeehaws who like to think they are protecting their homes against imagined swarthy marauders desperate to steal their flea-bitten sofas from their rotting front porches”. Many of the newspaper’s readers probably had a good laugh – and then wondered why it has gotten so difficult to seek sensible compromise.

There are countless dubious and polarizing claims made by the supporters of gun rights, too; examples include a recent NRA-backed tirade by Dana Loesch denouncing the “godless left”, or the constant onslaught of conspiracy theories spewed by Alex Jones and Glenn Beck. But when introducing new legislation, the burden of making educated and thoughtful arguments should rest on its proponents, not other citizens. When folks such as Bloomberg prescribe sweeping changes to the American society while demonstrating striking ignorance about the topics they want to regulate, they come across as elitist and flippant – and deservedly so.

Given how controversial the topic is, I think it’s wise to start an open, national conversation about the European model of gun control and the risks and benefits of living in an unarmed society. But it’s also likely that such a debate wouldn’t last very long. Progressive politicians like to say that the dialogue is impossible because of the undue influence of the National Rifle Association – but as I discussed in my earlier blog posts, the organization’s financial resources and power are often overstated: it does not even make it onto the list of top 100 lobbyists in Washington, and its support comes mostly from member dues, not from shadowy business interests or wealthy oligarchs. In reality, disarmament just happens to be a very unpopular policy in America today: the support for gun ownership is very strong and has been growing over the past 20 years – even though hunting is on the decline.

Perhaps it would serve the progressive movement better to embrace the gun culture – and then think of ways to curb its unwanted costs. Addressing inner-city violence, especially among the disadvantaged youth, would quickly bring the US homicide rate much closer to the rest of the highly developed world. But admitting the staggering scale of this social problem can be an uncomfortable and politically charged position to hold. For Democrats, it would be tantamount to singling out minorities. For Republicans, it would be just another expansion of the nanny state.

PS. If you are interested in a more systematic evaluation of the scale, the impact, and the politics of gun ownership in the United States, you may enjoy an earlier entry on this blog. Or, if you prefer to read my entire series comparing the life in Europe and in the US, try this link.

Poland vs the United States: American exceptionalism

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/07/poland-vs-united-states-american.html

This is the fourteenth article talking about Poland, Europe, and the United States. To explore the entire collection, start here.

This is destined to be the final entry in the series that opened with a chronicle of my journey from Poland to the United States, only to veer into some of the most interesting social differences between America and the old continent. There are many other topics I could still write about – anything from the school system, to religion, to the driving culture – but with my parental leave coming to an end, I decided to draw a line. I’m sure that this decision will come as a relief for those who read the blog for technical insights, rather than political commentary 🙂

The final topic I wanted to talk about is something that truly irks some of my European friends: the belief, held deeply by many Americans, that their country is the proverbial “city upon a hill” – a shining beacon of liberty and righteousness, blessed by the maker with the moral right to shape the world – be it by flexing its economic and diplomatic muscles, or with its sheer military might.

It is an interesting phenomenon, and one that certainly isn’t exclusive to the United States. In fact, expansive exceptionalism used to be a very strong theme in the European doctrine long before it emerged in other parts of the Western world. For one, it underpinned many of the British, French, Spanish, and Dutch colonial conquests over the past 500 years. The romanticized notion of Sonderweg played a menacing role in German political discourse, too – eventually culminating in the rise of the Nazi ideology and the onset of World War II. It wasn’t until the defeat of the Third Reich when Europe, faced with unspeakable destruction and unprecedented loss of life, made a concerted effort to root out many of its nationalist sentiments and embrace a more harmonious, collective path as a single European community.

America, in a way, experienced the opposite: although it has always celebrated its own rejection of feudalism and monarchism – and in that sense, it had a robust claim to being a pretty unique corner of the world – the country largely shied away from global politics, participating only very reluctantly in World War I, then hoping to wait out World War II up until being attacked by Japan. Its conviction about its special role on the world stage has solidified only after it paid a tremendous price to help defeat the Germans, to stop the march of the Red Army through the continent, and to build a prosperous and peaceful Europe; given the remarkable significance of this feat, the post-war sentiments in America may be not hard to understand. In that way, the roots of American exceptionalism differed from its European predecessors, being fueled by a fairly pure sense of righteousness – and not by anger, by a sense of injury, or by territorial demands.

Of course, the new superpower has also learned that its military might has its limits, facing humiliating defeats in some of the proxy wars with the Soviets and seeing an endless spiral of violence in the Middle East. The voices predicting its imminent demise, invariably present from the earliest days of the republic, have grown stronger and more confident over the past 50 years. But the country remains a military and economic powerhouse; and in some ways, its trigger-happy politicians provide a counterbalance to the other superpowers’ greater propensity to turn a blind eye to humanitarian crises and to genocide. It’s quite possible that without the United States arming its allies and tempering the appetites of Russia, North Korea, or China, the world would have been a less happy place. It’s just as likely that the Middle East would have been a happier one.

Some Europeans show indignation that Americans, with their seemingly know-it-all attitudes toward the rest of the world, still struggle to pinpoint Austria or Belgium on the map. It is certainly true that the media in the US pays little attention to the old continent. But deep down inside, European outlets don’t necessarily fare a lot better, often focusing its international coverage on the silly and the formulaic: when in Europe, you are far more likely to hear about a daring rescue of a cat stuck on a tree in Wyoming, or about the Creation Museum in Kentucky, than you are to learn anything substantive about Obamacare. (And speaking of Wyoming and Kentucky, pinpointing these places on the map probably wouldn’t be the European viewer’s strongest feat). In the end, Europeans who think they understand the intricacies of US politics are probably about as wrong as the average American making sweeping generalizations about Europe.

And on that intentionally self-deprecating note, it’s time to wrap the series up.

Poland vs the United States: American exceptionalism

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/07/poland-vs-united-states-american.html

This is the fourteenth article talking about Poland, Europe, and the United States. To explore the entire collection, start here.

This is destined to be the final entry in the series that opened with a chronicle of my journey from Poland to the United States, only to veer into some of the most interesting social differences between America and the old continent. There are many other topics I could still write about – anything from the school system, to religion, to the driving culture – but with my parental leave coming to an end, I decided to draw a line. I’m sure that this decision will come as a relief for those who read the blog for technical insights, rather than political commentary 🙂

The final topic I wanted to talk about is something that truly irks some of my European friends: the belief, held deeply by many Americans, that their country is the proverbial “city upon a hill” – a shining beacon of liberty and righteousness, blessed by the maker with the moral right to shape the world – be it by flexing its economic and diplomatic muscles, or with its sheer military might.

It is an interesting phenomenon, and one that certainly isn’t exclusive to the United States. In fact, expansive exceptionalism used to be a very strong theme in the European doctrine long before it emerged in other parts of the Western world. For one, it underpinned many of the British, French, Spanish, and Dutch colonial conquests over the past 500 years. The romanticized notion of Sonderweg played a menacing role in German political discourse, too – eventually culminating in the rise of the Nazi ideology and the onset of World War II. It wasn’t until the defeat of the Third Reich when Europe, faced with unspeakable destruction and unprecedented loss of life, made a concerted effort to root out many of its nationalist sentiments and embrace a more harmonious, collective path as a single European community.

America, in a way, experienced the opposite: although it has always celebrated its own rejection of feudalism and monarchism – and in that sense, it had a robust claim to being a pretty unique corner of the world – the country largely shied away from global politics, participating only very reluctantly in World War I, then hoping to wait out World War II up until being attacked by Japan. Its conviction about its special role on the world stage has solidified only after it paid a tremendous price to help defeat the Germans, to stop the march of the Red Army through the continent, and to build a prosperous and peaceful Europe; given the remarkable significance of this feat, the post-war sentiments in America may be not hard to understand. In that way, the roots of American exceptionalism differed from its European predecessors, being fueled by a fairly pure sense of righteousness – and not by anger, by a sense of injury, or by territorial demands.

Of course, the new superpower has also learned that its military might has its limits, facing humiliating defeats in some of the proxy wars with the Soviets and seeing an endless spiral of violence in the Middle East. The voices predicting its imminent demise, invariably present from the earliest days of the republic, have grown stronger and more confident over the past 50 years. But the country remains a military and economic powerhouse; and in some ways, its trigger-happy politicians provide a counterbalance to the other superpowers’ greater propensity to turn a blind eye to humanitarian crises and to genocide. It’s quite possible that without the United States arming its allies and tempering the appetites of Russia, North Korea, or China, the world would have been a less happy place. It’s just as likely that the Middle East would have been a happier one.

Some Europeans show indignation that Americans, with their seemingly know-it-all attitudes toward the rest of the world, still struggle to pinpoint Austria or Belgium on the map. It is certainly true that the media in the US pays little attention to the old continent. But deep down inside, European outlets don’t necessarily fare a lot better, often focusing its international coverage on the silly and the formulaic: when in Europe, you are far more likely to hear about a daring rescue of a cat stuck on a tree in Wyoming, or about the Creation Museum in Kentucky, than you are to learn anything substantive about Obamacare. (And speaking of Wyoming and Kentucky, pinpointing these places on the map probably wouldn’t be the European viewer’s strongest feat). In the end, Europeans who think they understand the intricacies of US politics are probably about as wrong as the average American making sweeping generalizations about Europe.

And on that intentionally self-deprecating note, it’s time to wrap the series up.

GCC, LLVM, Copyleft, Companies, and Non-Profits

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2014/01/26/llvm.html

[ Please keep in mind in reading this post that while both FSF and
Conservancy are mentioned, and
that I have leadership roles at both organizations, these opinions on
ebb.org, as always, are my own and
don’t necessarily reflect the view of FSF and/or Conservancy. ]

Most people know I’m a fan of RMS’
writing about Free Software and I agree with most (but not all) of his
beliefs about software freedom politics and strategy. I was delighted to
read RMS’ post
about LLVM on the GCC mailing list on Friday
. It’s clear and concise,
and, as usual, I agree with most (but not all) of it, and I encourage
people to read it. Meanwhile, upon reading comments on LWN on
this post
, I felt the need to add a few points to the discussion.

Firstly, I’m troubled to see so many developers, including
GCC developers,
conflating various social troubles in the GCC community with the choice of
license. I think it’s impossible to deny that culturally, the GCC
community faces challenges, like any community that has lasted for so long.
Indeed, there’s a long political history of GCC that even predates my
earliest involvement with the Free Software community (even though I’m now
considered an old-timer in Free Software in part because I played a small
role — as a young, inexperienced FSF
volunteer — in helping negotiate the EGCS
fork
back into the GCC mainline).

But none of these politics really relate to GCC’s license. The copyleft
was about ensuring that there were never proprietary improvements to the
compiler, and AFAIK no GCC developers ever wanted that. In fact, GCC was
ultimately the first major enforcement test of the GPL, and ironically that
test sent us on the trajectory that led to the current situation.

Specifically, as I’ve spoken about
in my
many talks
on
GPL compliance, the
earliest publicly discussed major GPL violation was by NeXT computing when
Steve Jobs attempted and failed (thanks to RMS’ GPL enforcement work) to
make the Objective C
front-end to GCC proprietary
. Everything for everyone involved would
have gone quite differently if that enforcement effort had failed.

As it stands, copyleft was upheld and worked. For years, until quite
recently (in context of the history of computing, anyway), Apple itself
used and relied on the Free Software GCC as its primary and preferred
Objective C compiler, because of that enforcement against NeXT so
long ago. But, that occurrence also likely solidified Jobs’ irrational
hatred of copyleft and software freedom, and Apple was on a mission to find
an alternative compiler — but writing a compiler is difficult and
takes time.

Meanwhile, I should point out that copyleft advocates sometimes conflate
issues in analyzing the situation with LLVM. I believe most LLVM
developers when they say that they don’t like proprietary software and that
they want to encourage software freedom. I really think they do. And, for
all of us, copyleft
isn’t a religion, or even a belief — it’s
a strategy to maximize software freedom, and no one
(AFAICT) has
said it’s the only viable strategy to do that. It’s quite
possible the strategy of LLVM developers of changing the APIs quickly to
thwart proprietarization
might work. I really doubt it, though, and here’s
why:

I’ll concede that LLVM was started with the best of academic intentions to
make better compiler technology and share it freely. (I’ve discussed this
issue at some length
with Chris Lattner
directly, and I believe he actually is someone who wants more software
freedom in the world, even if he disagrees with copyleft as a strategy.)
IMO, though, the problem we face is exploitation by various anti-copyleft,
software-freedom-unfriendly companies that seek to remove every copyleft
component from any software stack. Their reasons for pursuing that goal may or may not
be rational, but its collateral damage has already become clear: it’s
possible today to license proprietary improvements to LLVM that aren’t
released as Free Software. I predict this will become more common,
notwithstanding any technical efforts of LLVM developers to thwart it.
(Consider, by way of historical
example, that
proprietary combined works with Apache
web server continue
to this very day, despite Apache developers’ decades of
we’ll break APIs, so don’t keep your stuff
proprietary claims.)

Copyleft is always a trade-off between software freedom and adoption. I
don’t admonish people for picking the adoption side over the software
freedom side, but I do think as a community we should be honest with
ourselves that copyleft remains the best strategy to
prevent proprietary improvements and forks and no other strategy has been
as successful in reaching that goal. And, those who don’t pick copyleft have priorities other than
software freedom ranked higher in their goals.

As a penultimate point, I’ll reiterate something
that Joe Buck pointed out on the
LWN thread
: a lot of effort was put in to creating a licensing solution
that solved the copyleft concerns of GCC plugins. FSF’s worry for more
than a decade (reaching back into the late 1990s) was that a GCC plugin
architecture would allow writing to an output file GCC’s intermediate
representation, which would, in turn, allow a wholly separate program to
optimize the software by reading and writing that file format, and thus
circumvent the protections of copyleft.
The GCC
Runtime Library Exception (GCC RTL Exception)
is (in my biased opinion)
an innovative licensing solution that solves the problem
— the ironic
outcome: you are only permitted to perform proprietary optimization with
GCC on GPL’d software, but not on proprietary software.

The problem was that the GCC RTL

Exception came too late. While I led the GCC RTL Exception drafting
process, I don’t take the blame for delays. In fact, I fought for nearly a
year to prioritize the work when FSF’s outside law firm was focused on
other priorities and ignored my calls for urgency. I finally convinced
everyone, but the work got done far too late. (IMO, it should have been
timed for release in parallel with GPLv3
in June 2007.)

Finally, I want to reiterate that copyleft is a strategy, not a moral
principle. I respect the LLVM developers’ decision to use a
different strategy for software freedom, even if it isn’t my preferred
strategy. Indeed, I respect it so much that I supported
Conservancy’s offer
of membership to LLVM in Software Freedom Conservancy
. I still hope
the LLVM developers will take Conservancy up on this offer. I think that
regardless of a project’s preferred strategy for software freedom —
copyleft or non-copyleft — that it’s important for the developers
to have a not-for-profit charity as a gathering place for developers,
separate from their for-profit employer affiliations.

Undue for-profit corporate influence is the biggest problem that software
freedom faces today. Indeed, I don’t know a single developer in our
community who likes to see their work proprietarized. Developers,
generally speaking, want to share their code with other developers. It’s
lawyers and business people with dollar signs in their eyes who want to
make proprietary software. Those people sometimes convince developers to make
trade-offs (which I don’t agree with myself) to work on proprietary
software (— usually in exchange for funding some of their work time
on upstream Free Software). Meanwhile, those for-profit-corporate folks
frequently spread lies and half-truths about the copyleft side of the
community — in an effort to convince developers that their Free Software
projects “won’t survive” if those developers don’t follow the
exact plan The Company proposes. I’ve experienced these
manipulations myself — for example, in April 2013, a prominent corporate lawyer with
an interest in LLVM told me to my face that his company would continue
spreading false rumors that I’d use LLVM’s membership in
Conservancy to push the LLVM developers toward copyleft, despite
my public
statements to the contrary
. (Again, for the record, I have no such
intention and I’d be delighted to help LLVM be led in a non-profit home by its rightful
developer leaders, whichever Open Source and Free Software license they
chose.)

In short, the biggest threat to the future of software has always been
for-profit companies who wish to maximize profits by exploiting the code, developers and users while
limiting their software freedom. Such companies try every trick in pursuit of
that goal. As such, I prefer copyleft as a strategy. However, I don’t
necessarily admonish those who pick a different strategy. The reason that I
encourage membership of non-copylefted projects in Conservancy (and other
501(c)(3) charities) is to give those projects the benefits of a non-profit
home that maximize software freedom using the project’s chosen strategy,
whatever it may be.

The Biggest Myths

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/the-biggest-myths.html

Since we first proposed systemd
for inclusion in the distributions it has been frequently discussed in
many forums, mailing lists and conferences. In these discussions one
can often hear certain myths about systemd, that are repeated over and
over again, but certainly don’t gain any truth by constant
repetition. Let’s take the time to debunk a few of them:

Myth: systemd is monolithic.

If you build systemd with all configuration options enabled you
will build 69 individual binaries. These binaries all serve different
tasks, and are neatly separated for a number of reasons. For example,
we designed systemd with security in mind, hence most daemons run at
minimal privileges (using kernel capabilities, for example) and are
responsible for very specific tasks only, to minimize their security
surface and impact. Also, systemd parallelizes the boot more than any
prior solution. This parallization happens by running more processes
in parallel. Thus it is essential that systemd is nicely split up into
many binaries and thus processes. In fact, many of these
binaries[1] are separated out so nicely, that they are very
useful outside of systemd, too.

A package involving 69 individual binaries can hardly be called
monolithic. What is different from prior solutions however,
is that we ship more components in a single tarball, and maintain them
upstream in a single repository with a unified release cycle.

Myth: systemd is about speed.

Yes, systemd is fast (A
pretty complete userspace boot-up in ~900ms, anyone?
), but that’s
primarily just a side-effect of doing things right. In fact, we
never really sat down and optimized the last tiny bit of performance
out of systemd. Instead, we actually frequently knowingly picked the
slightly slower code paths in order to keep the code more
readable. This doesn’t mean being fast was irrelevant for us, but
reducing systemd to its speed is certainly quite a misconception,
since that is certainly not anywhere near the top of our list of
goals.

Myth: systemd’s fast boot-up is irrelevant for
servers.

That is just completely not true. Many administrators actually are
keen on reduced downtimes during maintenance windows. In High
Availability setups it’s kinda nice if the failed machine comes back
up really fast. In cloud setups with a large number of VMs or
containers the price of slow boots multiplies with the number of
instances. Spending minutes of CPU and IO on really slow boots of
hundreds of VMs or containers reduces your system’s density
drastically, heck, it even costs you more energy. Slow boots can be
quite financially expensive. Then, fast booting of containers allows
you to implement a logic such as socket
activated containers
, allowing you to drastically increase the
density of your cloud system.

Of course, in many server setups boot-up is indeed irrelevant, but
systemd is supposed to cover the whole range. And yes, I am aware
that often it is the server firmware that costs the most time at
boot-up, and the OS anyways fast compared to that, but well, systemd
is still supposed to cover the whole range (see above…), and no,
not all servers have such bad firmware, and certainly not VMs and
containers, which are servers of a kind, too.[2]

Myth: systemd is incompatible with shell scripts.

This is entirely bogus. We just don’t use them for the boot
process, because we believe they aren’t the best tool for that
specific purpose, but that doesn’t mean systemd was incompatible with
them. You can easily run shell scripts as systemd services, heck, you
can run scripts written in any language as systemd services,
systemd doesn’t care the slightest bit what’s inside your
executable. Moreover, we heavily use shell scripts for our own
purposes, for installing, building, testing systemd. And you can stick
your scripts in the early boot process, use them for normal services,
you can run them at latest shutdown, there are practically no
limits.

Myth: systemd is difficult.

This also is entire non-sense. A systemd platform is actually much
simpler than traditional Linuxes because it unifies
system objects and their dependencies as systemd units. The
configuration file language is very simple, and redundant
configuration files we got rid of. We provide uniform tools for much
of the configuration of the system. The system is much less
conglomerate than traditional Linuxes are. We also have pretty
comprehensive documentation (all linked
from the homepage
) about pretty much every detail of systemd, and
this not only covers admin/user-facing interfaces, but also developer
APIs.

systemd certainly comes with a learning curve. Everything
does. However, we like to believe that it is actually simpler to
understand systemd than a Shell-based boot for most people. Surprised
we say that? Well, as it turns out, Shell is not a pretty language to
learn, it’s syntax is arcane and complex. systemd unit files are
substantially easier to understand, they do not expose a programming
language, but are simple and declarative by nature. That all said, if
you are experienced in shell, then yes, adopting systemd will take a
bit of learning.

To make learning easy we tried hard to provide the maximum
compatibility to previous solutions. But not only that, on many
distributions you’ll find that some of the traditional tools will now
even tell you — while executing what you are asking for — how you
could do it with the newer tools instead, in a possibly nicer way.

Anyway, the take-away is probably that systemd is probably as
simple as such a system can be, and that we try hard to make it easy
to learn. But yes, if you know sysvinit then adopting systemd will
require a bit learning, but quite frankly if you mastered sysvinit,
then systemd should be easy for you.

Myth: systemd is not modular.

Not true at all. At compile time you have a number of
configure switches to select what you want to build, and what
not. And we
document
how you can select in even more detail what you need,
going beyond our configure switches.

This modularity is not totally unlike the one of the Linux kernel,
where you can select many features individually at compile time. If the
kernel is modular enough for you then systemd should be pretty close,
too.

Myth: systemd is only for desktops.

That is certainly not true. With systemd we try to cover pretty
much the same range as Linux itself does. While we care for desktop
uses, we also care pretty much the same way for server uses, and
embedded uses as well. You can bet that Red Hat wouldn’t make it a
core piece of RHEL7 if it wasn’t the best option for managing services
on servers.

People from numerous companies work on systemd. Car manufactureres
build it into cars, Red Hat uses it for a server operating system, and
GNOME uses many of its interfaces for improving the desktop. You find
it in toys, in space telescopes, and in wind turbines.

Most features I most recently worked on are probably relevant
primarily on servers, such as container
support
, resource
management
or the security
features
. We cover desktop systems pretty well already, and there
are number of companies doing systemd development for embedded, some
even offer consulting services in it.

Myth: systemd was created as result of the NIH syndrome.

This is not true. Before we began working on systemd we were
pushing for Canonical’s Upstart to be widely adopted (and Fedora/RHEL
used it too for a while). However, we eventually came to the
conclusion that its design was inherently flawed at its core (at least
in our eyes: most fundamentally, it leaves dependency management to
the admin/developer, instead of solving this hard problem in code),
and if something’s wrong in the core you better replace it, rather
than fix it. This was hardly the only reason though, other things that
came into play, such as the licensing/contribution agreement mess
around it. NIH wasn’t one of the reasons, though…[3]

Myth: systemd is a freedesktop.org project.

Well, systemd is certainly hosted at fdo, but freedesktop.org is
little else but a repository for code and documentation. Pretty much
any coder can request a repository there and dump his stuff there (as
long as it’s somewhat relevant for the infrastructure of free
systems). There’s no cabal involved, no “standardization” scheme, no
project vetting, nothing. It’s just a nice, free, reliable place to
have your repository. In that regard it’s a bit like SourceForge,
github, kernel.org, just not commercial and without over-the-top
requirements, and hence a good place to keep our stuff.

So yes, we host our stuff at fdo, but the implied assumption of
this myth in that there was a group of people who meet and then agree
on how the future free systems look like, is entirely bogus.

Myth: systemd is not UNIX.

There’s certainly some truth in that. systemd’s sources do not
contain a single line of code originating from original UNIX. However,
we derive inspiration from UNIX, and thus there’s a ton of UNIX in
systemd. For example, the UNIX idea of “everything is a file” finds
reflection in that in systemd all services are exposed at runtime in a
kernel file system, the cgroupfs. Then, one of the original
features of UNIX was multi-seat support, based on built-in terminal
support. Text terminals are hardly the state of the art how you
interface with your computer these days however. With systemd we
brought native multi-seat
support back, but this time with full support for today’s hardware,
covering graphics, mice, audio, webcams and more, and all that fully
automatic, hotplug-capable and without configuration. In fact the
design of systemd as a suite of integrated tools that each have their
individual purposes but when used together are more than just the sum
of the parts, that’s pretty much at the core of UNIX philosophy. Then,
the way our project is handled (i.e. maintaining much of the core OS
in a single git repository) is much closer to the BSD model (which is
a true UNIX, unlike Linux) of doing things (where most of the core OS
is kept in a single CVS/SVN repository) than things on Linux ever
were.

Ultimately, UNIX is something different for everybody. For us
systemd maintainers it is something we derive inspiration from. For
others it is a religion, and much like the other world religions there
are different readings and understandings of it. Some define UNIX
based on specific pieces of code heritage, others see it just as a set
of ideas, others as a set of commands or APIs, and even others as a
definition of behaviours. Of course, it is impossible to ever make all
these people happy.

Ultimately the question whether something is UNIX or not matters
very little. Being technically excellent is hardly exclusive to
UNIX. For us, UNIX is a major influence (heck, the biggest one), but
we also have other influences. Hence in some areas systemd will be
very UNIXy, and in others a little bit less.

Myth: systemd is complex.

There’s certainly some truth in that. Modern computers are complex
beasts, and the OS running on it will hence have to be complex
too. However, systemd is certainly not more complex than prior
implementations of the same components. Much rather, it’s simpler, and
has less redundancy (see above). Moreover, building a simple OS based
on systemd will involve much fewer packages than a traditional Linux
did. Fewer packages makes it easier to build your system, gets rid of
interdependencies and of much of the different behaviour of every
component involved.

Myth: systemd is bloated.

Well, bloated certainly has many different definitions. But in
most definitions systemd is probably the opposite of bloat. Since
systemd components share a common code base, they tend to share much
more code for common code paths. Here’s an example: in a traditional
Linux setup, sysvinit, start-stop-daemon, inetd, cron, dbus, all
implemented a scheme to execute processes with various configuration
options in a certain, hopefully clean environment. On systemd the code
paths for all of this, for the configuration parsing, as well as the
actual execution is shared. This means less code, less place for
mistakes, less memory and cache pressure, and is thus a very good
thing. And as a side-effect you actually get a ton more functionality
for it…

As mentioned above, systemd is also pretty modular. You can choose
at build time which components you need, and which you don’t
need. People can hence specifically choose the level of “bloat” they
want.

When you build systemd, it only requires three dependencies: glibc,
libcap and dbus. That’s it. It can make use of more dependencies, but
these are entirely optional.

So, yeah, whichever way you look at it, it’s really not
bloated.

Myth: systemd being Linux-only is not nice to the BSDs.

Completely wrong. The BSD folks are pretty much uninterested in
systemd. If systemd was portable, this would change nothing, they
still wouldn’t adopt it. And the same is true for the other Unixes in
the world. Solaris has SMF, BSD has their own “rc” system, and they
always maintained it separately from Linux. The init system is very
close to the core of the entire OS. And these other operating systems
hence define themselves among other things by their core
userspace. The assumption that they’d adopt our core userspace if we
just made it portable, is completely without any foundation.

Myth: systemd being Linux-only makes it impossible for Debian to adopt it as default.

Debian supports non-Linux kernels in their distribution. systemd
won’t run on those. Is that a problem though, and should that hinder
them to adopt system as default? Not really. The folks who ported
Debian to these other kernels were willing to invest time in a massive
porting effort, they set up test and build systems, and patched and
built numerous packages for their goal. The maintainance of both a
systemd unit file and a classic init script for the packaged services
is a negligable amount of work compared to that, especially since
those scripts more often than not exist already.

Myth: systemd could be ported to other kernels if its maintainers just wanted to.

That is simply not true. Porting systemd to other kernel is not
feasible. We just use too many Linux-specific interfaces. For a few
one might find replacements on other kernels, some features one might
want to turn off, but for most this is nor really possible. Here’s a
small, very incomprehensive list: cgroups, fanotify, umount2(),
/proc/self/mountinfo (including notification), /dev/swaps (same),
udev, netlink, the structure of /sys, /proc/$PID/comm,
/proc/$PID/cmdline, /proc/$PID/loginuid, /proc/$PID/stat,
/proc/$PID/session, /proc/$PID/exe, /proc/$PID/fd, tmpfs, devtmpfs,
capabilities, namespaces of all kinds, various prctl()s, numerous
ioctls, the mount() system call and its semantics, selinux, audit,
inotify, statfs, O_DIRECTORY, O_NOATIME, /proc/$PID/root, waitid(),
SCM_CREDENTIALS, SCM_RIGHTS, mkostemp(), /dev/input, …

And no, if you look at this list and pick out the few where you can
think of obvious counterparts on other kernels, then think again, and
look at the others you didn’t pick, and the complexity of replacing
them.

Myth: systemd is not portable for no reason.

Non-sense! We use the Linux-specific functionality because we need
it to implement what we want. Linux has so many features that
UNIX/POSIX didn’t have, and we want to empower the user with
them. These features are incredibly useful, but only if they are
actually exposed in a friendly way to the user, and that’s what we do
with systemd.

Myth: systemd uses binary configuration files.

No idea who came up with this crazy myth, but it’s absolutely not
true. systemd is configured pretty much exclusively via simple text
files. A few settings you can also alter with the kernel command line
and via environment variables. There’s nothing binary in its
configuration (not even XML). Just plain, simple, easy-to-read text
files.

Myth: systemd is a feature creep.

Well, systemd certainly covers more ground that it used to. It’s
not just an init system anymore, but the basic userspace building
block to build an OS from, but we carefully make sure to keep most of
the features optional. You can turn a lot off at compile time, and
even more at runtime. Thus you can choose freely how much feature
creeping you want.

Myth: systemd forces you to do something.

systemd is not the mafia. It’s Free Software, you can do with it
whatever you want, and that includes not using it. That’s pretty much
the opposite of “forcing”.

Myth: systemd makes it impossible to run syslog.

Not true, we carefully made sure when we introduced
the journal
that all data is also passed on to any syslog daemon
running. In fact, if something changed, then only that syslog gets
more complete data now than it got before, since we now cover early
boot stuff as well as STDOUT/STDERR of any system service.

Myth: systemd is incompatible.

We try very hard to provide the best possible compatibility with
sysvinit. In fact, the vast majority of init scripts should work just
fine on systemd, unmodified. However, there actually are indeed a few
incompatibilities, but we try to document
these
and explain what to do about them. Ultimately every system
that is not actually sysvinit itself will have a certain amount of
incompatibilities with it since it will not share the exect same code
paths.

It is our goal to ensure that differences between the various
distributions are kept at a minimum. That means unit files usually
work just fine on a different distribution than you wrote it on, which
is a big improvement over classic init scripts which are very hard to
write in a way that they run on multiple Linux distributions, due to
numerous incompatibilities between them.

Myth: systemd is not scriptable, because of its D-Bus use.

Not true. Pretty much every single D-Bus interface systemd provides
is also available in a command line tool, for example in systemctl,
loginctl,
timedatectl,
hostnamectl,
localectl
and suchlike. You can easily call these tools from shell scripts, they
open up pretty much the entire API from the command line with
easy-to-use commands.

That said, D-Bus actually has bindings for almost any scripting
language this world knows. Even from the shell you can invoke
arbitrary D-Bus methods with dbus-send
or gdbus. If
anything, this improves scriptability due to the good support of D-Bus
in the various scripting languages.

Myth: systemd requires you to use some arcane configuration
tools instead of allowing you to edit your configuration files
directly.

Not true at all. We offer some configuration tools, and using them
gets you a bit of additional functionality (for example, command line
completion for all settings!), but there’s no need at all to use
them. You can always edit the files in question directly if you wish,
and that’s fully supported. Of course sometimes you need to explicitly
reload configuration of some daemon after editing the configuration,
but that’s pretty much true for most UNIX services.

Myth: systemd is unstable and buggy.

Certainly not according to our data. We have been monitoring the
Fedora bug tracker (and some others) closely for a long long time. The
number of bugs is very low for such a central component of the OS,
especially if you discount the numerous RFE bugs we track for the
project. We are pretty good in keeping systemd out of the list of
blocker bugs of the distribution. We have a relatively fast
development cycle with mostly incremental changes to keep quality and
stability high.

Myth: systemd is not debuggable.

False. Some people try to imply that the shell was a good
debugger. Well, it isn’t really. In systemd we provide you with actual
debugging features instead. For example: interactive debugging,
verbose tracing, the ability to mask any component during boot, and
more. Also, we provide documentation
for it
.

It’s certainly well debuggable, we needed that for our own
development work, after all. But we’ll grant you one thing: it uses
different debugging tools, we believe more appropriate ones for the
purpose, though.

Myth: systemd makes changes for the changes’ sake.

Very much untrue. We pretty much exclusively have technical
reasons for the changes we make, and we explain them in the various
pieces of documentation, wiki pages, blog articles, mailing list
announcements. We try hard to avoid making incompatible changes, and
if we do we try to document the why and how in detail. And if you
wonder about something, just ask us!

Myth: systemd is a Red-Hat-only project, is private property
of some smart-ass developers, who use it to push their views to the
world.

Not true. Currently, there are 16 hackers with commit powers to the
systemd git tree. Of these 16 only six are employed by Red Hat. The 10
others are folks from ArchLinux, from Debian, from Intel, even from
Canonical, Mandriva, Pantheon and a number of community folks with
full commit rights. And they frequently commit big stuff, major
changes. Then, there are 374 individuals with patches in our tree, and
they too came from a number of different companies and backgrounds,
and many of those have way more than one patch in the tree. The
discussions about where we want to take systemd are done in the open,
on our IRC channel (#systemd on freenode, you are always
weclome), on our mailing
list
, and on public hackfests (such
as our next one in Brno
, you are invited). We regularly attend
various conferences, to collect feedback, to explain what we are doing
and why, like few others do. We maintain blogs, engage in social
networks (we actually
have some pretty interesting content on Google+
, and our Google+
Community is pretty alive, too
.), and try really hard to explain
the why and the how how we do things, and to listen to feedback and
figure out where the current issues are (for example, from that
feedback we compiled this lists of often heard myths about
systemd…).

What most systemd contributors probably share is a rough idea how a
good OS should look like, and the desire to make it happen. However,
by the very nature of the project being Open Source, and rooted in the
community systemd is just what people want it to be, and if it’s not
what they want then they can drive the direction with patches and
code, and if that’s not feasible, then there are numerous other
options to use, too, systemd is never exclusive.

One goal of systemd is to unify the dispersed Linux landscape a
bit. We try to get rid of many of the more pointless differences of
the various distributions in various areas of the core OS. As part of
that we sometimes adopt schemes that were previously used by only one
of the distributions and push it to a level where it’s the default of
systemd, trying to gently push everybody towards the same set of basic
configuration. This is never exclusive though, distributions can
continue to deviate from that if they wish, however, if they end-up
using the well-supported default their work becomes much easier and
they might gain a feature or two. Now, as it turns out, more
frequently than not we actually adopted schemes that where Debianisms,
rather than Fedoraisms/Redhatisms as best supported scheme by
systemd. For example, systems running systemd now generally store
their hostname in /etc/hostname, something that used to be
specific to Debian and now is used across distributions.

One thing we’ll grant you though, we sometimes can be
smart-asses. We try to be prepared whenever we open our mouth, in
order to be able to back-up with facts what we claim. That might make
us appear as smart-asses.

But in general, yes, some of the more influental contributors of
systemd work for Red Hat, but they are in the minority, and systemd is
a healthy, open community with different interests, different
backgrounds, just unified by a few rough ideas where the trip should
go, a community where code and its design counts, and certainly not
company affiliation.

Myth: systemd doesn’t support /usr split from the root directory.

Non-sense. Since its beginnings systemd supports the
–with-rootprefix= option to its configure script
which allows you to tell systemd to neatly split up the stuff needed
for early boot and the stuff needed for later on. All this logic is
fully present and we keep it up-to-date right there in systemd’s build
system.

Of course, we still don’t think that actually
booting with /usr unavailable is a good idea
, but we
support this just fine in our build system. This won’t fix the
inherent problems of the scheme that you’ll encounter all across the
board, but you can’t blame that on systemd, because in systemd we
support this just fine.

Myth: systemd doesn’t allow your to replace its components.

Not true, you can turn off and replace pretty much any part of
systemd, with very few exceptions. And those exceptions (such as
journald) generally allow you to run an alternative side by side to
it, while cooperating nicely with it.

Myth: systemd’s use of D-Bus instead of sockets makes it intransparent.

This claim is already contradictory in itself: D-Bus uses sockets
as transport, too. Hence whenever D-Bus is used to send something
around, a socket is used for that too. D-Bus is mostly a standardized
serialization of messages to send over these sockets. If anything this
makes it more transparent, since this serialization is well
documented, understood and there are numerous tracing tools and
language bindings for it. This is very much unlike the usual
homegrown protocols the various classic UNIX daemons use to
communicate locally.

Hmm, did I write I just wanted to debunk a “few” myths? Maybe these
were more than just a few… Anyway, I hope I managed to clear up a
couple of misconceptions. Thanks for your time.

Footnotes

[1] For example, systemd-detect-virt,
systemd-tmpfiles,
systemd-udevd are.

[2] Also, we are trying to do our little part on maybe
making this better. By exposing boot-time performance of the firmware
more prominently in systemd’s boot output we hope to shame the
firmware writers to clean up their stuff.

[3] And anyways, guess which project includes a library “libnih” — Upstart or systemd?[4]

[4] Hint: it’s not systemd!

The Biggest Myths

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/the-biggest-myths.html

Since we first proposed systemd
for inclusion in the distributions it has been frequently discussed in
many forums, mailing lists and conferences. In these discussions one
can often hear certain myths about systemd, that are repeated over and
over again, but certainly don’t gain any truth by constant
repetition. Let’s take the time to debunk a few of them:

  1. Myth: systemd is monolithic.

    If you build systemd with all configuration options enabled you
    will build 69 individual binaries. These binaries all serve different
    tasks, and are neatly separated for a number of reasons. For example,
    we designed systemd with security in mind, hence most daemons run at
    minimal privileges (using kernel capabilities, for example) and are
    responsible for very specific tasks only, to minimize their security
    surface and impact. Also, systemd parallelizes the boot more than any
    prior solution. This parallization happens by running more processes
    in parallel. Thus it is essential that systemd is nicely split up into
    many binaries and thus processes. In fact, many of these
    binaries[1] are separated out so nicely, that they are very
    useful outside of systemd, too.

    A package involving 69 individual binaries can hardly be called
    monolithic. What is different from prior solutions however,
    is that we ship more components in a single tarball, and maintain them
    upstream in a single repository with a unified release cycle.

  2. Myth: systemd is about speed.

    Yes, systemd is fast (A
    pretty complete userspace boot-up in ~900ms, anyone?
    ), but that’s
    primarily just a side-effect of doing things right. In fact, we
    never really sat down and optimized the last tiny bit of performance
    out of systemd. Instead, we actually frequently knowingly picked the
    slightly slower code paths in order to keep the code more
    readable. This doesn’t mean being fast was irrelevant for us, but
    reducing systemd to its speed is certainly quite a misconception,
    since that is certainly not anywhere near the top of our list of
    goals.

  3. Myth: systemd’s fast boot-up is irrelevant for
    servers.

    That is just completely not true. Many administrators actually are
    keen on reduced downtimes during maintenance windows. In High
    Availability setups it’s kinda nice if the failed machine comes back
    up really fast. In cloud setups with a large number of VMs or
    containers the price of slow boots multiplies with the number of
    instances. Spending minutes of CPU and IO on really slow boots of
    hundreds of VMs or containers reduces your system’s density
    drastically, heck, it even costs you more energy. Slow boots can be
    quite financially expensive. Then, fast booting of containers allows
    you to implement a logic such as socket
    activated containers
    , allowing you to drastically increase the
    density of your cloud system.

    Of course, in many server setups boot-up is indeed irrelevant, but
    systemd is supposed to cover the whole range. And yes, I am aware
    that often it is the server firmware that costs the most time at
    boot-up, and the OS anyways fast compared to that, but well, systemd
    is still supposed to cover the whole range (see above…), and no,
    not all servers have such bad firmware, and certainly not VMs and
    containers, which are servers of a kind, too.[2]

  4. Myth: systemd is incompatible with shell scripts.

    This is entirely bogus. We just don’t use them for the boot
    process, because we believe they aren’t the best tool for that
    specific purpose, but that doesn’t mean systemd was incompatible with
    them. You can easily run shell scripts as systemd services, heck, you
    can run scripts written in any language as systemd services,
    systemd doesn’t care the slightest bit what’s inside your
    executable. Moreover, we heavily use shell scripts for our own
    purposes, for installing, building, testing systemd. And you can stick
    your scripts in the early boot process, use them for normal services,
    you can run them at latest shutdown, there are practically no
    limits.

  5. Myth: systemd is difficult.

    This also is entire non-sense. A systemd platform is actually much
    simpler than traditional Linuxes because it unifies
    system objects and their dependencies as systemd units. The
    configuration file language is very simple, and redundant
    configuration files we got rid of. We provide uniform tools for much
    of the configuration of the system. The system is much less
    conglomerate than traditional Linuxes are. We also have pretty
    comprehensive documentation (all linked
    from the homepage
    ) about pretty much every detail of systemd, and
    this not only covers admin/user-facing interfaces, but also developer
    APIs.

    systemd certainly comes with a learning curve. Everything
    does. However, we like to believe that it is actually simpler to
    understand systemd than a Shell-based boot for most people. Surprised
    we say that? Well, as it turns out, Shell is not a pretty language to
    learn, it’s syntax is arcane and complex. systemd unit files are
    substantially easier to understand, they do not expose a programming
    language, but are simple and declarative by nature. That all said, if
    you are experienced in shell, then yes, adopting systemd will take a
    bit of learning.

    To make learning easy we tried hard to provide the maximum
    compatibility to previous solutions. But not only that, on many
    distributions you’ll find that some of the traditional tools will now
    even tell you — while executing what you are asking for — how you
    could do it with the newer tools instead, in a possibly nicer way.

    Anyway, the take-away is probably that systemd is probably as
    simple as such a system can be, and that we try hard to make it easy
    to learn. But yes, if you know sysvinit then adopting systemd will
    require a bit learning, but quite frankly if you mastered sysvinit,
    then systemd should be easy for you.

  6. Myth: systemd is not modular.

    Not true at all. At compile time you have a number of
    configure switches to select what you want to build, and what
    not. And we
    document
    how you can select in even more detail what you need,
    going beyond our configure switches.

    This modularity is not totally unlike the one of the Linux kernel,
    where you can select many features individually at compile time. If the
    kernel is modular enough for you then systemd should be pretty close,
    too.

  7. Myth: systemd is only for desktops.

    That is certainly not true. With systemd we try to cover pretty
    much the same range as Linux itself does. While we care for desktop
    uses, we also care pretty much the same way for server uses, and
    embedded uses as well. You can bet that Red Hat wouldn’t make it a
    core piece of RHEL7 if it wasn’t the best option for managing services
    on servers.

    People from numerous companies work on systemd. Car manufactureres
    build it into cars, Red Hat uses it for a server operating system, and
    GNOME uses many of its interfaces for improving the desktop. You find
    it in toys, in space telescopes, and in wind turbines.

    Most features I most recently worked on are probably relevant
    primarily on servers, such as container
    support
    , resource
    management
    or the security
    features
    . We cover desktop systems pretty well already, and there
    are number of companies doing systemd development for embedded, some
    even offer consulting services in it.

  8. Myth: systemd was created as result of the NIH syndrome.

    This is not true. Before we began working on systemd we were
    pushing for Canonical’s Upstart to be widely adopted (and Fedora/RHEL
    used it too for a while). However, we eventually came to the
    conclusion that its design was inherently flawed at its core (at least
    in our eyes: most fundamentally, it leaves dependency management to
    the admin/developer, instead of solving this hard problem in code),
    and if something’s wrong in the core you better replace it, rather
    than fix it. This was hardly the only reason though, other things that
    came into play, such as the licensing/contribution agreement mess
    around it. NIH wasn’t one of the reasons, though…[3]

  9. Myth: systemd is a freedesktop.org project.

    Well, systemd is certainly hosted at fdo, but freedesktop.org is
    little else but a repository for code and documentation. Pretty much
    any coder can request a repository there and dump his stuff there (as
    long as it’s somewhat relevant for the infrastructure of free
    systems). There’s no cabal involved, no “standardization” scheme, no
    project vetting, nothing. It’s just a nice, free, reliable place to
    have your repository. In that regard it’s a bit like SourceForge,
    github, kernel.org, just not commercial and without over-the-top
    requirements, and hence a good place to keep our stuff.

    So yes, we host our stuff at fdo, but the implied assumption of
    this myth in that there was a group of people who meet and then agree
    on how the future free systems look like, is entirely bogus.

  10. Myth: systemd is not UNIX.

    There’s certainly some truth in that. systemd’s sources do not
    contain a single line of code originating from original UNIX. However,
    we derive inspiration from UNIX, and thus there’s a ton of UNIX in
    systemd. For example, the UNIX idea of “everything is a file” finds
    reflection in that in systemd all services are exposed at runtime in a
    kernel file system, the cgroupfs. Then, one of the original
    features of UNIX was multi-seat support, based on built-in terminal
    support. Text terminals are hardly the state of the art how you
    interface with your computer these days however. With systemd we
    brought native multi-seat
    support back, but this time with full support for today’s hardware,
    covering graphics, mice, audio, webcams and more, and all that fully
    automatic, hotplug-capable and without configuration. In fact the
    design of systemd as a suite of integrated tools that each have their
    individual purposes but when used together are more than just the sum
    of the parts, that’s pretty much at the core of UNIX philosophy. Then,
    the way our project is handled (i.e. maintaining much of the core OS
    in a single git repository) is much closer to the BSD model (which is
    a true UNIX, unlike Linux) of doing things (where most of the core OS
    is kept in a single CVS/SVN repository) than things on Linux ever
    were.

    Ultimately, UNIX is something different for everybody. For us
    systemd maintainers it is something we derive inspiration from. For
    others it is a religion, and much like the other world religions there
    are different readings and understandings of it. Some define UNIX
    based on specific pieces of code heritage, others see it just as a set
    of ideas, others as a set of commands or APIs, and even others as a
    definition of behaviours. Of course, it is impossible to ever make all
    these people happy.

    Ultimately the question whether something is UNIX or not matters
    very little. Being technically excellent is hardly exclusive to
    UNIX. For us, UNIX is a major influence (heck, the biggest one), but
    we also have other influences. Hence in some areas systemd will be
    very UNIXy, and in others a little bit less.

  11. Myth: systemd is complex.

    There’s certainly some truth in that. Modern computers are complex
    beasts, and the OS running on it will hence have to be complex
    too. However, systemd is certainly not more complex than prior
    implementations of the same components. Much rather, it’s simpler, and
    has less redundancy (see above). Moreover, building a simple OS based
    on systemd will involve much fewer packages than a traditional Linux
    did. Fewer packages makes it easier to build your system, gets rid of
    interdependencies and of much of the different behaviour of every
    component involved.

  12. Myth: systemd is bloated.

    Well, bloated certainly has many different definitions. But in
    most definitions systemd is probably the opposite of bloat. Since
    systemd components share a common code base, they tend to share much
    more code for common code paths. Here’s an example: in a traditional
    Linux setup, sysvinit, start-stop-daemon, inetd, cron, dbus, all
    implemented a scheme to execute processes with various configuration
    options in a certain, hopefully clean environment. On systemd the code
    paths for all of this, for the configuration parsing, as well as the
    actual execution is shared. This means less code, less place for
    mistakes, less memory and cache pressure, and is thus a very good
    thing. And as a side-effect you actually get a ton more functionality
    for it…

    As mentioned above, systemd is also pretty modular. You can choose
    at build time which components you need, and which you don’t
    need. People can hence specifically choose the level of “bloat” they
    want.

    When you build systemd, it only requires three dependencies: glibc,
    libcap and dbus. That’s it. It can make use of more dependencies, but
    these are entirely optional.

    So, yeah, whichever way you look at it, it’s really not
    bloated.

  13. Myth: systemd being Linux-only is not nice to the BSDs.

    Completely wrong. The BSD folks are pretty much uninterested in
    systemd. If systemd was portable, this would change nothing, they
    still wouldn’t adopt it. And the same is true for the other Unixes in
    the world. Solaris has SMF, BSD has their own “rc” system, and they
    always maintained it separately from Linux. The init system is very
    close to the core of the entire OS. And these other operating systems
    hence define themselves among other things by their core
    userspace. The assumption that they’d adopt our core userspace if we
    just made it portable, is completely without any foundation.

  14. Myth: systemd being Linux-only makes it impossible for Debian to adopt it as default.

    Debian supports non-Linux kernels in their distribution. systemd
    won’t run on those. Is that a problem though, and should that hinder
    them to adopt system as default? Not really. The folks who ported
    Debian to these other kernels were willing to invest time in a massive
    porting effort, they set up test and build systems, and patched and
    built numerous packages for their goal. The maintainance of both a
    systemd unit file and a classic init script for the packaged services
    is a negligable amount of work compared to that, especially since
    those scripts more often than not exist already.

  15. Myth: systemd could be ported to other kernels if its maintainers just wanted to.

    That is simply not true. Porting systemd to other kernel is not
    feasible. We just use too many Linux-specific interfaces. For a few
    one might find replacements on other kernels, some features one might
    want to turn off, but for most this is nor really possible. Here’s a
    small, very incomprehensive list: cgroups, fanotify, umount2(),
    /proc/self/mountinfo
    (including notification), /dev/swaps (same),
    udev, netlink,
    the structure of /sys, /proc/$PID/comm,
    /proc/$PID/cmdline, /proc/$PID/loginuid, /proc/$PID/stat,
    /proc/$PID/session, /proc/$PID/exe, /proc/$PID/fd, tmpfs, devtmpfs,
    capabilities, namespaces of all kinds, various prctl()s, numerous
    ioctls,
    the mount() system call and its semantics, selinux, audit,
    inotify, statfs, O_DIRECTORY, O_NOATIME, /proc/$PID/root, waitid(),
    SCM_CREDENTIALS, SCM_RIGHTS, mkostemp(), /dev/input, ...

    And no, if you look at this list and pick out the few where you can
    think of obvious counterparts on other kernels, then think again, and
    look at the others you didn’t pick, and the complexity of replacing
    them.

  16. Myth: systemd is not portable for no reason.

    Non-sense! We use the Linux-specific functionality because we need
    it to implement what we want. Linux has so many features that
    UNIX/POSIX didn’t have, and we want to empower the user with
    them. These features are incredibly useful, but only if they are
    actually exposed in a friendly way to the user, and that’s what we do
    with systemd.

  17. Myth: systemd uses binary configuration files.

    No idea who came up with this crazy myth, but it’s absolutely not
    true. systemd is configured pretty much exclusively via simple text
    files. A few settings you can also alter with the kernel command line
    and via environment variables. There’s nothing binary in its
    configuration (not even XML). Just plain, simple, easy-to-read text
    files.

  18. Myth: systemd is a feature creep.

    Well, systemd certainly covers more ground that it used to. It’s
    not just an init system anymore, but the basic userspace building
    block to build an OS from, but we carefully make sure to keep most of
    the features optional. You can turn a lot off at compile time, and
    even more at runtime. Thus you can choose freely how much feature
    creeping you want.

  19. Myth: systemd forces you to do something.

    systemd is not the mafia. It’s Free Software, you can do with it
    whatever you want, and that includes not using it. That’s pretty much
    the opposite of “forcing”.

  20. Myth: systemd makes it impossible to run syslog.

    Not true, we carefully made sure when we introduced
    the journal
    that all data is also passed on to any syslog daemon
    running. In fact, if something changed, then only that syslog gets
    more complete data now than it got before, since we now cover early
    boot stuff as well as STDOUT/STDERR of any system service.

  21. Myth: systemd is incompatible.

    We try very hard to provide the best possible compatibility with
    sysvinit. In fact, the vast majority of init scripts should work just
    fine on systemd, unmodified. However, there actually are indeed a few
    incompatibilities, but we try to document
    these
    and explain what to do about them. Ultimately every system
    that is not actually sysvinit itself will have a certain amount of
    incompatibilities with it since it will not share the exect same code
    paths.

    It is our goal to ensure that differences between the various
    distributions are kept at a minimum. That means unit files usually
    work just fine on a different distribution than you wrote it on, which
    is a big improvement over classic init scripts which are very hard to
    write in a way that they run on multiple Linux distributions, due to
    numerous incompatibilities between them.

  22. Myth: systemd is not scriptable, because of its D-Bus use.

    Not true. Pretty much every single D-Bus interface systemd provides
    is also available in a command line tool, for example in systemctl,
    loginctl,
    timedatectl,
    hostnamectl,
    localectl
    and suchlike. You can easily call these tools from shell scripts, they
    open up pretty much the entire API from the command line with
    easy-to-use commands.

    That said, D-Bus actually has bindings for almost any scripting
    language this world knows. Even from the shell you can invoke
    arbitrary D-Bus methods with dbus-send
    or gdbus. If
    anything, this improves scriptability due to the good support of D-Bus
    in the various scripting languages.

  23. Myth: systemd requires you to use some arcane configuration
    tools instead of allowing you to edit your configuration files
    directly.

    Not true at all. We offer some configuration tools, and using them
    gets you a bit of additional functionality (for example, command line
    completion for all settings!), but there’s no need at all to use
    them. You can always edit the files in question directly if you wish,
    and that’s fully supported. Of course sometimes you need to explicitly
    reload configuration of some daemon after editing the configuration,
    but that’s pretty much true for most UNIX services.

  24. Myth: systemd is unstable and buggy.

    Certainly not according to our data. We have been monitoring the
    Fedora bug tracker (and some others) closely for a long long time. The
    number of bugs is very low for such a central component of the OS,
    especially if you discount the numerous RFE bugs we track for the
    project. We are pretty good in keeping systemd out of the list of
    blocker bugs of the distribution. We have a relatively fast
    development cycle with mostly incremental changes to keep quality and
    stability high.

  25. Myth: systemd is not debuggable.

    False. Some people try to imply that the shell was a good
    debugger. Well, it isn’t really. In systemd we provide you with actual
    debugging features instead. For example: interactive debugging,
    verbose tracing, the ability to mask any component during boot, and
    more. Also, we provide documentation
    for it
    .

    It’s certainly well debuggable, we needed that for our own
    development work, after all. But we’ll grant you one thing: it uses
    different debugging tools, we believe more appropriate ones for the
    purpose, though.

  26. Myth: systemd makes changes for the changes’ sake.

    Very much untrue. We pretty much exclusively have technical
    reasons for the changes we make, and we explain them in the various
    pieces of documentation, wiki pages, blog articles, mailing list
    announcements. We try hard to avoid making incompatible changes, and
    if we do we try to document the why and how in detail. And if you
    wonder about something, just ask us!

  27. Myth: systemd is a Red-Hat-only project, is private property
    of some smart-ass developers, who use it to push their views to the
    world.

    Not true. Currently, there are 16 hackers with commit powers to the
    systemd git tree. Of these 16 only six are employed by Red Hat. The 10
    others are folks from ArchLinux, from Debian, from Intel, even from
    Canonical, Mandriva, Pantheon and a number of community folks with
    full commit rights. And they frequently commit big stuff, major
    changes. Then, there are 374 individuals with patches in our tree, and
    they too came from a number of different companies and backgrounds,
    and many of those have way more than one patch in the tree. The
    discussions about where we want to take systemd are done in the open,
    on our IRC channel (#systemd on freenode, you are always
    weclome), on our mailing
    list
    , and on public hackfests (such
    as our next one in Brno
    , you are invited). We regularly attend
    various conferences, to collect feedback, to explain what we are doing
    and why, like few others do. We maintain blogs, engage in social
    networks (we actually
    have some pretty interesting content on Google+
    , and our Google+
    Community is pretty alive, too
    .), and try really hard to explain
    the why and the how how we do things, and to listen to feedback and
    figure out where the current issues are (for example, from that
    feedback we compiled this lists of often heard myths about
    systemd…).

    What most systemd contributors probably share is a rough idea how a
    good OS should look like, and the desire to make it happen. However,
    by the very nature of the project being Open Source, and rooted in the
    community systemd is just what people want it to be, and if it’s not
    what they want then they can drive the direction with patches and
    code, and if that’s not feasible, then there are numerous other
    options to use, too, systemd is never exclusive.

    One goal of systemd is to unify the dispersed Linux landscape a
    bit. We try to get rid of many of the more pointless differences of
    the various distributions in various areas of the core OS. As part of
    that we sometimes adopt schemes that were previously used by only one
    of the distributions and push it to a level where it’s the default of
    systemd, trying to gently push everybody towards the same set of basic
    configuration. This is never exclusive though, distributions can
    continue to deviate from that if they wish, however, if they end-up
    using the well-supported default their work becomes much easier and
    they might gain a feature or two. Now, as it turns out, more
    frequently than not we actually adopted schemes that where Debianisms,
    rather than Fedoraisms/Redhatisms as best supported scheme by
    systemd. For example, systems running systemd now generally store
    their hostname in /etc/hostname, something that used to be
    specific to Debian and now is used across distributions.

    One thing we’ll grant you though, we sometimes can be
    smart-asses. We try to be prepared whenever we open our mouth, in
    order to be able to back-up with facts what we claim. That might make
    us appear as smart-asses.

    But in general, yes, some of the more influental contributors of
    systemd work for Red Hat, but they are in the minority, and systemd is
    a healthy, open community with different interests, different
    backgrounds, just unified by a few rough ideas where the trip should
    go, a community where code and its design counts, and certainly not
    company affiliation.

  28. Myth: systemd doesn’t support /usr split from the root directory.

    Non-sense. Since its beginnings systemd supports the
    --with-rootprefix= option to its configure script
    which allows you to tell systemd to neatly split up the stuff needed
    for early boot and the stuff needed for later on. All this logic is
    fully present and we keep it up-to-date right there in systemd’s build
    system.

    Of course, we still don’t think that actually
    booting with /usr unavailable is a good idea
    , but we
    support this just fine in our build system. This won’t fix the
    inherent problems of the scheme that you’ll encounter all across the
    board, but you can’t blame that on systemd, because in systemd we
    support this just fine.

  29. Myth: systemd doesn’t allow your to replace its components.

    Not true, you can turn off and replace pretty much any part of
    systemd, with very few exceptions. And those exceptions (such as
    journald) generally allow you to run an alternative side by side to
    it, while cooperating nicely with it.

  30. Myth: systemd’s use of D-Bus instead of sockets makes it intransparent.

    This claim is already contradictory in itself: D-Bus uses sockets
    as transport, too. Hence whenever D-Bus is used to send something
    around, a socket is used for that too. D-Bus is mostly a standardized
    serialization of messages to send over these sockets. If anything this
    makes it more transparent, since this serialization is well
    documented, understood and there are numerous tracing tools and
    language bindings for it. This is very much unlike the usual
    homegrown protocols the various classic UNIX daemons use to
    communicate locally.

Hmm, did I write I just wanted to debunk a “few” myths? Maybe these
were more than just a few… Anyway, I hope I managed to clear up a
couple of misconceptions. Thanks for your time.

Footnotes

[1] For example, systemd-detect-virt,
systemd-tmpfiles,
systemd-udevd are.

[2] Also, we are trying to do our little part on maybe
making this better. By exposing boot-time performance of the firmware
more prominently in systemd’s boot output we hope to shame the
firmware writers to clean up their stuff.

[3] And anyways, guess which project includes a library “libnih” — Upstart or systemd?[4]

[4] Hint: it’s not systemd!

The Biggest Myths

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/the-biggest-myths.html

Since we first proposed systemd
for inclusion in the distributions it has been frequently discussed in
many forums, mailing lists and conferences. In these discussions one
can often hear certain myths about systemd, that are repeated over and
over again, but certainly don’t gain any truth by constant
repetition. Let’s take the time to debunk a few of them:

  1. Myth: systemd is monolithic.

    If you build systemd with all configuration options enabled you
    will build 69 individual binaries. These binaries all serve different
    tasks, and are neatly separated for a number of reasons. For example,
    we designed systemd with security in mind, hence most daemons run at
    minimal privileges (using kernel capabilities, for example) and are
    responsible for very specific tasks only, to minimize their security
    surface and impact. Also, systemd parallelizes the boot more than any
    prior solution. This parallization happens by running more processes
    in parallel. Thus it is essential that systemd is nicely split up into
    many binaries and thus processes. In fact, many of these
    binaries[1] are separated out so nicely, that they are very
    useful outside of systemd, too.

    A package involving 69 individual binaries can hardly be called
    monolithic. What is different from prior solutions however,
    is that we ship more components in a single tarball, and maintain them
    upstream in a single repository with a unified release cycle.

  2. Myth: systemd is about speed.

    Yes, systemd is fast (A
    pretty complete userspace boot-up in ~900ms, anyone?
    ), but that’s
    primarily just a side-effect of doing things right. In fact, we
    never really sat down and optimized the last tiny bit of performance
    out of systemd. Instead, we actually frequently knowingly picked the
    slightly slower code paths in order to keep the code more
    readable. This doesn’t mean being fast was irrelevant for us, but
    reducing systemd to its speed is certainly quite a misconception,
    since that is certainly not anywhere near the top of our list of
    goals.

  3. Myth: systemd’s fast boot-up is irrelevant for
    servers.

    That is just completely not true. Many administrators actually are
    keen on reduced downtimes during maintenance windows. In High
    Availability setups it’s kinda nice if the failed machine comes back
    up really fast. In cloud setups with a large number of VMs or
    containers the price of slow boots multiplies with the number of
    instances. Spending minutes of CPU and IO on really slow boots of
    hundreds of VMs or containers reduces your system’s density
    drastically, heck, it even costs you more energy. Slow boots can be
    quite financially expensive. Then, fast booting of containers allows
    you to implement a logic such as socket
    activated containers
    , allowing you to drastically increase the
    density of your cloud system.

    Of course, in many server setups boot-up is indeed irrelevant, but
    systemd is supposed to cover the whole range. And yes, I am aware
    that often it is the server firmware that costs the most time at
    boot-up, and the OS anyways fast compared to that, but well, systemd
    is still supposed to cover the whole range (see above…), and no,
    not all servers have such bad firmware, and certainly not VMs and
    containers, which are servers of a kind, too.[2]

  4. Myth: systemd is incompatible with shell scripts.

    This is entirely bogus. We just don’t use them for the boot
    process, because we believe they aren’t the best tool for that
    specific purpose, but that doesn’t mean systemd was incompatible with
    them. You can easily run shell scripts as systemd services, heck, you
    can run scripts written in any language as systemd services,
    systemd doesn’t care the slightest bit what’s inside your
    executable. Moreover, we heavily use shell scripts for our own
    purposes, for installing, building, testing systemd. And you can stick
    your scripts in the early boot process, use them for normal services,
    you can run them at latest shutdown, there are practically no
    limits.

  5. Myth: systemd is difficult.

    This also is entire non-sense. A systemd platform is actually much
    simpler than traditional Linuxes because it unifies
    system objects and their dependencies as systemd units. The
    configuration file language is very simple, and redundant
    configuration files we got rid of. We provide uniform tools for much
    of the configuration of the system. The system is much less
    conglomerate than traditional Linuxes are. We also have pretty
    comprehensive documentation (all linked
    from the homepage
    ) about pretty much every detail of systemd, and
    this not only covers admin/user-facing interfaces, but also developer
    APIs.

    systemd certainly comes with a learning curve. Everything
    does. However, we like to believe that it is actually simpler to
    understand systemd than a Shell-based boot for most people. Surprised
    we say that? Well, as it turns out, Shell is not a pretty language to
    learn, it’s syntax is arcane and complex. systemd unit files are
    substantially easier to understand, they do not expose a programming
    language, but are simple and declarative by nature. That all said, if
    you are experienced in shell, then yes, adopting systemd will take a
    bit of learning.

    To make learning easy we tried hard to provide the maximum
    compatibility to previous solutions. But not only that, on many
    distributions you’ll find that some of the traditional tools will now
    even tell you — while executing what you are asking for — how you
    could do it with the newer tools instead, in a possibly nicer way.

    Anyway, the take-away is probably that systemd is probably as
    simple as such a system can be, and that we try hard to make it easy
    to learn. But yes, if you know sysvinit then adopting systemd will
    require a bit learning, but quite frankly if you mastered sysvinit,
    then systemd should be easy for you.

  6. Myth: systemd is not modular.

    Not true at all. At compile time you have a number of
    configure switches to select what you want to build, and what
    not. And we
    document
    how you can select in even more detail what you need,
    going beyond our configure switches.

    This modularity is not totally unlike the one of the Linux kernel,
    where you can select many features individually at compile time. If the
    kernel is modular enough for you then systemd should be pretty close,
    too.

  7. Myth: systemd is only for desktops.

    That is certainly not true. With systemd we try to cover pretty
    much the same range as Linux itself does. While we care for desktop
    uses, we also care pretty much the same way for server uses, and
    embedded uses as well. You can bet that Red Hat wouldn’t make it a
    core piece of RHEL7 if it wasn’t the best option for managing services
    on servers.

    People from numerous companies work on systemd. Car manufactureres
    build it into cars, Red Hat uses it for a server operating system, and
    GNOME uses many of its interfaces for improving the desktop. You find
    it in toys, in space telescopes, and in wind turbines.

    Most features I most recently worked on are probably relevant
    primarily on servers, such as container
    support
    , resource
    management
    or the security
    features
    . We cover desktop systems pretty well already, and there
    are number of companies doing systemd development for embedded, some
    even offer consulting services in it.

  8. Myth: systemd was created as result of the NIH syndrome.

    This is not true. Before we began working on systemd we were
    pushing for Canonical’s Upstart to be widely adopted (and Fedora/RHEL
    used it too for a while). However, we eventually came to the
    conclusion that its design was inherently flawed at its core (at least
    in our eyes: most fundamentally, it leaves dependency management to
    the admin/developer, instead of solving this hard problem in code),
    and if something’s wrong in the core you better replace it, rather
    than fix it. This was hardly the only reason though, other things that
    came into play, such as the licensing/contribution agreement mess
    around it. NIH wasn’t one of the reasons, though…[3]

  9. Myth: systemd is a freedesktop.org project.

    Well, systemd is certainly hosted at fdo, but freedesktop.org is
    little else but a repository for code and documentation. Pretty much
    any coder can request a repository there and dump his stuff there (as
    long as it’s somewhat relevant for the infrastructure of free
    systems). There’s no cabal involved, no “standardization” scheme, no
    project vetting, nothing. It’s just a nice, free, reliable place to
    have your repository. In that regard it’s a bit like SourceForge,
    github, kernel.org, just not commercial and without over-the-top
    requirements, and hence a good place to keep our stuff.

    So yes, we host our stuff at fdo, but the implied assumption of
    this myth in that there was a group of people who meet and then agree
    on how the future free systems look like, is entirely bogus.

  10. Myth: systemd is not UNIX.

    There’s certainly some truth in that. systemd’s sources do not
    contain a single line of code originating from original UNIX. However,
    we derive inspiration from UNIX, and thus there’s a ton of UNIX in
    systemd. For example, the UNIX idea of “everything is a file” finds
    reflection in that in systemd all services are exposed at runtime in a
    kernel file system, the cgroupfs. Then, one of the original
    features of UNIX was multi-seat support, based on built-in terminal
    support. Text terminals are hardly the state of the art how you
    interface with your computer these days however. With systemd we
    brought native multi-seat
    support back, but this time with full support for today’s hardware,
    covering graphics, mice, audio, webcams and more, and all that fully
    automatic, hotplug-capable and without configuration. In fact the
    design of systemd as a suite of integrated tools that each have their
    individual purposes but when used together are more than just the sum
    of the parts, that’s pretty much at the core of UNIX philosophy. Then,
    the way our project is handled (i.e. maintaining much of the core OS
    in a single git repository) is much closer to the BSD model (which is
    a true UNIX, unlike Linux) of doing things (where most of the core OS
    is kept in a single CVS/SVN repository) than things on Linux ever
    were.

    Ultimately, UNIX is something different for everybody. For us
    systemd maintainers it is something we derive inspiration from. For
    others it is a religion, and much like the other world religions there
    are different readings and understandings of it. Some define UNIX
    based on specific pieces of code heritage, others see it just as a set
    of ideas, others as a set of commands or APIs, and even others as a
    definition of behaviours. Of course, it is impossible to ever make all
    these people happy.

    Ultimately the question whether something is UNIX or not matters
    very little. Being technically excellent is hardly exclusive to
    UNIX. For us, UNIX is a major influence (heck, the biggest one), but
    we also have other influences. Hence in some areas systemd will be
    very UNIXy, and in others a little bit less.

  11. Myth: systemd is complex.

    There’s certainly some truth in that. Modern computers are complex
    beasts, and the OS running on it will hence have to be complex
    too. However, systemd is certainly not more complex than prior
    implementations of the same components. Much rather, it’s simpler, and
    has less redundancy (see above). Moreover, building a simple OS based
    on systemd will involve much fewer packages than a traditional Linux
    did. Fewer packages makes it easier to build your system, gets rid of
    interdependencies and of much of the different behaviour of every
    component involved.

  12. Myth: systemd is bloated.

    Well, bloated certainly has many different definitions. But in
    most definitions systemd is probably the opposite of bloat. Since
    systemd components share a common code base, they tend to share much
    more code for common code paths. Here’s an example: in a traditional
    Linux setup, sysvinit, start-stop-daemon, inetd, cron, dbus, all
    implemented a scheme to execute processes with various configuration
    options in a certain, hopefully clean environment. On systemd the code
    paths for all of this, for the configuration parsing, as well as the
    actual execution is shared. This means less code, less place for
    mistakes, less memory and cache pressure, and is thus a very good
    thing. And as a side-effect you actually get a ton more functionality
    for it…

    As mentioned above, systemd is also pretty modular. You can choose
    at build time which components you need, and which you don’t
    need. People can hence specifically choose the level of “bloat” they
    want.

    When you build systemd, it only requires three dependencies: glibc,
    libcap and dbus. That’s it. It can make use of more dependencies, but
    these are entirely optional.

    So, yeah, whichever way you look at it, it’s really not
    bloated.

  13. Myth: systemd being Linux-only is not nice to the BSDs.

    Completely wrong. The BSD folks are pretty much uninterested in
    systemd. If systemd was portable, this would change nothing, they
    still wouldn’t adopt it. And the same is true for the other Unixes in
    the world. Solaris has SMF, BSD has their own “rc” system, and they
    always maintained it separately from Linux. The init system is very
    close to the core of the entire OS. And these other operating systems
    hence define themselves among other things by their core
    userspace. The assumption that they’d adopt our core userspace if we
    just made it portable, is completely without any foundation.

  14. Myth: systemd being Linux-only makes it impossible for Debian to adopt it as default.

    Debian supports non-Linux kernels in their distribution. systemd
    won’t run on those. Is that a problem though, and should that hinder
    them to adopt system as default? Not really. The folks who ported
    Debian to these other kernels were willing to invest time in a massive
    porting effort, they set up test and build systems, and patched and
    built numerous packages for their goal. The maintainance of both a
    systemd unit file and a classic init script for the packaged services
    is a negligable amount of work compared to that, especially since
    those scripts more often than not exist already.

  15. Myth: systemd could be ported to other kernels if its maintainers just wanted to.

    That is simply not true. Porting systemd to other kernel is not
    feasible. We just use too many Linux-specific interfaces. For a few
    one might find replacements on other kernels, some features one might
    want to turn off, but for most this is nor really possible. Here’s a
    small, very incomprehensive list: cgroups, fanotify, umount2(),
    /proc/self/mountinfo
    (including notification), /dev/swaps (same),
    udev, netlink,
    the structure of /sys, /proc/$PID/comm,
    /proc/$PID/cmdline, /proc/$PID/loginuid, /proc/$PID/stat,
    /proc/$PID/session, /proc/$PID/exe, /proc/$PID/fd, tmpfs, devtmpfs,
    capabilities, namespaces of all kinds, various prctl()s, numerous
    ioctls,
    the mount() system call and its semantics, selinux, audit,
    inotify, statfs, O_DIRECTORY, O_NOATIME, /proc/$PID/root, waitid(),
    SCM_CREDENTIALS, SCM_RIGHTS, mkostemp(), /dev/input, ...

    And no, if you look at this list and pick out the few where you can
    think of obvious counterparts on other kernels, then think again, and
    look at the others you didn’t pick, and the complexity of replacing
    them.

  16. Myth: systemd is not portable for no reason.

    Non-sense! We use the Linux-specific functionality because we need
    it to implement what we want. Linux has so many features that
    UNIX/POSIX didn’t have, and we want to empower the user with
    them. These features are incredibly useful, but only if they are
    actually exposed in a friendly way to the user, and that’s what we do
    with systemd.

  17. Myth: systemd uses binary configuration files.

    No idea who came up with this crazy myth, but it’s absolutely not
    true. systemd is configured pretty much exclusively via simple text
    files. A few settings you can also alter with the kernel command line
    and via environment variables. There’s nothing binary in its
    configuration (not even XML). Just plain, simple, easy-to-read text
    files.

  18. Myth: systemd is a feature creep.

    Well, systemd certainly covers more ground that it used to. It’s
    not just an init system anymore, but the basic userspace building
    block to build an OS from, but we carefully make sure to keep most of
    the features optional. You can turn a lot off at compile time, and
    even more at runtime. Thus you can choose freely how much feature
    creeping you want.

  19. Myth: systemd forces you to do something.

    systemd is not the mafia. It’s Free Software, you can do with it
    whatever you want, and that includes not using it. That’s pretty much
    the opposite of “forcing”.

  20. Myth: systemd makes it impossible to run syslog.

    Not true, we carefully made sure when we introduced
    the journal
    that all data is also passed on to any syslog daemon
    running. In fact, if something changed, then only that syslog gets
    more complete data now than it got before, since we now cover early
    boot stuff as well as STDOUT/STDERR of any system service.

  21. Myth: systemd is incompatible.

    We try very hard to provide the best possible compatibility with
    sysvinit. In fact, the vast majority of init scripts should work just
    fine on systemd, unmodified. However, there actually are indeed a few
    incompatibilities, but we try to document
    these
    and explain what to do about them. Ultimately every system
    that is not actually sysvinit itself will have a certain amount of
    incompatibilities with it since it will not share the exect same code
    paths.

    It is our goal to ensure that differences between the various
    distributions are kept at a minimum. That means unit files usually
    work just fine on a different distribution than you wrote it on, which
    is a big improvement over classic init scripts which are very hard to
    write in a way that they run on multiple Linux distributions, due to
    numerous incompatibilities between them.

  22. Myth: systemd is not scriptable, because of its D-Bus use.

    Not true. Pretty much every single D-Bus interface systemd provides
    is also available in a command line tool, for example in systemctl,
    loginctl,
    timedatectl,
    hostnamectl,
    localectl
    and suchlike. You can easily call these tools from shell scripts, they
    open up pretty much the entire API from the command line with
    easy-to-use commands.

    That said, D-Bus actually has bindings for almost any scripting
    language this world knows. Even from the shell you can invoke
    arbitrary D-Bus methods with dbus-send
    or gdbus. If
    anything, this improves scriptability due to the good support of D-Bus
    in the various scripting languages.

  23. Myth: systemd requires you to use some arcane configuration
    tools instead of allowing you to edit your configuration files
    directly.

    Not true at all. We offer some configuration tools, and using them
    gets you a bit of additional functionality (for example, command line
    completion for all settings!), but there’s no need at all to use
    them. You can always edit the files in question directly if you wish,
    and that’s fully supported. Of course sometimes you need to explicitly
    reload configuration of some daemon after editing the configuration,
    but that’s pretty much true for most UNIX services.

  24. Myth: systemd is unstable and buggy.

    Certainly not according to our data. We have been monitoring the
    Fedora bug tracker (and some others) closely for a long long time. The
    number of bugs is very low for such a central component of the OS,
    especially if you discount the numerous RFE bugs we track for the
    project. We are pretty good in keeping systemd out of the list of
    blocker bugs of the distribution. We have a relatively fast
    development cycle with mostly incremental changes to keep quality and
    stability high.

  25. Myth: systemd is not debuggable.

    False. Some people try to imply that the shell was a good
    debugger. Well, it isn’t really. In systemd we provide you with actual
    debugging features instead. For example: interactive debugging,
    verbose tracing, the ability to mask any component during boot, and
    more. Also, we provide documentation
    for it
    .

    It’s certainly well debuggable, we needed that for our own
    development work, after all. But we’ll grant you one thing: it uses
    different debugging tools, we believe more appropriate ones for the
    purpose, though.

  26. Myth: systemd makes changes for the changes’ sake.

    Very much untrue. We pretty much exclusively have technical
    reasons for the changes we make, and we explain them in the various
    pieces of documentation, wiki pages, blog articles, mailing list
    announcements. We try hard to avoid making incompatible changes, and
    if we do we try to document the why and how in detail. And if you
    wonder about something, just ask us!

  27. Myth: systemd is a Red-Hat-only project, is private property
    of some smart-ass developers, who use it to push their views to the
    world.

    Not true. Currently, there are 16 hackers with commit powers to the
    systemd git tree. Of these 16 only six are employed by Red Hat. The 10
    others are folks from ArchLinux, from Debian, from Intel, even from
    Canonical, Mandriva, Pantheon and a number of community folks with
    full commit rights. And they frequently commit big stuff, major
    changes. Then, there are 374 individuals with patches in our tree, and
    they too came from a number of different companies and backgrounds,
    and many of those have way more than one patch in the tree. The
    discussions about where we want to take systemd are done in the open,
    on our IRC channel (#systemd on freenode, you are always
    weclome), on our mailing
    list
    , and on public hackfests (such
    as our next one in Brno
    , you are invited). We regularly attend
    various conferences, to collect feedback, to explain what we are doing
    and why, like few others do. We maintain blogs, engage in social
    networks (we actually
    have some pretty interesting content on Google+
    , and our Google+
    Community is pretty alive, too
    .), and try really hard to explain
    the why and the how how we do things, and to listen to feedback and
    figure out where the current issues are (for example, from that
    feedback we compiled this lists of often heard myths about
    systemd…).

    What most systemd contributors probably share is a rough idea how a
    good OS should look like, and the desire to make it happen. However,
    by the very nature of the project being Open Source, and rooted in the
    community systemd is just what people want it to be, and if it’s not
    what they want then they can drive the direction with patches and
    code, and if that’s not feasible, then there are numerous other
    options to use, too, systemd is never exclusive.

    One goal of systemd is to unify the dispersed Linux landscape a
    bit. We try to get rid of many of the more pointless differences of
    the various distributions in various areas of the core OS. As part of
    that we sometimes adopt schemes that were previously used by only one
    of the distributions and push it to a level where it’s the default of
    systemd, trying to gently push everybody towards the same set of basic
    configuration. This is never exclusive though, distributions can
    continue to deviate from that if they wish, however, if they end-up
    using the well-supported default their work becomes much easier and
    they might gain a feature or two. Now, as it turns out, more
    frequently than not we actually adopted schemes that where Debianisms,
    rather than Fedoraisms/Redhatisms as best supported scheme by
    systemd. For example, systems running systemd now generally store
    their hostname in /etc/hostname, something that used to be
    specific to Debian and now is used across distributions.

    One thing we’ll grant you though, we sometimes can be
    smart-asses. We try to be prepared whenever we open our mouth, in
    order to be able to back-up with facts what we claim. That might make
    us appear as smart-asses.

    But in general, yes, some of the more influental contributors of
    systemd work for Red Hat, but they are in the minority, and systemd is
    a healthy, open community with different interests, different
    backgrounds, just unified by a few rough ideas where the trip should
    go, a community where code and its design counts, and certainly not
    company affiliation.

  28. Myth: systemd doesn’t support /usr split from the root directory.

    Non-sense. Since its beginnings systemd supports the
    --with-rootprefix= option to its configure script
    which allows you to tell systemd to neatly split up the stuff needed
    for early boot and the stuff needed for later on. All this logic is
    fully present and we keep it up-to-date right there in systemd’s build
    system.

    Of course, we still don’t think that actually
    booting with /usr unavailable is a good idea
    , but we
    support this just fine in our build system. This won’t fix the
    inherent problems of the scheme that you’ll encounter all across the
    board, but you can’t blame that on systemd, because in systemd we
    support this just fine.

  29. Myth: systemd doesn’t allow your to replace its components.

    Not true, you can turn off and replace pretty much any part of
    systemd, with very few exceptions. And those exceptions (such as
    journald) generally allow you to run an alternative side by side to
    it, while cooperating nicely with it.

  30. Myth: systemd’s use of D-Bus instead of sockets makes it intransparent.

    This claim is already contradictory in itself: D-Bus uses sockets
    as transport, too. Hence whenever D-Bus is used to send something
    around, a socket is used for that too. D-Bus is mostly a standardized
    serialization of messages to send over these sockets. If anything this
    makes it more transparent, since this serialization is well
    documented, understood and there are numerous tracing tools and
    language bindings for it. This is very much unlike the usual
    homegrown protocols the various classic UNIX daemons use to
    communicate locally.

Hmm, did I write I just wanted to debunk a “few” myths? Maybe these
were more than just a few… Anyway, I hope I managed to clear up a
couple of misconceptions. Thanks for your time.

Footnotes

[1] For example, systemd-detect-virt,
systemd-tmpfiles,
systemd-udevd are.

[2] Also, we are trying to do our little part on maybe
making this better. By exposing boot-time performance of the firmware
more prominently in systemd’s boot output we hope to shame the
firmware writers to clean up their stuff.

[3] And anyways, guess which project includes a library “libnih” — Upstart or systemd?[4]

[4] Hint: it’s not systemd!