Tag Archives: hbo

A bit more on firearms in the US

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/06/a-bit-more-on-firearms-in-us.html

This is the fifth article in a short series about Poland, Europe, and the United States. To explore the entire series, start here.

Perhaps not surprisingly, my previous blog post sparked several interesting discussions with my Polish friends who took a more decisive view of the social costs of firearm ownership, or who saw the Second Amendment as a barbaric construct with no place in today’s world. Their opinions reminded me of my own attitude some ten years ago; in this brief follow-up, I wanted to share several data points that convinced me to take a more measured stance.

Let’s start with the basics: most estimates place the number of guns in the United States at 300 to 350 million – that’s roughly one firearm per every single resident. In Gallup polls, some 40-50% of all households report having a gun, frequently more than one. The demographics of firearm ownership are more uniform than stereotypes may imply; there is some variance across regions, political affiliations, and genders – but for most part, it tends to fall within fairly narrow bands.

An overwhelming majority of gun owners cite personal safety as the leading motive for purchasing a firearm; hunting and recreation activities come strong second. The defensive aspect of firearm ownership is of special note, because it can potentially provide a very compelling argument for protecting the right to bear arms even if it’s a socially unwelcome practice, or if it comes at an elevated cost to the nation as a whole.

The self-defense argument is sometimes dismissed as pure fantasy, with many eminent pundits citing one questionable statistic to support this view: the fairly low number of justifiable homicides in the country. Despite its strong appeal to ideologues, the metric does not stand up to scrutiny: all available data implies that most encounters where a gun is pulled by a would-be victim will not end with the assailant getting killed; it’s overwhelmingly more likely that the bad guy would hastily retreat, be detained at gunpoint, or suffer non-fatal injuries. In fact, even in the unlikely case that a firearm is actually discharged with the intent to kill or maim, somewhere around 70-80% of victims survive.

In reality, we have no single, elegant, and reliable source of data about the frequency with which firearms are used to deter threats; the results of scientific polls probably offer the most comprehensive view, but are open to interpretation and their results vary significantly depending on sampling methods and questions asked. That said, a recent meta-analysis from Centers for Disease Control and Prevention provided some general bounds:


“Defensive use of guns by crime victims is a common occurrence, although the exact number remains disputed (Cook and Ludwig, 1996; Kleck, 2001a). Almost all national survey estimates indicate that defensive gun uses by victims are at least as common as offensive uses by criminals, with estimates of annual uses ranging from about 500,000 to more than 3 million.”

An earlier but probably similarly unbiased estimate from US Dept of Justice puts the number at approximately 1.5 million uses a year.

The CDC study also goes on to say:


“A different issue is whether defensive uses of guns, however numerous or rare they may be, are effective in preventing injury to the gun-wielding crime victim. Studies that directly assessed the effect of actual defensive uses of guns (i.e., incidents in which a gun was “used” by the crime victim in the sense of attacking or threatening an offender) have found consistently lower injury rates among gun-using crime victims compared with victims who used other self-protective strategies.”

An argument can be made that the availability of firearms translates to higher rates of violent crime, thus elevating the likelihood of encounters where a defensive firearm would be useful – feeding into an endless cycle of escalating violence. That said, such an effect does not seem to be particularly evident. For example, the United States comes out reasonably well in statistics related to assault, rape, and robbery; on these fronts, America looks less violent than the UK or a bunch of other OECD countries with low firearm ownership rates.

But there is an exception: one area where the United States clearly falls behind other highly developed nations are homicides. The per-capita figures are almost three times as high as in much of the European Union. And indeed, the bulk of intentional homicides – some 11 thousand deaths a year – trace back to firearms.

We tend to instinctively draw a connection to guns, but the origins of this tragic situation may be more elusive than they appear. For one, non-gun-related homicides happen in the US at a higher rate than in many other countries, too; Americans just seem to be generally more keen on killing each other than people in places such as Europe, Australia, or Canada. In addition, no convincing pattern emerges when comparing overall homicide rates across states with permissive and restrictive gun ownership laws. Some of the lowest per-capita homicide figures can be found in extremely gun-friendly states such as Idaho, Utah, or Vermont; whereas highly-regulated Washington D.C., Maryland, Illinois, and California all rank pretty high. There is, however, fairly strong correlation between gun and non-gun homicide rates across the country – suggesting that common factors such as population density, urban poverty, and drug-related gang activities play a far more significant role in violent crime than the ease of legally acquiring a firearm. It’s tragic but worth noting that a strikingly disproportionate percentage of homicides involves both victims and perpetrators that belong to socially disadvantaged and impoverished minorities. Another striking pattern is that up to about a half of all gun murders are related to or committed under the influence of illicit drugs.

Now, international comparisons show general correlation between gun ownership and some types of crime, but it’s difficult to draw solid conclusions from that: there are countless other ways to explain why crime rates may be low in the wealthy European states, and high in Venezuela, Mexico, Honduras, or South Africa; compensating for these factors is theoretically possible, but requires making far-fetched assumptions that are hopelessly vulnerable to researcher bias. Comparing European countries is easier, but yields inconclusive results: gun ownership in Poland is almost twenty times lower than in neighboring Germany and ten times lower than in Czech Republic – but you certainly wouldn’t able to tell that from national crime stats.

When it comes to gun control, one CDC study on the topic concluded with:


“The Task Force found insufficient evidence to determine the effectiveness of any of the firearms laws or combinations of laws reviewed on violent outcomes.”

This does not imply that such approaches are necessarily ineffective; for example, it seems pretty reasonable to assume that well-designed background checks or modest waiting periods do save lives. Similarly, safe storage requirements would likely prevent dozens of child deaths every year, at the cost of rendering firearms less available for home defense. But for the hundreds of sometimes far-fetched gun control proposals introduced every year on federal and state level, emotions often take place of real data, poisoning the debate around gun laws and ultimately bringing little or no public benefit. The heated assault weapon debate is one such red herring: although modern semi-automatic rifles look sinister, they are far more common in movies than on the streets; in reality, all kinds of rifles account only for somewhere around 4% of firearm homicides, and AR-15s are only a tiny fraction of that – likely claiming about as many lives as hammers, ladders, or swimming pools. The efforts to close the “gun show loophole” seem fairly sensible at the surface, too, but are of similarly uncertain merit; instead of gun shows, criminals depend on friends, family, and on more than 200,000 guns that stolen from their rightful owners every year. When breaking into a random home yields a 40-50% chance of scoring a firearm, it’s not hard to see why.

Another oddball example of simplistic legislative zeal are the attempts to mandate costly gun owner liability insurance, based on drawing an impassioned but flawed parallel between firearms and cars; what undermines this argument is that car accidents are commonplace, while gun handling mishaps – especially ones that injure others – are rare. We also have proposals to institute $100 ammunition purchase permits, to prohibit ammo sales over the Internet, or to impose a hefty per-bullet tax. Many critics feel that such laws seem to be geared not toward addressing any specific dangers, but toward making firearms more expensive and burdensome to own – slowly eroding the constitutional rights of the less wealthy folks. They also see hypocrisy in the common practice of making retired police officers and many high-ranking government officials exempt from said laws.

Regardless of individual merits of the regulations, it’s certainly true that with countless pieces of sometimes obtuse and poorly-written federal, state, and municipal statutes introduced every year, it’s increasingly easy for people to unintentionally run afoul of the rules. In California, the law as written today implies that any legal permanent resident in good standing can own a gun, but that only US citizens can transport it by car. Given that Californians are also generally barred from carrying firearms on foot in many populated areas, non-citizen residents are seemingly expected to teleport between the gun store, their home, and the shooting range. With many laws hastily drafted in the days after mass shootings and other tragedies, such gems are commonplace. The federal Gun-Free School Zones Act imposes special restrictions on gun ownership within 1,000 feet of a school and slaps harsh penalties for as little carrying it in an unlocked container from one’s home to a car parked in the driveway. In many urban areas, a lot of people either live within such a school zone or can’t conceivably avoid it when going about their business; GFSZA violations are almost certainly common and are policed only selectively.

Meanwhile, with sharp declines in crime continuing for the past 20 years, the public opinion is increasingly in favor of broad, reasonably policed gun ownership; for example, more than 70% respondents to one Gallup poll are against the restrictive handgun bans of the sort attempted in Chicago, San Francisco, or Washington D.C.; and in a recent Rasmussen poll, only 22% say that they would feel safer in a neighborhood where people are not allowed to keep guns. In fact, responding to the media’s undue obsession with random of acts of violence against law-abiding citizens, and worried about the historically very anti-gun views of the sitting president, Americans are buying a lot more firearms than ever before. Even the National Rifle Association – a staunchly conservative organization vilified by gun control advocates and mainstream pundits – enjoys a pretty reasonable approval rating across many demographics: 58% overall and 78% in households with a gun.

And here’s the kicker: despite its reputation for being a political arm of firearm manufacturers, the NRA is funded largely through individual memberships, small-scale donations, and purchase round-ups; organizational donations add up to about 5% of their budget – and if you throw in advertising income, the total still stays under 15%. That makes it quite unlike most of the other large-scale lobbying groups that Democrats aren’t as keen on naming-and-shaming on the campaign trail. The NRA’s financial muscle is also frequently overstated; it doesn’t even make it onto the list of top 100 lobbyists in Washington – and gun control advocacy groups, backed by activist billionaires such as Michael Bloomberg, now frequently outspend the pro-gun crowd. Of course, it would be better for the association’s socially conservative and unnecessarily polarizing rhetoric – sometimes veering onto the topics of abortion or video games – to be offset by the voice of other, more liberal groups. But ironically, organizations such as American Civil Liberties Union – well-known for fearlessly defending controversial speech – prefer to avoid the Second Amendment; they do so not because the latter concept has lesser constitutional standing, but because supporting it would not sit well with their own, progressive support base.

America’s attitude toward guns is a choice, not a necessity. It is also true that gun violence is a devastating problem; and that the emotional horror and lasting social impact of incidents such as school shootings can’t be possibly captured in any cold, dry statistic alone. But there is also nuance and reason to the gun control debate that can be hard to see for newcomers from more firearm-averse parts of the world.

For the next article in the series, click here. Alternatively, if you prefer to keep reading about firearms, go here for an overview of the gun control debate in the US.

Our docker & screen development environment

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2015/05/05/our-docker--screen-development-environment/

The raintank software stack is comprised of a number of different components that all work together to deliver our platform. Some of our components include:
front-end visualization and dashboarding (based on Grafana) network performance data “collectors” event and metric processing and persisting core RESTful API To support these services we have a number of shared services including:
Elasticsearch InfluxDB MySQL RabbitMQ Redis As the stack has grown it has become increasingly more difficult to manage our development environments.

On journeys

Post Syndicated from Michal Zalewski original http://lcamtuf.blogspot.com/2015/03/on-journeys.html

– 1 –

Poland is an ancient country whose history is deeply intertwined with that of the western civilization. In its glory days, the Polish-Lithuanian Commonwealth sprawled across vast expanses of land in central Europe, from Black Sea to Baltic Sea. But over the past two centuries, it suffered a series of military defeats and political partitions at the hands of its closest neighbors: Russia, Austria, Prussia, and – later – Germany.

After more than a hundred years of foreign rule, Poland re-emerged as an independent state in 1918, only to face the armies of Nazi Germany at the onset of World War II. With Poland’s European allies reneging on their earlier military guarantees, the fierce fighting left the country in ruins. Some six million people have died within its borders – more than ten times the death toll in France or in the UK. Warsaw was reduced to a sea of rubble, with perhaps one in ten buildings still standing by the end of the war.

With the collapse of the Third Reich, Franklin D. Roosevelt, Winston Churchill, and Joseph Stalin held a meeting in Yalta to decide the new order for war-torn Europe. At Stalin’s behest, Poland and its neighboring countries were placed under Soviet political and military control, forming what has become known as the Eastern Bloc.

Over the next several decades, the Soviet satellite states experienced widespread repression and economic decline. But weakened by the expense of the Cold War, the communist chokehold on the region eventually began to wane. In Poland, even the introduction of martial law in 1981 could not put an end to sweeping labor unrest. Narrowly dodging the specter of Soviet intervention, the country regained its independence in 1989 and elected its first democratic government; many other Eastern Bloc countries soon followed suit.

Ever since then, Poland has enjoyed a period of unprecedented growth and has emerged as one of the more robust capitalist democracies in the region. In just two decades, it shed many of its backwardly, state-run heavy industries and adopted a modern, service-oriented economy. But the effects of the devastating war and the lost decades under communist rule still linger on – whether you look at the country’s infrastructure, at its socrealist cityscapes, at its political traditions, or at the depressingly low median wage.

When thinking about the American involvement in the Cold War, people around the world may recall Vietnam, Bay of Pigs, or the proxy wars fought in the Middle East. But in Poland and many of its neighboring states, the picture you remember the most is the fall of the Berlin Wall.

– 2 –

I was born in Warsaw in the winter of 1981, at the onset of martial law, with armored vehicles rolling onto Polish streets. My mother, like many of her generation, moved to the capital in the sixties as a part of an effort to rebuild and repopulate the war-torn city. My grandma would tell eerie stories of Germans and Soviets marching through their home village somewhere in the west. I liked listening to the stories; almost every family in Poland had some to tell.

I did not get to know my father. I knew his name; he was a noted cinematographer who worked on big-ticket productions back in the day. He left my mother when I was very young and never showed interest in staying in touch. He had a wife and other children, so it might have been that.

Compared to him, mom hasn’t done well for herself. We ended up in social housing in one of the worst parts of the city, on the right bank of the Vistula river. My early memories from school are that of classmates sniffing glue from crumpled grocery bags. I remember my family waiting in lines for rationed toilet paper and meat. As a kid, you don’t think about it much.

The fall of communism came suddenly. I have a memory of grandma listening to broadcasts from Radio Free Europe, but I did not understand what they were all about. I remember my family cheering one afternoon, transfixed to a black-and-white TV screen. I recall my Russian language class morphing into English; I had my first taste of bananas and grapefruits. There is the image of the monument of Feliks Dzierżyński coming down. I remember being able to go to a better school on the other side of Warsaw – and getting mugged many times on the way.

The transformation brought great wealth to some, but many others have struggled to find their place in the fledgling and sometimes ruthless capitalist economy. Well-educated and well read, my mom ended up in the latter pack, at times barely making ends meet. I think she was in part a victim of circumstance, and in part a slave to way of thinking that did not permit the possibility of taking chances or pursuing happiness.

– 3 –

Mother always frowned upon popular culture, seeing it as unworthy of an educated mind. For a time, she insisted that I only listen to classical music. She angrily shunned video games, comic books, and cartoons. I think she perceived technology as trivia; the only field of science she held in high regard was abstract mathematics, perhaps for its detachment from the mundane world. She hoped that I would learn Latin, a language she could read and write; that I would practice drawing and painting; or that I would read more of the classics of modernist literature.

Of course, I did almost none of that. I hid my grunge rock tapes between Tchaikovsky, listened to the radio under the sheets, and watched the reruns of The A-Team while waiting for her to come back from work. I liked electronics and chemistry a lot more than math. And when I laid my hands on my first computer – an 8-bit relic of British engineering from 1982 – I soon knew that these machines, in their incredible complexity and flexibility, were what I wanted to spend my time on.

I suspected I could become a competent programmer, but never had enough faith in my skill. Yet, in learning about computers, I realized that I had a knack for understanding complex systems and poking holes in how they work. With a couple of friends, we joined the nascent information security community in Europe, comparing notes on mailing lists. Before long, we were taking on serious consulting projects for banks and the government – usually on weekends and after school, but sometimes skipping a class or two. Well, sometimes more than that.

All of the sudden, I was facing an odd choice. I could stop, stay in school and try to get a degree – going back every night to a cramped apartment, my mom sleeping on a folding bed in the kitchen, my personal space limited to a bare futon and a tiny desk. Or, I could seize the moment and try to make it on my own, without hoping that one day, my family would be able to give me a head start.

I moved out, dropped out of school, and took on a full-time job. It paid somewhere around $12,000 a year – a pittance anywhere west of the border, but a solid wage in Poland even today. Not much later, I was making two times as much, about the upper end of what one could hope for in this line of work. I promised myself to keep taking courses after hours, but I wasn’t good at sticking to the plan. I moved in with my girlfriend, and at the age of 19, I felt for the first time that things were going to be all right.

– 4 –

Growing up in Europe, you get used to the barrage of low-brow swipes taken at the United States. Your local news will never pass up the opportunity to snicker about the advances of creationism somewhere in Kentucky. You can stay tuned for a panel of experts telling you about the vastly inferior schools, the medieval justice system, and the striking social inequality on the other side of the pond. You don’t doubt their words – but deep down inside, no matter how smug the critics are, or how seemingly convincing their arguments, the American culture still draws you in.

My moment of truth came in the summer of 2000. A company from Boston asked me if I’d like to talk about a position on their research team; I looked at the five-digit figure and could not believe my luck. Moving to the US was an unreasonable risk for a kid who could barely speak English and had no safety net to fall back to. But that did not matter: I knew I had no prospects of financial independence in Poland – and besides, I simply needed to experience the New World through my own eyes.

Of course, even with a job offer in hand, getting into the United States is not an easy task. An engineering degree and a willing employer opens up a straightforward path; it is simple enough that some companies would abuse the process to source cheap labor for menial, low-level jobs. With a visa tied to the petitioning company, such captive employees could not seek better wages or more rewarding work.

But without a degree, the options shrink drastically. For me, the only route would be a seldom-granted visa reserved for extraordinary skill – meant for the recipients of the Nobel Prize and other folks who truly stand out in their field of expertise. The attorneys looked over my publication record, citations, and the supporting letters from other well-known people in the field. Especially given my age, they thought we had a good shot. A few stressful months later, it turned out that they were right.

On the week of my twentieth birthday, I packed two suitcases and boarded a plane to Boston. My girlfriend joined me, miraculously securing a scholarship at a local university to continue her physics degree; her father helped her with some of the costs. We had no idea what we were doing; we had perhaps few hundred bucks on us, enough to get us through the first couple of days. Four thousand miles away from our place of birth, we were starting a brand new life.

– 5 –

The cultural shock gets you, but not in the sense you imagine. You expect big contrasts, a single eye-opening day to remember for the rest of your life. But driving down a highway in the middle of a New England winter, I couldn’t believe how ordinary the world looked: just trees, boxy buildings, and pavements blanketed with dirty snow.

Instead of a moment of awe, you drown in a sea of small, inconsequential things, draining your energy and making you feel helpless and lost. It’s how you turn on the shower; it’s where you can find a grocery store; it’s what they meant by that incessant “paper or plastic” question at the checkout line. It’s how you get a mailbox key, how you make international calls, it’s how you pay your bills with a check. It’s the rules at the roundabout, it’s your social security number, it’s picking the right toll lane, it’s getting your laundry done. It’s setting up a dial-up account and finding the food you like in the sea of unfamiliar brands. It’s doing all this without Google Maps or a Facebook group to connect with other expats nearby.

The other thing you don’t expect is losing touch with your old friends; you can call or e-mail them every day, but your social frames of reference begin to drift apart, leaving less and less to talk about. The acquaintances you make in the office will probably never replace the folks you grew up with. We managed, but we weren’t prepared for that.

– 6 –

In the summer, we had friends from Poland staying over for a couple of weeks. By the end of their trip, they asked to visit New York City one more time; we liked the Big Apple, so we took them on a familiar ride down I-95. One of them went to see the top of World Trade Center; the rest of us just walked around, grabbing something to eat before we all headed back. A few days later, we were all standing in front of a TV, watching September 11 unfold in real time.

We felt horror and outrage. But when we roamed the unsettlingly quiet streets of Boston, greeted by flags and cardboard signs urging American drivers to honk, we understood that we were strangers a long way from home – and that our future in this country hanged in the balance more than we would have thought.

Permanent residency is a status that gives a foreigner the right to live in the US and do almost anything they please – change jobs, start a business, or live off one’s savings all the same. For many immigrants, the pursuit of this privilege can take a decade or more; for some others, it stays forever out of reach, forcing them to abandon the country in a matter of days as their visas expire or companies fold. With my O-1 visa, I always counted myself among the lucky ones. Sure, it tied me to an employer, but I figured that sorting it out wouldn’t be a big deal.

That proved to be a mistake. In the wake of 9/11, an agency known as Immigration and Naturalization Services was being dismantled and replaced by a division within the Department of Homeland Security. My own seemingly straightforward immigration petition ended up somewhere in the bureaucratic vacuum that formed in between the two administrative bodies. I waited patiently, watching the deepening market slump, and seeing my employer’s prospects get dimmer and dimmer every month. I was ready for the inevitable, with other offers in hand, prepared to make my move perhaps the very first moment I could. But the paperwork just would not come through. With the Boston office finally shutting down, we packed our bags and booked flights. We faced the painful admission that for three years, we chased nothing but a pipe dream. The only thing we had to show for it were two adopted cats, now sitting frightened somewhere in the cargo hold.

The now-worthless approval came through two months later; the lawyers, cheerful as ever, were happy to send me a scan. The hollowed-out remnants of my former employer were eventually bought by Symantec – the very place from where I had my backup offer in hand.

– 7 –

In a way, Europe’s obsession with America’s flaws made it easier to come home without ever explaining how the adventure really played out. When asked, I could just wing it: a mention of the death penalty or permissive gun laws would always get you a knowing nod, allowing the conversation to move on.

Playing to other people’s preconceptions takes little effort; lying to yourself calls for more skill. It doesn’t help that when you come back after three years away from home, you notice all the small annoyances that you used to simply tune out. Back then, Warsaw still had a run-down vibe: the dilapidated road from the airport; the drab buildings on the other side of the river; the uneven pavements littered with dog poop; the dirty walls at my mother’s place, with barely any space to turn. You can live with it, of course – but it’s a reminder that you settled for less, and it’s a sensation that follows you every step of the way.

But more than the sights, I couldn’t forgive myself something else: that I was coming back home with just loose change in my pocket. There are some things that a failed communist state won’t teach you, and personal finance is one of them; I always looked at money just as a reward for work, something you get to spend to brighten your day. The indulgences were never extravagant: perhaps I would take the cab more often, or have take-out every day. But no matter how much I made, I kept living paycheck-to-paycheck – the only way I knew, the way our family always did.

– 8 –

With a three-year stint in the US on your resume, you don’t have a hard time finding a job in Poland. You face the music in a different way. I ended up with a salary around a fourth of what I used to make in Massachusetts, but I simply decided not to think about it much. I wanted to settle down, work on interesting projects, marry my girlfriend, have a child. I started doing consulting work whenever I could, setting almost all the proceeds aside.

After four years with T-Mobile in Poland, I had enough saved to get us through a year or so – and in a way, it changed the way I looked at my work. Being able to take on ambitious challenges and learn new things started to matter more than jumping ships for a modest salary bump. Burned by the folly of pursuing riches in a foreign land, I put a premium on boring professional growth.

Comically, all this introspection made me realize that from where I stood, I had almost nowhere left to go. Sure, Poland had telcos, refineries, banks – but they all consumed the technologies developed elsewhere, shipped here in a shrink-wrapped box; as far as their IT went, you could hardly tell the companies apart. To be a part of the cutting edge, you had to pack your bags, book a flight, and take a jump into the unknown. I sure as heck wasn’t ready for that again.

And then, out of the blue, Google swooped in with an offer to work for them from the comfort of my home, dialing in for a videoconference every now and then. The starting pay was about the same, but I had no second thoughts. I didn’t say it out loud, but deep down inside, I already knew what needed to happen next.

– 9 –

We moved back to the US in 2009, two years after taking the job, already on the hook for a good chunk of Google’s product security and with the comfort of knowing where we stood. In a sense, my motive was petty: you could call it a desire to vindicate a failed adolescent dream. But in many other ways, I have grown fond of the country that shunned us once before; and I wanted our children to grow up without ever having to face the tough choices and the uncertain prospects I had to deal with in my earlier years.

This time, we knew exactly what to do: a quick stop at a grocery store on a way from the airport, followed by e-mail to our immigration folks to get the green card paperwork out the door. A bit more than half a decade later, we were standing in a theater in Campbell, reciting the Oath of Allegiance and clinging on to our new certificates of US citizenship.

The ceremony closed a long and interesting chapter in my life. But more importantly, standing in that hall with people from all over the globe made me realize that my story is not extraordinary; many of them had lived through experiences far more harrowing and captivating than mine. If anything, my tale is hard to tell apart from that of countless other immigrants from the former Eastern Bloc. By some estimates, in the US alone, the Polish diaspora is about 9 million strong.

I know that the Poland of today is not the Poland I grew up in. It’s not not even the Poland I came back to in 2003; the gap to Western Europe is shrinking every single year. But I am grateful to now live in a country that welcomes more immigrants than any other place on Earth – and at the end of their journey, makes many of them them feel at home. It also makes me realize how small and misguided must be the conversations we are having about immigration – not just here, but all over the developed world.

To explore other articles in this short series about Poland, click here. You can also directly proceed to the next entry here.

Grafana 2.0-Beta1 Released

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2015/03/30/grafana-2.0-beta1-released/

Today is an exciting day for the Grafana project, Grafana 2.0-beta1 is released! So much hard work during the last 5 months has gone in to making this release.
Grafana 2.0 comes with its own backend server (written in Go). So upgrading requires a little more than for previous upgrades. Dashboards are 100% backward compatible.
We provide deb, rpm and regular binary tar packages for download. If your platform is not included, check the Building from source guide.

Grafana 1.9.0-rc1 Released!

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2014/11/17/grafana-1.9.0-rc1-released/

New release today! v1.9.0-rc1 is now available for download.
play.grafana.org is updated to run 1.9.0-rc1. There is also a New features in v1.9 dashboard that highlights some of the new features.
No breaking changes, so update should be problem free. But this is a release candidate so there could be some bugs.
Release highlights 1) Singlestat panel This release marks the introduction of a new panel!
A multitude of options makes this panel pretty flexible.

Grafana 1.8.0-rc1 Released!

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2014/09/11/grafana-1.8.0-rc1-released/

New release today! v1.8.0-rc1 is now available for download.
This is probably the biggest release since the first version was released. So many new features, enhancements and general UI improvements that this will likely be a very long blog post. play.grafana.org is updated to run 1.8.0-rc1, now with InfluxDB demo dashboards. There is also a New features in v1.8 dashboard that highlights some of the new features.
Release highlights 1) Mixed graph styling You can override any display setting for a specific series using the series name or a regex pattern.

Grafana 1.7.0-rc1 Released!

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2014/08/05/grafana-1.7.0-rc1-released/

New release today! v1.7.0-rc1 is now available for download.
This is a big release with a many new features and improvements. There are also a large number of architectural changes (hence the release candidate status).
Release highlights InfluxDB can now be used as dashboard storage InfluxDB can be used as a data source for annotations InfluxDB series aliasing improvements (alias patterns) Elasticsearch can be used as an annotation data source New graph legend options: Align as table and Right side.

Grafana 1.6.0 released.

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2014/06/16/grafana-1.6.0-released./

New release today! v1.6.0 is now available for download.
This is a pretty big release with a many of small improvements and new features.
OpenTSDB This release marks the first release with initial support for OpenTSDB as a metrics datasource. Checkout the docs article for how to configure opentsdb.
Dashboard editing improvements Adding, moving and changing row settings as well as adding panels is much improved. The row edit icon on the right of the row has been changed into a menu with quick options to move the panel up/down, set height, add panel and delete.

Grafana 1.5.4 released.

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2014/05/13/grafana-1.5.4-released./

New release today! v1.5.4 is now available for download.
Some small new features and enhancements. Check the download page for a complete changelog.
One notable change is the unsaved changes tracking. You now get a warning when moving to another dashboard without having saved your changes, or when you close tab / window.
The progress for bigger new features and improvements has been slow. There are a number of pull requests that are wating for review.

Shuffle Sharding: Massive and Magical Fault Isolation

Post Syndicated from Colm MacCarthaigh original https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/

A standard deck of cards has 52 different playing cards and 2 jokers. If we shuffle a deck thoroughly and deal out a four card hand, there are over 300,000 different hands. Another way to say the same thing is that if we put the cards back, shuffle and deal again, the odds are worse than 1 in 300,000 that we’ll deal the same hand again. It’s very unlikely.

It’s also unlikely, less than a 1/4 chance, that even just one of the cards will match between the two hands. And to complete the picture, there’s a less than a 1/40 chance that two cards will match, and much less than a 1/1000 chance that three cards will be the same.

In our last post I promised to cover more about how Route 53 Infima can be used to isolate faults that are request-related, such as user or customer specific problems. Route 53 Infima’s Shuffle Sharding takes this pattern of rapidly diminishing likelihood for an increasing number of matches – a pattern which underlies many card games, lotteries and even bingo – and combines it with traditional horizontal scaling to produce a kind of fault isolation that can seem almost magical.

Traditional Horizontal Scaling

All but the smallest services usually run on more than one instance (though there are some impressive exceptions). Using multiple instances means that we can have active redundancy: when one instance has a problem the others can take over. Typically traffic and requests are spread across all of the healthy instances.

Though this pattern is great for balancing traffic and for handling occasional instance-level failure, it’s terrible if there’s something harmful about the requests themselves: every instance will be impacted. If the service is serving many customers for example, then one busy customer may swamp everyone else. Throttling requests on a per-customer basis can help, but even throttling mechanisms can themselves be overwhelmed.

Worse still, throttling won’t help if the problem is some kind of “poison request”. If a particular request happens to trigger a bug that causes the system to fail over, then the caller may trigger a cascading failure by repeatedly trying the same request against instance after instance until they have all fallen over.

Sharding and Bulkheads

One fault isolating improvement we can make upon traditional horizontal scaling is to use sharding. Sharding is a technique traditionally used with data storage and indexing systems. Instead of spreading traffic from all customers across every instance, we can divide the instances into shards. For example if we have eight instances for our service, we might create four shards of two instances each (two instances for some redundancy within each shard).

Next we face the decision of how to shard. One common way is by customer id, assigning customers to particular shards, but other sharding choices are viable such as by operation type or by resource identifier. You can even choose to do multidimensional sharding, and have customer-resource pairs select a shard, or customer-resource-operation-type.

Whatever makes the most sense for a given service depends on its innards and its particular mix of risks, but it’s usually possible to find some combination of id or operation type that will make a big difference if it can be isolated.

With this kind of sharding in place, if there is a problem caused by the requests, then we get the same kind of bulkhead effect we have seen before; the problem can be contained to one shard. So in our example above, with four shards then around one quarter of customers (or whichever other dimension has been chosen) may be impacted by a problem triggered by one customer. That’s considerably better than all customers being impacted.

If customers (or objects) are given specific DNS names to use (just as customers are given unique DNS names with many AWS services) then DNS can be used to keep per-customer cleanly separated across shards.

Shuffle Sharding

With sharding, we are able to reduce customer impact in direct proportion to the number of instances we have. Even if we had 100 shards, 1% of customers would still experience impact in the event of a problem. One sensible solution for this is to build monitoring systems that can detect these problems and once detected re-shard the problem requests to their own fully isolated shard. This is great, but it’s a reactive measure and can usually only mitigate impact, rather than avoid it in the first place.

With Shuffle Sharding, we can do better. The basic idea of Shuffle Sharding is to generate shards as we might deal hands from a deck of cards. Take the eight instances example. Previously we divided it into four shards of two instances. With Shuffle Sharding the shards contain two random instances, and the shards, just like our hands of cards, may have some overlap.

By choosing two instances from eight there are 56 potential shuffle shards, much more than the four simple shards we had before.

At first, it may seem as if these Shuffle Shards are less suited to isolating faults; in the above example diagram, two shuffle shards share instance 5, and so a problem affecting that instance may impact both shards. The key to resolving this is to make the client fault tolerant. By having simple retry logic in the client that causes it to try every endpoint in a Shuffle Shard, until one succeeds, we get a dramatic bulkhead effect.

With the client trying each instance in the shard, then a customer who is causing a problem to Shuffle Shard 1, may impact both instance 3 and instance 5 and so become impacted, but the customers using Shuffle Shard 2 should experience only negligible (if any) impact if the client retries have been carefully tested and implemented to handle this kind of partial degradation correctly. Thus the real impact is constrained to 1/56th of the overall shuffle shards.

1/56 impact is a great improvement on 1/4, but we can do even better still. Before, with simple sharding we needed to put two instances in each shard to have some redundancy. With Shuffle Sharding – as in traditional N+1 horizontal scaling – we have more instances available. We can shard to as many instances as we are willing to retry. With 3 retries – a common retry value – we can use four instances in total per shuffle shard.

With four instances per shuffle shard, we can reduce the impact to 1/1680 of our total customer base and we’ve made the “noisy neighbor” problem much more manageable.

Infima and Shuffle Sharding

The Route Infima library includes two kinds of Shuffle sharding. The first kind is simple stateless shuffle sharding. Stateless shuffle sharding uses hashing, much like a bloom filter does, to take a customer, object or other identifiers and turn it into shuffle shard pattern. This technique results in some probability of overlap between customers, just as when we deal hands from a deck of cards. But by being stateless, this kind of shuffle sharding can be easily used, even directly in calling clients.

The second kind of Shuffle Sharding included is Stateful Searching Shuffle Sharding. These shuffle shards are generated using random assignment, again like hands from a deck of cards, but there is built in support for checking each newly generated shard against every previously assigned shard in order to make guarantees about overlap. For example we might choose to give every shuffle shard 4 out of 20 endpoints, but require that no two shuffle shards ever share more than two particular endpoints.

Both kinds of shuffle sharding in Infima are compartmentalization aware. For example, we can ensure that shuffle shards also make use of every availability zone. So our instances could be in 2 availability zones, 4 in each one. Infima can make sure to choose 2 endpoints from each zone, rather than simply 2 at random (which might choose both from one availability zone).

Lastly, Infima also makes it easy to use Shuffle Shards along with RubberTrees, so that endpoints can be easily expressed in DNS using Route 53. For example, every customer can be supplied their own DNS name, which maps to a shuffle shard which is handled by a RubberTree.

Post-script

The two general principles at work are that it can often be better to use many smaller things as it lowers the cost of capacity buffers and makes the impact of any contention small, and that it can be beneficial to allow shards to partially overlap in their membership, in return for an exponential increase in the number of shards the system can support.

Those principles mean that Shuffle Sharding is a general-purpose technique, and you can also choose to Shuffle Shard across many kinds of resources, including pure in-memory data-structures such as queues, rate-limiters, locks and other contended resources.

As it happens, Amazon Route 53, CloudFront and other AWS services use compartmentalization, per-customer Shuffle Sharding and more to provide fault isolation, and we will be sharing some more details about how some of that works in a future blog post.

Update from the author: an earlier version of this blog post used an incorrect figure for the number of 4-card hands from a 52-card deck (I wrote 7 million, based on permutations, instead of 300,000 based on combinations).

– Colm MacCárthaigh

PulseAudio and Jack

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/when-pa-and-when-not.html

#nocomments yes

One thing became very clear to me during my trip to the Linux Audio Conference 2010
in Utrecht: even many pro audio folks are not sure what Jack does that PulseAudio doesn’t do and what
PulseAudio does that Jack doesn’t do; why they are not competing, why
you cannot replace one by the other, and why merging them (at least in
the short term) might not make immediate sense. In other words, why
millions of phones on this world run PulseAudio and not Jack, and why
a music studio running PulseAudio is crack.

To light this up a bit and for future reference I’ll try to explain in the
following text why there is this seperation between the two systems and why this isn’t
necessarily bad. This is mostly a written up version of (parts of) my slides
from LAC
, so if you attended that event you might find little new, but I hope
it is interesting nonetheless.

This is mostly written from my perspective as a hacker working on
consumer audio stuff (more specifically having written most of
PulseAudio), but I am sure most pro audio folks would agree with the
points I raise here, and have more things to add. What I explain below
is in no way comprehensive, just a list of a couple of points I think
are the most important, as they touch the very core of both
systems (and we ignore all the toppings here, i.e. sound effects, yadda, yadda).

First of all let’s clear up the background of the sound server use cases here:

Consumer Audio (i.e. PulseAudio)Pro Audio (i.e. Jack)
Reducing power usage is a defining requirement, most systems are battery powered (Laptops, Cell Phones).Power usage usually not an issue, power comes out of the wall.
Must support latencies low enough for telephony and
games. Also covers high latency uses, such as movie and music playback
(2s of latency is a good choice).
Minimal latencies are a
definining requirement.
System is highly dynamic, with applications starting/stopping, hardware added and removed all the time.System is usually static in its configuration during operation.
User is usually not proficient in the technologies used.[1]User is usually a professional and knows audio technology and computers well.
User is not necessarily the administrator of his machine, might have limited access.User usually administrates his own machines, has root privileges.
Audio is just one use of the system among many, and often just a background job.Audio is the primary purpose of the system.
Hardware tends to have limited resources and be crappy and cheap.Hardware is powerful, expensive and high quality.

Of course, things are often not as black and white like this, there are uses
that fall in the middle of these two areas.

From the table above a few conclusions may be drawn:

  • A consumer sound system must support both low and high latency operation.
    Since low latencies mean high CPU load and hence high power
    consumption[2] (Heisenberg…), a system should always run with the
    highest latency latency possible, but the lowest latency necessary.
  • Since the consumer system is highly dynamic in its use latencies must be
    adjusted dynamically too. That makes a design such as PulseAudio’s timer-based scheduling important.
  • A pro audio system’s primary optimization target is low latency. Low
    power usage, dynamic changeble configuration (i.e. a short drop-out while you
    change your pipeline is acceptable) and user-friendliness may be sacrificed for
    that.
  • For large buffer sizes a zero-copy design suggests itself: since data
    blocks are large the cache pressure can be considerably reduced by zero-copy
    designs. Only for large buffers the cost of passing pointers around is
    considerable smaller than the cost of passing around the data itself (or the
    other way round: if your audio data has the same size as your pointers, then
    passing pointers around is useless extra work).
  • On a resource constrained system the ideal audio pipeline does not touch
    and convert the data passed along it unnecessarily. That makes it important to
    support natively the sample types and interleaving modes of the audio source or
    destination.
  • A consumer system needs to simplify the view on the hardware, hide the its
    complexity: hide redundant mixer elements, or merge them while making use of
    the hardware capabilities, and extending it in software so that the same
    functionality is provided on all hardware. A production system should not hide
    or simplify the hardware functionality.
  • A consumer system should not drop-out when a client misbehaves or the
    configuration changes (OTOH if it happens in exceptions it is not disastrous
    either). A synchronous pipeline is hence not advisable, clients need to supply
    their data asynchronously.
  • In a pro audio system a drop-out during reconfiguration is acceptable,
    during operation unacceptable.
  • In consumer audio we need to make compromises on resource usage,
    which pro audio does not have to commit to. Example: a pro audio
    system can issue memlock() with little limitations since the
    hardware is powerful (i.e. a lot of RAM available) and audio is the
    primary purpose. A consumer audio system cannot do that because that
    call practically makes memory unavailable to other applications,
    increasing their swap pressure. And since audio is not the primary
    purpose of the system and resources limited we hence need to find a
    different way.

Jack has been designed for low latencies, where synchronous
operation is advisable, meaning that a misbehaving client call stall
the entire pipeline. Changes of the pipeline or latencies usually
result in drop-outs in one way or the other, since the entire pipeline
is reconfigured, from the hardware to the various clients. Jack only
supports FLOAT32 samples and non-interleaved audio channels (and that
is a good thing). Jack does not employ reference-counted zero-copy
buffers. It does not try to simplify the hardware mixer in any
way.

PulseAudio OTOH can deal with varying latancies, dynamically
adjusting to the lowest latencies any of the connected clients
needs
. Client communication is fully asynchronous, a single client
cannot stall the entire pipeline. PulseAudio supports a variety of PCM
formats and channel setups. PulseAudio’s design is heavily based on
reference-counted zero-copy buffers that are passed around, even
between processes, instead of the audio data itself. PulseAudio tries
to simplify the hardware mixer as suggested above.

Now, the two paragraphs above hopefully show how Jack is more
suitable for the pro audio use case and PulseAudio more for the
consumer audio use case. One question asks itself though: can we marry
the two approaches? Yes, we probably can, MacOS has a unified approach
for both uses. However, it is not clear this would be a good
idea. First of all, a system with the complexities introduced by
sample format/channel mapping conversion, as well as dynamically
changing latencies and pipelines, and asynchronous behaviour would
certainly be much less attractive to pro audio developers. In fact,
that Jack limits itself to synchronous, FLOAT32-only,
non-interleaved-only audio streams is one of the big features of its
design. Marrying the two approaches would corrupt that. A merged
solution would probably not have a good stand in the community.

But it goes even further than this: what would the use case for
this be? After all, most of the time, you don’t want your event
sounds, your Youtube, your VoIP and your Rhythmbox mixed into the new
record you are producing. Hence a clear seperation between the two
worlds might even be handy?

Also, let’s not forget that we lack the manpower to even create
such an audio chimera.

So, where to from here? Well, I think we should put the focus on
cooperation instead of amalgamation: teach PulseAudio to go out of the
way as soon as Jack needs access to the device, and optionally make
PulseAudio a normal JACK client while both are running. That way, the
user has the option to use the PulseAudio supplied streams, but
normally does not see them in his pipeline. The first part of this has
already been implemented: Jack2 and PulseAudio do not fight for the
audio device, a friendly handover takes place. Jack takes precedence,
PulseAudio takes the back seat. The second part is still missing: you
still have to manually hookup PulseAudio to Jack if you are interested
in its streams. If both are implemented starting Jack basically has
the effect of replacing PulseAudio’s core with the Jack core, while
still providing full compatibility with PulseAudio clients.

And that I guess is all I have to say on the entire Jack and
PulseAudio story.

Oh, one more thing, while we are at clearing things up: some news
sites claim that PulseAudio’s not necessarily stellar reputation in
some parts of the community comes from Ubuntu and other distributions
having integrated it too early. Well, let me stress here explicitly,
that while they might have made a mistake or two in packaging
PulseAudio and I publicly pointed that out (and probably not in a too
friendly way), I do believe that the point in time they adopted it was
right. Why? Basically, it’s a chicken and egg problem. If it is not
used in the distributions it is not tested, and there is no pressure
to get fixed what then turns out to be broken: in PulseAudio itself,
and in both the layers on top and below of it. Don’t forget that
pushing a new layer into an existing stack will break a lot of
assumptions that the neighboring layers made. Doing this must
break things. Most Free Software projects could probably use more
developers, and that is particularly true for Audio on Linux. And
given that that is how it is, pushing the feature in at that point in
time was the right thing to do. Or in other words, if the features are
right, and things do work correctly as far as the limited test base
the developers control shows, then one day you need to push into the
distributions, even if this might break setups and software that
previously has not been tested, unless you want to stay stuck in your
development indefinitely. So yes, Ubuntu, I think you did well with
adopting PulseAudio when you did.

Footnotes

[1] Side note: yes, consumers tend not to know what dB is, and expect
volume settings in “percentages”, a mostly meaningless unit in
audio. This even spills into projects like VLC or Amarok which expose
linear volume controls (which is a really bad idea).

[2] In case you are wondering why that is the case: if the latency is
low the buffers must be sized smaller. And if the buffers are sized smaller
then the CPU will have to wake up more often to fill them up for the same
playback time. This drives up the CPU load since less actual payload can be
processed for the amount of housekeeping that the CPU has to do during each
buffer iteration. Also, frequent wake-ups make it impossible for the CPU to go
to deeper sleep states. Sleep states are the primary way for modern CPUs
to save power.

The Highest Man in Spain

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/photos/canaries-360.html

Ever wanted to know what’s the view like being the highest person in all of Spain? — No? Hmm, can’t help you then. — Otherwise:

Pico del Teide

That’s on the summit of Pico del Teide at 3718m, on Tenerife island. Unless you leave solid ground this is as high as you can get in Spain. 163m lower it’s a bit more obvious that the Teide is a volcano:

Pico del Teide

And coming down to the surrounding caldera it’s even more obvious:

Pico del Teide

Pico del Teide

Pico del Teide

On a ridge next to the caldera you find the Teide Observatory:

Teide Observatory

The caldera is covered in old lava flows:

Caldera

Caldera

Vulcanism has created various interesting rock formations in the caldera:

Roques

Roques

Tenerife is not just about the Teide and its dusty caldera. In the north of the island you find the Anaga mountain range:

Tenerife North

Neighboring Gran Canaria was where our little trip started and ended, right after the Gran Canaria Desktop Summit. Gran Canaria has no Teide but a very impressive landscape nonetheless:

Roque Nublo

That’s the view from the Roque Nublo, the island’s most famous landmark. The rock itself is visible here (on the left):

Roque Nublo