Tag Archives: amf

PulseAudio 9.0 is out

Post Syndicated from corbet original http://lwn.net/Articles/692988/rss

The PulseAudio 9.0 release is out. Changes include improvements to
automatic routing, beamforming support, use of the Linux memfd mechanism for transport, higher
sample-rate support, and more; see the
release notes
for details.

See also: this
article from Arun Raghavan
on how the beamforming feature works.
The basic idea is that if you have a number of microphones (a mic
array) in some known arrangement, it is possible to ‘point’ or steer the
array in a particular direction, so sounds coming from that direction are
made louder, while sounds from other directions are rendered softer
(attenuated).

Graphical fidelity is ruining video games

Post Syndicated from Eevee original https://eev.ee/blog/2016/06/22/graphical-fidelity-is-ruining-video-games/

I’m almost 30, so I have to start practicing being crotchety.

Okay, maybe not all video games, but something curious has definitely happened here. Please bear with me for a moment.

Discovering Doom

Surprise! This is about Doom again.

Last month, I sat down and played through the first episode of Doom 1 for the first time. Yep, the first time. I’ve mentioned before that I was introduced to Doom a bit late, and mostly via Doom 2. I’m familiar with a decent bit of Doom 1, but I’d never gotten around to actually playing through any of it.

I might be almost unique in playing Doom 1 for the first time decades after it came out, while already being familiar with the series overall. I didn’t experience Doom 1 only in contrast to modern games, but in contrast to later games using the same engine.

It was very interesting to experience Romero’s design sense in one big chunk, rather than sprinkled around as it is in Doom 2. Come to think of it, Doom 1’s first episode is the only contiguous block of official Doom maps to have any serious consistency: it sticks to a single dominant theme and expands gradually in complexity as you play through it. Episodes 2 and 3, as well of most of Doom 2, are dominated by Sandy Petersen’s more haphazard and bizarre style. Episode 4 and Final Doom, if you care to count them, are effectively just map packs.

It was also painfully obvious just how new this kind of game was. I’ve heard Romero stress the importance of contrast in floor height (among other things) so many times, and yet Doom 1 is almost comically flat. There’s the occasional lift or staircase, sure, but the spaces generally feel like they’re focused around a single floor height with the occasional variation. Remember, floor height was a new thing — id had just finished making Wolfenstein 3D, where the floor and ceiling were completely flat and untextured.

The game was also clearly designed for people who had never played this kind of game. There was much more ammo than I could possibly carry; I left multiple shell boxes behind on every map. The levels were almost comically easy, even on UV, and I’m not particularly good at shooters. It was a very stark contrast to when I played partway through The Plutonia Experiment a few years ago and had to rely heavily on quicksaving.

Seeing Doom 1 from a Doom 2 perspective got me thinking about how design sensibilities in shooters have morphed over time. And then I realized something: I haven’t enjoyed an FPS since Quake 2.

Or… hang on. That’s not true. I enjoy Splatoon (except when I lose). I loved the Metroid Prime series. I played Team Fortress 2 for quite a while.

On the other hand, I found Half-Life 2 a little boring, I lost interest in Doom 3 before even reaching Hell, and I bailed on Quake 4 right around the extremely hammy spoiler plot thing. I loved Fallout, but I couldn’t stand Fallout 3. Uncharted is pretty to watch, but looks incredibly tedious to play. I never cared about Halo. I don’t understand the appeal of Counterstrike or Call of Duty.

If I made a collage of screenshots of these two sets of games, you’d probably spot the pattern pretty quickly. It seems I can’t stand games with realistic graphics.

I have a theory about this.

The rise of realism

Quake introduced the world to “true” 3D — an environment made out of arbitrary shapes, not just floors and walls. (I’m sure there were other true-3D games before it, but I challenge you to name one off the top of your head.)

Before Quake, games couldn’t even simulate a two-story building, which ruled out most realistic architecture. Walls that slid sideways were virtually unique to Hexen (and, for some reason, the much earlier Wolfenstein 3D). So level designers built slightly more abstract spaces instead. Consider this iconic room from the very beginning of Doom’s E1M1.

What is this room? This is supposed to be a base of some kind, but who would build this room just to store a single armored vest? Up a flight of stairs, on a dedicated platform, and framed by glowing pillars? This is completely ridiculous.

But nobody thinks like that, and even the people who do, don’t really care too much. It’s a room with a clear design idea and a clear gameplay purpose: to house the green armor. It doesn’t matter that this would never be a real part of a base. The game exists in its own universe, and it establishes early on that these are the rules of that universe. Sometimes a fancy room exists just to give the player a thing.

At the same time, the room still resembles a base. I can take for granted, in the back of my head, that someone deliberately placed this armor here for storage. It’s off the critical path, too, so it doesn’t quit feel like it was left specifically for me to pick up. The world is designed for the player, but it doesn’t feel that way — the environment implies, however vaguely, that other stuff is going on here.


Fast forward twenty years. Graphics and physics technology have vastly improved, to the point that we can now roughly approximate a realistic aesthetic in real-time. A great many games thus strive to do exactly that.

And that… seems like a shame. The better a game emulates reality, the less of a style it has. I can’t even tell Call of Duty and Battlefield apart.

That’s fine, though, right? It’s just an aesthetic thing. It doesn’t really affect the game.

It totally affects the game

Everything looks the same

Realism” generally means “ludicrous amounts of detail” — even moreso if the environments are already partially-destroyed, which is a fairly common trope I’ll be touching on a lot here.

When everything is highly-detailed, screenshots may look very good, but gameplay suffers because the player can no longer tell what’s important. The tendency for everything to have a thick coating of sepia certainly doesn’t help.

Look at that Call of Duty screenshot again. What in this screenshot is actually important? What here matters to you as a player? As far as I can tell, the only critical objects are:

  • Your current weapon

That’s it. The rocks and grass and billboards and vehicles and Hollywood sign might look very nice (by which I mean, “look like those things look”), but they aren’t important to the game at all. This might as well be a completely empty hallway.

To be fair, I haven’t played the game, so for all I know there’s a compelling reason to collect traffic cones. Otherwise, this screenshot is 100% noise. Everything in it serves only to emphasize that you’re in a realistic environment.

Don’t get me wrong, setting the scene is important, but something has been missed here. Detail catches the eye, and this screenshot is nothing but detail. None of it is relevant. If there were ammo lying around, would you even be able to find it?

Ah, but then, modern realistic games either do away with ammo pickups entirely or make them glow so you can tell they’re there. You know, for the realism.

(Speaking of glowing: something I always found ridiculous was how utterly bland the imp fireballs look in Doom 3 and 4. We have these amazing lighting engines, and the best we can do for a fireball is a solid pale orange circle? How do modern fireballs look less interesting than a Doom 1 fireball sprite?)

Even Fallout 2 bugged me a little with this; the world was full of shelves and containers, but it seemed almost all of them were completely empty. Fallout 1 had tons of loot waiting to be swiped from shelves, but someone must’ve decided that was a little silly and cut down on it in Fallout 2. So then, what’s the point of having so many shelves? They encourage the player to explore, then offer no reward whatsoever most of the time.

Environments are boring and static

Fallout 3 went right off the rails, filling the world with tons of (gray) detail, none of which I could interact with. I was barely finished with the first settlement before I gave up on the game because of how empty it felt. Everywhere was detailed as though it were equally important, but most of it was static decorations. From what I’ve seen, Fallout 4 is even worse.

Our graphical capabilities have improved much faster than our ability to actually simulate all the junk we’re putting on the screen. Hey, there’s a car! Can I get in it? Can I drive it? No, I can only bump into an awkwardly-shaped collision box drawn around it. So what’s the point of having a car, an object that — in the real world — I’m accustomed to being able to use?

And yet… a game that has nothing to do with driving a car doesn’t need you to be able to drive a car. Games are games, not perfect simulations of reality. They have rules, a goal, and a set of things the player is able to do. There’s no reason to make the player able to do everything if it has no bearing on what the game’s about.

This puts “realistic” games in an awkward position. How do they solve it?

One good example that comes to mind is Portal, which was rendered realistically, but managed to develop a style from the limited palette it used in the actual play areas. It didn’t matter that you couldn’t interact with the world in any way other than portaling walls and lifting cubes, because for the vast majority of the game, you only encountered walls and cubes! Even the “behind the scenes” parts at the end were mostly architecture, not objects, and I’m not particularly bothered that I can’t interact with a large rusty pipe.

The standouts were the handful of offices you managed to finagle your way into, which were of course full of files and computers and other desktop detritus. Everything in an office is — necessarily! — something a human can meaningfully interact with, but the most you can do in Portal is drop a coffee cup on the floor. It’s all the more infuriating if you consider that the plot might have been explained by the information in those files or on those computers. Portal 2 was in fact a little worse about this, as you spent much more time outside of the controlled test areas.

I think Left 4 Dead may have also avoided this problem by forcing the players to be moving constantly — you don’t notice that you can’t get in a car if you’re running for your life. The only time the players can really rest is in a safe house, which are generally full of objects the players can pick up and use.

Progression feels linear and prescripted

Ah, but the main draw of Portal is one of my favorite properties of games: you could manipulate the environment itself. It’s the whole point of the game, even. And it seems to be conspicuously missing from many modern “realistic” games, partly because real environments are just static, but also in large part because… of the graphics!

Rendering a very complex scene is hard, so modern map formats do a whole lot of computing stuff ahead of time. (For similar reasons, albeit more primitive ones, vanilla Doom can’t move walls sideways.) Having any of the environment actually move or change is thus harder, so it tends to be reserved for fancy cutscenes when you press the button that lets you progress. And because grandiose environmental changes aren’t very realistic, that button often just opens a door or blows something up.

It feels hamfisted, like someone carefully set it all up just for me. Obviously someone did, but the last thing I want is to be reminded of that. I’m reminded very strongly of Half-Life 2, which felt like one very long corridor punctuated by the occasional overt physics puzzle. Contrast with Doom, where there are buttons all over the place and they just do things without drawing any particular attention to the results. Mystery switches are sometimes a problem, but for better or worse, Doom’s switches always feel like something I’m doing to the game, rather than the game waiting for me to come along so it can do some preordained song and dance.

I miss switches. Real switches, not touchscreens. Big chunky switches that take up half a wall.

It’s not just the switches, though. Several of Romero’s maps from episode 1 are shaped like a “horseshoe”, which more or less means that you can see the exit from the beginning (across some open plaza). More importantly, the enemies at the exit can see you, and will be shooting at you for much of the level.

That gives you choices, even within the limited vocabulary of Doom. Do you risk wasting ammo trying to take them out from a distance, or do you just dodge their shots all throughout the level? It’s up to you! You get to decide how to play the game, naturally, without choosing from a How Do You Want To Play The Game menu. Hell, Doom has entire speedrun categories focused around combat — Tyson for only using the fist and pistol, pacifist for never attacking a monster at all.

You don’t see a lot of that any more. Rendering an entire large area in a polygon-obsessed game is, of course, probably not going to happen — whereas the Doom engine can handle it just fine. I’ll also hazard a guess and say that having too much enemy AI going at once and/or rendering too many highly-detailed enemies at once is too intensive. Or perhaps balancing and testing multiple paths is too complicated.

Or it might be the same tendency I see in modding scenes: the instinct to obsessively control the player’s experience, to come up with a perfectly-crafted gameplay concept and then force the player to go through it exactly as it was conceived. Even Doom 4, from what I can see, has a shocking amount of “oh no the doors are locked, kill all the monsters to unlock them!” nonsense. Why do you feel the need to force the player to shoot the monsters? Isn’t that the whole point of the game? Either the player wants to do it and the railroading is pointless, or the player doesn’t want to do it and you’re making the game actively worse for them!

Something that struck me in Doom’s E1M7 was that, at a certain point, you run back across half the level and there are just straggler monsters all over the place. They all came out of closets when you picked up something, of course, but they also milled around while waiting for you to find them. They weren’t carefully scripted to teleport around you in a fixed pattern when you showed up; they were allowed to behave however they want, following the rules of the game.

Whatever the cause, something has been lost. The entire point of games is that they’re an interactive medium — the player has some input, too.

Exploration is discouraged

I haven’t played through too many recent single-player shooters, but I get the feeling that branching paths (true nonlinearity) and sprawling secrets have become less popular too. I’ve seen a good few people specifically praise Doom 4 for having them, so I assume the status quo is to… not.

That’s particularly sad off the back of Doom episode 1, which has sprawling secrets that often feel like an entire hidden part of the base. In several levels, merely getting outside qualifies as a secret. There are secrets within secrets. There are locked doors inside secrets. It’s great.

And these are real secrets, not three hidden coins in a level and you need to find so many of them to unlock more levels. The rewards are heaps of resources, not a fixed list of easter eggs to collect. Sometimes they’re not telegraphed at all; sometimes you need to do something strange to open them. Doom has a secret you open by walking up to one of two pillars with a heart on it. Doom 2 has a secret you open by run-jumping onto a light fixture, and another you open by “using” a torch and shooting some eyes in the wall.

I miss these, too. Finding one can be a serious advantage, and you can feel genuinely clever for figuring them out, yet at the same time you’re not permanently missing out on anything if you don’t find them all.

I can imagine why these might not be so common any more. If decorating an area is expensive and complicated, you’re not going to want to build large areas off the critical path. In Doom, though, you can make a little closet containing a powerup in about twenty seconds.

More crucially, many of the Doom secrets require the player to notice a detail that’s out of place — and that’s much easier to set up in a simple world like Doom. In a realistic world where every square inch is filled with clutter, how could anyone possibly notice a detail out of place? How can a designer lay any subtle hints at all, when even the core gameplay elements have to glow for anyone to pick them out from background noise?

This might be the biggest drawback to extreme detail: it ultimately teaches the player to ignore the detail, because very little of it is ever worth exploring. After running into enough invisible walls, you’re going to give up on straying from the beaten path.

We wind up with a world where players are trained to look for whatever glows, and completely ignore everything else. At which point… why are we even bothering?

There are no surprises

Realistic” graphics mean a “realistic” world, and let’s face it, the real world can be a little dull. That’s why we invented video games, right?

Doom has a very clear design vocabulary. Here are some demons. They throw stuff at you; don’t get hit by it. Here are some guns, which you can all hold at once, because those are the rules. Also here’s a glowing floating sphere that gives you a lot of health.

What is a megasphere, anyway? Does it matter? It’s a thing in the game with very clearly-defined rules. It’s good; pick it up.

You can’t do that in a “realistic” game. (Or maybe you can, but we seem to be trying to avoid it.) You can’t just pick up a pair of stereoscopic glasses to inexplicably get night vision for 30 seconds; you need to have some night-vision goggles with batteries and it’s a whole thing. You can’t pick up health kits that heal you; you have to be wearing regenerative power armor and pick up energy cells. Even Doom 4 seems to be uncomfortable leaving brightly flashing keycards lying around — instead you retrieve them from the corpses of people wearing correspondingly-colored armor.

Everything needs an explanation, which vastly reduces the chances of finding anything too surprising or new.

I’m told that Call of Duty is the most popular vidya among the millenials, so I went to look at its weapons:

  • Gun
  • Fast gun
  • Long gun
  • Different gun

How exciting! If you click through each of those gun categories, you can even see the list of unintelligible gun model numbers, which are exactly what gets me excited about a game.

I wonder if those model numbers are real or not. I’m not sure which would be worse.

Get off my lawn

So my problem is that striving for realism is incredibly boring and counter-productive. I don’t even understand the appeal; if I wanted reality, I could look out my window.

Realism” actively sabotages games. I can judge Doom or Mario or Metroid or whatever as independent universes with their own rules, because that’s what they are. A game that’s trying to mirror reality, I can only compare to reality — and it’ll be a very pale imitation.

It comes down to internal consistency. Doom and Team Fortress 2 and Portal and Splatoon and whatever else are pretty upfront about what they’re offering: you have a gun, you can shoot it, also you can run around and maybe press some buttons if you’re lucky. That’s exactly what you get. It’s right there on the box, even.

Then I load Fallout 3, and it tries to look like the real world, and it does a big song and dance asking me for my stats “in-world”, and it tries to imply I can roam this world and do anything I want and forge my own destiny. Then I get into the game, and it turns out I can pretty much just shoot, pick from dialogue trees, and make the occasional hamfisted moral choice. The gameplay doesn’t live up to what the environment tried to promise. The controls don’t even live up to what the environment tried to promise.

The great irony is that “realism” is harshly limiting, even as it grows ever more expensive and elaborate. I’m reminded of the Fat Man in Fallout 3, the gun that launches “mini nukes”. If that weapon had been in Fallout 1 or 2, I probably wouldn’t think twice about it. But in the attempted “realistic” world of Fallout 3, I have to judge it as though it were trying to be a real thing — because it is! — and that makes it sound completely ridiculous.

(It may sound like I’m picking on Fallout 3 a lot here, but to its credit, it actually had enough stuff going on that it stands out to me. I barely remember anything about Doom 3 or Quake 4, and when I think of Half-Life 2 I mostly imagine indistinct crumbling hallways or a grungy river that never ends.)

I’ve never felt this way about series that ignored realism and went for their own art style. Pikmin 3 looks very nice, but I never once felt that I ought to be able to do anything other than direct Pikmin around. Metroid Prime looks great too and has some “realistic” touches, but it still has a very distinct aesthetic, and it manages to do everything important with a relatively small vocabulary — even plentiful secrets.

I just don’t understand the game industry (and game culture)’s fanatical obsession with realistic graphics. They make games worse. It’s entirely possible to have an art style other than “get a lot of unpaid interns to model photos of rocks”, even for a mind-numbingly bland army man simulator. Please feel free to experiment a little more. I would love to see more weird and abstract worlds that follow their own rules and drag you down the rabbit hole with them.

Making it easier to deploy TPMTOTP on non-EFI systems

Post Syndicated from Matthew Garrett original http://mjg59.dreamwidth.org/41458.html

I’ve been working on TPMTOTP a little this weekend. I merged a pull request that adds command-line argument handling, which includes the ability to choose the set of PCRs you want to seal to without rebuilding the tools, and also lets you print the base32 encoding of the secret rather than the qr code so you can import it into a wider range of devices. More importantly it also adds support for setting the expected PCR values on the command line rather than reading them out of the TPM, so you can now re-seal the secret against new values before rebooting.

I also wrote some new code myself. TPMTOTP is designed to be usable in the initramfs, allowing you to validate system state before typing in your passphrase. Unfortunately the initramfs itself is one of the things that’s measured. So, you end up with something of a chicken and egg problem – TPMTOTP needs access to the secret, and the obvious thing to do is to put the secret in the initramfs. But the secret is sealed against the hash of the initramfs, and so you can’t generate the secret until after the initramfs. Modify the initramfs to insert the secret and you change the hash, so the secret is no longer released. Boo.

On EFI systems you can handle this by sticking the secret in an EFI variable (there’s some special-casing in the code to deal with the additional metadata on the front of things you read out of efivarfs). But that’s not terribly useful if you’re not on an EFI system. Thankfully, there’s a way around this. TPMs have a small quantity of nvram built into them, so we can stick the secret there. If you pass the -n argument to sealdata, that’ll happen. The unseal apps will attempt to pull the secret out of nvram before falling back to looking for a file, so things should just magically work.

I think it’s pretty feature complete now, other than TPM2 support? That’s on my list.

comment count unavailable comments

Will Spark Power the Data behind Precision Medicine?

Post Syndicated from Christopher Crosbie original https://blogs.aws.amazon.com/bigdata/post/Tx1GE3J0NATVJ39/Will-Spark-Power-the-Data-behind-Precision-Medicine

Christopher Crosbie is a Healthcare and Life Science Solutions Architect with Amazon Web Services.
This post was co-authored by Ujjwal Ratan, a Solutions Architect with Amazon Web Services.

———————————

“And that’s the promise of precision medicine — delivering the right treatments, at the right time, every time to the right person.“ (President Obama, 2015 State of the Union address)

The promise of precision medicine that President Obama envisions with this statement is a far-reaching goal that will require sweeping changes to the ways physicians treat patients, health data is collected, and global collaborative research is performed. Precision medicine typically describes an approach for treating and preventing disease that takes into account a patient’s individual variation in genes, lifestyle, and environment. Achieving this mission relies on the intersection of several technology innovations and a major restructuring of health data to focus on the genetic makeup of an individual.

The healthcare ecosystem has chosen a variety of tools and techniques for working with big data, but one tool that comes up again and again in many of the architectures we design and review is Spark on Amazon EMR.

Spark is already known for being a major player in big data analysis, but it is additionally uniquely capable in advancing genomics algorithms given the complex nature of genomics research. This post introduces gene analysis using Spark on EMR and ADAM, for those new to precision medicine.

Development of precision medicine

Data-driven tailored treatments have been commonplace for certain treatments like blood transfusions for a long time. Historically, however, most treatment plans are deeply subjective, due to the many disparate pieces of information that physicians must tie together to make a health plan based on the individual’s specific conditions.

To move past the idiosyncratic nature of most medical treatments, we need to amass properly collected and curated biological data to compare and correlate outcomes and biomarkers across varying patient populations.

The data of precision medicine inherently involves the data representation of large volumes of living, mobile, and irrationally complex humans. The recent blog post How The Healthcare of Tomorrow is Being Delivered Today detailed the ways in which AWS underlies some of the most advanced and innovative big data technologies in precision medicine. Technologies like these will enable the research and analysis of the data structures necessary to create true individualized care.

Genomics data sets require exploration

The study of genomics dates back much further than the Obama era. The field benefits from the results of a prodigious amount of research spanning Gregor Mendel’s pea pods in the 1860s to the Human Genome Project of the 1990s.

As with most areas of science, building knowledge often goes hand in hand with legacy analysis features that turn out to be outdated as the discipline evolves. The generation of scientists following Mendel, for example, significantly altered his calculations due to the wide adoption of the p-value in statistics.

The anachronism for many of the most common genomics algorithms today is the failure to properly optimize for cloud technology. Memory requirements are often limited to a single compute node or expect a POSIX file system. These tools may have been sufficient for the computing task involved in analyzing the genome of a single person. However, a shift to cloud computing will be necessary as we move to the full-scale population studies that will be required to develop novel methods in precision medicine.

Many existing genomics algorithms must be refactored to address the scale at which research is done today. The Broad Institute of MIT and Harvard, a leading genomic research center, recognized this and moved many of the algorithms that could be pulled apart and parallelized into a MapReduce paradigm within a Genome Analysis Toolkit (GATK). Despite the MapReduce migration and many Hadoop-based projects such as BioPig, the bioinformatics community did not fully embrace the Hadoop ecosystem due to the sequential nature of the reads and the overhead associated with splitting the MapReduce tasks.

Precision medicine is also going to rely heavily on referencing public data sets. Generally available, open data sets are often downloaded and copied among many researcher centers. These multiple copies of the same data create an inefficiency for researchers that is addressed through the AWS Public Data Sets program. This program allows researchers to leverage popular genomics data sets like TCGA and ICGC without having to pay for raw storage.

Migrating genetics to Spark on Amazon EMR

The transitioning of genomics algorithms that are popular today to Spark is one path that scientists are taking to capture the distributed processing capabilities of the cloud. However, precision medicine will require an abundance of exploration and new approaches. Many of these are already being built on top of Spark on EMR.

This is because Spark on EMR provides much of the functionality that aligns well with the goals of the precision medicine. For instance, using Amazon S3 as an extension of HDFS storage makes it easy to share the results of your analysis with collaborators from all over the world by simply allowing access to an S3 URL. It also provides the ability for researchers to adjust their cluster to the algorithm they are trying to build instead of adjusting their algorithm to the cluster to which they have access. Competition for compute resources with other cluster users is another drawback that can be mitigated with a move towards EMR.

The introduction of Spark has now overcome many of the previous limitations associated with parallelizing the data based on index files that comprise the standard Variant Control Files (VCF) and Binary Alignment Map (BAM) formats used by genomics researchers. The AMPLab at UC Berkeley, through projects like ADAM, demonstrated the value of a Spark infrastructure for genomics data processing.

Moreover, genomics scientists began identifying the need to get away from the undifferentiated heavy lifting of developing custom distributed system code. These developments motivated the Broad to develop the next version of the GATK with an option to run in the cloud on Spark. The upcoming GATK is a major step forward for the scientific community since it will soon be able to incorporate many of the features of EMR, such as on-demand cluster of various types and Amazon S3–backed storage.

Although Spark on EMR provides many infrastructure advantages, Spark still speaks the languages that are popular with the research community. Languages like SparkR on EMR make for an easy transition into the cloud. That way when there are issues that arise such as needing to unpack Gzip-compressed files to make them split-able, familiar R code can be used to make the transformation.

What’s under the hood of ADAM?

ADAM is an in-memory MapReduce system. ADAM is the result of contributions from universities, biotech, and pharma companies. It is entirely open source under the Apache 2 license. ADAM uses Apache Avro and Parquet file formats for data storage to provide a schema-based description of genomic data. This eliminates the dependencies on format-based libraries, which in the past has created incompatibilities.

Apache Parquet is a columnar storage format that is well suited for genomic data. The primary reason for this is because genomic data consists of a large number of similar data items that is align well with columnar storage. By using Apache Parquet as its storage format, ADAM makes querying and storing genomic data highly efficient. It limits the I/O to data actually needed by loading only the columns that need to be accessed. Because of its columnar nature, storing data in Parquet saves space as a result of better compression ratios.

The columnar Parquet file format enables up to 25% improvement in storage volume compared to compressed genomics file formats like BAM. Moreover, the in-memory caching using Spark and the ability to parallelize data processing over multiple nodes have been known to provide up to 50 times better performance on average on a 100-node cluster.

In addition to using the Parquet format for columnar storage, ADAM makes use of a new schema for genomics data referred to as bdg-formats, a project that provides schemas for describing common genomic data types such as variants, assemblies, and genotypes. It uses Apache Avro as its base framework and as a result, works well with common programming languages and platforms. By using the Apache Avro-based schema as its core data structure, workflows built using ADAM are flexible and easy to maintain using familiar technologies.

ADAM on Amazon EMR

The figure above represents a typical genomic workload designed on AWS using ADAM. File formats like SAM/BAM and VCF are uploaded as objects on Amazon S3.

Amazon EMR provides a way to run Spark compute jobs with ADAM and other applications on an as-needed basis while keeping the genomics data itself stored in separate, cheap object storage.

Your first genomics analysis with ADAM on Amazon EMR

To install ADAM on EMR, launch an EMR cluster from the AWS Management Console. Make sure you select the option to install Spark, as shown in the screen shot below.

While launching the EMR cluster, you should configure the EMR master security groups to allow you access to port 22 so you can SSH to the master node. The master node should be able to communicate to the Internet to download the necessary packages and build ADAM.

ADAM installation steps

During the creation of a cluster, a shell script known as a bootstrap action can automate the installation process of ADAM. In case you want to install ADAM manually, the following instructions walk through what the script does.

After the EMR cluster with Apache Spark is up and running, you can log in into the master node using SSH and install ADAM. ADAM requires Maven to build the packages; after you install Maven, you can clone ADAM from the ADAM GitHub repository.

SSH into the master node of the EMR cluster.

Install Maven by downloading the apache libraries from its website as shown below:

/* create a new directory to install maven */
mkdir maven
cd maven
/* download the maven zip from the apache mirror website */
echo "copying the maven zip file from the apache mirror"
wget http://apachemirror.ovidiudan.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
echo "unzipping the file"
tar -zxvf apache-maven-3.3.9-bin.tar.gz
/* export the “MAVEN_HOME” path */
echo "exporting MAVEN_HOME"
export PATH=$HOME/maven/apache-maven-3.3.9/bin:$PATH

Before cloning ADAM from GitHub, install GIT on the the master node of the EMR cluster. This can be done by running the following command:

/* install git in the home directory */
cd $HOME
sudo yum install git

Clone the ADAM repository from GitHub and begin your build.

/* clone the ADAM repository from GitHub and build the package */
git clone https://github.com/bigdatagenomics/adam.git
cd adam

Finally, begin your build

export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
mvn clean package -DskipTests

On completion, you see a message confirming that ADAM was built successfully.

Analysis of genomic data using ADAM

After installation, ADAM provides a set of functions to transform and analyze genomic data sets. The functions range from actions to count K-mers to conversion operations to convert file standard genomics formats like SAM, BAM or VCF to ADAM Parquet. A list of ADAM functions can be viewed by invoking adam-submit from the command line.

For the purposes of this analysis, you use a VCF file from an AWS public data set, specifically the 1000 genome project.

This is a project that aims to build the most detailed map of human genetic variation available. When complete, the 1000 genome project will have genomic data from over 2,661 people. Amazon S3 hosts the initial pilot data for this project in a public S3 bucket. The latest data set available to everyone hosts data for approximately 1700 people and is more than 200 TB in size.

From the master node of the EMR cluster, you can connect to the S3 bucket and download the file to the master node. The first step is to copy the file into HDFS so it can be accessed using ADAM. This can be done by running the following commands:

//copying a vcf file from S3 to master node
$ aws s3 cp s3://1000genomes/phase1/analysis_results/integrated_call_sets/ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf.gz /home/hadoop/

// Unzip the file
$ gunzip ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf.gz

//copying file to hdfs
$ hadoop fs -put /home/hadoop/ ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf /user/hadoop/

The command generates more than 690 ADAM Parquet files. The following screenshot shows a subset of these files.

You can now process the ADAM Parquet files as regular Parquet files in Apache Spark using Scala. ADAM provides its users its own shell, which is invoked from the command line using adam-shell. In this example, read the ADAM  Parquet file as a data frame and print its schema. Then, go on to register the data frame as a temp table and use it to query the genomic data sets.

val gnomeDF = sqlContext.read.parquet("/user/hadoop/adamfiles/ ")
gnomeDF.printSchema()
gnomeDF.registerTempTable("gnome")
val gnome_data = sqlContext.sql("select count(*) from gnome")
gnome_data.show()

Here’s how to use ADAM interactive shell to perform a reads count and K-mer count. A k-mer count returns all of the strings of length k from this DNA sequence. This is an important first step to understanding all of the possible DNA sequences (of length 20) that are contained in this file.

import org.bdgenomics.adam.rdd.ADAMContext._
val reads = sc.loadAlignments("part-r-00000.gz.parquet").cache()
reads.count()
val gnomeDF = sqlContext.read.parquet("part-r-00000.gz.parquet")
gnomeDF.printSchema()
println(reads.first)

/*
The following command cuts reads into _k_-mers, and then counts the number of
occurrences of each _k_-mer
*/
val Kmers = reads.adamCountKmers(20).cache()

kmers.count()

Conclusion  

Spark on Amazon EMR provides unique advantages over traditional genomic processing and may become a necessary tool as genomics moves into the scale of population-based studies required for precision medicine. Innovative companies like Human Longevity Inc. have discussed at re:Invent  how they use AWS tools such as Spark on EMR with Amazon Redshift to build a platform that is pushing precision medicine forward.

Of course, simply distributing compute resources will not solve all of the complexities associated with understanding the human condition. There is an old adage that reminds us that nine women cannot be used to make a baby in one month. Often in biology problems, there is a need to wait for one step to be completed before the next can be undergone. This procedure does not necessarily lend itself well to Spark, which benefits from distributing many small tasks at once. There are still many areas such as sequence assembly that may not have an easy transition to Spark. However, Spark on Amazon EMR will be a very interesting project to watch on our move toward precision medicine.

If you have questions or suggestions, please leave a comment below.

Special thanks to Angel Pizaro for his help on this post!

———————————–

Related

Extending Seven Bridges Genomics with Amazon Redshift and R

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

 

Will Spark Power the Data behind Precision Medicine?

Post Syndicated from Christopher Crosbie original https://blogs.aws.amazon.com/bigdata/post/Tx1GE3J0NATVJ39/Will-Spark-Power-the-Data-behind-Precision-Medicine

Christopher Crosbie is a Healthcare and Life Science Solutions Architect with Amazon Web Services.
This post was co-authored by Ujjwal Ratan, a Solutions Architect with Amazon Web Services.

———————————

“And that’s the promise of precision medicine — delivering the right treatments, at the right time, every time to the right person.“ (President Obama, 2015 State of the Union address)

The promise of precision medicine that President Obama envisions with this statement is a far-reaching goal that will require sweeping changes to the ways physicians treat patients, health data is collected, and global collaborative research is performed. Precision medicine typically describes an approach for treating and preventing disease that takes into account a patient’s individual variation in genes, lifestyle, and environment. Achieving this mission relies on the intersection of several technology innovations and a major restructuring of health data to focus on the genetic makeup of an individual.

The healthcare ecosystem has chosen a variety of tools and techniques for working with big data, but one tool that comes up again and again in many of the architectures we design and review is Spark on Amazon EMR.

Spark is already known for being a major player in big data analysis, but it is additionally uniquely capable in advancing genomics algorithms given the complex nature of genomics research. This post introduces gene analysis using Spark on EMR and ADAM, for those new to precision medicine.

Development of precision medicine

Data-driven tailored treatments have been commonplace for certain treatments like blood transfusions for a long time. Historically, however, most treatment plans are deeply subjective, due to the many disparate pieces of information that physicians must tie together to make a health plan based on the individual’s specific conditions.

To move past the idiosyncratic nature of most medical treatments, we need to amass properly collected and curated biological data to compare and correlate outcomes and biomarkers across varying patient populations.

The data of precision medicine inherently involves the data representation of large volumes of living, mobile, and irrationally complex humans. The recent blog post How The Healthcare of Tomorrow is Being Delivered Today detailed the ways in which AWS underlies some of the most advanced and innovative big data technologies in precision medicine. Technologies like these will enable the research and analysis of the data structures necessary to create true individualized care.

Genomics data sets require exploration

The study of genomics dates back much further than the Obama era. The field benefits from the results of a prodigious amount of research spanning Gregor Mendel’s pea pods in the 1860s to the Human Genome Project of the 1990s.

As with most areas of science, building knowledge often goes hand in hand with legacy analysis features that turn out to be outdated as the discipline evolves. The generation of scientists following Mendel, for example, significantly altered his calculations due to the wide adoption of the p-value in statistics.

The anachronism for many of the most common genomics algorithms today is the failure to properly optimize for cloud technology. Memory requirements are often limited to a single compute node or expect a POSIX file system. These tools may have been sufficient for the computing task involved in analyzing the genome of a single person. However, a shift to cloud computing will be necessary as we move to the full-scale population studies that will be required to develop novel methods in precision medicine.

Many existing genomics algorithms must be refactored to address the scale at which research is done today. The Broad Institute of MIT and Harvard, a leading genomic research center, recognized this and moved many of the algorithms that could be pulled apart and parallelized into a MapReduce paradigm within a Genome Analysis Toolkit (GATK). Despite the MapReduce migration and many Hadoop-based projects such as BioPig, the bioinformatics community did not fully embrace the Hadoop ecosystem due to the sequential nature of the reads and the overhead associated with splitting the MapReduce tasks.

Precision medicine is also going to rely heavily on referencing public data sets. Generally available, open data sets are often downloaded and copied among many researcher centers. These multiple copies of the same data create an inefficiency for researchers that is addressed through the AWS Public Data Sets program. This program allows researchers to leverage popular genomics data sets like TCGA and ICGC without having to pay for raw storage.

Migrating genetics to Spark on Amazon EMR

The transitioning of genomics algorithms that are popular today to Spark is one path that scientists are taking to capture the distributed processing capabilities of the cloud. However, precision medicine will require an abundance of exploration and new approaches. Many of these are already being built on top of Spark on EMR.

This is because Spark on EMR provides much of the functionality that aligns well with the goals of the precision medicine. For instance, using Amazon S3 as an extension of HDFS storage makes it easy to share the results of your analysis with collaborators from all over the world by simply allowing access to an S3 URL. It also provides the ability for researchers to adjust their cluster to the algorithm they are trying to build instead of adjusting their algorithm to the cluster to which they have access. Competition for compute resources with other cluster users is another drawback that can be mitigated with a move towards EMR.

The introduction of Spark has now overcome many of the previous limitations associated with parallelizing the data based on index files that comprise the standard Variant Control Files (VCF) and Binary Alignment Map (BAM) formats used by genomics researchers. The AMPLab at UC Berkeley, through projects like ADAM, demonstrated the value of a Spark infrastructure for genomics data processing.

Moreover, genomics scientists began identifying the need to get away from the undifferentiated heavy lifting of developing custom distributed system code. These developments motivated the Broad to develop the next version of the GATK with an option to run in the cloud on Spark. The upcoming GATK is a major step forward for the scientific community since it will soon be able to incorporate many of the features of EMR, such as on-demand cluster of various types and Amazon S3–backed storage.

Although Spark on EMR provides many infrastructure advantages, Spark still speaks the languages that are popular with the research community. Languages like SparkR on EMR make for an easy transition into the cloud. That way when there are issues that arise such as needing to unpack Gzip-compressed files to make them split-able, familiar R code can be used to make the transformation.

What’s under the hood of ADAM?

ADAM is an in-memory MapReduce system. ADAM is the result of contributions from universities, biotech, and pharma companies. It is entirely open source under the Apache 2 license. ADAM uses Apache Avro and Parquet file formats for data storage to provide a schema-based description of genomic data. This eliminates the dependencies on format-based libraries, which in the past has created incompatibilities.

Apache Parquet is a columnar storage format that is well suited for genomic data. The primary reason for this is because genomic data consists of a large number of similar data items that is align well with columnar storage. By using Apache Parquet as its storage format, ADAM makes querying and storing genomic data highly efficient. It limits the I/O to data actually needed by loading only the columns that need to be accessed. Because of its columnar nature, storing data in Parquet saves space as a result of better compression ratios.

The columnar Parquet file format enables up to 25% improvement in storage volume compared to compressed genomics file formats like BAM. Moreover, the in-memory caching using Spark and the ability to parallelize data processing over multiple nodes have been known to provide up to 50 times better performance on average on a 100-node cluster.

In addition to using the Parquet format for columnar storage, ADAM makes use of a new schema for genomics data referred to as bdg-formats, a project that provides schemas for describing common genomic data types such as variants, assemblies, and genotypes. It uses Apache Avro as its base framework and as a result, works well with common programming languages and platforms. By using the Apache Avro-based schema as its core data structure, workflows built using ADAM are flexible and easy to maintain using familiar technologies.

ADAM on Amazon EMR

The figure above represents a typical genomic workload designed on AWS using ADAM. File formats like SAM/BAM and VCF are uploaded as objects on Amazon S3.

Amazon EMR provides a way to run Spark compute jobs with ADAM and other applications on an as-needed basis while keeping the genomics data itself stored in separate, cheap object storage.

Your first genomics analysis with ADAM on Amazon EMR

To install ADAM on EMR, launch an EMR cluster from the AWS Management Console. Make sure you select the option to install Spark, as shown in the screen shot below.

While launching the EMR cluster, you should configure the EMR master security groups to allow you access to port 22 so you can SSH to the master node. The master node should be able to communicate to the Internet to download the necessary packages and build ADAM.

ADAM installation steps

During the creation of a cluster, a shell script known as a bootstrap action can automate the installation process of ADAM. In case you want to install ADAM manually, the following instructions walk through what the script does.

After the EMR cluster with Apache Spark is up and running, you can log in into the master node using SSH and install ADAM. ADAM requires Maven to build the packages; after you install Maven, you can clone ADAM from the ADAM GitHub repository.

SSH into the master node of the EMR cluster.

Install Maven by downloading the apache libraries from its website as shown below:

/* create a new directory to install maven */
mkdir maven
cd maven
/* download the maven zip from the apache mirror website */
echo "copying the maven zip file from the apache mirror"
wget http://apachemirror.ovidiudan.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
echo "unzipping the file"
tar -zxvf apache-maven-3.3.9-bin.tar.gz
/* export the “MAVEN_HOME” path */
echo "exporting MAVEN_HOME"
export PATH=$HOME/maven/apache-maven-3.3.9/bin:$PATH

Before cloning ADAM from GitHub, install GIT on the the master node of the EMR cluster. This can be done by running the following command:

/* install git in the home directory */
cd $HOME
sudo yum install git

Clone the ADAM repository from GitHub and begin your build.

/* clone the ADAM repository from GitHub and build the package */
git clone https://github.com/bigdatagenomics/adam.git
cd adam

Finally, begin your build

export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
mvn clean package -DskipTests

On completion, you see a message confirming that ADAM was built successfully.

Analysis of genomic data using ADAM

After installation, ADAM provides a set of functions to transform and analyze genomic data sets. The functions range from actions to count K-mers to conversion operations to convert file standard genomics formats like SAM, BAM or VCF to ADAM Parquet. A list of ADAM functions can be viewed by invoking adam-submit from the command line.

For the purposes of this analysis, you use a VCF file from an AWS public data set, specifically the 1000 genome project.

This is a project that aims to build the most detailed map of human genetic variation available. When complete, the 1000 genome project will have genomic data from over 2,661 people. Amazon S3 hosts the initial pilot data for this project in a public S3 bucket. The latest data set available to everyone hosts data for approximately 1700 people and is more than 200 TB in size.

From the master node of the EMR cluster, you can connect to the S3 bucket and download the file to the master node. The first step is to copy the file into HDFS so it can be accessed using ADAM. This can be done by running the following commands:

//copying a vcf file from S3 to master node
$ aws s3 cp s3://1000genomes/phase1/analysis_results/integrated_call_sets/ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf.gz /home/hadoop/

// Unzip the file
$ gunzip ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf.gz

//copying file to hdfs
$ hadoop fs -put /home/hadoop/ ALL.chr1.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf /user/hadoop/

The command generates more than 690 ADAM Parquet files. The following screenshot shows a subset of these files.

You can now process the ADAM Parquet files as regular Parquet files in Apache Spark using Scala. ADAM provides its users its own shell, which is invoked from the command line using adam-shell. In this example, read the ADAM  Parquet file as a data frame and print its schema. Then, go on to register the data frame as a temp table and use it to query the genomic data sets.

val gnomeDF = sqlContext.read.parquet("/user/hadoop/adamfiles/ ")
gnomeDF.printSchema()
gnomeDF.registerTempTable("gnome")
val gnome_data = sqlContext.sql("select count(*) from gnome")
gnome_data.show()

Here’s how to use ADAM interactive shell to perform a reads count and K-mer count. A k-mer count returns all of the strings of length k from this DNA sequence. This is an important first step to understanding all of the possible DNA sequences (of length 20) that are contained in this file.

import org.bdgenomics.adam.rdd.ADAMContext._
val reads = sc.loadAlignments("part-r-00000.gz.parquet").cache()
reads.count()
val gnomeDF = sqlContext.read.parquet("part-r-00000.gz.parquet")
gnomeDF.printSchema()
println(reads.first)

/*
The following command cuts reads into _k_-mers, and then counts the number of
occurrences of each _k_-mer
*/
val Kmers = reads.adamCountKmers(20).cache()

kmers.count()

Conclusion  

Spark on Amazon EMR provides unique advantages over traditional genomic processing and may become a necessary tool as genomics moves into the scale of population-based studies required for precision medicine. Innovative companies like Human Longevity Inc. have discussed at re:Invent  how they use AWS tools such as Spark on EMR with Amazon Redshift to build a platform that is pushing precision medicine forward.

Of course, simply distributing compute resources will not solve all of the complexities associated with understanding the human condition. There is an old adage that reminds us that nine women cannot be used to make a baby in one month. Often in biology problems, there is a need to wait for one step to be completed before the next can be undergone. This procedure does not necessarily lend itself well to Spark, which benefits from distributing many small tasks at once. There are still many areas such as sequence assembly that may not have an easy transition to Spark. However, Spark on Amazon EMR will be a very interesting project to watch on our move toward precision medicine.

If you have questions or suggestions, please leave a comment below.

Special thanks to Angel Pizaro for his help on this post!

———————————–

Related

Extending Seven Bridges Genomics with Amazon Redshift and R

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

 

How to Detect and Automatically Revoke Unintended IAM Access with Amazon CloudWatch Events

Post Syndicated from Mustafa Torun original https://blogs.aws.amazon.com/security/post/Tx2XGINRKCK9XNS/How-to-Detect-and-Automatically-Revoke-Unintended-IAM-Access-with-Amazon-CloudWa

AWS Identity and Access Management (IAM) enables you to create IAM users and roles in your account, each with a specific set of permissions. For example, you can create administrative users who have access to all AWS APIs (also called actions), and you can create other users who have access to only a specific subset of those APIs. In the case of IAM APIs, only your administrative users should have access to them.

If your account is shared across departments in your organization, monitoring the permissions of yocompur users can become a challenge as the number of users grows. For example, what if a user is granted unintended IAM API access and the user begins making API calls? In this post, I will show a solution that detects API callers who should not have IAM access and automatically revokes those permissions with the help of Amazon CloudWatch Events.

Solution overview

CloudWatch Events enables you to create rules to deliver events (including API call events) to targets such as AWS Lambda functions, Amazon Kinesis streams, and Amazon SNS topics. As the following diagram of this solution shows, (1) IAM sends API call events (including the ones that failed with a 4xx client error) to CloudWatch Events. Then, (2) a CloudWatch Events rule I create delivers all the calls to IAM APIs as events to a Lambda function. If the caller is not part of the admins group, (3) the Lambda function attaches a policy to the IAM user that denies (and effectively revokes) IAM access for the user.

To automatically detect and revoke unintended IAM API access, I will:

  1. Create an IAM policy that will deny access to any IAM API.
  2. Create an IAM role that allows access to IAM APIs in order to use it in my Lambda function as an execution role.
  3. Create a Lambda function that will receive the API call event, check whether it represents a policy violation, and, if so, assume the role created in the previous step. The function then can make IAM calls, and attach the policy created in Step 1 to the user who should not have IAM access.
  4. Create a CloudWatch Events rule that matches AWS API call events and invokes the Lambda function created in the previous step.

Note that this process is a reactive approach (a complement to the proactive approach) for “just in case” situations. It is one way to make sure that all IAM users are created with locked-down policies and the admin group reviews any change.

The rest of this blog post details the steps of this solution’s deployment.

Deploying the solution

I will follow the four solution steps in this section to use CloudWatch Events to detect and automatically revoke IAM access for a user who should not have had IAM access, starting with the creation of an IAM policy that denies access to any IAM API.

Step 1: Create an IAM policy that will deny access to any IAM API

I start by creating an IAM policy that denies access to IAM. Note that explicit Deny policies have precedence over explicit Allow policies. For further details about IAM policy evaluation logic, see IAM Policy Evaluation Logic.

First, I put the following policy document into a file named access_policy1.json.

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "DenyIAMAccess",
       "Effect": "Deny",
       "Action": "iam:*",
       "Resource": "*"
     }
   ]
}

Then, I run the following AWS CLI command. (To learn more about setting up AWS CLI, see Getting Set Up with the AWS Command Line Interface.)

$ aws iam create-policy 
--policy-name DenyIAMAccess 
--policy-document file://access_policy1.json 

That is all it takes to create an IAM policy that denies access to IAM. However, if I wanted to use the IAM console instead to create the IAM policy, I would have done the following:

  1. Go to IAM console and click Policies on the left pane.
  2. Click Create Policy and click Select next to Policy Generator.
  3. On the next page:

    1. For Effect, select Deny.
    2. For AWS Service, select AWS Identity and Access Management.
    3. For Actions, select All Actions.
    4. For Amazon Resource Name (ARN), type * (an asterisk).
  4. Click Add Statement, and then click Next Step.
  5. Name the new policy, and then click Create Policy to finish.

Step 2: Create an IAM role for the Lambda function

In this step, I create an execution IAM role for my Lambda function. The role will allow Lambda to perform IAM actions on my behalf while the function is being executed.

In the Lambda function, I use the AttachUserPolicy, ListGroupsForUser, and PutUserPolicy IAM APIs, and Lambda uses CreateLogGroup, CreateLogStream, and PutLogEvents  CloudWatch Logs APIs to write logs while the Lambda function is running. I put the following access policy document in a file named access_policy2.json.

{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "AllowIAMForLambdaPolicy",
       "Effect": "Allow",
       "Action": [
         "iam:AttachUserPolicy",
         "iam:ListGroupsForUser",
         "iam:PutUserPolicy",
         "logs:CreateLogGroup",
         "logs:CreateLogStream",
         "logs:PutLogEvents"
       ],
       "Resource": "*"
     }
   ]
}

Next, I put the following trust policy document in a file named trust_policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Finally, I run the following CLI commands to create my role. Here and throughout this post, remember to replace the placeholder account ID with your own account ID.                                               

$ aws iam create-policy 
--policy-name AllowIAMAndLogsForLambdaPolicy 
--policy-document file://access_policy2.json

$ aws iam create-role 
--role-name RoleThatAllowsIAMAndLogsForLambda 
--assume-role-policy-document file://trust_policy.json

$ aws iam attach-role-policy 
--role-name RoleThatAllowsIAMandLogsforLambda 
--policy-arn arn:aws:iam::123456789012:policy/AllowIAMAndLogsForLambdaPolicy

If you would prefer to use the IAM console to create the IAM role, do the following:

  1. Create an IAM policy with the access policy document shown at the beginning of this step (the policy-creation steps are the same steps shown in Step 1).
  2. In the left pane of the IAM console, click Roles. On the next page, click Create New Role.
  3. After you name the role and click Next Step, on the Select Role Type page, click Select next to AWS Lambda. Click Next Step.
  4. On the Attach Policy page, select the new IAM policy you just created, and then click Next Step.
  5. On the Review page, review the new IAM role you created, and then click Create Role to finish.

Step 3: Create the Lambda function

In this step, I create the Lambda function that will process the event and decide whether there is a violation. An AWS API call event in CloudWatch Events looks like the following (I omitted the request and response bodies to save space).

{
   "version": "0",
   "id": "4223e154-e763-4770-8108-83b7461eb51b",
   "detail-type": "AWS API Call via CloudTrail",
   "source": "aws.iam",
   "account": "123456789012",
   "time": "2016-01-21T17:22:36Z",
   "region": "us-east-1",
   "resources": [],
   "detail": {
     "eventVersion": "1.02",
     "userIdentity": {
       "type": "IAMUser",
       "principalId": "AIDACKCEVSQ6C2EXAMPLE",
       "arn": "arn:aws:iam::123456789012:user/some-user",
       "accountId": "123456789012",
       "accessKeyId": "AKIAIOSFODNN7EXAMPLE”,
       "userName": "some-user"
     },
     ...
   }
}

I will use the following Lambda function that revokes IAM access for any user that is not part of the admins group.

var aws = require('aws-sdk');
var iam = new aws.IAM();

// ARN of the policy that denies all IAM access
var policyArn = 'arn:aws:iam::123456789012:policy/DenyIAMAccess';

// Name of the group for admin users
var adminGroupName = 'admins';

exports.handler = function(event, context) {
     // Print the incoming Amazon CloudWatch Events event.
     console.log('Received event:', JSON.stringify(event, null, 2));

     // If the caller is not an IAM user, do nothing.
     if (event.detail.userIdentity.type != 'IAMUser') {
         console.log('This call was not made by an IAM user!');
         context.done();
     } else {
         // Get the user name from the event.
         var userName = event.detail.userIdentity.userName;    

         // List the groups for the user. If the user is not part of the
         // 'admins' group, revoke their IAM access.
         iam.listGroupsForUser({UserName: userName}, function(error, data) {
             if (error) {
                 console.log(error);
                 context.fail('Could not list groups for the user!');
             } else {
                 for (var i = 0; i < data.Groups.length; i++) {
                     if (data.Groups[i].GroupName == adminGroupName) {
                         console.log('IAM user ' + userName + ' is part of the admins group.');
                         context.done()
                     }
                 }
                 revokeAccess();
             }
         });
     }

     // Revoke IAM access by attaching a Deny policy to the unauthorized
     // IAM user. If the IAM user already has more than the allowed number
     // of managed policies attached, add an inline policy to
     // deny access.
     function revokeAccess() {
         iam.attachUserPolicy(
             {UserName: userName, PolicyArn: policyArn},
             function(error, data) {
                 if (error) {
                     if (error.code == 'LimitExceeded') {
                         revokeAccessByInlinePolicy();
                     } else {
                         console.log(error);
                     }
                 } else {
                     console.log('Revoked IAM access for IAM user ' + userName);
                     context.done(); 
                 }
             }
         );
     }

     // Revoke IAM access by adding an inline policy.
     function revokeAccessByInlinePolicy() {
         var policyDocument =
             '{' +
                 '"Version": "2012-10-17",' +
                 '"Statement": [' +
                     '{' +
                         '"Effect": "Deny",' +
                         '"Action": "iam:*",' +
                         '"Resource": "*"' +
                     '}' +
                 ']' +
             '}';

         iam.putUserPolicy(
             {UserName: userName, PolicyName: 'DenyIAM', PolicyDocument: policyDocument},
             function(error, data) {
                 if (error) {
                     console.log(error);
                     context.fail('Failed to add inline DenyIAM policy!');
                 } else {
                     console.log('Revoked access via inline policy for IAM user ' + userName);
                     context.done(); 
                 }
             }
         );
     }
}

To create the preceding Lambda function, do the following:

  1. Go to the Lambda console and ensure you are in the US East (N. Virginia) Region (us-east-1). IAM is a global service and its AWS API call events are only available in that region.
  2. Click Create a Lambda Function, and then select the hello-world blueprint. (You can use the Filter box to search for the blueprint, if it does not appear on your first page of blueprints.)
  3. Give the function a name and a description. I named mine RevokeIAMAccess.
  4. Select Node.js for Runtime.
  5. Copy the code in the preceding Lambda function and paste it in the Lambda function code editor.
  6. In the Lambda function handler and role section, select from the drop-down list the IAM role you created earlier in this post in Step 2. I selected RoleThatAllowsIAMAndLogsforLambda.
  7. Increase Timeout to 30 seconds.
  8. To finish, click Next and then click Create Function.

Step 4: Create a CloudWatch Events rule

Now, I will create the CloudWatch Events rule that will be triggered when an AWS API call event is received from IAM. I will use AWS CLI to create the rule. A rule in CloudWatch Events must have an event pattern. Event patterns lets you define the pattern to look for in an incoming event. If the pattern matches the event, the rule is triggered. First, I put the following event pattern in a file named event_pattern1.json.

{
   "detail-type": [
     "AWS API Call via CloudTrail"
   ],
   "detail": {
     "eventSource": [
       "iam.amazonaws.com"
     ]
   }
}

Next, I run the following AWS CLI commands to create the CloudWatch Events rule, and add the Lambda function to it as a target.

$ aws events put-rule 
--name DetectIAMCallsAndRevokeAccess 
--event-pattern file://event_pattern1.json

$ aws events put-targets 
--rule DetectIAMCallsAndRevokeAccess 
--targets Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:RevokeIAMAccess

Finally, I run the following AWS CLI command to allow CloudWatch Events to invoke my Lambda function for me.

$ aws lambda add-permission 
--function-name RevokeIAMAccess 
--statement-id AllowCloudWatchEventsToInvoke 
--action 'lambda:InvokeFunction' 
--principal events.amazonaws.com 
--source-arn arn:aws:events:us-east-1:123456789012:rule/DetectIAMCallsAndRevokeAccess

If you would prefer to use the CloudWatch Events console to create this rule, do the following:

  1. Select the US East (N. Virginia) Region (us-east-1). Go to the CloudWatch console, and click Rules under Events in the left pane.
  2. Click Create rule. Select AWS API Call from the first Event selector drop-down list. If you see a warning that you don’t have CloudTrail events turned on, this means the AWS API call events are not available for your account in CloudWatch Events. See Creating and Updating Your Trail to enable it.
  3. Select IAM from the Service name drop-down list. Leave the Any operation option selected.
  4. Click Add target under Targets, and then select Lambda function from the first drop-down list. From the Function drop-down list, select the function you created earlier that revokes the IAM access.
  5. Leave the Matched event option selected under Configure input.
  6. Click Configure details. Name the rule and add a description of the rule. I named my rule DetectIAMCallsAndRevokeAccess.

From now on, whenever a user who was given unintended access to IAM APIs makes an API call against IAM, this setup will detect those events and automatically revoke the user’s IAM access. Note that the new IAM policy might not take effect for a few minutes after you create it, because of eventual consistency. Therefore, it is possible that a user will continue to be able to make successive calls to IAM for a short time after the first detection.

The power of the event pattern: How to reduce the number of calls to the Lambda function

Let me revisit the Lambda function I created in Step 3. It has the following If statement so that it processes only API call events originated by an API call made against IAM by a user.

// If the caller is not IAM user then do nothing.
if (event.detail.userIdentity.type!= 'IAMUser') {
    context.succeed('This call was not made by an IAM user!');
}

I can delete this statement from my Lambda function and let CloudWatch Events do the work for me by using the following event pattern in my rule.

{
   "detail-type": [
     "AWS API Call via CloudTrail"
   ],
   "detail": {
     "eventSource": [
       "iam.amazonaws.com"
     ],
     "userIdentity": {
       "type": [
         "IAMUser"
       ]
     }
   }
}

This event pattern ensures that my rule matches only AWS API call events that are made against IAM by an IAM user in my account. This will reduce the number of calls to my Lambda function. To learn more about event patterns, see Events and Event Patterns.

Conclusion

In this post, I have shown how you can help detect unintended IAM access and automatically revoke it with the help of Amazon CloudWatch Events. Keep in mind that IAM API call events and Lambda functions are only a small set of the events and targets that are available in CloudWatch Events. To learn more, see Using CloudWatch Events.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about this post or how to implement the solution described, please start a new thread on the CloudWatch forum.

– Mustafa

How to Detect and Automatically Revoke Unintended IAM Access with Amazon CloudWatch Events

Post Syndicated from Mustafa Torun original https://blogs.aws.amazon.com/security/post/Tx2XGINRKCK9XNS/How-to-Detect-and-Automatically-Revoke-Unintended-IAM-Access-with-Amazon-CloudWa

AWS Identity and Access Management (IAM) enables you to create IAM users and roles in your account, each with a specific set of permissions. For example, you can create administrative users who have access to all AWS APIs (also called actions), and you can create other users who have access to only a specific subset of those APIs. In the case of IAM APIs, only your administrative users should have access to them.

If your account is shared across departments in your organization, monitoring the permissions of yocompur users can become a challenge as the number of users grows. For example, what if a user is granted unintended IAM API access and the user begins making API calls? In this post, I will show a solution that detects API callers who should not have IAM access and automatically revokes those permissions with the help of Amazon CloudWatch Events.

Solution overview

CloudWatch Events enables you to create rules to deliver events (including API call events) to targets such as AWS Lambda functions, Amazon Kinesis streams, and Amazon SNS topics. As the following diagram of this solution shows, (1) IAM sends API call events (including the ones that failed with a 4xx client error) to CloudWatch Events. Then, (2) a CloudWatch Events rule I create delivers all the calls to IAM APIs as events to a Lambda function. If the caller is not part of the admins group, (3) the Lambda function attaches a policy to the IAM user that denies (and effectively revokes) IAM access for the user.

To automatically detect and revoke unintended IAM API access, I will:

Create an IAM policy that will deny access to any IAM API.

Create an IAM role that allows access to IAM APIs in order to use it in my Lambda function as an execution role.

Create a Lambda function that will receive the API call event, check whether it represents a policy violation, and, if so, assume the role created in the previous step. The function then can make IAM calls, and attach the policy created in Step 1 to the user who should not have IAM access.

Create a CloudWatch Events rule that matches AWS API call events and invokes the Lambda function created in the previous step.

Note that this process is a reactive approach (a complement to the proactive approach) for “just in case” situations. It is one way to make sure that all IAM users are created with locked-down policies and the admin group reviews any change.

The rest of this blog post details the steps of this solution’s deployment.

Deploying the solution

I will follow the four solution steps in this section to use CloudWatch Events to detect and automatically revoke IAM access for a user who should not have had IAM access, starting with the creation of an IAM policy that denies access to any IAM API.

Step 1: Create an IAM policy that will deny access to any IAM API

I start by creating an IAM policy that denies access to IAM. Note that explicit Deny policies have precedence over explicit Allow policies. For further details about IAM policy evaluation logic, see IAM Policy Evaluation Logic.

First, I put the following policy document into a file named access_policy1.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyIAMAccess",
      "Effect": "Deny",
      "Action": "iam:*",
      "Resource": "*"
    }
  ]
}

Then, I run the following AWS CLI command. (To learn more about setting up AWS CLI, see Getting Set Up with the AWS Command Line Interface.)

$ aws iam create-policy
–policy-name DenyIAMAccess
–policy-document file://access_policy1.json

That is all it takes to create an IAM policy that denies access to IAM. However, if I wanted to use the IAM console instead to create the IAM policy, I would have done the following:

Go to IAM console and click Policies on the left pane.

Click Create Policy and click Select next to Policy Generator.

On the next page:

For Effect, select Deny.

For AWS Service, select AWS Identity and Access Management.

For Actions, select All Actions.

For Amazon Resource Name (ARN), type * (an asterisk).

Click Add Statement, and then click Next Step.

Name the new policy, and then click Create Policy to finish.

Step 2: Create an IAM role for the Lambda function

In this step, I create an execution IAM role for my Lambda function. The role will allow Lambda to perform IAM actions on my behalf while the function is being executed.

In the Lambda function, I use the AttachUserPolicy, ListGroupsForUser, and PutUserPolicy IAM APIs, and Lambda uses CreateLogGroup, CreateLogStream, and PutLogEvents  CloudWatch Logs APIs to write logs while the Lambda function is running. I put the following access policy document in a file named access_policy2.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowIAMForLambdaPolicy",
      "Effect": "Allow",
      "Action": [
        "iam:AttachUserPolicy",
        "iam:ListGroupsForUser",
        "iam:PutUserPolicy",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Next, I put the following trust policy document in a file named trust_policy.json.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}

Finally, I run the following CLI commands to create my role. Here and throughout this post, remember to replace the placeholder account ID with your own account ID.                                               

$ aws iam create-policy
–policy-name AllowIAMAndLogsForLambdaPolicy
–policy-document file://access_policy2.json

$ aws iam create-role
–role-name RoleThatAllowsIAMAndLogsForLambda
–assume-role-policy-document file://trust_policy.json

$ aws iam attach-role-policy
–role-name RoleThatAllowsIAMandLogsforLambda
–policy-arn arn:aws:iam::123456789012:policy/AllowIAMAndLogsForLambdaPolicy

If you would prefer to use the IAM console to create the IAM role, do the following:

Create an IAM policy with the access policy document shown at the beginning of this step (the policy-creation steps are the same steps shown in Step 1).

In the left pane of the IAM console, click Roles. On the next page, click Create New Role.

After you name the role and click Next Step, on the Select Role Type page, click Select next to AWS Lambda. Click Next Step.

On the Attach Policy page, select the new IAM policy you just created, and then click Next Step.

On the Review page, review the new IAM role you created, and then click Create Role to finish.

Step 3: Create the Lambda function

In this step, I create the Lambda function that will process the event and decide whether there is a violation. An AWS API call event in CloudWatch Events looks like the following (I omitted the request and response bodies to save space).

{
  "version": "0",
  "id": "4223e154-e763-4770-8108-83b7461eb51b",
  "detail-type": "AWS API Call via CloudTrail",
  "source": "aws.iam",
  "account": "123456789012",
  "time": "2016-01-21T17:22:36Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "eventVersion": "1.02",
    "userIdentity": {
      "type": "IAMUser",
      "principalId": "AIDACKCEVSQ6C2EXAMPLE",
      "arn": "arn:aws:iam::123456789012:user/some-user",
      "accountId": "123456789012",
      "accessKeyId": "AKIAIOSFODNN7EXAMPLE”,
      "userName": "some-user"
    },
    …
  }
}

I will use the following Lambda function that revokes IAM access for any user that is not part of the admins group.

var aws = require(‘aws-sdk’);
var iam = new aws.IAM();

// ARN of the policy that denies all IAM access
var policyArn = ‘arn:aws:iam::123456789012:policy/DenyIAMAccess’;

// Name of the group for admin users
var adminGroupName = ‘admins’;

exports.handler = function(event, context) {
// Print the incoming Amazon CloudWatch Events event.
console.log(‘Received event:’, JSON.stringify(event, null, 2));

// If the caller is not an IAM user, do nothing.
if (event.detail.userIdentity.type != ‘IAMUser’) {
console.log(‘This call was not made by an IAM user!’);
context.done();
} else {
// Get the user name from the event.
var userName = event.detail.userIdentity.userName;

// List the groups for the user. If the user is not part of the
// ‘admins’ group, revoke their IAM access.
iam.listGroupsForUser({UserName: userName}, function(error, data) {
if (error) {
console.log(error);
context.fail(‘Could not list groups for the user!’);
} else {
for (var i = 0; i < data.Groups.length; i++) {
if (data.Groups[i].GroupName == adminGroupName) {
console.log(‘IAM user ‘ + userName + ‘ is part of the admins group.’);
context.done()
}
}
revokeAccess();
}
});
}

// Revoke IAM access by attaching a Deny policy to the unauthorized
// IAM user. If the IAM user already has more than the allowed number
// of managed policies attached, add an inline policy to
// deny access.
function revokeAccess() {
iam.attachUserPolicy(
{UserName: userName, PolicyArn: policyArn},
function(error, data) {
if (error) {
if (error.code == ‘LimitExceeded’) {
revokeAccessByInlinePolicy();
} else {
console.log(error);
}
} else {
console.log(‘Revoked IAM access for IAM user ‘ + userName);
context.done();
}
}
);
}

// Revoke IAM access by adding an inline policy.
function revokeAccessByInlinePolicy() {
var policyDocument =
‘{‘ +
‘"Version": "2012-10-17",’ +
‘"Statement": [‘ +
‘{‘ +
‘"Effect": "Deny",’ +
‘"Action": "iam:*",’ +
‘"Resource": "*"’ +
‘}’ +
‘]’ +
‘}’;

iam.putUserPolicy(
{UserName: userName, PolicyName: ‘DenyIAM’, PolicyDocument: policyDocument},
function(error, data) {
if (error) {
console.log(error);
context.fail(‘Failed to add inline DenyIAM policy!’);
} else {
console.log(‘Revoked access via inline policy for IAM user ‘ + userName);
context.done();
}
}
);
}
}

To create the preceding Lambda function, do the following:

Go to the Lambda console and ensure you are in the US East (N. Virginia) Region (us-east-1). IAM is a global service and its AWS API call events are only available in that region.

Click Create a Lambda Function, and then select the hello-world blueprint. (You can use the Filter box to search for the blueprint, if it does not appear on your first page of blueprints.)

Give the function a name and a description. I named mine RevokeIAMAccess.

Select Node.js for Runtime.

Copy the code in the preceding Lambda function and paste it in the Lambda function code editor.

In the Lambda function handler and role section, select from the drop-down list the IAM role you created earlier in this post in Step 2. I selected RoleThatAllowsIAMAndLogsforLambda.

Increase Timeout to 30 seconds.

To finish, click Next and then click Create Function.

Step 4: Create a CloudWatch Events rule

Now, I will create the CloudWatch Events rule that will be triggered when an AWS API call event is received from IAM. I will use AWS CLI to create the rule. A rule in CloudWatch Events must have an event pattern. Event patterns lets you define the pattern to look for in an incoming event. If the pattern matches the event, the rule is triggered. First, I put the following event pattern in a file named event_pattern1.json.

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "iam.amazonaws.com"
    ]
  }
}

Next, I run the following AWS CLI commands to create the CloudWatch Events rule, and add the Lambda function to it as a target.

$ aws events put-rule
–name DetectIAMCallsAndRevokeAccess
–event-pattern file://event_pattern1.json

$ aws events put-targets
–rule DetectIAMCallsAndRevokeAccess
–targets Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:RevokeIAMAccess

Finally, I run the following AWS CLI command to allow CloudWatch Events to invoke my Lambda function for me.

$ aws lambda add-permission
–function-name RevokeIAMAccess
–statement-id AllowCloudWatchEventsToInvoke
–action ‘lambda:InvokeFunction’
–principal events.amazonaws.com
–source-arn arn:aws:events:us-east-1:123456789012:rule/DetectIAMCallsAndRevokeAccess

If you would prefer to use the CloudWatch Events console to create this rule, do the following:

Select the US East (N. Virginia) Region (us-east-1). Go to the CloudWatch console, and click Rules under Events in the left pane.

Click Create rule. Select AWS API Call from the first Event selector drop-down list. If you see a warning that you don’t have CloudTrail events turned on, this means the AWS API call events are not available for your account in CloudWatch Events. See Creating and Updating Your Trail to enable it.

Select IAM from the Service name drop-down list. Leave the Any operation option selected.

Click Add target under Targets, and then select Lambda function from the first drop-down list. From the Function drop-down list, select the function you created earlier that revokes the IAM access.

Leave the Matched event option selected under Configure input.

Click Configure details. Name the rule and add a description of the rule. I named my rule DetectIAMCallsAndRevokeAccess.

From now on, whenever a user who was given unintended access to IAM APIs makes an API call against IAM, this setup will detect those events and automatically revoke the user’s IAM access. Note that the new IAM policy might not take effect for a few minutes after you create it, because of eventual consistency. Therefore, it is possible that a user will continue to be able to make successive calls to IAM for a short time after the first detection.

The power of the event pattern: How to reduce the number of calls to the Lambda function

Let me revisit the Lambda function I created in Step 3. It has the following If statement so that it processes only API call events originated by an API call made against IAM by a user.

// If the caller is not IAM user then do nothing.
if (event.detail.userIdentity.type!= ‘IAMUser’) {
context.succeed(‘This call was not made by an IAM user!’);
}

I can delete this statement from my Lambda function and let CloudWatch Events do the work for me by using the following event pattern in my rule.

{
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"iam.amazonaws.com"
],
"userIdentity": {
"type": [
"IAMUser"
]
}
}
}

This event pattern ensures that my rule matches only AWS API call events that are made against IAM by an IAM user in my account. This will reduce the number of calls to my Lambda function. To learn more about event patterns, see Events and Event Patterns.

Conclusion

In this post, I have shown how you can help detect unintended IAM access and automatically revoke it with the help of Amazon CloudWatch Events. Keep in mind that IAM API call events and Lambda functions are only a small set of the events and targets that are available in CloudWatch Events. To learn more, see Using CloudWatch Events.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about this post or how to implement the solution described, please start a new thread on the CloudWatch forum.

– Mustafa

Filling in the holes in Linux boot chain measurement, and the TPM measurement log

Post Syndicated from Matthew Garrett original http://mjg59.dreamwidth.org/37656.html

When I wrote about TPM attestation via 2FA, I mentioned that you needed a bootloader that actually performed measurement. I’ve now written some patches for Shim and Grub that do so.The Shim code does a couple of things. The obvious one is to measure the second-stage bootloader into PCR 9. The perhaps less expected one is to measure the contents of the MokList and MokSBState UEFI variables into PCR 14. This means that if you’re happy simply running a system with your own set of signing keys and just want to ensure that your secure boot configuration hasn’t been compromised, you can simply seal to PCR 7 (which will contain the UEFI Secure Boot state as defined by the UEFI spec) and PCR 14 (which will contain the additional state used by Shim) and ignore all the others.The grub code is a little more complicated because there’s more ways to get it to execute code. Right now I’ve gone for a fairly extreme implementation. On BIOS systems, the grub stage 1 and 2 will be measured into PCR 9[1]. That’s the only BIOS-specific part of things. From then on, any grub modules that are loaded will also be measured into PCR 9. The full kernel image will be measured into PCR10, and the full initramfs will be measured into PCR11. The command line passed to the kernel is in PCR12. Finally, each command executed by grub (including those in the config file) is measured into PCR 13.That’s quite a lot of measurement, and there are probably fairly reasonable circumstances under which you won’t want to pay attention to all of those PCRs. But you’ve probably also noticed that several different things may be measured into the same PCR, and that makes it more difficult to figure out what’s going on. Thankfully, the spec designers have a solution to this in the form of the TPM measurement log.Rather than merely extending a PCR with a new hash, software can extend the measurement log at the same time. This is stored outside the TPM and so isn’t directly cryptographically protected. In the simplest form, it contains a hash and some form of description of the event associated with that hash. If you replay those hashes you should end up with the same value that’s in the TPM, so for attestation purposes you can perform that verification and then merely check that specific log values you care about are correct. This makes it possible to have a system perform an attestation to a remote server that contains a full list of the grub commands that it ran and for that server to make its attestation decision based on a subset of those.No promises as yet about PCR allocation being final or these patches ever going anywhere in their current form, but it seems reasonable to get them out there so people can play. Let me know if you end up using them![1] The code for this is derived from the old Trusted Grub patchset, by way of Sirrix AG’s Trusted Grub 2 tree.comment count unavailable comments

Using AWS OpsWorks to Customize and Automate App Deployment on Windows

Post Syndicated from Daniel Huesch original http://blogs.aws.amazon.com/application-management/post/Tx1BBK730NQL2F4/Using-AWS-OpsWorks-to-Customize-and-Automate-App-Deployment-on-Windows

Using OpsWorks and Chef on Windows helps you optimize your use of Windows by reliably automating configuration tasks to enforce instance compliance. Automating instance configuration enables software engineering best practices, like code-reviews and continuous integration, and allows smaller, faster delivery than with manual configuration. With automated instance configuration, you depend less on golden images or manual changes. OpsWorks also ships with features that ease operational tasks, like user management or scaling your infrastructure by booting additional machines.

In this post, I show how you can use OpsWorks to customize your instances and deploy your apps on Windows. To show how easy application management is when using OpsWorks, we will deploy a Node.JS app to Microsoft Internet Information Server (IIS). You can find both the cookbooks and the app source code in the Amazon Web Services – Labs repository on GitHub.

To follow this example, you need to understand how OpsWorks uses recipes and lifecycle events. For more information, see What is AWS OpsWorks?.

Create the Stack and Instance

First, let’s create a stack in the OpsWorks console. Navigate to the Add Stack page at https://console.aws.amazon.com/opsworks/home?#/stack/new.

For the name, type Node.JS Windows Demo. For the Default operating system, choose Microsoft Windows Server 2012 R2 Base.

Next, configure the cookbook source. Choose Advanced to display the Configuration Management section. Enable Use custom Chef cookbooks, choose the Git version control system as the Repository type, and enter https://github.com/awslabs/opsworks-windows-demo-cookbooks.git as the Repository URL. The cookbooks that we just configured describe how a Node.JS app is installed.

Choose Add Stack to create the stack.

In the Layers section, choose Add a Layer. Choose any name and short name you like. For Security Groups, choose AWS-OpsWorks-Web-Server as an additional security group. This ensures that HTTP traffic is allowed and you can connect to the demo app with your browser. For this example, you don’t need RDP access, so you can safely ignore the warning. Choose Add Layer.

Before we add instances to this layer, we have to wire up the Chef code with the lifecycle events. To do that, edit the layer recipes. On the Layers page, under more layers, choose Recipes. On the Layer more layers page, choose Edit. For the Setup lifecycle event, choose webserver_nodejs::setup, and for the Deploy event, choose webserver_nodejs::deploy.

Confirm your changes by choosing Save.

Now we are ready to add an instance. Switch to the Instances page by choosing Instances, and then choose Add an Instance. OpsWorks suggests a Hostname based on the layer name; for this example, webserver-node1. Choose Add Instance to confirm.

Don’t start the instance yet. We need to tell OpsWorks about the app that we want to deploy first. This ensures that the app is deployed when the instance starts. (When an instance executes the Setup event during boot, OpsWorks deploys apps automatically. Later, you can deploy new and existing apps to any running instances.)

Create the App

In the left pane, choose Apps to switch to the Apps page, and then choose Add an app. Give the app a name, choose Git as the Repository type, and enter https://github.com/awslabs/opsworks-windows-demo-nodejs.git as the Repository URL. The demo app supports environment variables, so you can use the APP_ADMIN_EMAIL key to set the mail address displayed on the demo app’s front page. Use any value you like.

To save the app, choose Add App. Return to the Instances page. Now start the instance.

OpsWorks reports setup progress, switching from “requested” to “pending,” and then to “booting.” When the instance is “booting,” it is running on Amazon EC2. After the OpsWorks agent is installed and has picked up its first lifecycle event, the status changes to “running_setup.” After OpsWorks processes the Setup lifecycle event, it shows the instance as “online.” When an instance reaches the online state, OpsWorks fires a Configure lifecycle event to inform all instances in the stack about the new instance.

Booting a new instance can take a few minutes, the total time depending on the instance size and your Chef recipes. While you wait, get a cup of coffee or tea. Then, let’s take a look at the code that Chef will execute and the general structure of the opsworks-windows-example-cookbooks.

Cookbook Deep Dive – Setup Event

Let’s take a look at the webserver_nodejs::setup recipe, which we wired to the Setup event when we chose layer settings:

# Recipes to install software on initial boot

include_recipe "opsworks_iis"
include_recipe "opsworks_nodejs"
include_recipe "opsworks_iisnode"

This recipe simply includes other recipes. As you can see, Chef installs the IIS, Node.JS, and the IIS-to-Node.JS bridge iisnode. By taking a closer look at the opsworks_nodejs cookbook, we can learn how to install apps. The general folder structure of a Chef cookbook is:

opsworks_nodejs
├── README.md
├── attributes
│ └── default.rb
├── definitions
│ └── opsworks_nodejs_npm.rb
├── metadata.rb
└── recipes
└── default.rb

The opsworks_nodejs cookbook uses attributes to define settings that you can override when using the cookbook. This makes it easy to update to a new version of Node.JS or npm, the node package manager, by just setting the attribute to another value.

The file opsworks_nodejs/attributes/default.rb defines these version attributes:

default["opsworks_nodejs"]["node_version"] = "0.12.7"
default["opsworks_nodejs"]["npm_version"] = "2.13.0"

The default recipe in the opsworks_nodejs cookbook uses the node_version and npm_version. It uses node_version as part of the download URL construction and npm_version in the batch code.

version = node["opsworks_nodejs"]["node_version"]

download_file = "node-v#{version}-x86.msi"
download_path = ::File.join(Chef::Config["file_cache_path"], download_file)

remote_file download_path do
source "https://nodejs.org/dist/v#{version}/#{download_file}"
retries 2
end

windows_package "nodejs" do
source download_path
end

batch "install npm version #{node[‘opsworks_nodejs’][‘npm_version’]}" do
code ""%programfiles(x86)%\nodejs\npm" -g install [email protected]#{node[‘opsworks_nodejs’][‘npm_version’]}"
end

The cookbook installs Node.JS in two steps. First, it uses the Chef remote_file resource to download the installation package from the official Node.JS website and save it to the local disk. The cookbook also sets the retries attribute to enable retries, so the code is more resilient to short-term networking issues.

After the cookbook saves the file, the windows_package resource installs the MSI. Then, the cookbook installs the requested npm version using the batch resource.

Chef resources provide many attributes for fine-tuning their behavior. For more information, see the Chef Resources Reference.

Cookbook Deep Dive – Deploy Event

As I mentioned, OpsWorks doesn’t prepopulate your Chef run with recipes or resources. This gives you fine-grained control and complete flexibility over how to deploy your app. However, there are some common tasks, like checking your app out of Git or downloading it from Amazon S3. To make performing common tasks easier, the example cookbooks ship with a custom Chef resource that handles these steps for you, opsworks_scm_checkout.

As it does with the Setup recipe, OpsWorks uses the webserver_nodejs::deploy recipe only to include other recipes. The opsworks_app_nodejs cookbook’s default recipe does the heavy lifting.

The slimmed-down version of the recipe looks like the following.

apps = search(:aws_opsworks_app, "deploy:true")

apps.each do |app|
opsworks_scm_checkout app["shortname"] do

end

directory app_deploy do

end

# Copy app to deployment directory
batch "copy #{app["shortname"]}" do
code "Robocopy.exe …"
end

# Run ‘npm install’
opsworks_nodejs_npm app["shortname"] do
cwd app_deploy
end

template "#{app[‘shortname’]}/web.config" do

variables({ :environment => app["environment"]})
end

powershell_script "register #{app["shortname"]}" do

end
end

Reviewing the code will help you understand how to write your own deployment cookbooks, so let’s walk through it. First, we use Chef search to fetch information about the apps to be deployed. The AWS OpsWorks User Guide lists all exposed attributes.

For each app, Chef then executes the following steps:

·      It checks out the app to the local file system using the opsworks_scm_checkout resource.

·      It creates the target directory and copies the app to that directory. To transfer only the files that have changed, it uses Robocopy.

·      Running npm install, it downloads all required third-party libraries. It does this by using the opsworks_nodejs_npm custom resource, which is defined in the opsworks_nodejs cookbook.

·      The web.config file is generated using a template, taking OpsWorks environment variables into account. The file configures the IIS-to-Node.JS integration.

·      A Windows PowerShell run registers the IIS site and brings the app online.

Check Out the Node.JS Demo App

Your booted instance should be online now:

To access the demo app, under Public IP, click the IP address. The app allows you to leave comments that other visitors can see.

Notice that the page is not static, but includes information about the hostname, the browser you used to request it, the system time, and the Node.JS version used. At the bottom of the page, you can see that the APP_ADMIN_EMAIL environment variable you configured in the app was picked up.

Leave a comment on the app, and choose Send. Because this is a minimal app for demo purposes only, the comment is saved on the instance in a plain JSON file.

To see the details of the customizations OpsWorks just applied to your instance, choose the hostname on the Layer overview page. On the very bottom of the Instance detail page, you will find the last 30 commands the instance received. To see the corresponding Chef log, choose show.

Conclusion

Using OpsWorks with Chef 12 on Windows gives you a reliable framework for customizing your Windows Server 2012 R2 instances and managing your apps. By attaching Chef recipes to lifecycle events, you can customize your instances. You can use open source community cookbooks or create your own cookbooks. Chef’s support for PowerShell allows you to reuse automation for Windows. OpsWorks’ built-in operational tools, like user management, permissions, and temporary RDP access help you manage day-to-day operations.

The example showed how to deploy a Node.JS app using IIS. By referring to typical cookbooks that were included in the example like opsworks_nodejs, which is used to install Node.JS, or opsworks_app_nodejs, which is used to deploy the app, you can learn how to write your own cookbooks.

Using AWS OpsWorks to Customize and Automate App Deployment on Windows

Post Syndicated from Daniel Huesch original http://blogs.aws.amazon.com/application-management/post/Tx1BBK730NQL2F4/Using-AWS-OpsWorks-to-Customize-and-Automate-App-Deployment-on-Windows

Using OpsWorks and Chef on Windows helps you optimize your use of Windows by reliably automating configuration tasks to enforce instance compliance. Automating instance configuration enables software engineering best practices, like code-reviews and continuous integration, and allows smaller, faster delivery than with manual configuration. With automated instance configuration, you depend less on golden images or manual changes. OpsWorks also ships with features that ease operational tasks, like user management or scaling your infrastructure by booting additional machines.

In this post, I show how you can use OpsWorks to customize your instances and deploy your apps on Windows. To show how easy application management is when using OpsWorks, we will deploy a Node.JS app to Microsoft Internet Information Server (IIS). You can find both the cookbooks and the app source code in the Amazon Web Services – Labs repository on GitHub.

To follow this example, you need to understand how OpsWorks uses recipes and lifecycle events. For more information, see What is AWS OpsWorks?.

Create the Stack and Instance

First, let’s create a stack in the OpsWorks console. Navigate to the Add Stack page at https://console.aws.amazon.com/opsworks/home?#/stack/new.

For the name, type Node.JS Windows Demo. For the Default operating system, choose Microsoft Windows Server 2012 R2 Base.

Next, configure the cookbook source. Choose Advanced to display the Configuration Management section. Enable Use custom Chef cookbooks, choose the Git version control system as the Repository type, and enter https://github.com/awslabs/opsworks-windows-demo-cookbooks.git as the Repository URL. The cookbooks that we just configured describe how a Node.JS app is installed.

Choose Add Stack to create the stack.

In the Layers section, choose Add a Layer. Choose any name and short name you like. For Security Groups, choose AWS-OpsWorks-Web-Server as an additional security group. This ensures that HTTP traffic is allowed and you can connect to the demo app with your browser. For this example, you don’t need RDP access, so you can safely ignore the warning. Choose Add Layer.

Before we add instances to this layer, we have to wire up the Chef code with the lifecycle events. To do that, edit the layer recipes. On the Layers page, under more layers, choose Recipes. On the Layer more layers page, choose Edit. For the Setup lifecycle event, choose webserver_nodejs::setup, and for the Deploy event, choose webserver_nodejs::deploy.

Confirm your changes by choosing Save.

Now we are ready to add an instance. Switch to the Instances page by choosing Instances, and then choose Add an Instance. OpsWorks suggests a Hostname based on the layer name; for this example, webserver-node1. Choose Add Instance to confirm.

Don’t start the instance yet. We need to tell OpsWorks about the app that we want to deploy first. This ensures that the app is deployed when the instance starts. (When an instance executes the Setup event during boot, OpsWorks deploys apps automatically. Later, you can deploy new and existing apps to any running instances.)

Create the App

In the left pane, choose Apps to switch to the Apps page, and then choose Add an app. Give the app a name, choose Git as the Repository type, and enter https://github.com/awslabs/opsworks-windows-demo-nodejs.git as the Repository URL. The demo app supports environment variables, so you can use the APP_ADMIN_EMAIL key to set the mail address displayed on the demo app’s front page. Use any value you like.

To save the app, choose Add App. Return to the Instances page. Now start the instance.

OpsWorks reports setup progress, switching from “requested” to “pending,” and then to “booting.” When the instance is “booting,” it is running on Amazon EC2. After the OpsWorks agent is installed and has picked up its first lifecycle event, the status changes to “running_setup.” After OpsWorks processes the Setup lifecycle event, it shows the instance as “online.” When an instance reaches the online state, OpsWorks fires a Configure lifecycle event to inform all instances in the stack about the new instance.

Booting a new instance can take a few minutes, the total time depending on the instance size and your Chef recipes. While you wait, get a cup of coffee or tea. Then, let’s take a look at the code that Chef will execute and the general structure of the opsworks-windows-example-cookbooks.

Cookbook Deep Dive – Setup Event

Let’s take a look at the webserver_nodejs::setup recipe, which we wired to the Setup event when we chose layer settings:

# Recipes to install software on initial boot

include_recipe "opsworks_iis"
include_recipe "opsworks_nodejs"
include_recipe "opsworks_iisnode"

This recipe simply includes other recipes. As you can see, Chef installs the IIS, Node.JS, and the IIS-to-Node.JS bridge iisnode. By taking a closer look at the opsworks_nodejs cookbook, we can learn how to install apps. The general folder structure of a Chef cookbook is:

opsworks_nodejs
├── README.md
├── attributes
│ └── default.rb
├── definitions
│ └── opsworks_nodejs_npm.rb
├── metadata.rb
└── recipes
└── default.rb

The opsworks_nodejs cookbook uses attributes to define settings that you can override when using the cookbook. This makes it easy to update to a new version of Node.JS or npm, the node package manager, by just setting the attribute to another value.

The file opsworks_nodejs/attributes/default.rb defines these version attributes:

default["opsworks_nodejs"]["node_version"] = "0.12.7"
default["opsworks_nodejs"]["npm_version"] = "2.13.0"

The default recipe in the opsworks_nodejs cookbook uses the node_version and npm_version. It uses node_version as part of the download URL construction and npm_version in the batch code.

version = node["opsworks_nodejs"]["node_version"]

download_file = "node-v#{version}-x86.msi"
download_path = ::File.join(Chef::Config["file_cache_path"], download_file)

remote_file download_path do
source "https://nodejs.org/dist/v#{version}/#{download_file}"
retries 2
end

windows_package "nodejs" do
source download_path
end

batch "install npm version #{node[‘opsworks_nodejs’][‘npm_version’]}" do
code ""%programfiles(x86)%\nodejs\npm" -g install [email protected]#{node[‘opsworks_nodejs’][‘npm_version’]}"
end

The cookbook installs Node.JS in two steps. First, it uses the Chef remote_file resource to download the installation package from the official Node.JS website and save it to the local disk. The cookbook also sets the retries attribute to enable retries, so the code is more resilient to short-term networking issues.

After the cookbook saves the file, the windows_package resource installs the MSI. Then, the cookbook installs the requested npm version using the batch resource.

Chef resources provide many attributes for fine-tuning their behavior. For more information, see the Chef Resources Reference.

Cookbook Deep Dive – Deploy Event

As I mentioned, OpsWorks doesn’t prepopulate your Chef run with recipes or resources. This gives you fine-grained control and complete flexibility over how to deploy your app. However, there are some common tasks, like checking your app out of Git or downloading it from Amazon S3. To make performing common tasks easier, the example cookbooks ship with a custom Chef resource that handles these steps for you, opsworks_scm_checkout.

As it does with the Setup recipe, OpsWorks uses the webserver_nodejs::deploy recipe only to include other recipes. The opsworks_app_nodejs cookbook’s default recipe does the heavy lifting.

The slimmed-down version of the recipe looks like the following.

apps = search(:aws_opsworks_app, "deploy:true")

apps.each do |app|
opsworks_scm_checkout app["shortname"] do

end

directory app_deploy do

end

# Copy app to deployment directory
batch "copy #{app["shortname"]}" do
code "Robocopy.exe …"
end

# Run ‘npm install’
opsworks_nodejs_npm app["shortname"] do
cwd app_deploy
end

template "#{app[‘shortname’]}/web.config" do

variables({ :environment => app["environment"]})
end

powershell_script "register #{app["shortname"]}" do

end
end

Reviewing the code will help you understand how to write your own deployment cookbooks, so let’s walk through it. First, we use Chef search to fetch information about the apps to be deployed. The AWS OpsWorks User Guide lists all exposed attributes.

For each app, Chef then executes the following steps:

·      It checks out the app to the local file system using the opsworks_scm_checkout resource.

·      It creates the target directory and copies the app to that directory. To transfer only the files that have changed, it uses Robocopy.

·      Running npm install, it downloads all required third-party libraries. It does this by using the opsworks_nodejs_npm custom resource, which is defined in the opsworks_nodejs cookbook.

·      The web.config file is generated using a template, taking OpsWorks environment variables into account. The file configures the IIS-to-Node.JS integration.

·      A Windows PowerShell run registers the IIS site and brings the app online.

Check Out the Node.JS Demo App

Your booted instance should be online now:

To access the demo app, under Public IP, click the IP address. The app allows you to leave comments that other visitors can see.

Notice that the page is not static, but includes information about the hostname, the browser you used to request it, the system time, and the Node.JS version used. At the bottom of the page, you can see that the APP_ADMIN_EMAIL environment variable you configured in the app was picked up.

Leave a comment on the app, and choose Send. Because this is a minimal app for demo purposes only, the comment is saved on the instance in a plain JSON file.

To see the details of the customizations OpsWorks just applied to your instance, choose the hostname on the Layer overview page. On the very bottom of the Instance detail page, you will find the last 30 commands the instance received. To see the corresponding Chef log, choose show.

Conclusion

Using OpsWorks with Chef 12 on Windows gives you a reliable framework for customizing your Windows Server 2012 R2 instances and managing your apps. By attaching Chef recipes to lifecycle events, you can customize your instances. You can use open source community cookbooks or create your own cookbooks. Chef’s support for PowerShell allows you to reuse automation for Windows. OpsWorks’ built-in operational tools, like user management, permissions, and temporary RDP access help you manage day-to-day operations.

The example showed how to deploy a Node.JS app using IIS. By referring to typical cookbooks that were included in the example like opsworks_nodejs, which is used to install Node.JS, or opsworks_app_nodejs, which is used to deploy the app, you can learn how to write your own cookbooks.

Using CloudWatch Logs with Amazon EC2 Running Microsoft Windows Server

Post Syndicated from Mats Lanner original http://blogs.aws.amazon.com/application-management/post/Tx1KG4IKXZ94QFK/Using-CloudWatch-Logs-with-Amazon-EC2-Running-Microsoft-Windows-Server

Now Amazon EC2 running Microsoft Windows Server provides enhanced log support for Amazon CloudWatch Logs. You can monitor the operations and performance of your EC2 for Windows instances and applications in near real-time using standard log and performance data sources including:

Event Tracing for Windows log sources

IIS request logs

Performance Counters

Text-based log files

Windows Event Logs

EC2 Windows integrates with CloudWatch Logs using a plug-in for the EC2Config service that is installed on all new EC2 Windows instances by default. To follow along with the example in this post, you will need to download the latest version of the EC2Config service software from http://aws.amazon.com/developertools/5562082477397515.

For this blog post, I’m using a SQL Server 2014 instance just so that uploading of SQL Server logs can be demonstrated.

Permissions

While the CloudWatch plug-in for EC2Config supports explicit credentials, it’s strongly recommended to use IAM Roles for EC2 which makes it possible to associate specific permissions with an EC2 instance when it’s launched.

A sample policy allows the necessary CloudWatch and CloudWatch logs actions (to learn more about adding a policy to an existing IAM role, see IAM Users and Groups and Managing IAM Policies in Using IAM) is this:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1414002531000",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1414002720000",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": [
"*"
]
}
]
}

CloudWatch Plug-In Configuration

Detailed information on how to configure CloudWatch can be found here: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#send_logs_to_cwl. In this blog post I will focus on a few specific examples around Event Tracing for Windows (ETW) and SQL Server error logs.

Enable the Plug-In

The CloudWatch plug-in for EC2Config is disabled by default, so the first step that needs to be taken is to enable it. The easiest way to do this is to run EC2ConfigService Settings on the instance where you want to enable the plug-in:

Enabling CloudWatch support

Check the Enable CloudWatch Logs integration checkbox and click the OK button to save the changes. Note that the EC2Config service must be restarted for this to take effect, but we can hold off on doing that in this case as we will first configure the log sources that should be uploaded to CloudWatch Logs and the Performance Counters that should be uploaded to CloudWatch. For information on how to programatically enable the integration, see the documentation: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#enable_cwl_integration

Configuration

The CloudWatch plug-in is configured using the file %PROGRAMFILES%AmazonEc2ConfigServiceSettingsAWS.EC2.Windows.CloudWatch.json. This file contains the settings for CloudWatch, CloudWatch Logs, the log sources and performance counters that should be uploaded. For the purposes of this post I’m going to use the default settings for CloudWatch which means that data will be uploaded to us-east-1 using the CloudWatch namespace Windows/Default. I will, however, customize the configuration for CloudWatch Logs to demonstrate how different log sources can be uploaded to different log groups and log streams.

CloudWatch Logs

For this example I will use two log groups and log streams for the different types of log data that will be configured later, so replace the default CloudWatch Logs configuration:

{
  "Id": "CloudWatchLogs",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "Default-Log-Group",
    "LogStream": "{instance_id}"
  }
},

With:

{
  "Id": "CloudWatchLogsSystem",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "System-Logs",
    "LogStream": "{instance_id}"
  }
},
{
  "Id": "CloudWatchLogsSQL",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "SQL-Logs",
    "LogStream": "{instance_id}"
  }
},

This configuration creates two CloudWatch Logs components, one that will upload log data to the log group System-Logs (which will be used for Event Logs and ETW) and one that will upload log data to the log group SQL-Logs (which will be used to upload SQL Server log files).

Log Sources

The default configuration file contains sample configurations for all supported log sources. For this post I will define a new ETW log source to capture group policy processing and a custom log file source to capture SQL Server error log files.

Event Tracing for Windows

The default configuration file contains a sample ETW log source with the id ETW, modify this log source to look like this:

{
  "Id": "ETW",
  "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "LogName": "Microsoft-Windows-GroupPolicy/Operational",
    "Levels": "7"
  }
},

This change tells the plug-in to upload all log entries from the ETW source that shows up as Applications and Services Logs | Microsoft | Windows | GroupPolicy | Operational in the Windows Event Viewer.

Custom Log Files

The custom log file support can handle almost any text-based log file. To configure upload of SQL Server error logs, change the sample CustomLogs log source to look like this:

{
  "Id": "CustomLogs",
  "FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "LogDirectoryPath": "C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Log",
    "TimestampFormat": "yyyy-MM-dd HH:mm:ss.ff",
    "Encoding": "UTF-16",
    "Filter": "ERRORLOG*",
    "CultureName": "en-US",
    "TimeZoneKind": "Local"
  }
},

This configuration tells the plug-in to look for log files that start with the file name ERRORLOG (e.g. ERRORLOG, ERRORLOG.1, etc.) in the directory C:Program FilesMicrosoft SQL ServerMSSQL12.MSSQLSERVERMSSQLLog. Finally, the timestamp format setting has been updated to match the timestamp used by SQL Server and the encoding for the file set to reflect that SQL Server uses UTF-16 encoding for the log file.

Performance Counters

In addition to log data, you can also send Windows Performance Counters to CloudWatch as custom metrics. This makes it possible to monitor specific performance indicators from inside an instance and allows you to create alarms based on this data.

The default configuration file contains a sample performance counter that uploads the available memory performance counter:

{
  "Id": "PerformanceCounter",
  "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "CategoryName": "Memory",
    "CounterName": "Available MBytes",
    "InstanceName": "",
    "MetricName": "Memory",
    "Unit": "Megabytes",
    "DimensionName": "",
    "DimensionValue": ""
  }
},

Add an additional performance counter to this – the amount of free space available on the C drive on the instance. Update the configuration file to also include the Logical Disk | Free Megabytes performance counter:

{
  "Id": "PerformanceCounterDisk",
  "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "CategoryName": "LogicalDisk",
    "CounterName": "Free Megabytes",
    "InstanceName": "C:",
    "MetricName": "FreeDisk",
    "Unit": "Megabytes",
    "DimensionName": "",
    "DimensionValue": ""
  }
},

Putting it All Together

The final step in configuring the plug-in is to define what data should be sent where. Towards the end of the default configuration file you will find a section looking like this:

"Flows": {
  "Flows":
  [
    "(ApplicationEventLog,SystemEventLog),CloudWatchLogs"
  ]
}

Detailed information on how this setting works is available here: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#configure_log_flow. To use the configuration defined above, change this section to this:

"Flows": {
  "Flows":
  [
    "(SystemEventLog,ETW),CloudWatchLogsSystem",
    "CustomLogs,CloudWatchLogsSQL",
    "(PerformanceCounter,PerformanceCounterDisk),CloudWatch"
  ]
}

The flow definition uses the component identifiers to specify how data should flow from the source to the destination. In the example above the flow is defined as:

Send log data from the System event log and the ETW log source log group System-Logs

Send SQL Server error log file to the log group SQL-Logs

Send the two performance counters to the Windows/Default metric namespace

Now that the configuration is complete, save the configuration file and use the Service Manager to restart the Ec2Config service, or from the command line run the commands:

C:> net stop ec2config

C:> net start ec2config

Once the Ec2Config service has restarted, it will start sending both the configured log data and performance counters to CloudWatch Logs and CloudWatch. Note that it will take a few minutes before the first data appears in the console.

Access Log Data in CloudWatch Logs

Shortly after starting the Ec2Config service, you will be able to see the new log groups created in the console at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:. In our example, we can see both the System-Logs and SQL-Logs log groups and the log data from our example instance in the instance-specific log stream:

Uploaded SQL Server log data.

With the log data in place, you can now create metric filters to start monitoring the events that are relevant to you. More information on how to configure metric filters can be found here: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/MonitoringPolicyExamples.html

Access Performance Metrics in CloudWatch

With the configuration defined above, you can access the uploaded performance counters in the console at https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#metrics:metricFilter=Pattern%253DWindows%252FDefault:

Performance counter data uploaded to CloudWatch.

At this point you can configure alarms on the metrics (for instance configure an alarm if the FreeDisk metrics goes below 20GB).

 

Using CloudWatch Logs with Amazon EC2 Running Microsoft Windows Server

Post Syndicated from Mats Lanner original http://blogs.aws.amazon.com/application-management/post/Tx1KG4IKXZ94QFK/Using-CloudWatch-Logs-with-Amazon-EC2-Running-Microsoft-Windows-Server

Now Amazon EC2 running Microsoft Windows Server provides enhanced log support for Amazon CloudWatch Logs. You can monitor the operations and performance of your EC2 for Windows instances and applications in near real-time using standard log and performance data sources including:

Event Tracing for Windows log sources

IIS request logs

Performance Counters

Text-based log files

Windows Event Logs

EC2 Windows integrates with CloudWatch Logs using a plug-in for the EC2Config service that is installed on all new EC2 Windows instances by default. To follow along with the example in this post, you will need to download the latest version of the EC2Config service software from http://aws.amazon.com/developertools/5562082477397515.

For this blog post, I’m using a SQL Server 2014 instance just so that uploading of SQL Server logs can be demonstrated.

Permissions

While the CloudWatch plug-in for EC2Config supports explicit credentials, it’s strongly recommended to use IAM Roles for EC2 which makes it possible to associate specific permissions with an EC2 instance when it’s launched.

A sample policy allows the necessary CloudWatch and CloudWatch logs actions (to learn more about adding a policy to an existing IAM role, see IAM Users and Groups and Managing IAM Policies in Using IAM) is this:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1414002531000",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1414002720000",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": [
"*"
]
}
]
}

CloudWatch Plug-In Configuration

Detailed information on how to configure CloudWatch can be found here: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#send_logs_to_cwl. In this blog post I will focus on a few specific examples around Event Tracing for Windows (ETW) and SQL Server error logs.

Enable the Plug-In

The CloudWatch plug-in for EC2Config is disabled by default, so the first step that needs to be taken is to enable it. The easiest way to do this is to run EC2ConfigService Settings on the instance where you want to enable the plug-in:

Enabling CloudWatch support

Check the Enable CloudWatch Logs integration checkbox and click the OK button to save the changes. Note that the EC2Config service must be restarted for this to take effect, but we can hold off on doing that in this case as we will first configure the log sources that should be uploaded to CloudWatch Logs and the Performance Counters that should be uploaded to CloudWatch. For information on how to programatically enable the integration, see the documentation: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#enable_cwl_integration

Configuration

The CloudWatch plug-in is configured using the file %PROGRAMFILES%AmazonEc2ConfigServiceSettingsAWS.EC2.Windows.CloudWatch.json. This file contains the settings for CloudWatch, CloudWatch Logs, the log sources and performance counters that should be uploaded. For the purposes of this post I’m going to use the default settings for CloudWatch which means that data will be uploaded to us-east-1 using the CloudWatch namespace Windows/Default. I will, however, customize the configuration for CloudWatch Logs to demonstrate how different log sources can be uploaded to different log groups and log streams.

CloudWatch Logs

For this example I will use two log groups and log streams for the different types of log data that will be configured later, so replace the default CloudWatch Logs configuration:

{
  "Id": "CloudWatchLogs",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "Default-Log-Group",
    "LogStream": "{instance_id}"
  }
},

With:

{
  "Id": "CloudWatchLogsSystem",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "System-Logs",
    "LogStream": "{instance_id}"
  }
},
{
  "Id": "CloudWatchLogsSQL",
  "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "AccessKey": "",
    "SecretKey": "",
    "Region": "us-east-1",
    "LogGroup": "SQL-Logs",
    "LogStream": "{instance_id}"
  }
},

This configuration creates two CloudWatch Logs components, one that will upload log data to the log group System-Logs (which will be used for Event Logs and ETW) and one that will upload log data to the log group SQL-Logs (which will be used to upload SQL Server log files).

Log Sources

The default configuration file contains sample configurations for all supported log sources. For this post I will define a new ETW log source to capture group policy processing and a custom log file source to capture SQL Server error log files.

Event Tracing for Windows

The default configuration file contains a sample ETW log source with the id ETW, modify this log source to look like this:

{
  "Id": "ETW",
  "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "LogName": "Microsoft-Windows-GroupPolicy/Operational",
    "Levels": "7"
  }
},

This change tells the plug-in to upload all log entries from the ETW source that shows up as Applications and Services Logs | Microsoft | Windows | GroupPolicy | Operational in the Windows Event Viewer.

Custom Log Files

The custom log file support can handle almost any text-based log file. To configure upload of SQL Server error logs, change the sample CustomLogs log source to look like this:

{
  "Id": "CustomLogs",
  "FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "LogDirectoryPath": "C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Log",
    "TimestampFormat": "yyyy-MM-dd HH:mm:ss.ff",
    "Encoding": "UTF-16",
    "Filter": "ERRORLOG*",
    "CultureName": "en-US",
    "TimeZoneKind": "Local"
  }
},

This configuration tells the plug-in to look for log files that start with the file name ERRORLOG (e.g. ERRORLOG, ERRORLOG.1, etc.) in the directory C:Program FilesMicrosoft SQL ServerMSSQL12.MSSQLSERVERMSSQLLog. Finally, the timestamp format setting has been updated to match the timestamp used by SQL Server and the encoding for the file set to reflect that SQL Server uses UTF-16 encoding for the log file.

Performance Counters

In addition to log data, you can also send Windows Performance Counters to CloudWatch as custom metrics. This makes it possible to monitor specific performance indicators from inside an instance and allows you to create alarms based on this data.

The default configuration file contains a sample performance counter that uploads the available memory performance counter:

{
  "Id": "PerformanceCounter",
  "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "CategoryName": "Memory",
    "CounterName": "Available MBytes",
    "InstanceName": "",
    "MetricName": "Memory",
    "Unit": "Megabytes",
    "DimensionName": "",
    "DimensionValue": ""
  }
},

Add an additional performance counter to this – the amount of free space available on the C drive on the instance. Update the configuration file to also include the Logical Disk | Free Megabytes performance counter:

{
  "Id": "PerformanceCounterDisk",
  "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
  "Parameters": {
    "CategoryName": "LogicalDisk",
    "CounterName": "Free Megabytes",
    "InstanceName": "C:",
    "MetricName": "FreeDisk",
    "Unit": "Megabytes",
    "DimensionName": "",
    "DimensionValue": ""
  }
},

Putting it All Together

The final step in configuring the plug-in is to define what data should be sent where. Towards the end of the default configuration file you will find a section looking like this:

"Flows": {
  "Flows":
  [
    "(ApplicationEventLog,SystemEventLog),CloudWatchLogs"
  ]
}

Detailed information on how this setting works is available here: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html#configure_log_flow. To use the configuration defined above, change this section to this:

"Flows": {
  "Flows":
  [
    "(SystemEventLog,ETW),CloudWatchLogsSystem",
    "CustomLogs,CloudWatchLogsSQL",
    "(PerformanceCounter,PerformanceCounterDisk),CloudWatch"
  ]
}

The flow definition uses the component identifiers to specify how data should flow from the source to the destination. In the example above the flow is defined as:

Send log data from the System event log and the ETW log source log group System-Logs

Send SQL Server error log file to the log group SQL-Logs

Send the two performance counters to the Windows/Default metric namespace

Now that the configuration is complete, save the configuration file and use the Service Manager to restart the Ec2Config service, or from the command line run the commands:

C:> net stop ec2config

C:> net start ec2config

Once the Ec2Config service has restarted, it will start sending both the configured log data and performance counters to CloudWatch Logs and CloudWatch. Note that it will take a few minutes before the first data appears in the console.

Access Log Data in CloudWatch Logs

Shortly after starting the Ec2Config service, you will be able to see the new log groups created in the console at https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logs:. In our example, we can see both the System-Logs and SQL-Logs log groups and the log data from our example instance in the instance-specific log stream:

Uploaded SQL Server log data.

With the log data in place, you can now create metric filters to start monitoring the events that are relevant to you. More information on how to configure metric filters can be found here: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/MonitoringPolicyExamples.html

Access Performance Metrics in CloudWatch

With the configuration defined above, you can access the uploaded performance counters in the console at https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#metrics:metricFilter=Pattern%253DWindows%252FDefault:

Performance counter data uploaded to CloudWatch.

At this point you can configure alarms on the metrics (for instance configure an alarm if the FreeDisk metrics goes below 20GB).

 

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.So I had to write one on my own.The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutilsIt is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)It is easy to use this module. You just have to do something like this:var amfUtils = require(‘node-amfutils’);var buffer = amfUtils.amf0Encode([{ a: “xxx”},b: null]);

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.So I had to write one on my own.The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutilsIt is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)It is easy to use this module. You just have to do something like this:var amfUtils = require(‘node-amfutils’);var buffer = amfUtils.amf0Encode([{ a: “xxx”},b: null]);

Plumbers Conference 2011

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/lpc2011.html

The Linux Plumbers
Conference 2011 in Santa Rosa, CA, USA
is coming nearer (Sep. 7-9).
Together with Kay Sievers I am running the Boot&Init track, and together with
Mark Brown the Audio track.

For both tracks we still need proposals. So if you haven’t submitted
anything yet, please consider doing so. And that quickly. i.e. if you can
arrange for it, last sunday would be best, since that was actually the final
deadline. However, the submission form is still open, so if you submit
something really, really quickly we’ll ignore the absence of time travel and the calendar for a bit. So, go,
submit something. Now.

What are we looking for? Well, here’s what I just posted on the audio
related mailing lists
:

So, please consider submitting something if you haven't done so yet. We
are looking for all kinds of technical talks covering everything audio
plumbing related: audio drivers, audio APIs, sound servers, pro audio,
consumer audio. If you can propose something audio related -- like talks
on media controller routing, on audio for ASOC/Embedded, submit
something! If you care for low-latency audio, submit something. If you
care about the Linux audio stack in general, submit something.

LPC is probably the most relevant technical conference on the general
Linux platform, so be sure that if you want your project, your work,
your ideas to be heard then this is the right forum for everything
related to the Linux stack. And the Audio track covers everything in our
Audio Stack, regardless whether it is pro or consumer audio.

And here’s what I posted to the init
related lists
:

So, please consider submitting something if you haven't done so yet. We
are looking for all kinds of technical talks covering everything from
the BIOS (i.e. CoreBoot and friends), over boot loaders (i.e. GRUB and
friends), to initramfs (i.e. Dracut and friends) and init systems
(i.e. systemd and friends). If you have something smart to say about any
of these areas or maybe about related tools (i.e. you wrote a fancy new
tool to measure boot performance) or fancy boot schemes in your
favourite Linux based OS (i.e. the new Meego zero second boot ;-)) then
don't hesitate to submit something on the LPC web site, in the Boot&Init
track!

And now, quickly, go to the
LPC website
and post your session proposal in the Audio resp. Boot&Init; track! Thank you!

Plumbers Conference 2011

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/lpc2011.html

The Linux Plumbers
Conference 2011 in Santa Rosa, CA, USA
is coming nearer (Sep. 7-9).
Together with Kay Sievers I am running the Boot&Init track, and together with
Mark Brown the Audio track.

For both tracks we still need proposals. So if you haven’t submitted
anything yet, please consider doing so. And that quickly. i.e. if you can
arrange for it, last sunday would be best, since that was actually the final
deadline. However, the submission form is still open, so if you submit
something really, really quickly we’ll ignore the absence of time travel and the calendar for a bit. So, go,
submit something. Now.

What are we looking for? Well, here’s what I just posted on the audio
related mailing lists
:

So, please consider submitting something if you haven’t done so yet. We
are looking for all kinds of technical talks covering everything audio
plumbing related: audio drivers, audio APIs, sound servers, pro audio,
consumer audio. If you can propose something audio related — like talks
on media controller routing, on audio for ASOC/Embedded, submit
something! If you care for low-latency audio, submit something. If you
care about the Linux audio stack in general, submit something.

LPC is probably the most relevant technical conference on the general
Linux platform, so be sure that if you want your project, your work,
your ideas to be heard then this is the right forum for everything
related to the Linux stack. And the Audio track covers everything in our
Audio Stack, regardless whether it is pro or consumer audio.

And here’s what I posted to the init
related lists
:

So, please consider submitting something if you haven’t done so yet. We
are looking for all kinds of technical talks covering everything from
the BIOS (i.e. CoreBoot and friends), over boot loaders (i.e. GRUB and
friends), to initramfs (i.e. Dracut and friends) and init systems
(i.e. systemd and friends). If you have something smart to say about any
of these areas or maybe about related tools (i.e. you wrote a fancy new
tool to measure boot performance) or fancy boot schemes in your
favourite Linux based OS (i.e. the new Meego zero second boot ;-)) then
don’t hesitate to submit something on the LPC web site, in the Boot&Init
track!

And now, quickly, go to the
LPC website
and post your session proposal in the Audio resp. Boot&Init; track! Thank you!

Plumbers Conference 2011

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/lpc2011.html

The Linux Plumbers
Conference 2011 in Santa Rosa, CA, USA
is coming nearer (Sep. 7-9).
Together with Kay Sievers I am running the Boot&Init track, and together with
Mark Brown the Audio track.

For both tracks we still need proposals. So if you haven’t submitted
anything yet, please consider doing so. And that quickly. i.e. if you can
arrange for it, last sunday would be best, since that was actually the final
deadline. However, the submission form is still open, so if you submit
something really, really quickly we’ll ignore the absence of time travel and the calendar for a bit. So, go,
submit something. Now.

What are we looking for? Well, here’s what I just posted on the audio
related mailing lists
:

So, please consider submitting something if you haven't done so yet. We
are looking for all kinds of technical talks covering everything audio
plumbing related: audio drivers, audio APIs, sound servers, pro audio,
consumer audio. If you can propose something audio related -- like talks
on media controller routing, on audio for ASOC/Embedded, submit
something! If you care for low-latency audio, submit something. If you
care about the Linux audio stack in general, submit something.

LPC is probably the most relevant technical conference on the general
Linux platform, so be sure that if you want your project, your work,
your ideas to be heard then this is the right forum for everything
related to the Linux stack. And the Audio track covers everything in our
Audio Stack, regardless whether it is pro or consumer audio.

And here’s what I posted to the init
related lists
:

So, please consider submitting something if you haven't done so yet. We
are looking for all kinds of technical talks covering everything from
the BIOS (i.e. CoreBoot and friends), over boot loaders (i.e. GRUB and
friends), to initramfs (i.e. Dracut and friends) and init systems
(i.e. systemd and friends). If you have something smart to say about any
of these areas or maybe about related tools (i.e. you wrote a fancy new
tool to measure boot performance) or fancy boot schemes in your
favourite Linux based OS (i.e. the new Meego zero second boot ;-)) then
don't hesitate to submit something on the LPC web site, in the Boot&Init
track!

And now, quickly, go to the
LPC website
and post your session proposal in the Audio resp. Boot&Init; track! Thank you!