Tag Archives: amf

FOSS Project Spotlight: LinuxBoot (Linux Journal)

Post Syndicated from jake original https://lwn.net/Articles/747380/rss

Linux Journal takes a look at the newly announced LinuxBoot project. LWN covered a related talk back in November. “Modern firmware generally consists of two main parts: hardware initialization (early stages) and OS loading (late stages). These parts may be divided further depending on the implementation, but the overall flow is similar across boot firmware. The late stages have gained many capabilities over the years and often have an environment with drivers, utilities, a shell, a graphical menu (sometimes with 3D animations) and much more. Runtime components may remain resident and active after firmware exits. Firmware, which used to fit in an 8 KiB ROM, now contains an OS used to boot another OS and doesn’t always stop running after the OS boots. LinuxBoot replaces the late stages with a Linux kernel and initramfs, which are used to load and execute the next stage, whatever it may be and wherever it may come from. The Linux kernel included in LinuxBoot is called the ‘boot kernel’ to distinguish it from the ‘target kernel’ that is to be booted and may be something other than Linux.

Wanted: Sales Engineer

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-sales-engineer/

At inception, Backblaze was a consumer company. Thousands upon thousands of individuals came to our website and gave us $5/mo to keep their data safe. But, we didn’t sell business solutions. It took us years before we had a sales team. In the last couple of years, we’ve released products that businesses of all sizes love: Backblaze B2 Cloud Storage and Backblaze for Business Computer Backup. Those businesses want to integrate Backblaze deeply into their infrastructure, so it’s time to hire our first Sales Engineer!

Company Description:
Founded in 2007, Backblaze started with a mission to make backup software elegant and provide complete peace of mind. Over the course of almost a decade, we have become a pioneer in robust, scalable low cost cloud backup. Recently, we launched B2 – robust and reliable object storage at just $0.005/gb/mo. Part of our differentiation is being able to offer the lowest price of any of the big players while still being profitable.

We’ve managed to nurture a team oriented culture with amazingly low turnover. We value our people and their families. Don’t forget to check out our “About Us” page to learn more about the people and some of our perks.

We have built a profitable, high growth business. While we love our investors, we have maintained control over the business. That means our corporate goals are simple – grow sustainably and profitably.

Some Backblaze Perks:

  • Competitive healthcare plans
  • Competitive compensation and 401k
  • All employees receive Option grants
  • Unlimited vacation days
  • Strong coffee
  • Fully stocked Micro kitchen
  • Catered breakfast and lunches
  • Awesome people who work on awesome projects
  • Childcare bonus
  • Normal work hours
  • Get to bring your pets into the office
  • San Mateo Office – located near Caltrain and Highways 101 & 280.

Backblaze B2 cloud storage is a building block for almost any computing service that requires storage. Customers need our help integrating B2 into iOS apps to Docker containers. Some customers integrate directly to the API using the programming language of their choice, others want to solve a specific problem using ready made software, already integrated with B2.

At the same time, our computer backup product is deepening it’s integration into enterprise IT systems. We are commonly asked for how to set Windows policies, integrate with Active Directory, and install the client via remote management tools.

We are looking for a sales engineer who can help our customers navigate the integration of Backblaze into their technical environments.

Are you 1/2” deep into many different technologies, and unafraid to dive deeper?

Can you confidently talk with customers about their technology, even if you have to look up all the acronyms right after the call?

Are you excited to setup complicated software in a lab and write knowledge base articles about your work?

Then Backblaze is the place for you!

Enough about Backblaze already, what’s in it for me?
In this role, you will be given the opportunity to learn about the technologies that drive innovation today; diverse technologies that customers are using day in and out. And more importantly, you’ll learn how to learn new technologies.

Just as an example, in the past 12 months, we’ve had the opportunity to learn and become experts in these diverse technologies:

  • How to setup VM servers for lab environments, both on-prem and using cloud services.
  • Create an automatically “resetting” demo environment for the sales team.
  • Setup Microsoft Domain Controllers with Active Directory and AD Federation Services.
  • Learn the basics of OAUTH and web single sign on (SSO).
  • Archive video workflows from camera to media asset management systems.
  • How upload/download files from Javascript by enabling CORS.
  • How to install and monitor online backup installations using RMM tools, like JAMF.
  • Tape (LTO) systems. (Yes – people still use tape for storage!)

How can I know if I’ll succeed in this role?

You have:

  • Confidence. Be able to ask customers questions about their environments and convey to them your technical acumen.
  • Curiosity. Always want to learn about customers’ situations, how they got there and what problems they are trying to solve.
  • Organization. You’ll work with customers, integration partners, and Backblaze team members on projects of various lengths. You can context switch and either have a great memory or keep copious notes. Your checklists have their own checklists.

You are versed in:

  • The fundamentals of Windows, Linux and Mac OS X operating systems. You shouldn’t be afraid to use a command line.
  • Building, installing, integrating and configuring applications on any operating system.
  • Debugging failures – reading logs, monitoring usage, effective google searching to fix problems excites you.
  • The basics of TCP/IP networking and the HTTP protocol.
  • Novice development skills in any programming/scripting language. Have basic understanding of data structures and program flow.
  • Your background contains:

  • Bachelor’s degree in computer science or the equivalent.
  • 2+ years of experience as a pre or post-sales engineer.
  • The right extra credit:
    There are literally hundreds of previous experiences you can have had that would make you perfect for this job. Some experiences that we know would be helpful for us are below, but make sure you tell us your stories!

  • Experience using or programming against Amazon S3.
  • Experience with large on-prem storage – NAS, SAN, Object. And backing up data on such storage with tools like Veeam, Veritas and others.
  • Experience with photo or video media. Media archiving is a key market for Backblaze B2.
  • Program arduinos to automatically feed your dog.
  • Experience programming against web or REST APIs. (Point us towards your projects, if they are open source and available to link to.)
  • Experience with sales tools like Salesforce.
  • 3D print door stops.
  • Experience with Windows Servers, Active Directory, Group policies and the like.
  • What’s it like working with the Sales team?
    The Backblaze sales team collaborates. We help each other out by sharing ideas, templates, and our customer’s experiences. When we talk about our accomplishments, there is no “I did this,” only “we”. We are truly a team.

    We are honest to each other and our customers and communicate openly. We aim to have fun by embracing crazy ideas and creative solutions. We try to think not outside the box, but with no boxes at all. Customers are the driving force behind the success of the company and we care deeply about their success.

    If this all sounds like you:

    1. Send an email to [email protected] with the position in the subject line.
    2. Tell us a bit about your Sales Engineering experience.
    3. Include your resume.

    The post Wanted: Sales Engineer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

    Backing Up the Modern Enterprise with Backblaze for Business

    Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/endpoint-backup-solutions/

    Endpoint backup diagram

    Organizations of all types and sizes need reliable and secure backup. Whether they have as few as 3 or as many as 300,000 computer users, an organization’s computer data is a valuable business asset that needs to be protected.

    Modern organizations are changing how they work and where they work, which brings new challenges to making sure that company’s data assets are not only available, but secure. Larger organizations have IT departments that are prepared to address these needs, but often times in smaller and newer organizations the challenge falls upon office management who might not be as prepared or knowledgeable to face a work environment undergoing dramatic changes.

    Whether small or large, local or world-wide, for-profit or non-profit, organizations need a backup strategy and solution that matches the new ways of working in the enterprise.

    The Enterprise Has Changed, and So Has Data Use

    More and more, organizations are working in the cloud. These days organizations can operate just fine without their own file servers, database servers, mail servers, or other IT infrastructure that used to be standard for all but the smallest organization.

    The reality is that for most organizations, though, it’s a hybrid work environment, with a combination of cloud-based and PC and Macintosh-based applications. Legacy apps aren’t going away any time soon. They will be with us for a while, with their accompanying data scattered amongst all the desktops, laptops and other endpoints in corporate headquarters, home offices, hotel rooms, and airport waiting areas.

    In addition, the modern workforce likely combines regular full-time employees, remote workers, contractors, and sometimes interns, volunteers, and other temporary workers who also use company IT assets.

    The Modern Enterprise Brings New Challenges for IT

    These changes in how enterprises work present a problem for anyone tasked with making sure that data — no matter who uses it or where it lives — is adequately backed-up. Cloud-based applications, when properly used and managed, can be adequately backed up, provided that users are connected to the internet and data transfers occur regularly — which is not always the case. But what about the data on the laptops, desktops, and devices used by remote employees, contractors, or just employees whose work keeps them on the road?

    The organization’s backup solution must address all the needs of the modern organization or enterprise using both cloud and PC and Mac-based applications, and not be constrained by employee or computer location.

    A Ten-Point Checklist for the Modern Enterprise for Backing Up

    What should the modern enterprise look for when evaluating a backup solution?

    1) Easy to deploy to workers’ computers

    Whether installed by the computer user or an IT person locally or remotely, the backup solution must be easy to implement quickly with minimal demands on the user or administrator.

    2) Fast and unobtrusive client software

    Backups should happen in the background by efficient (native) PC and Macintosh software clients that don’t consume valuable processing power or take memory away from applications the user needs.

    3) Easy to configure

    The backup solutions must be easy to configure for both the user and the IT professional. Ease-of-use means less time to deploy, configure, and manage.

    4) Defaults to backing up all valuable data

    By default, the solution backs up commonly used files and folders or directories, including desktops. Some backup solutions are difficult and intimidating because they require that the user chose what needs to be backed up, often missing files and folders/directories that contain valuable data.

    5) Works automatically in the background

    Backups should happen automatically, no matter where the computer is located. The computer user, especially the remote or mobile one, shouldn’t be required to attach cables or drives, or remember to initiate backups. A working solution backs up automatically without requiring action by the user or IT administrator.

    6) Data restores are fast and easy

    Whether it’s a single file, directory, or an entire system that must be restored, a user or IT sysadmin needs to be able to restore backed up data as quickly as possible. In cases of large restores to remote locations, the ability to send a restore via physical media is a must.

    7) No limitations on data

    Throttling, caps, and data limits complicate backups and require guesses about how much storage space will be needed.

    8) Safe & Secure

    Organizations require that their data is secure during all phases of initial upload, storage, and restore.

    9) Easy-to-manage

    The backup solution needs to provide a clear and simple web management interface for all functions. Designing for ease-of-use leads to efficiency in management and operation.

    10) Affordable and transparent pricing

    Backup costs should be predictable, understandable, and without surprises.

    Two Scenarios for the Modern Enterprise

    Enterprises exist in many forms and types, but wanting to meet the above requirements is common across all of them. Below, we take a look at two common scenarios showing how enterprises face these challenges. Three case studies are available that provide more information about how Backblaze customers have succeeded in these environments.

    Enterprise Profile 1

    The needs of a smaller enterprise differ from those of larger, established organizations. This organization likely doesn’t have anyone who is devoted full-time to IT. The job of on-boarding new employees and getting them set up with a computer likely falls upon an executive assistant or office manager. This person might give new employees a checklist with the software and account information and lets users handle setting up the computer themselves.

    Organizations in this profile need solutions that are easy to install and require little to no configuration. Backblaze, by default, backs up all user data, which lets the organization be secure in knowing all the data will be backed up to the cloud — including files left on the desktop. Combined with Backblaze’s unlimited data policy, organizations have a truly “set it and forget it” platform.

    Customizing Groups To Meet Teams’ Needs

    The Groups feature of Backblaze for Business allows an organization to decide whether an individual client’s computer will be Unmanaged (backups and restores under the control of the worker), or Managed, in which an administrator can monitor the status and frequency of backups and handle restores should they become necessary. One group for the entire organization might be adequate at this stage, but the organization has the option to add additional groups as it grows and needs more flexibility and control.

    The organization, of course, has the choice of managing and monitoring users using Groups. With Backblaze’s Groups, organizations can set user-based access rules, which allows the administrator to create restores for lost files or entire computers on an employee’s behalf, to centralize billing for all client computers in the organization, and to redeploy a recovered computer or new computer with the backed up data.

    Restores

    In this scenario, the decision has been made to let each user manage her own backups, including restores, if necessary, of individual files or entire systems. If a restore of a file or system is needed, the restore process is easy enough for the user to handle it by herself.

    Case Study 1

    Read about how PagerDuty uses Backblaze for Business in a mixed enterprise of cloud and desktop/laptop applications.

    PagerDuty Case Study

    In a common approach, the employee can retrieve an accidentally deleted file or an earlier version of a document on her own. The Backblaze for Business interface is easy to navigate and was designed with feedback from thousands of customers over the course of a decade.

    In the event of a lost, damaged, or stolen laptop,  administrators of Managed Groups can  initiate the restore, which could be in the form of a download of a restore ZIP file from the web management console, or the overnight shipment of a USB drive directly to the organization or user.

    Enterprise Profile 2

    This profile is for an organization with a full-time IT staff. When a new worker joins the team, the IT staff is tasked with configuring the computer and delivering it to the new employee.

    Backblaze for Business Groups

    Case Study 2

    Global charitable organization charity: water uses Backblaze for Business to back up workers’ and volunteers’ laptops as they travel to developing countries in their efforts to provide clean and safe drinking water.

    charity: water Case Study

    This organization can take advantage of additional capabilities in Groups. A Managed Group makes sense in an organization with a geographically dispersed work force as it lets IT ensure that workers’ data is being regularly backed up no matter where they are. Billing can be company-wide or assigned to individual departments or geographical locations. The organization has the choice of how to divide the organization into Groups (location, function, subsidiary, etc.) and whether the Group should be Managed or Unmanaged. Using Managed Groups might be suitable for most of the organization, but there are exceptions in which sensitive data might dictate using an Unmanaged Group, such as could be the case with HR, the executive team, or finance.

    Deployment

    By Invitation Email, Link, or Domain

    Backblaze for Business allows a number of options for deploying the client software to workers’ computers. Client installation is fast and easy on both Windows and Macintosh, so sending email invitations to users or automatically enrolling users by domain or invitation link, is a common approach.

    By Remote Deployment

    IT might choose to remotely and silently deploy Backblaze for Business across specific Groups or the entire organization. An administrator can silently deploy the Backblaze backup client via the command-line, or use common RMM (Remote Monitoring and Management) tools such as Jamf and Munki.

    Restores

    Case Study 3

    Read about how Bright Bear Technology Solutions, an IT Managed Service Provider (MSP), uses the Groups feature of Backblaze for Business to manage customer backups and restores, deploy Backblaze licenses to their customers, and centralize billing for all their client-based backup services.

    Bright Bear Case Study

    Some organizations are better equipped to manage or assist workers when restores become necessary. Individual users will be pleased to discover they can roll-back files to an earlier version if they wish, but IT will likely manage any complete system restore that involves reconfiguring a computer after a repair or requisitioning an entirely new system when needed.

    This organization might chose to retain a client’s entire computer backup for archival purposes, using Backblaze B2 as the cloud storage solution. This is another advantage of having a cloud storage provider that combines both endpoint backup and cloud object storage among its services.

    The Next Step: Server Backup & Data Archiving with B2 Cloud Storage

    As organizations grow, they have increased needs for cloud storage beyond Macintosh and PC data backup. Backblaze’s object cloud storage, Backblaze B2, provides low-cost storage and archiving of records, media, and server data that can grow with the organization’s size and needs.

    B2 Cloud Storage is available through the same Backblaze management console as Backblaze Computer Backup. This means that Admins have one console for billing, monitoring, deployment, and role provisioning. B2 is priced at 1/4 the cost of Amazon S3, or $0.005 per month per gigabyte (which equals $5/month per terabyte).

    Why Modern Enterprises Chose Backblaze

    Backblaze for Business

    Businesses and organizations select Backblaze for Business for backup because Backblaze is designed to meet the needs of the modern enterprise. Backblaze customers are part of a a platform that has a 10+ year track record of innovation and over 400 petabytes of customer data already under management.

    Backblaze’s backup model is proven through head-to-head comparisons to back up data that other backup solutions overlook in their default configurations — including valuable files that are needed after an accidental deletion, theft, or computer failure.

    Backblaze is the only enterprise-level backup company that provides TOTP (Time-based One-time Password) via both SMS and Authentication app to all accounts at no incremental charge. At just $50/year/computer, Backblaze is affordable for any size of enterprise.

    Modern Enterprises can Meet The Challenge of The Changing Data Environment

    With the right backup solution and strategy, the modern enterprise will be prepared to ensure that its data is protected from accident, disaster, or theft, whether its data is in one office or dispersed among many locations, and remote and mobile employees.

    Backblaze for Business is an affordable solution that enables organizations to meet the evolving data demands facing the modern enterprise.

    The post Backing Up the Modern Enterprise with Backblaze for Business appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

    Backblaze Release 5.1 – RMM Compatibility for Mass Deployments

    Post Syndicated from Yev original https://www.backblaze.com/blog/rmm-for-mass-deployments/

    diagram of Backblaze remote monitoring and management

    Introducing Backblaze Computer Backup Release 5.1

    This is a relatively minor release in terms of the core Backblaze Computer Backup service functionality, but is a big deal for Backblaze for Business as we’ve updated our Mac and PC clients to be RMM (Remote Monitoring and Management) compatible.

    What Is New?

    • Updated Mac and PC clients to better handle large file uploads
    • Updated PC downloader to improve stability
    • Added RMM support for PC and Mac clients

    What Is RMM?

    RMM stands for “Remote Monitoring and Management.” It’s a way to administer computers that might be distributed geographically, without having access to the actual machine. If you are a systems administrator working with anywhere from a few distributed computers to a few thousand, you’re familiar with RMM and how it makes life easier.

    The new clients allow administrators to deploy Backblaze Computer Backup through most “silent” installation/mass deployment tools. Two popular RMM tools are Munki and Jamf. We’ve written up knowledge base articles for both of these.

    munki logo jamf logo
    Learn more about Munki Learn more about Jamf

    Do I Need To Use RMM Tools?

    No — unless you are a systems administrator or someone who is deploying Backblaze to a lot of people all at once, you do not have to worry about RMM support.

    Release Version Number:

    Mac:  5.1.0
    PC:  5.1.0

    Availability:

    October 12, 2017

    Upgrade Methods:

    • “Check for Updates” on the Backblaze Client (right click on the Backblaze icon and then select “Check for Updates”)
    • Download from: https://secure.backblaze.com/update.htm
    • Auto-update will begin in a couple of weeks
    Mac backup update PC backup update
    Updating Backblaze on Mac Updating Backblaze on Windows

    Questions:

    If you have any questions, please contact Backblaze Support at www.backblaze.com/help.

    The post Backblaze Release 5.1 – RMM Compatibility for Mass Deployments appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

    Avoiding TPM PCR fragility using Secure Boot

    Post Syndicated from Matthew Garrett original https://mjg59.dreamwidth.org/48897.html

    In measured boot, each component of the boot process is “measured” (ie, hashed and that hash recorded) in a register in the Trusted Platform Module (TPM) build into the system. The TPM has several different registers (Platform Configuration Registers, or PCRs) which are typically used for different purposes – for instance, PCR0 contains measurements of various system firmware components, PCR2 contains any option ROMs, PCR4 contains information about the partition table and the bootloader. The allocation of these is defined by the PC Client working group of the Trusted Computing Group. However, once the boot loader takes over, we’re outside the spec[1].

    One important thing to note here is that the TPM doesn’t actually have any ability to directly interfere with the boot process. If you try to boot modified code on a system, the TPM will contain different measurements but boot will still succeed. What the TPM can do is refuse to hand over secrets unless the measurements are correct. This allows for configurations where your disk encryption key can be stored in the TPM and then handed over automatically if the measurements are unaltered. If anybody interferes with your boot process then the measurements will be different, the TPM will refuse to hand over the key, your disk will remain encrypted and whoever’s trying to compromise your machine will be sad.

    The problem here is that a lot of things can affect the measurements. Upgrading your bootloader or kernel will do so. At that point if you reboot your disk fails to unlock and you become unhappy. To get around this your update system needs to notice that a new component is about to be installed, generate the new expected hashes and re-seal the secret to the TPM using the new hashes. If there are several different points in the update where this can happen, this can quite easily go wrong. And if it goes wrong, you’re back to being unhappy.

    Is there a way to improve this? Surprisingly, the answer is “yes” and the people to thank are Microsoft. Appendix A of a basically entirely unrelated spec defines a mechanism for storing the UEFI Secure Boot policy and used keys in PCR 7 of the TPM. The idea here is that you trust your OS vendor (since otherwise they could just backdoor your system anyway), so anything signed by your OS vendor is acceptable. If someone tries to boot something signed by a different vendor then PCR 7 will be different. If someone disables secure boot, PCR 7 will be different. If you upgrade your bootloader or kernel, PCR 7 will be the same. This simplifies things significantly.

    I’ve put together a (not well-tested) patchset for Shim that adds support for including Shim’s measurements in PCR 7. In conjunction with appropriate firmware, it should then be straightforward to seal secrets to PCR 7 and not worry about things breaking over system updates. This makes tying things like disk encryption keys to the TPM much more reasonable.

    However, there’s still one pretty major problem, which is that the initramfs (ie, the component responsible for setting up the disk encryption in the first place) isn’t signed and isn’t included in PCR 7[2]. An attacker can simply modify it to stash any TPM-backed secrets or mount the encrypted filesystem and then drop to a root prompt. This, uh, reduces the utility of the entire exercise.

    The simplest solution to this that I’ve come up with depends on how Linux implements initramfs files. In its simplest form, an initramfs is just a cpio archive. In its slightly more complicated form, it’s a compressed cpio archive. And in its peak form of evolution, it’s a series of compressed cpio archives concatenated together. As the kernel reads each one in turn, it extracts it over the previous ones. That means that any files in the final archive will overwrite files of the same name in previous archives.

    My proposal is to generate a small initramfs whose sole job is to get secrets from the TPM and stash them in the kernel keyring, and then measure an additional value into PCR 7 in order to ensure that the secrets can’t be obtained again. Later disk encryption setup will then be able to set up dm-crypt using the secret already stored within the kernel. This small initramfs will be built into the signed kernel image, and the bootloader will be responsible for appending it to the end of any user-provided initramfs. This means that the TPM will only grant access to the secrets while trustworthy code is running – once the secret is in the kernel it will only be available for in-kernel use, and once PCR 7 has been modified the TPM won’t give it to anyone else. A similar approach for some kernel command-line arguments (the kernel, module-init-tools and systemd all interpret the kernel command line left-to-right, with later arguments overriding earlier ones) would make it possible to ensure that certain kernel configuration options (such as the iommu) weren’t overridable by an attacker.

    There’s obviously a few things that have to be done here (standardise how to embed such an initramfs in the kernel image, ensure that luks knows how to use the kernel keyring, teach all relevant bootloaders how to handle these images), but overall this should make it practical to use PCR 7 as a mechanism for supporting TPM-backed disk encryption secrets on Linux without introducing a hug support burden in the process.

    [1] The patchset I’ve posted to add measured boot support to Grub use PCRs 8 and 9 to measure various components during the boot process, but other bootloaders may have different policies.

    [2] This is because most Linux systems generate the initramfs locally rather than shipping it pre-built. It may also get rebuilt on various userspace updates, even if the kernel hasn’t changed. Including it in PCR 7 would entirely break the fragility guarantees and defeat the point of all of this.

    comment count unavailable comments

    Avoiding TPM PCR fragility using Secure Boot

    Post Syndicated from Matthew Garrett original http://mjg59.dreamwidth.org/48897.html

    In measured boot, each component of the boot process is “measured” (ie, hashed and that hash recorded) in a register in the Trusted Platform Module (TPM) build into the system. The TPM has several different registers (Platform Configuration Registers, or PCRs) which are typically used for different purposes – for instance, PCR0 contains measurements of various system firmware components, PCR2 contains any option ROMs, PCR4 contains information about the partition table and the bootloader. The allocation of these is defined by the PC Client working group of the Trusted Computing Group. However, once the boot loader takes over, we’re outside the spec[1].

    One important thing to note here is that the TPM doesn’t actually have any ability to directly interfere with the boot process. If you try to boot modified code on a system, the TPM will contain different measurements but boot will still succeed. What the TPM can do is refuse to hand over secrets unless the measurements are correct. This allows for configurations where your disk encryption key can be stored in the TPM and then handed over automatically if the measurements are unaltered. If anybody interferes with your boot process then the measurements will be different, the TPM will refuse to hand over the key, your disk will remain encrypted and whoever’s trying to compromise your machine will be sad.

    The problem here is that a lot of things can affect the measurements. Upgrading your bootloader or kernel will do so. At that point if you reboot your disk fails to unlock and you become unhappy. To get around this your update system needs to notice that a new component is about to be installed, generate the new expected hashes and re-seal the secret to the TPM using the new hashes. If there are several different points in the update where this can happen, this can quite easily go wrong. And if it goes wrong, you’re back to being unhappy.

    Is there a way to improve this? Surprisingly, the answer is “yes” and the people to thank are Microsoft. Appendix A of a basically entirely unrelated spec defines a mechanism for storing the UEFI Secure Boot policy and used keys in PCR 7 of the TPM. The idea here is that you trust your OS vendor (since otherwise they could just backdoor your system anyway), so anything signed by your OS vendor is acceptable. If someone tries to boot something signed by a different vendor then PCR 7 will be different. If someone disables secure boot, PCR 7 will be different. If you upgrade your bootloader or kernel, PCR 7 will be the same. This simplifies things significantly.

    I’ve put together a (not well-tested) patchset for Shim that adds support for including Shim’s measurements in PCR 7. In conjunction with appropriate firmware, it should then be straightforward to seal secrets to PCR 7 and not worry about things breaking over system updates. This makes tying things like disk encryption keys to the TPM much more reasonable.

    However, there’s still one pretty major problem, which is that the initramfs (ie, the component responsible for setting up the disk encryption in the first place) isn’t signed and isn’t included in PCR 7[2]. An attacker can simply modify it to stash any TPM-backed secrets or mount the encrypted filesystem and then drop to a root prompt. This, uh, reduces the utility of the entire exercise.

    The simplest solution to this that I’ve come up with depends on how Linux implements initramfs files. In its simplest form, an initramfs is just a cpio archive. In its slightly more complicated form, it’s a compressed cpio archive. And in its peak form of evolution, it’s a series of compressed cpio archives concatenated together. As the kernel reads each one in turn, it extracts it over the previous ones. That means that any files in the final archive will overwrite files of the same name in previous archives.

    My proposal is to generate a small initramfs whose sole job is to get secrets from the TPM and stash them in the kernel keyring, and then measure an additional value into PCR 7 in order to ensure that the secrets can’t be obtained again. Later disk encryption setup will then be able to set up dm-crypt using the secret already stored within the kernel. This small initramfs will be built into the signed kernel image, and the bootloader will be responsible for appending it to the end of any user-provided initramfs. This means that the TPM will only grant access to the secrets while trustworthy code is running – once the secret is in the kernel it will only be available for in-kernel use, and once PCR 7 has been modified the TPM won’t give it to anyone else. A similar approach for some kernel command-line arguments (the kernel, module-init-tools and systemd all interpret the kernel command line left-to-right, with later arguments overriding earlier ones) would make it possible to ensure that certain kernel configuration options (such as the iommu) weren’t overridable by an attacker.

    There’s obviously a few things that have to be done here (standardise how to embed such an initramfs in the kernel image, ensure that luks knows how to use the kernel keyring, teach all relevant bootloaders how to handle these images), but overall this should make it practical to use PCR 7 as a mechanism for supporting TPM-backed disk encryption secrets on Linux without introducing a hug support burden in the process.

    [1] The patchset I’ve posted to add measured boot support to Grub use PCRs 8 and 9 to measure various components during the boot process, but other bootloaders may have different policies.

    [2] This is because most Linux systems generate the initramfs locally rather than shipping it pre-built. It may also get rebuilt on various userspace updates, even if the kernel hasn’t changed. Including it in PCR 7 would entirely break the fragility guarantees and defeat the point of all of this.

    comment count unavailable comments

    Manage Kubernetes Clusters on AWS Using Kops

    Post Syndicated from Arun Gupta original https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/

    Any containerized application typically consists of multiple containers. There is a container for the application itself, one for database, possibly another for web server, and so on. During development, its normal to build and test this multi-container application on a single host. This approach works fine during early dev and test cycles but becomes a single point of failure for production where the availability of the application is critical. In such cases, this multi-container application is deployed on multiple hosts. There is a need for an external tool to manage such a multi-container multi-host deployment. Container orchestration frameworks provides the capability of cluster management, scheduling containers on different hosts, service discovery and load balancing, crash recovery and other related functionalities. There are multiple options for container orchestration on Amazon Web Services: Amazon ECS, Docker for AWS, and DC/OS.

    Another popular option for container orchestration on AWS is Kubernetes. There are multiple ways to run a Kubernetes cluster on AWS. This multi-part blog series provides a brief overview and explains some of these approaches in detail. This first post explains how to create a Kubernetes cluster on AWS using kops.

    Kubernetes and Kops overview

    Kubernetes is an open source, container orchestration platform. Applications packaged as Docker images can be easily deployed, scaled, and managed in a Kubernetes cluster. Some of the key features of Kubernetes are:

    • Self-healing
      Failed containers are restarted to ensure that the desired state of the application is maintained. If a node in the cluster dies, then the containers are rescheduled on a different node. Containers that do not respond to application-defined health check are terminated, and thus rescheduled.
    • Horizontal scaling
      Number of containers can be easily scaled up and down automatically based upon CPU utilization, or manually using a command.
    • Service discovery and load balancing
      Multiple containers can be grouped together discoverable using a DNS name. The service can be load balanced with integration to the native LB provided by the cloud provider.
    • Application upgrades and rollbacks
      Applications can be upgraded to a newer version without an impact to the existing one. If something goes wrong, Kubernetes rolls back the change.

    Kops, short for Kubernetes Operations, is a set of tools for installing, operating, and deleting Kubernetes clusters in the cloud. A rolling upgrade of an older version of Kubernetes to a new version can also be performed. It also manages the cluster add-ons. After the cluster is created, the usual kubectl CLI can be used to manage resources in the cluster.

    Download Kops and Kubectl

    There is no need to download the Kubernetes binary distribution for creating a cluster using kops. However, you do need to download the kops CLI. It then takes care of downloading the right Kubernetes binary in the cloud, and provisions the cluster.

    The different download options for kops are explained at github.com/kubernetes/kops#installing. On MacOS, the easiest way to install kops is using the brew package manager.

    brew update && brew install kops

    The version of kops can be verified using the kops version command, which shows:

    Version 1.6.1

    In addition, download kubectl. This is required to manage the Kubernetes cluster. The latest version of kubectl can be downloaded using the following command:

    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl

    Make sure to include the directory where kubectl is downloaded in your PATH.

    IAM user permission

    The IAM user to create the Kubernetes cluster must have the following permissions:

    • AmazonEC2FullAccess
    • AmazonRoute53FullAccess
    • AmazonS3FullAccess
    • IAMFullAccess
    • AmazonVPCFullAccess

    Alternatively, a new IAM user may be created and the policies attached as explained at github.com/kubernetes/kops/blob/master/docs/aws.md#setup-iam-user.

    Create an Amazon S3 bucket for the Kubernetes state store

    Kops needs a “state store” to store configuration information of the cluster.  For example, how many nodes, instance type of each node, and Kubernetes version. The state is stored during the initial cluster creation. Any subsequent changes to the cluster are also persisted to this store as well. As of publication, Amazon S3 is the only supported storage mechanism. Create a S3 bucket and pass that to the kops CLI during cluster creation.

    This post uses the bucket name kubernetes-aws-io. Bucket names must be unique; you have to use a different name. Create an S3 bucket:

    aws s3api create-bucket --bucket kubernetes-aws-io

    I strongly recommend versioning this bucket in case you ever need to revert or recover a previous version of the cluster. This can be enabled using the AWS CLI as well:

    aws s3api put-bucket-versioning --bucket kubernetes-aws-io --versioning-configuration Status=Enabled

    For convenience, you can also define KOPS_STATE_STORE environment variable pointing to the S3 bucket. For example:

    export KOPS_STATE_STORE=s3://kubernetes-aws-io

    This environment variable is then used by the kops CLI.

    DNS configuration

    As of Kops 1.6.1, a top-level domain or a subdomain is required to create the cluster. This domain allows the worker nodes to discover the master and the master to discover all the etcd servers. This is also needed for kubectl to be able to talk directly with the master.

    This domain may be registered with AWS, in which case a Route 53 hosted zone is created for you. Alternatively, this domain may be at a different registrar. In this case, create a Route 53 hosted zone. Specify the name server (NS) records from the created zone as NS records with the domain registrar.

    This post uses a kubernetes-aws.io domain registered at a third-party registrar.

    Generate a Route 53 hosted zone using the AWS CLI. Download jq to run this command:

    ID=$(uuidgen) && \
    aws route53 create-hosted-zone \
    --name cluster.kubernetes-aws.io \
    --caller-reference $ID \
    | jq .DelegationSet.NameServers

    This shows an output such as the following:

    [
    "ns-94.awsdns-11.com",
    "ns-1962.awsdns-53.co.uk",
    "ns-838.awsdns-40.net",
    "ns-1107.awsdns-10.org"
    ]

    Create NS records for the domain with your registrar. Different options on how to configure DNS for the cluster are explained at github.com/kubernetes/kops/blob/master/docs/aws.md#configure-dns.

    Experimental support to create a gossip-based cluster was added in Kops 1.6.2. This post uses a DNS-based approach, as that is more mature and well tested.

    Create the Kubernetes cluster

    The Kops CLI can be used to create a highly available cluster, with multiple master nodes spread across multiple Availability Zones. Workers can be spread across multiple zones as well. Some of the tasks that happen behind the scene during cluster creation are:

    • Provisioning EC2 instances
    • Setting up AWS resources such as networks, Auto Scaling groups, IAM users, and security groups
    • Installing Kubernetes.

    Start the Kubernetes cluster using the following command:

    kops create cluster \
    --name cluster.kubernetes-aws.io \
    --zones us-west-2a \
    --state s3://kubernetes-aws-io \
    --yes

    In this command:

    • --zones
      Defines the zones in which the cluster is going to be created. Multiple comma-separated zones can be specified to span the cluster across multiple zones.
    • --name
      Defines the cluster’s name.
    • --state
      Points to the S3 bucket that is the state store.
    • --yes
      Immediately creates the cluster. Otherwise, only the cloud resources are created and the cluster needs to be started explicitly using the command kops update --yes. If the cluster needs to be edited, then the kops edit cluster command can be used.

    This starts a single master and two worker node Kubernetes cluster. The master is in an Auto Scaling group and the worker nodes are in a separate group. By default, the master node is m3.medium and the worker node is t2.medium. Master and worker nodes are assigned separate IAM roles as well.

    Wait for a few minutes for the cluster to be created. The cluster can be verified using the command kops validate cluster --state=s3://kubernetes-aws-io. It shows the following output:

    Using cluster from kubectl context: cluster.kubernetes-aws.io
    
    Validating cluster cluster.kubernetes-aws.io
    
    INSTANCE GROUPS
    NAME                 ROLE      MACHINETYPE    MIN    MAX    SUBNETS
    master-us-west-2a    Master    m3.medium      1      1      us-west-2a
    nodes                Node      t2.medium      2      2      us-west-2a
    
    NODE STATUS
    NAME                                           ROLE      READY
    ip-172-20-38-133.us-west-2.compute.internal    node      True
    ip-172-20-38-177.us-west-2.compute.internal    master    True
    ip-172-20-46-33.us-west-2.compute.internal     node      True
    
    Your cluster cluster.kubernetes-aws.io is ready

    It shows the different instances started for the cluster, and their roles. If multiple cluster states are stored in the same bucket, then --name <NAME> can be used to specify the exact cluster name.

    Check all nodes in the cluster using the command kubectl get nodes:

    NAME                                          STATUS         AGE       VERSION
    ip-172-20-38-133.us-west-2.compute.internal   Ready,node     14m       v1.6.2
    ip-172-20-38-177.us-west-2.compute.internal   Ready,master   15m       v1.6.2
    ip-172-20-46-33.us-west-2.compute.internal    Ready,node     14m       v1.6.2

    Again, the internal IP address of each node, their current status (master or node), and uptime are shown. The key information here is the Kubernetes version for each node in the cluster, 1.6.2 in this case.

    The kubectl value included in the PATH earlier is configured to manage this cluster. Resources such as pods, replica sets, and services can now be created in the usual way.

    Some of the common options that can be used to override the default cluster creation are:

    • --kubernetes-version
      The version of Kubernetes cluster. The exact versions supported are defined at github.com/kubernetes/kops/blob/master/channels/stable.
    • --master-size and --node-size
      Define the instance of master and worker nodes.
    • --master-count and --node-count
      Define the number of master and worker nodes. By default, a master is created in each zone specified by --master-zones. Multiple master nodes can be created by a higher number using --master-count or specifying multiple Availability Zones in --master-zones.

    A three-master and five-worker node cluster, with master nodes spread across different Availability Zones, can be created using the following command:

    kops create cluster \
    --name cluster2.kubernetes-aws.io \
    --zones us-west-2a,us-west-2b,us-west-2c \
    --node-count 5 \
    --state s3://kubernetes-aws-io \
    --yes

    Both the clusters are sharing the same state store but have different names. This also requires you to create an additional Amazon Route 53 hosted zone for the name.

    By default, the resources required for the cluster are directly created in the cloud. The --target option can be used to generate the AWS CloudFormation scripts instead. These scripts can then be used by the AWS CLI to create resources at your convenience.

    Get a complete list of options for cluster creation with kops create cluster --help.

    More details about the cluster can be seen using the command kubectl cluster-info:

    Kubernetes master is running at https://api.cluster.kubernetes-aws.io
    KubeDNS is running at https://api.cluster.kubernetes-aws.io/api/v1/proxy/namespaces/kube-system/services/kube-dns
    
    To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

    Check the client and server version using the command kubectl version:

    Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

    Both client and server version are 1.6 as shown by the Major and Minor attribute values.

    Upgrade the Kubernetes cluster

    Kops can be used to create a Kubernetes 1.4.x, 1.5.x, or an older version of the 1.6.x cluster using the --kubernetes-version option. The exact versions supported are defined at github.com/kubernetes/kops/blob/master/channels/stable.

    Or, you may have used kops to create a cluster a while ago, and now want to upgrade to the latest recommended version of Kubernetes. Kops supports rolling cluster upgrades where the master and worker nodes are upgraded one by one.

    As of kops 1.6.1, upgrading a cluster is a three-step process.

    First, check and apply the latest recommended Kubernetes update.

    kops upgrade cluster \
    --name cluster2.kubernetes-aws.io \
    --state s3://kubernetes-aws-io \
    --yes

    The --yes option immediately applies the changes. Not specifying the --yes option shows only the changes that are applied.

    Second, update the state store to match the cluster state. This can be done using the following command:

    kops update cluster \
    --name cluster2.kubernetes-aws.io \
    --state s3://kubernetes-aws-io \
    --yes

    Lastly, perform a rolling update for all cluster nodes using the kops rolling-update command:

    kops rolling-update cluster \
    --name cluster2.kubernetes-aws.io \
    --state s3://kubernetes-aws-io \
    --yes

    Previewing the changes before updating the cluster can be done using the same command but without specifying the --yes option. This shows the following output:

    NAME                 STATUS        NEEDUPDATE    READY    MIN    MAX    NODES
    master-us-west-2a    NeedsUpdate   1             0        1      1      1
    nodes                NeedsUpdate   2             0        2      2      2

    Using --yes updates all nodes in the cluster, first master and then worker. There is a 5-minute delay between restarting master nodes, and a 2-minute delay between restarting nodes. These values can be altered using --master-interval and --node-interval options, respectively.

    Only the worker nodes may be updated by using the --instance-group node option.

    Delete the Kubernetes cluster

    Typically, the Kubernetes cluster is a long-running cluster to serve your applications. After its purpose is served, you may delete it. It is important to delete the cluster using the kops command. This ensures that all resources created by the cluster are appropriately cleaned up.

    The command to delete the Kubernetes cluster is:

    kops delete cluster --state=s3://kubernetes-aws-io --yes

    If multiple clusters have been created, then specify the cluster name as in the following command:

    kops delete cluster cluster2.kubernetes-aws.io --state=s3://kubernetes-aws-io --yes

    Conclusion

    This post explained how to manage a Kubernetes cluster on AWS using kops. Kubernetes on AWS users provides a self-published list of companies using Kubernetes on AWS.

    Try starting a cluster, create a few Kubernetes resources, and then tear it down. Kops on AWS provides a more comprehensive tutorial for setting up Kubernetes clusters. Kops docs are also helpful for understanding the details.

    In addition, the Kops team hosts office hours to help you get started, from guiding you with your first pull request. You can always join the #kops channel on Kubernetes slack to ask questions. If nothing works, then file an issue at github.com/kubernetes/kops/issues.

    Future posts in this series will explain other ways of creating and running a Kubernetes cluster on AWS.

    — Arun

    Building High-Throughput Genomic Batch Workflows on AWS: Batch Layer (Part 3 of 4)

    Post Syndicated from Andy Katz original https://aws.amazon.com/blogs/compute/building-high-throughput-genomic-batch-workflows-on-aws-batch-layer-part-3-of-4/

    Aaron Friedman is a Healthcare and Life Sciences Partner Solutions Architect at AWS

    Angel Pizarro is a Scientific Computing Technical Business Development Manager at AWS

    This post is the third in a series on how to build a genomics workflow on AWS. In Part 1, we introduced a general architecture, shown below, and highlighted the three common layers in a batch workflow:

    • Job
    • Batch
    • Workflow

    In Part 2, you built a Docker container for each job that needed to run as part of your workflow, and stored them in Amazon ECR.

    In Part 3, you tackle the batch layer and build a scalable, elastic, and easily maintainable batch engine using AWS Batch.

    AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. It dynamically provisions the optimal quantity and type of compute resources (for example, CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs that you submit. With AWS Batch, you do not need to install and manage your own batch computing software or server clusters, which allows you to focus on analyzing results, such as those of your genomic analysis.

    Integrating applications into AWS Batch

    If you are new to AWS Batch, we recommend reading Setting Up AWS Batch to ensure that you have the proper permissions and AWS environment.

    After you have a working environment, you define several types of resources:

    • IAM roles that provide service permissions
    • A compute environment that launches and terminates compute resources for jobs
    • A custom Amazon Machine Image (AMI)
    • A job queue to submit the units of work and to schedule the appropriate resources within the compute environment to execute those jobs
    • Job definitions that define how to execute an application

    After the resources are created, you’ll test the environment and create an AWS Lambda function to send generic jobs to the queue.

    This genomics workflow covers the basic steps. For more information, see Getting Started with AWS Batch.

    Creating the necessary IAM roles

    AWS Batch simplifies batch processing by managing a number of underlying AWS services so that you can focus on your applications. As a result, you create IAM roles that give the service permissions to act on your behalf. In this section, deploy the AWS CloudFormation template included in the GitHub repository and extract the ARNs for later use.

    To deploy the stack, go to the top level in the repo with the following command:

    aws cloudformation create-stack --template-body file://batch/setup/iam.template.yaml --stack-name iam --capabilities CAPABILITY_NAMED_IAM

    You can capture the output from this stack in the Outputs tab in the CloudFormation console:

    Creating the compute environment

    In AWS Batch, you will set up a managed compute environments. Managed compute environments automatically launch and terminate compute resources on your behalf based on the aggregate resources needed by your jobs, such as vCPU and memory, and simple boundaries that you define.

    When defining your compute environment, specify the following:

    • Desired instance types in your environment
    • Min and max vCPUs in the environment
    • The Amazon Machine Image (AMI) to use
    • Percentage value for bids on the Spot Market and VPC subnets that can be used.

    AWS Batch then provisions an elastic and heterogeneous pool of Amazon EC2 instances based on the aggregate resource requirements of jobs sitting in the RUNNABLE state. If a mix of CPU and memory-intensive jobs are ready to run, AWS Batch provisions the appropriate ratio and size of CPU and memory-optimized instances within your environment. For this post, you will use the simplest configuration, in which instance types are set to "optimal" allowing AWS Batch to choose from the latest C, M, and R EC2 instance families.

    While you could create this compute environment in the console, we provide the following CLI commands. Replace the subnet IDs and key name with your own private subnets and key, and the image-id with the image you will build in the next section.

    ACCOUNTID=<your account id>
    SERVICEROLE=<from output in CloudFormation template>
    IAMFLEETROLE=<from output in CloudFormation template>
    JOBROLEARN=<from output in CloudFormation template>
    SUBNETS=<comma delimited list of subnets>
    SECGROUPS=<your security groups>
    SPOTPER=50 # percentage of on demand
    IMAGEID=<ami-id corresponding to the one you created>
    INSTANCEROLE=<from output in CloudFormation template>
    REGISTRY=${ACCOUNTID}.dkr.ecr.us-east-1.amazonaws.com
    KEYNAME=<your key name>
    MAXCPU=1024 # max vCPUs in compute environment
    ENV=myenv
    
    # Creates the compute environment
    aws batch create-compute-environment --compute-environment-name genomicsEnv-$ENV --type MANAGED --state ENABLED --service-role ${SERVICEROLE} --compute-resources type=SPOT,minvCpus=0,maxvCpus=$MAXCPU,desiredvCpus=0,instanceTypes=optimal,imageId=$IMAGEID,subnets=$SUBNETS,securityGroupIds=$SECGROUPS,ec2KeyPair=$KEYNAME,instanceRole=$INSTANCEROLE,bidPercentage=$SPOTPER,spotIamFleetRole=$IAMFLEETROLE

    Creating the custom AMI for AWS Batch

    While you can use default Amazon ECS-optimized AMIs with AWS Batch, you can also provide your own image in managed compute environments. We will use this feature to provision additional scratch EBS storage on each of the instances that AWS Batch launches and also to encrypt both the Docker and scratch EBS volumes.

    AWS Batch has the same requirements for your AMI as Amazon ECS. To build the custom image, modify the default Amazon ECS-Optimized Amazon Linux AMI in the following ways:

    • Attach a 1 TB scratch volume to /dev/sdb
    • Encrypt the Docker and new scratch volumes
    • Mount the scratch volume to /docker_scratch by modifying /etcfstab

    The first two tasks can be addressed when you create the custom AMI in the console. Spin up a small t2.micro instance, and proceed through the standard EC2 instance launch.

    After your instance has launched, record the IP address and then SSH into the instance. Copy and paste the following code:

    sudo yum -y update
    sudo parted /dev/xvdb mklabel gpt
    sudo parted /dev/xvdb mkpart primary 0% 100%
    sudo mkfs -t ext4 /dev/xvdb1
    sudo mkdir /docker_scratch
    sudo echo -e '/dev/xvdb1\t/docker_scratch\text4\tdefaults\t0\t0' | sudo tee -a /etc/fstab
    sudo mount -a

    This auto-mounts your scratch volume to /docker_scratch, which is your scratch directory for batch processing. Next, create your new AMI and record the image ID.

    Creating the job queues

    AWS Batch job queues are used to coordinate the submission of batch jobs. Your jobs are submitted to job queues, which can be mapped to one or more compute environments. Job queues have priority relative to each other. You can also specify the order in which they consume resources from your compute environments.

    In this solution, use two job queues. The first is for high priority jobs, such as alignment or variant calling. Set this with a high priority (1000) and map back to the previously created compute environment. Next, set a second job queue for low priority jobs, such as quality statistics generation. To create these compute environments, enter the following CLI commands:

    aws batch create-job-queue --job-queue-name highPriority-${ENV} --compute-environment-order order=0,computeEnvironment=genomicsEnv-${ENV}  --priority 1000 --state ENABLED
    aws batch create-job-queue --job-queue-name lowPriority-${ENV} --compute-environment-order order=0,computeEnvironment=genomicsEnv-${ENV}  --priority 1 --state ENABLED

    Creating the job definitions

    To run the Isaac aligner container image locally, supply the Amazon S3 locations for the FASTQ input sequences, the reference genome to align to, and the output BAM file. For more information, see tools/isaac/README.md.

    The Docker container itself also requires some information on a suitable mountable volume so that it can read and write files temporary files without running out of space.

    Note: In the following example, the FASTQ files as well as the reference files to run are in a publicly available bucket.

    FASTQ1=s3://aws-batch-genomics-resources/fastq/SRR1919605_1.fastq.gz
    FASTQ2=s3://aws-batch-genomics-resources/fastq/SRR1919605_2.fastq.gz
    REF=s3://aws-batch-genomics-resources/reference/isaac/
    BAM=s3://mybucket/genomic-workflow/test_results/bam/
    
    mkdir ~/scratch
    
    docker run --rm -ti -v $(HOME)/scratch:/scratch $REPO_URI --bam_s3_folder_path $BAM \
    --fastq1_s3_path $FASTQ1 \
    --fastq2_s3_path $FASTQ2 \
    --reference_s3_path $REF \
    --working_dir /scratch 

    Locally running containers can typically expand their CPU and memory resource headroom. In AWS Batch, the CPU and memory requirements are hard limits and are allocated to the container image at runtime.

    Isaac is a fairly resource-intensive algorithm, as it creates an uncompressed index of the reference genome in memory to match the query DNA sequences. The large memory space is shared across multiple CPU threads, and Isaac can scale almost linearly with the number of CPU threads given to it as a parameter.

    To fit these characteristics, choose an optimal instance size to maximize the number of CPU threads based on a given large memory footprint, and deploy a Docker container that uses all of the instance resources. In this case, we chose a host instance with 80+ GB of memory and 32+ vCPUs. The following code is example JSON that you can pass to the AWS CLI to create a job definition for Isaac.

    aws batch register-job-definition --job-definition-name isaac-${ENV} --type container --retry-strategy attempts=3 --container-properties '
    {"image": "'${REGISTRY}'/isaac",
    "jobRoleArn":"'${JOBROLEARN}'",
    "memory":80000,
    "vcpus":32,
    "mountPoints": [{"containerPath": "/scratch", "readOnly": false, "sourceVolume": "docker_scratch"}],
    "volumes": [{"name": "docker_scratch", "host": {"sourcePath": "/docker_scratch"}}]
    }'

    You can copy and paste the following code for the other three job definitions:

    aws batch register-job-definition --job-definition-name strelka-${ENV} --type container --retry-strategy attempts=3 --container-properties '
    {"image": "'${REGISTRY}'/strelka",
    "jobRoleArn":"'${JOBROLEARN}'",
    "memory":32000,
    "vcpus":32,
    "mountPoints": [{"containerPath": "/scratch", "readOnly": false, "sourceVolume": "docker_scratch"}],
    "volumes": [{"name": "docker_scratch", "host": {"sourcePath": "/docker_scratch"}}]
    }'
    
    aws batch register-job-definition --job-definition-name snpeff-${ENV} --type container --retry-strategy attempts=3 --container-properties '
    {"image": "'${REGISTRY}'/snpeff",
    "jobRoleArn":"'${JOBROLEARN}'",
    "memory":10000,
    "vcpus":4,
    "mountPoints": [{"containerPath": "/scratch", "readOnly": false, "sourceVolume": "docker_scratch"}],
    "volumes": [{"name": "docker_scratch", "host": {"sourcePath": "/docker_scratch"}}]
    }'
    
    aws batch register-job-definition --job-definition-name samtoolsStats-${ENV} --type container --retry-strategy attempts=3 --container-properties '
    {"image": "'${REGISTRY}'/samtools_stats",
    "jobRoleArn":"'${JOBROLEARN}'",
    "memory":10000,
    "vcpus":4,
    "mountPoints": [{"containerPath": "/scratch", "readOnly": false, "sourceVolume": "docker_scratch"}],
    "volumes": [{"name": "docker_scratch", "host": {"sourcePath": "/docker_scratch"}}]
    }'

    The value for "image" comes from the previous post on creating a Docker image and publishing to ECR. The value for jobRoleArn you can find from the output of the CloudFormation template that you deployed earlier. In addition to providing the number of CPU cores and memory required by Isaac, you also give it a storage volume for scratch and staging. The volume comes from the previously defined custom AMI.

    Testing the environment

    After you have created the Isaac job definition, you can submit the job using the AWS Batch submitJob API action. While the base mappings for Docker run are taken care of in the job definition that you just built, the specific job parameters should be specified in the container overrides section of the API call. Here’s what this would look like in the CLI, using the same parameters as in the bash commands shown earlier:

    aws batch submit-job --job-name testisaac --job-queue highPriority-${ENV} --job-definition isaac-${ENV}:1 --container-overrides '{
    "command": [
    			"--bam_s3_folder_path", "s3://mybucket/genomic-workflow/test_batch/bam/",
                "--fastq1_s3_path", "s3://aws-batch-genomics-resources/fastq/ SRR1919605_1.fastq.gz",
                "--fastq2_s3_path", "s3://aws-batch-genomics-resources/fastq/SRR1919605_2.fastq.gz",
                "--reference_s3_path", "s3://aws-batch-genomics-resources/reference/isaac/",
                "--working_dir", "/scratch",
    			"—cmd_args", " --exome ",]
    }'

    When you execute a submitJob call, jobId is returned. You can then track the progress of your job using the describeJobs API action:

    aws batch describe-jobs –jobs <jobId returned from submitJob>

    You can also track the progress of all of your jobs in the AWS Batch console dashboard.

    To see exactly where a RUNNING job is at, use the link in the AWS Batch console to direct you to the appropriate location in CloudWatch logs.

    Completing the batch environment setup

    To finish, create a Lambda function to submit a generic AWS Batch job.

    In the Lambda console, create a Python 2.7 Lambda function named batchSubmitJob. Copy and paste the following code. This is similar to the batch-submit-job-python27 Lambda blueprint. Use the LambdaBatchExecutionRole that you created earlier. For more information about creating functions, see Step 2.1: Create a Hello World Lambda Function.

    from __future__ import print_function
    
    import json
    import boto3
    
    batch_client = boto3.client('batch')
    
    def lambda_handler(event, context):
        # Log the received event
        print("Received event: " + json.dumps(event, indent=2))
        # Get parameters for the SubmitJob call
        # http://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html
        job_name = event['jobName']
        job_queue = event['jobQueue']
        job_definition = event['jobDefinition']
        
        # containerOverrides, dependsOn, and parameters are optional
        container_overrides = event['containerOverrides'] if event.get('containerOverrides') else {}
        parameters = event['parameters'] if event.get('parameters') else {}
        depends_on = event['dependsOn'] if event.get('dependsOn') else []
        
        try:
            response = batch_client.submit_job(
                dependsOn=depends_on,
                containerOverrides=container_overrides,
                jobDefinition=job_definition,
                jobName=job_name,
                jobQueue=job_queue,
                parameters=parameters
            )
            
            # Log response from AWS Batch
            print("Response: " + json.dumps(response, indent=2))
            
            # Return the jobId
            event['jobId'] = response['jobId']
            return event
        
        except Exception as e:
            print(e)
            message = 'Error getting Batch Job status'
            print(message)
            raise Exception(message)

    Conclusion

    In part 3 of this series, you successfully set up your data processing, or batch, environment in AWS Batch. We also provided a Python script in the corresponding GitHub repo that takes care of all of the above CLI arguments for you, as well as building out the job definitions for all of the jobs in the workflow: Isaac, Strelka, SAMtools, and snpEff. You can check the script’s README for additional documentation.

    In Part 4, you’ll cover the workflow layer using AWS Step Functions and AWS Lambda.

    Please leave any questions and comments below.

    ISP Bombarded With 82,000+ Demands to Reveal Alleged Pirates

    Post Syndicated from Andy original https://torrentfreak.com/isp-bombarded-with-82000-demands-to-reveal-alleged-pirates-170513/

    It was once a region where people could share files without fear of reprisal, but over the years Scandinavia has become a hotbed of ‘pirate’ prosecutions.

    Sweden, in particular, has seen many sites shut down and their operators sentenced, notably those behind The Pirate Bay but also more recent cases such as those against DreamFilm and Swefilmer.

    To this backdrop, members of the public have continued to share files, albeit in decreasing numbers. However, at the same time copyright trolls have hit countries like Sweden, Finland, and Denmark, hoping to scare alleged file-sharers into cash settlements.

    This week regional ISP Telia revealed that the activity has already reached epidemic proportions.

    Under the EU IPR Enforcement Directive (IPRED), Internet service providers are required to hand over the personal details of suspected pirates to copyright holders, if local courts deem that appropriate. Telia says it is now being bombarded with such demands.

    “Telia must adhere to court decisions. At the same time we have a commitment to respect the privacy of our customers and therefore to be transparent,” the company says.

    “While in previous years Telia has normally received less than ten such [disclosure] requests per market, per year, lately the number of requests has increased significantly.”

    The scale is huge. The company reports that in Sweden during the past year alone, it has been ordered to hand over the identities of subscribers behind more than 45,000 IP addresses.

    In Finland during the same period, court orders covered almost 37,000 IP addresses. Four court orders in Denmark currently require the surrendering of data on “hundreds” of customers.

    Telia says that a Danish law firm known as Njord Law is behind many of the demands. The company is connected to international copyright trolls operating out of the United States, United Kingdom, and elsewhere.

    “A Danish law firm (NJORD Law firm), representing the London-based copyright holder Copyright Management Services Ltd, was recently (2017-01-31) granted a court order forcing Telia Sweden to disclose to the law firm the subscriber identities behind 25,000 IP-addresses,” the company notes.

    Copyright Management Services Ltd was incorporated in the UK during October 2014. Its sole director is Patrick Achache, who also operates German-based BitTorrent tracking company MaverickEye. Both are part of the notorious international trolling operation Guardaley.

    Copyright Management Services, which is based at the same London address as fellow UK copyright-trolling partner Hatton and Berkeley, filed accounts in June 2016 claiming to be a dormant company. Other than that, it has never filed any financial information.

    Copyright Management Services will be legally required to publish more detailed accounts next time around, since the company is now clearly trading, but its role in this operation is far from clear. For its part, Telia hopes the court has done the necessary checking when handing information over to partner firm, Njord Law.

    “Telia assumes that the courts perform adequate assessments of the evidence provided by the above law firm, and also that the courts conduct a sufficient assessment of proportionality between copyright and privacy,” the company says.

    “Telia does not know what the above law firm intends to do with the large amount of customer data which they are now collecting.”

    While that statement from Telia is arguably correct, it doesn’t take a genius to work out where this is going. Every time that these companies can match an IP address to an account holder, they will receive a letter in the mail demanding a cash settlement. Anything that substantially deviates from this outcome would be a very surprising development indeed.

    In the meantime, Jon Karlung, the outspoken boss of ISP Bahnhof, has pointed out that if Telia didn’t store customer IP addresses in the first place, it wouldn’t have anything to hand out to copyright trolls.

    “Bahnhof does not store this data – and we can’t give out something we do not have. The same logic should apply to Telia,” he said.

    Bahnhof says it stores customer data including IP addresses for 24 hours, just long enough to troubleshoot technical issues but nowhere near long enough to be useful to trolls.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

    Four Men Jailed For Running Pirate Movie Sites

    Post Syndicated from Andy original https://torrentfreak.com/four-men-jailed-running-pirate-movie-sites-170510/

    In the wake of December 2014 action that closed down The Pirate Bay for weeks, Swedish police turned their focus to one of the country’s top streaming portals, Dreamfilm.se.

    The site had been growing in popularity for a while, and together with defunct streaming site Swefilmer, whose admins also went on trial recently, the site accounted for up to 25% of online viewing in Sweden.

    “After an administrator was detained and interrogated, it has been mutually agreed that dreamfilm.se will be shut down for good,” the site said in a January 2015 statement.

    While the site later came back to life under a new name, Swedish police kept up the pressure. In February 2015, several more sites bit the dust including the country’s second largest torrent site Tankefetast, torrent site PirateHub, and streaming portal Tankefetast Play (TFPlay).

    Image previously released by Tankafetasttankafetast

    It took more than two years, but recently the key people behind the sites had their day in court. According to IDG, all of the men admitted to being involved in Dreamfilm, but none accepted they had committed any crimes.

    Yesterday the Linköping District Court handed down its decision and it’s particularly bad news for those involved. Aged between 21 and 31-years-old, the men were sentenced to between six and 10 months in jail and ordered to pay damages of around $147,000 to the film industry.

    A 23-year-old man who founded Dreamfilm back in 2012 was handed the harshest sentence of 10 months. He was due to receive a sentence of one year in jail but due to his age at the time of some of the offenses, the Court chose to impose a slightly lower term.

    A member of the Pirate Party who reportedly handled advertising and helped to administer the site, was sentenced to eight months in prison. Two other men who worked in technical roles were told to serve between six and 10 months.

    Image published by Dreamfilm after the raiddreamfilm

    Anti-piracy outfit Rights Alliance, which as usual was deeply involved in the prosecution, says that the sites were significant players in the pirate landscape.

    “The network that included Dream Movie, Tankafetast, TF Play and Piratehub was one of Europe’s leading players for illegal file sharing and streaming. The coordination of the network was carried out by two of the convicted,” the group said.

    “This case is an example of how organized commercial piracy used Sweden as a base and target for its operations. They are well organized and earn a lot of money and the risks are considered small and punishments low in Sweden,” lawyer Henrik Pontén said.

    While lenient sentences are now clearly off the agenda, the convicted men still have a chance to appeal. It is not yet clear whether they will do so. In the meantime the Dreamfilm.se domain will be seized until the District Court decision becomes final.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

    Intel’s remote AMT vulnerablity

    Post Syndicated from Matthew Garrett original https://mjg59.dreamwidth.org/48429.html

    Intel just announced a vulnerability in their Active Management Technology stack. Here’s what we know so far.

    Background

    Intel chipsets for some years have included a Management Engine, a small microprocessor that runs independently of the main CPU and operating system. Various pieces of software run on the ME, ranging from code to handle media DRM to an implementation of a TPM. AMT is another piece of software running on the ME, albeit one that takes advantage of a wide range of ME features.

    Active Management Technology

    AMT is intended to provide IT departments with a means to manage client systems. When AMT is enabled, any packets sent to the machine’s wired network port on port 16992 or 16993 will be redirected to the ME and passed on to AMT – the OS never sees these packets. AMT provides a web UI that allows you to do things like reboot a machine, provide remote install media or even (if the OS is configured appropriately) get a remote console. Access to AMT requires a password – the implication of this vulnerability is that that password can be bypassed.

    Remote management

    AMT has two types of remote console: emulated serial and full graphical. The emulated serial console requires only that the operating system run a console on that serial port, while the graphical environment requires drivers on the OS side requires that the OS set a compatible video mode but is also otherwise OS-independent[2]. However, an attacker who enables emulated serial support may be able to use that to configure grub to enable serial console. Remote graphical console seems to be problematic under Linux but some people claim to have it working, so an attacker would be able to interact with your graphical console as if you were physically present. Yes, this is terrifying.

    Remote media

    AMT supports providing an ISO remotely. In older versions of AMT (before 11.0) this was in the form of an emulated IDE controller. In 11.0 and later, this takes the form of an emulated USB device. The nice thing about the latter is that any image provided that way will probably be automounted if there’s a logged in user, which probably means it’s possible to use a malformed filesystem to get arbitrary code execution in the kernel. Fun!

    The other part of the remote media is that systems will happily boot off it. An attacker can reboot a system into their own OS and examine drive contents at their leisure. This doesn’t let them bypass disk encryption in a straightforward way[1], so you should probably enable that.

    How bad is this

    That depends. Unless you’ve explicitly enabled AMT at any point, you’re probably fine. The drivers that allow local users to provision the system would require administrative rights to install, so as long as you don’t have them installed then the only local users who can do anything are the ones who are admins anyway. If you do have it enabled, though…

    How do I know if I have it enabled?

    Yeah this is way more annoying than it should be. First of all, does your system even support AMT? AMT requires a few things:

    1) A supported CPU
    2) A supported chipset
    3) Supported network hardware
    4) The ME firmware to contain the AMT firmware

    Merely having a “vPRO” CPU and chipset isn’t sufficient – your system vendor also needs to have licensed the AMT code. Under Linux, if lspci doesn’t show a communication controller with “MEI” or “HECI” in the description, AMT isn’t running and you’re safe. If it does show an MEI controller, that still doesn’t mean you’re vulnerable – AMT may still not be provisioned. If you reboot you should see a brief firmware splash mentioning the ME. Hitting ctrl+p at this point should get you into a menu which should let you disable AMT.

    How about over Wifi?

    Turning on AMT doesn’t automatically turn it on for wifi. AMT will also only connect itself to networks it’s been explicitly told about. Where things get more confusing is that once the OS is running, responsibility for wifi is switched from the ME to the OS and it forwards packets to AMT. I haven’t been able to find good documentation on whether having AMT enabled for wifi results in the OS forwarding packets to AMT on all wifi networks or only ones that are explicitly configured.

    What do we not know?

    We have zero information about the vulnerability, other than that it allows unauthenticated access to AMT. One big thing that’s not clear at the moment is whether this affects all AMT setups, setups that are in Small Business Mode, or setups that are in Enterprise Mode. If the latter, the impact on individual end-users will be basically zero – Enterprise Mode involves a bunch of effort to configure and nobody’s doing that for their home systems. If it affects all systems, or just systems in Small Business Mode, things are likely to be worse.
    We now know that the vulnerability exists in all configurations.

    What should I do?

    Make sure AMT is disabled. If it’s your own computer, you should then have nothing else to worry about. If you’re a Windows admin with untrusted users, you should also disable or uninstall LMS by following these instructions.

    Does this mean every Intel system built since 2008 can be taken over by hackers?

    No. Most Intel systems don’t ship with AMT. Most Intel systems with AMT don’t have it turned on.

    Does this allow persistent compromise of the system?

    Not in any novel way. An attacker could disable Secure Boot and install a backdoored bootloader, just as they could with physical access.

    But isn’t the ME a giant backdoor with arbitrary access to RAM?

    Yes, but there’s no indication that this vulnerability allows execution of arbitrary code on the ME – it looks like it’s just (ha ha) an authentication bypass for AMT.

    Is this a big deal anyway?

    Yes. Fixing this requires a system firmware update in order to provide new ME firmware (including an updated copy of the AMT code). Many of the affected machines are no longer receiving firmware updates from their manufacturers, and so will probably never get a fix. Anyone who ever enables AMT on one of these devices will be vulnerable. That’s ignoring the fact that firmware updates are rarely flagged as security critical (they don’t generally come via Windows update), so even when updates are made available, users probably won’t know about them or install them.

    Avoiding this kind of thing in future

    Users ought to have full control over what’s running on their systems, including the ME. If a vendor is no longer providing updates then it should at least be possible for a sufficiently desperate user to pay someone else to do a firmware build with the appropriate fixes. Leaving firmware updates at the whims of hardware manufacturers who will only support systems for a fraction of their useful lifespan is inevitably going to end badly.

    How certain are you about any of this?

    Not hugely – the quality of public documentation on AMT isn’t wonderful, and while I’ve spent some time playing with it (and related technologies) I’m not an expert. If anything above seems inaccurate, let me know and I’ll fix it.

    [1] Eh well. They could reboot into their own OS, modify your initramfs (because that’s not signed even if you’re using UEFI Secure Boot) such that it writes a copy of your disk passphrase to /boot before unlocking it, wait for you to type in your passphrase, reboot again and gain access. Sealing the encryption key to the TPM would avoid this.

    [2] Updated after this comment – I thought I’d fixed this before publishing but left that claim in by accident.

    (Updated to add the section on wifi)

    (Updated to typo replace LSM with LMS)

    (Updated to indicate that the vulnerability affects all configurations)

    comment count unavailable comments

    Intel’s remote AMT vulnerablity

    Post Syndicated from Matthew Garrett original http://mjg59.dreamwidth.org/48429.html

    Intel just announced a vulnerability in their Active Management Technology stack. Here’s what we know so far.

    Background

    Intel chipsets for some years have included a Management Engine, a small microprocessor that runs independently of the main CPU and operating system. Various pieces of software run on the ME, ranging from code to handle media DRM to an implementation of a TPM. AMT is another piece of software running on the ME, albeit one that takes advantage of a wide range of ME features.

    Active Management Technology

    AMT is intended to provide IT departments with a means to manage client systems. When AMT is enabled, any packets sent to the machine’s wired network port on port 16992 will be redirected to the ME and passed on to AMT – the OS never sees these packets. AMT provides a web UI that allows you to do things like reboot a machine, provide remote install media or even (if the OS is configured appropriately) get a remote console. Access to AMT requires a password – the implication of this vulnerability is that that password can be bypassed.

    Remote management

    AMT has two types of remote console: emulated serial and full graphical. The emulated serial console requires only that the operating system run a console on that serial port, while the graphical environment requires drivers on the OS side. However, an attacker who enables emulated serial support may be able to use that to configure grub to enable serial console. Remote graphical console seems to be problematic under Linux but some people claim to have it working, so an attacker would be able to interact with your graphical console as if you were physically present. Yes, this is terrifying.

    Remote media

    AMT supports providing an ISO remotely. In older versions of AMT (before 11.0) this was in the form of an emulated IDE controller. In 11.0 and later, this takes the form of an emulated USB device. The nice thing about the latter is that any image provided that way will probably be automounted if there’s a logged in user, which probably means it’s possible to use a malformed filesystem to get arbitrary code execution in the kernel. Fun!

    The other part of the remote media is that systems will happily boot off it. An attacker can reboot a system into their own OS and examine drive contents at their leisure. This doesn’t let them bypass disk encryption in a straightforward way[1], so you should probably enable that.

    How bad is this

    That depends. Unless you’ve explicitly enabled AMT at any point, you’re probably fine. The drivers that allow local users to provision the system would require administrative rights to install, so as long as you don’t have them installed then the only local users who can do anything are the ones who are admins anyway. If you do have it enabled, though…

    How do I know if I have it enabled?

    Yeah this is way more annoying than it should be. First of all, does your system even support AMT? AMT requires a few things:

    1) A supported CPU
    2) A supported chipset
    3) Supported network hardware
    4) The ME firmware to contain the AMT firmware

    Merely having a “vPRO” CPU and chipset isn’t sufficient – your system vendor also needs to have licensed the AMT code. Under Linux, if lspci doesn’t show a communication controller with “MEI” in the description, AMT isn’t running and you’re safe. If it does show an MEI controller, that still doesn’t mean you’re vulnerable – AMT may still not be provisioned. If you reboot you should see a brief firmware splash mentioning the ME. Hitting ctrl+p at this point should get you into a menu which should let you disable AMT.

    What do we not know?

    We have zero information about the vulnerability, other than that it allows unauthenticated access to AMT. One big thing that’s not clear at the moment is whether this affects all AMT setups, setups that are in Small Business Mode, or setups that are in Enterprise Mode. If the latter, the impact on individual end-users will be basically zero – Enterprise Mode involves a bunch of effort to configure and nobody’s doing that for their home systems. If it affects all systems, or just systems in Small Business Mode, things are likely to be worse.

    What should I do?

    Make sure AMT is disabled. If it’s your own computer, you should then have nothing else to worry about. If you’re a Windows admin with untrusted users, you should also disable or uninstall LSM by following these instructions.

    Does this mean every Intel system built since 2008 can be taken over by hackers?

    No. Most Intel systems don’t ship with AMT. Most Intel systems with AMT don’t have it turned on.

    Does this allow persistent compromise of the system?

    Not in any novel way. An attacker could disable Secure Boot and install a backdoored bootloader, just as they could with physical access.

    But isn’t the ME a giant backdoor with arbitrary access to RAM?

    Yes, but there’s no indication that this vulnerability allows execution of arbitrary code on the ME – it looks like it’s just (ha ha) an authentication bypass for AMT.

    Is this a big deal anyway?

    Yes. Fixing this requires a system firmware update in order to provide new ME firmware (including an updated copy of the AMT code). Many of the affected machines are no longer receiving firmware updates from their manufacturers, and so will probably never get a fix. Anyone who ever enables AMT on one of these devices will be vulnerable. That’s ignoring the fact that firmware updates are rarely flagged as security critical (they don’t generally come via Windows update), so even when updates are made available, users probably won’t know about them or install them.

    Avoiding this kind of thing in future

    Users ought to have full control over what’s running on their systems, including the ME. If a vendor is no longer providing updates then it should at least be possible for a sufficiently desperate user to pay someone else to do a firmware build with the appropriate fixes. Leaving firmware updates at the whims of hardware manufacturers who will only support systems for a fraction of their useful lifespan is inevitably going to end badly.

    How certain are you about any of this?

    Not hugely – the quality of public documentation on AMT isn’t wonderful, and while I’ve spent some time playing with it (and related technologies) I’m not an expert. If anything above seems inaccurate, let me know and I’ll fix it.

    [1] Eh well. They could reboot into their own OS, modify your initramfs (because that’s not signed even if you’re using UEFI Secure Boot) such that it writes a copy of your disk passphrase to /boot before unlocking it, wait for you to type in your passphrase, reboot again and gain access. Sealing the encryption key to the TPM would avoid this.

    comment count unavailable comments

    Pirate Site Operators Caught By Money Trail, Landmark Trial Hears

    Post Syndicated from Andy original https://torrentfreak.com/pirate-site-operators-caught-by-money-trail-landmark-trial-hears-170411/

    Founded half a decade ago, Swefilmer grew to become Sweden’s most popular movie and TV show streaming site. At one stage, Swefilmer and fellow streaming site Dreamfilm were said to account for 25% of all web TV viewing in Sweden.

    In 2015, local man Ola Johansson took to the Internet to reveal that he’d been raided by the police under suspicion of being involved in running the site. In March 2016, a Turkish national was arrested in Germany on a secret European arrest warrant.

    After a couple of false starts, one last June and another this January, the case finally got underway yesterday in Sweden.

    The pair stand accused of the unlawful distribution of around 1,400 movies, owned by a dozen studios including Warner, Disney and Fox. Investigators tested 67 of the titles and ten had been made available online before their DVD release.

    Anti-piracy group Rights Alliance claims that the site generated a lot of money from advertising without paying for the appropriate licenses. On the table are potential convictions for copyright infringement and money laundering.

    Follow the money

    In common with so many file-sharing related cases, it’s clear that the men in this case were tracked down from traces left online. Those included IP address evidence and money trails from both advertising revenues and site donations.

    According to Sveriges Radio who were in court yesterday, police were able to trace two IP addresses used to operate Swefilmer back to Turkey.

    In an effort to trace the bank account used by the site to hold funds, the prosecutor then sought assistance from Turkish authorities. After obtaining the name of the 26-year-old, the prosecutor was then able to link that with advertising revenue generated by the site.

    Swefilmer also had a PayPal account used to receive donations and payments for VIP memberships. That account was targeted by an investigator from Rights Alliance who donated money via the same method. That allowed the group to launch an investigation with the payment processor.

    The PayPal inquiry appears to have been quite fruitful. The receipt from the donation revealed the account name and from there PayPal apparently gave up the email and bank account details connected to the account. These were linked to the 26-year-old by the prosecutor.

    Advertising

    The site’s connections with its advertisers also proved useful to investigators. The prosecution claimed that Swefilmer received its first payment in 2013 and its last in 2015. The money generated, some $1.5m (14m kronor), was deposited in a bank account operated by the 26-year-old by a Stockholm-based ad company.

    The court heard that while the CEO of the advertising company had been questioned in connection with the case, he is not suspected of crimes.

    Connecting the site’s operators

    While the exact mechanism is unclear, investigators from Rights Alliance managed to find an IP address used by the 22-year-old. This IP was then traced back to his parents’ home in Kungsbacka, Sweden. The same IP address was used to access the man’s Facebook page.

    In court, the prosecution read out chat conversations between both men. They revealed that the men knew each other only through chat and that the younger man believed the older was from Russia.

    The prosecution’s case is that the 26-year-old was the ring-leader and that his colleague was a minor player. With that in mind, the latter is required to pay back around $4,000, which is the money he earned from the site.

    For the older man, the situation is much more serious. The prosecution is seeking all of the money the site made from advertising, a cool $1.5m.

    The case was initially set to go ahead last year but was postponed pending a ruling from the European Court of Justice. Last September, the Court determined that it was illegal to link to copyrighted material if profit was being made.

    Claes Kennedy, the lawyer for the 22-year-old, insists that his client did nothing wrong. His actions took place before the ECJ’s ruling so should be determined legal, he says.

    The case continues.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

    Scaling Your Desktop Application Streams with Amazon AppStream 2.0

    Post Syndicated from Bryan Liston original https://aws.amazon.com/blogs/compute/scaling-your-desktop-application-streams-with-amazon-appstream-2-0/

    Want to stream desktop applications to a web browser, without rewriting them? Amazon AppStream 2.0 is a fully managed, secure, application streaming service. An easy way to learn what the service does is to try out the end-user experience, at no cost.

    In this post, I describe how you can scale your AppStream 2.0 environment, and achieve some cost optimizations. I also add some setup and monitoring tips.

    AppStream 2.0 workflow

    You import your applications into AppStream 2.0 using an image builder. The image builder allows you to connect to a desktop experience from within the AWS Management Console, and then install and test your apps. Then, create an image that is a snapshot of the image builder.

    After you have an image containing your applications, select an instance type and launch a fleet of streaming instances. Each instance in the fleet is used by only one user, and you match the instance type used in the fleet to match the needed application performance. Finally, attach the fleet to a stack to set up user access. The following diagram shows the role of each resource in the workflow.

    Figure 1: Describing an AppStream 2.0 workflow

    appstreamscaling_1.png

    Setting up AppStream 2.0

    To get started, set up an example AppStream 2.0 stack or use the Quick Links on the console. For this example, I named my stack ds-sample, selected a sample image, and chose the stream.standard.medium instance type. You can explore the resources that you set up in the AWS console, or use the describe-stacks and describe-fleets commands as follows:

    Figure 2: Describing an AppStream 2.0 stack

    appstreamscaling_1.png

    Figure 3: Describing an AppStream 2.0 fleet

    appstreamscaling_2.43%20AM

    To set up user access to your streaming environment, you can use your existing SAML 2.0 compliant directory. Your users can then use their existing credentials to log in. Alternatively, to quickly test a streaming connection, or to start a streaming session from your own website, you can create a streaming URL. In the console, choose Stacks, Actions, Create URL, or call create-streaming-url as follows:

    Figure 4: Creating a streaming URL

    appstreamscaling_3.png

    You can paste the streaming URL into a browser, and open any of the displayed applications.

    appstreamscaling_4.30%20PM

    Now that you have a sample environment set up, here are a few tips on scaling.

    Scaling and cost optimization for AppStream 2.0

    To provide an instant-on streaming connection, the instances in an AppStream 2.0 fleet are always running. You are charged for running instances, and each running instance can serve exactly one user at any time. To optimize your costs, match the number of running instances to the number of users who want to stream apps concurrently. This section walks through three options for doing this:

    • Fleet Auto Scaling
    • Fixed fleets based on a schedule
    • Fleet Auto Scaling with schedules

    Fleet Auto Scaling

    To dynamically update the number of running instances, you can use Fleet Auto Scaling. This feature allows you to scale the size of the fleet automatically between a minimum and maximum value based on demand. This is useful if you have user demand that changes constantly, and you want to scale your fleet automatically to match this demand. For examples about setting up and managing scaling policies, see Fleet Auto Scaling.

    You can trigger changes to the fleet through the available Amazon CloudWatch metrics:

    • CapacityUtilization – the percentage of running instances already used.
    • AvailableCapacity – the number of instances that are unused and can receive connections from users.
    • InsufficientCapacityError – an error that is triggered when there is no available running instance to match a user’s request.

    You can create and attach scaling policies using the AWS SDK or AWS Management Console. I find it convenient to set up the policies using the console. Use the following steps:

    1. In the AWS Management Console, open AppStream 2.0.
    2. Choose Fleets, select a fleet, and choose Scaling Policies.
    3. For Minimum capacity and Maximum capacity, enter values for the fleet.

    Figure 5: Fleets tab for setting scaling policies

    appstreamscaling_5.png

    1. Create scale out and scale in policies by choosing Add Policy in each section.

    Figure 6: Adding a scale out policy

    appstreamscaling_6.png

    Figure 7: Adding a scale in policy

    appstreamscaling_7.png

    After you create the policies, they are displayed as part of your fleet details.

    appstreamscaling_8.png

    The scaling policies are triggered by CloudWatch alarms. These alarms are automatically created on your behalf when you create the scaling policies using the console. You can view and modify the alarms via the CloudWatch console.

    Figure 8: CloudWatch alarms for triggering fleet scaling

    appstreamscaling_9.png

    Fixed fleets based on a schedule

    An alternative option to optimize costs and respond to predictable demand is to fix the number of running instances based on the time of day or day of the week. This is useful if you have a fixed number of users signing in at different times of the day― scenarios such as a training classes, call center shifts, or school computer labs. You can easily set the number of instances that are running using the AppStream 2.0 update-fleet command. Update the Desired value for the compute capacity of your fleet. The number of Running instances changes to match the Desired value that you set, as follows:

    Figure 9: Updating desired capacity for your fleet

    appstreamscaling_10.png

    Set up a Lambda function to update your fleet size automatically. Follow the example below to set up your own functions. If you haven’t used Lambda before, see Step 2: Create a HelloWorld Lambda Function and Explore the Console.

    To create a function to change the fleet size

    1. In the Lambda console, choose Create a Lambda function.
    2. Choose the Blank Function blueprint. This gives you an empty blueprint to which you can add your code.
    3. Skip the trigger section for now. Later on, you can add a trigger based on time, or any other input.
    4. In the Configure function section:
      1. Provide a name and description.
      2. For Runtime, choose Node.js 4.3.
      3. Under Lambda function handler and role, choose Create a custom role.
      4. In the IAM wizard, enter a role name, for example Lambda-AppStream-Admin. Leave the defaults as is.
      5. After the IAM role is created, attach an AppStream 2.0 managed policy “AmazonAppStreamFullAccess” to the role. For more information, see Working with Managed Policies. This allows Lambda to call the AppStream 2.0 API on your behalf. You can edit and attach your own IAM policy, to limit access to only actions you would like to permit. To learn more, see Controlling Access to Amazon AppStream 2.0.
      6. Leave the default values for the rest of the fields, and choose Next, Create function.
    5. To change the AppStream 2.0 fleet size, choose Code and add some sample code, as follows:
      'use strict';
      
      /**
      This AppStream2 Update-Fleet blueprint sets up a schedule for a streaming fleet
      **/
      
      const AWS = require('aws-sdk');
      const appstream = new AWS.AppStream();
      const fleetParams = {
        Name: 'ds-sample-fleet', /* required */
        ComputeCapacity: {
          DesiredInstances: 1 /* required */
      
        }
      };
      
      exports.handler = (event, context, callback) => {
          console.log('Received event:', JSON.stringify(event, null, 2));
      
          var resource = event.resources[0];
          var increase = resource.includes('weekday-9am-increase-capacity')
      
          try {
              if (increase) {
                  fleetParams.ComputeCapacity.DesiredInstances = 3
              } else {
                  fleetParams.ComputeCapacity.DesiredInstances = 1
              }
              appstream.updateFleet(fleetParams, (error, data) => {
                  if (error) {
                      console.log(error, error.stack);
                      return callback(error);
                  }
                  console.log(data);
                  return callback(null, data);
              });
          } catch (error) {
              console.log('Caught Error: ', error);
              callback(error);
          }
      };

    6. Test the code. Choose Test and use the “Hello World” test template. The first time you do this, choose Save and Test. Create a test input like the following to trigger the scaling update.

      appstreamscaling_11.png

    7. You see output text showing the result of the update-fleet call. You can also use the CLI to check the effect of executing the Lambda function.

    Next, to set up a time-based schedule, set a trigger for invoking the Lambda function.

    To set a trigger for the Lambda function

    1. Choose Triggers, Add trigger.
    2. Choose CloudWatch Events – Schedule.
    3. Enter a rule name, such as “weekday-9am-increase-capacity”, and a description. For Schedule expression, choose cron. You can edit the value for the cron later.
    4. After the trigger is created, open the event weekday-9am-increase-capacity.
    5. In the CloudWatch console, edit the event details. To scale out the fleet at 9 am on a weekday, you can adjust the time to be: 00 17 ? * MON-FRI *. (If you’re not in Seattle (Pacific Time Zone), change this to another specific time zone).
    6. You can also add another event that triggers at the end of a weekday.

    appstreamscaling_12.png

    This setup now triggers scale-out and scale-in automatically, based on the time schedule that you set.

    Fleet Auto Scaling with schedules

    You can choose to combine both the fleet scaling and time-based schedule approaches to manage more complex scenarios. This is useful to manage the number of running instances based on business and non-business hours, and still respond to changes in demand. You could programmatically change the minimum and maximum sizes for your fleet based on time of day or day of week, and apply the default scale-out or scale-in policies. This allows you to respond to predictable minimum demand based on a schedule.

    For example, at the start of a work day, you might expect a certain number of users to request streaming connections at one time. You wouldn’t want to wait for the fleet to scale out and meet this requirement. However, during the course of the day, you might expect the demand to scale in or out, and would want to match the fleet size to this demand.

    To achieve this, set up the scaling polices via the console, and create a Lambda function to trigger changes to the minimum, maximum, and desired capacity for your fleet based on a schedule. Replace the code for the Lambda function that you created earlier with the following code:

    'use strict';
    
    /**
    This AppStream2 Update-Fleet function sets up a schedule for a streaming fleet
    **/
    
    const AWS = require('aws-sdk');
    const appstream = new AWS.AppStream();
    const applicationAutoScaling = new AWS.ApplicationAutoScaling();
    
    const fleetParams = {
      Name: 'ds-sample-fleet', /* required */
      ComputeCapacity: {
        DesiredInstances: 1 /* required */
      }
    };
    
    var scalingParams = {
      ResourceId: 'fleet/ds-sample-fleet', /* required - fleet name*/
      ScalableDimension: 'appstream:fleet:DesiredCapacity', /* required */
      ServiceNamespace: 'appstream', /* required */
      MaxCapacity: 1,
      MinCapacity: 6,
      RoleARN: 'arn:aws:iam::659382443255:role/service-role/ApplicationAutoScalingForAmazonAppStreamAccess'
    };
    
    exports.handler = (event, context, callback) => {
        
        console.log('Received this event now:', JSON.stringify(event, null, 2));
        
        var resource = event.resources[0];
        var increase = resource.includes('weekday-9am-increase-capacity')
    
        try {
            if (increase) {
                //usage during business hours - start at capacity of 10 and scale
                //if required. This implies at least 10 users can connect instantly. 
                //More users can connect as the scaling policy triggers addition of
                //more instances. Maximum cap is 20 instances - fleet will not scale
                //beyond 20. This is the cap for number of users.
                fleetParams.ComputeCapacity.DesiredInstances = 10
                scalingParams.MinCapacity = 10
                scalingParams.MaxCapacity = 20
            } else {
                //usage during non-business hours - start at capacity of 1 and scale
                //if required. This implies only 1 user can connect instantly. 
                //More users can connect as the scaling policy triggers addition of
                //more instances. 
                fleetParams.ComputeCapacity.DesiredInstances = 1
                scalingParams.MinCapacity = 1
                scalingParams.MaxCapacity = 10
            }
            
            //Update minimum and maximum capacity used by the scaling policies
            applicationAutoScaling.registerScalableTarget(scalingParams, (error, data) => {
                 if (error) console.log(error, error.stack); 
                 else console.log(data);                     
                });
                
            //Update the desired capacity for the fleet. This sets 
            //the number of running instances to desired number of instances
            appstream.updateFleet(fleetParams, (error, data) => {
                if (error) {
                    console.log(error, error.stack);
                    return callback(error);
                }
    
                console.log(data);
                return callback(null, data);
            });
                
        } catch (error) {
            console.log('Caught Error: ', error);
            callback(error);
        }
    };
    

    Note: To successfully execute this code, you need to add IAM policies to the role used by the Lambda function. The policies allow Lambda to call the Application Auto Scaling service on your behalf.

    Figure 10: Inline policies for using Application Auto Scaling with Lambda

    {
    "Version": "2012-10-17",
    "Statement": [
       {
          "Effect": "Allow", 
             "Action": [
                "iam:PassRole"
             ],
             "Resource": "*"
       }
    ]
    }
    {
    "Version": "2012-10-17",
    "Statement": [
       {
          "Effect": "Allow", 
             "Action": [
                "application-autoscaling:*"
             ],
             "Resource": "*"
       }
    ]
    }

    Monitoring usage

    After you have set up scaling for your fleet, you can use CloudWatch metrics with AppStream 2.0, and create a dashboard for monitoring. This helps optimize your scaling policies over time based on the amount of usage that you see.

    For example, if you were very conservative with your initial set up and over-provisioned resources, you might see long periods of low fleet utilization. On the other hand, if you set the fleet size too low, you would see high utilization or errors from insufficient capacity, which would block users’ connections. You can view CloudWatch metrics for up to 15 months, and drive adjustments to your fleet scaling policy.

    Figure 11: Dashboard with custom Amazon CloudWatch metrics

    appstreamscaling_13.53%20PM

    Summary

    These are just a few ideas for scaling AppStream 2.0 and optimizing your costs. Let us know if these are useful, and if you would like to see similar posts. If you have comments about the service, please post your feedback on the AWS forum for AppStream 2.0.

    Landmark Movie Streaming Trial Gets Underway in Sweden

    Post Syndicated from Andy original https://torrentfreak.com/landmark-movie-streaming-trial-gets-underway-in-sweden-170116/

    swefilmlogoFounded half a decade ago, Swefilmer grew to become Sweden’s most popular movie and TV show streaming site. Together with Dreamfilm, another site operating in the same niche, Swefilmer is said to have accounted for 25% of all web TV viewing in Sweden.

    In the summer of 2015, local man Ola Johansson revealed that he’d been raided by the police under suspicion of being involved in running the site. In March 2015, a Turkish national was arrested in Germany on a secret European arrest warrant. The now 26-year-old was accused of receiving donations from users and setting up Swefilmer’s deals with advertisers.

    In a subsequent indictment filed at the Varberg District Court, the men were accused of copyright infringement offenses relating to the unlawful distribution of more than 1,400 movies. However, just hours after the trial got underway last June, it was suspended, when a lawyer for one of the men asked to wait for an important EU copyright case to run its course.

    That case, between Dutch blog GeenStijl.nl and Playboy, had seen a Dutch court ask the EU Court of Justice to rule whether unauthorized links to copyrighted content could be seen as a ‘communication to the public’ under Article 3(1) of the Copyright Directive, and whether those links facilitated copyright infringement.

    Last September, the European Court of Justice ruled that it is usually acceptable to link to copyrighted content without permission when people are not aware content is infringing and when they do so on a non-profit basis. In commercial cases, the rules are much more strict.

    The Swefilmer siteswefilmer

    In light of that ruling, the pair return to the Varberg District Court today, accused of making more than $1.5m from their activities between November 2013 and June 2015.

    While Swedish prosecutions against sites such as The Pirate Bay have made global headlines, the case against Swefilmer is the first of its kind against a stream-links portal. Prosecutor Anna Ginner and Rights Alliance lawyer Henrik Pontén believe they have the evidence needed to take down the pair.

    “Swefilmer is a typical example of how a piracy operation looks today: fully commercial, well organized and great efforts expended to conceal itself. This applies particularly to the principal of the site,” Pontén told IDG.

    According to Ginner, the pair ran an extensive operation and generated revenues from a number of advertising companies. They did not act alone but the duo were the ones that were identified by, among other things, their IP addresses.

    The 26-year-old, who was arrested in Germany, was allegedly the money man who dealt with the advertisers. In addition to copyright infringement offenses, he stands accused of money laundering.

    According to IDG, he will plead not guilty. His lawyer gave no hints why but suggested the reasons will become evident during the trial.

    The younger man, who previously self-identified as Ola Johansson, is accused of being the day-to-day operator of the site, which included uploading movies to other sites where Swefilmer linked. He is said to have received a modest sum for his work, around $3,800.

    “I think what’s interesting for the Swedish court is that this case has such clear elements of organized crime compared to what we have seen before,” Anna Ginner concludes.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

    Hollywood Lawsuit Expands Pirate Bay & ExtraTorrent Web Blockade

    Post Syndicated from Andy original https://torrentfreak.com/hollywood-lawsuit-expands-pirate-bay-extratorrent-web-blockade-170113/

    FCT tyIn an expansion of their site-blocking campaign, several Hollywood studios under the banner of the MPA applied to a court in Norway during 2015 to have seven ‘pirate’ sites blocked at the ISP level.

    Warner Bros, Paramount, Twentieth Century Fox, Universal, Sony, Disney, Columbia and several local industry groups argued that The Pirate Bay, ExtraTorrent, Viooz, PrimeWire, Swefilmer, DreamFilm and Movie4K, infringe their copyrights.

    Local ISPs including Telenor, TeliaSonera, NextGenTel and Altibox were named as defendants in the case, which was handled by the Oslo District Court. In September 2015 the Court handed down its ruling, ordering the ISPs to block the seven sites listed in the lawsuit.

    While many observers believed the studios would be back immediately with more blocking requests, things went quiet for more than a year. Only recently did they return with fresh requests against more ISPs.

    The Motion Picture Association filed a new lawsuit against seven Internet service providers, named by Tek.no as Eidsiva Bredbånd, Lynet Internett, Breiband.no, Enivest, Neas Bredbånd, Tussa IKT and Opennet Norge.

    Like before, the MPA and the studios wanted the ISPs to restrict access to a number of ‘pirate’ sites including The Pirate Bay, Extratorrent, Viooz, SweFilmer, DreamFilm and Movie4K, which were all named in the original complaint. Additional sites named in the new complaint include Watch Series, Putlocker, TUBE+, Couch Tuner, Watch32, Project Free TV and Watch Free.

    It now appears that the MPA found a sympathetic ear at the Oslo District Court, which has just issued a ruling in the studios’ favor. Local media says the ruling is considerably shorter than the one handed down in 2015, which seems to indicate that the process has now been streamlined.

    According to Tek, the Court made its decision based on the fact that the sites in question published links to copyrighted material already available online. The Court determined that the sites published the content to a “new public”, noting that they had not changed their modus operandi since the original ruling.

    The ISPs were ordered to implement DNS blockades covering the domains currently in use. As illustrated on a a number of previous occasions, including in Norway itself, DNS blocks are the weakest form of blocking and are easily circumvented by switching to a different provider, such as Google.

    Torgeir Waterhouse of Internet interest group ICT Norway says that while the DNS blocks might only amount to a “speed bump”, it’s more important to make it easier to access legal services.

    “We must make it easy to pay, easy to access and easy to develop new services. Ultimately it is this that determines the levels of revenue for producers. Revenue does not occur ‘automagically’ by preventing their access to an illegal service,” he says.

    Waterhouse, who has campaigned on several intellectual property issues, also criticized the authorities for holding ISPs responsible for the actions of others.

    “The government should make a law that says what they really want, namely that parts of the Internet should not be accessible to the population. They should have the courage to stand up and say that this is what they want, not create the impression that Internet service providers are doing something wrong,” he adds

    In common with the 2015 ruling, the sites detailed in the lawsuit were all ordered to pay court costs. None of the sites’ owners appeared in Court so it’s unlikely that any will pay the $1,800 they each now owe.

    Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

    The “cryptsetup initrd root shell” vulnerability

    Post Syndicated from corbet original http://lwn.net/Articles/706444/rss

    Hector Marco and Ismael Ripoll report
    a discouraging vulnerability in many encrypted disk setups: simply running
    up too many password failures will eventually result in a root shell.
    This vulnerability allows to obtain a root initramfs shell on
    affected systems. The vulnerability is very reliable because it doesn’t
    depend on specific systems or configurations. Attackers can copy, modify or
    destroy the hard disc as well as set up the network to exfiltrate
    data. This vulnerability is specially serious in environments like
    libraries, ATMs, airport machines, labs, etc, where the whole boot process
    is protect (password in BIOS and GRUB) and we only have a keyboard or/and a
    mouse.

    The NSA Is Hoarding Vulnerabilities

    Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/08/the_nsa_is_hoar.html

    The National Security Agency is lying to us. We know that because of data stolen from an NSA server was dumped on the Internet. The agency is hoarding information about security vulnerabilities in the products you use, because it wants to use it to hack others’ computers. Those vulnerabilities aren’t being reported, and aren’t getting fixed, making your computers and networks unsafe.

    On August 13, a group calling itself the Shadow Brokers released 300 megabytes of NSA cyberweapon code on the Internet. Near as we experts can tell, the NSA network itself wasn’t hacked; what probably happened was that a “staging server” for NSA cyberweapons — that is, a server the NSA was making use of to mask its surveillance activities — was hacked in 2013.

    The NSA inadvertently resecured itself in what was coincidentally the early weeks of the Snowden document release. The people behind the link used casual hacker lingo, and made a weird, implausible proposal involving holding a bitcoin auction for the rest of the data: “!!! Attention government sponsors of cyber warfare and those who profit from it !!!! How much you pay for enemies cyber weapons?”

    Still, most people believe the hack was the work of the Russian government and the data release some sort of political message. Perhaps it was a warning that if the US government exposes the Russians as being behind the hack of the Democratic National Committee — or other high-profile data breaches — the Russians will expose NSA exploits in turn.

    But what I want to talk about is the data. The sophisticated cyberweapons in the data dump include vulnerabilities and “exploit code” that can be deployed against common Internet security systems. Products targeted include those made by Cisco, Fortinet, TOPSEC, Watchguard, and Juniper — systems that are used by both private and government organizations around the world. Some of these vulnerabilities have been independently discovered and fixed since 2013, and some had remained unknown until now.

    All of them are examples of the NSA — despite what it and other representatives of the US government say — prioritizing its ability to conduct surveillance over our security. Here’s one example. Security researcher Mustafa al-Bassam found an attack tool codenamed BENIGHCERTAIN that tricks certain Cisco firewalls into exposing some of their memory, including their authentication passwords. Those passwords can then be used to decrypt virtual private network, or VPN, traffic, completely bypassing the firewalls’ security. Cisco hasn’t sold these firewalls since 2009, but they’re still in use today.

    Vulnerabilities like that one could have, and should have, been fixed years ago. And they would have been, if the NSA had made good on its word to alert American companies and organizations when it had identified security holes.

    Over the past few years, different parts of the US government have repeatedly assured us that the NSA does not hoard “zero days” ­ the term used by security experts for vulnerabilities unknown to software vendors. After we learned from the Snowden documents that the NSA purchases zero-day vulnerabilities from cyberweapons arms manufacturers, the Obama administration announced, in early 2014, that the NSA must disclose flaws in common software so they can be patched (unless there is “a clear national security or law enforcement” use).

    Later that year, National Security Council cybersecurity coordinator and special adviser to the president on cybersecurity issues Michael Daniel insisted that US doesn’t stockpile zero-days (except for the same narrow exemption). An official statement from the White House in 2014 said the same thing.

    The Shadow Brokers data shows this is not true. The NSA hoards vulnerabilities.

    Hoarding zero-day vulnerabilities is a bad idea. It means that we’re all less secure. When Edward Snowden exposed many of the NSA’s surveillance programs, there was considerable discussion about what the agency does with vulnerabilities in common software products that it finds. Inside the US government, the system of figuring out what to do with individual vulnerabilities is called the Vulnerabilities Equities Process (VEP). It’s an inter-agency process, and it’s complicated.

    There is a fundamental tension between attack and defense. The NSA can keep the vulnerability secret and use it to attack other networks. In such a case, we are all at risk of someone else finding and using the same vulnerability. Alternatively, the NSA can disclose the vulnerability to the product vendor and see it gets fixed. In this case, we are all secure against whoever might be using the vulnerability, but the NSA can’t use it to attack other systems.

    There are probably some overly pedantic word games going on. Last year, the NSA said that it discloses 91 percent of the vulnerabilities it finds. Leaving aside the question of whether that remaining 9 percent represents 1, 10, or 1,000 vulnerabilities, there’s the bigger question of what qualifies in the NSA’s eyes as a “vulnerability.”

    Not all vulnerabilities can be turned into exploit code. The NSA loses no attack capabilities by disclosing the vulnerabilities it can’t use, and doing so gets its numbers up; it’s good PR. The vulnerabilities we care about are the ones in the Shadow Brokers data dump. We care about them because those are the ones whose existence leaves us all vulnerable.

    Because everyone uses the same software, hardware, and networking protocols, there is no way to simultaneously secure our systems while attacking their systems ­ whoever “they” are. Either everyone is more secure, or everyone is more vulnerable.

    Pretty much uniformly, security experts believe we ought to disclose and fix vulnerabilities. And the NSA continues to say things that appear to reflect that view, too. Recently, the NSA told everyone that it doesn’t rely on zero days — very much, anyway.

    Earlier this year at a security conference, Rob Joyce, the head of the NSA’s Tailored Access Operations (TAO) organization — basically the country’s chief hacker — gave a rare public talk, in which he said that credential stealing is a more fruitful method of attack than are zero days: “A lot of people think that nation states are running their operations on zero days, but it’s not that common. For big corporate networks, persistence and focus will get you in without a zero day; there are so many more vectors that are easier, less risky, and more productive.”

    The distinction he’s referring to is the one between exploiting a technical hole in software and waiting for a human being to, say, get sloppy with a password.

    A phrase you often hear in any discussion of the Vulnerabilities Equities Process is NOBUS, which stands for “nobody but us.” Basically, when the NSA finds a vulnerability, it tries to figure out if it is unique in its ability to find it, or whether someone else could find it, too. If it believes no one else will find the problem, it may decline to make it public. It’s an evaluation prone to both hubris and optimism, and many security experts have cast doubt on the very notion that there is some unique American ability to conduct vulnerability research.

    The vulnerabilities in the Shadow Brokers data dump are definitely not NOBUS-level. They are run-of-the-mill vulnerabilities that anyone — another government, cybercriminals, amateur hackers — could discover, as evidenced by the fact that many of them were discovered between 2013, when the data was stolen, and this summer, when it was published. They are vulnerabilities in common systems used by people and companies all over the world.

    So what are all these vulnerabilities doing in a secret stash of NSA code that was stolen in 2013? Assuming the Russians were the ones who did the stealing, how many US companies did they hack with these vulnerabilities? This is what the Vulnerabilities Equities Process is designed to prevent, and it has clearly failed.

    If there are any vulnerabilities that — according to the standards established by the White House and the NSA — should have been disclosed and fixed, it’s these. That they have not been during the three-plus years that the NSA knew about and exploited them — despite Joyce’s insistence that they’re not very important — demonstrates that the Vulnerable Equities Process is badly broken.

    We need to fix this. This is exactly the sort of thing a congressional investigation is for. This whole process needs a lot more transparency, oversight, and accountability. It needs guiding principles that prioritize security over surveillance. A good place to start are the recommendations by Ari Schwartz and Rob Knake in their report: these include a clearly defined and more public process, more oversight by Congress and other independent bodies, and a strong bias toward fixing vulnerabilities instead of exploiting them.

    And as long as I’m dreaming, we really need to separate our nation’s intelligence-gathering mission from our computer security mission: we should break up the NSA. The agency’s mission should be limited to nation state espionage. Individual investigation should be part of the FBI, cyberwar capabilities should be within US Cyber Command, and critical infrastructure defense should be part of DHS’s mission.

    I doubt we’re going to see any congressional investigations this year, but we’re going to have to figure this out eventually. In my 2014 book Data and Goliath, I write that “no matter what cybercriminals do, no matter what other countries do, we in the US need to err on the side of security by fixing almost all the vulnerabilities we find…” Our nation’s cybersecurity is just too important to let the NSA sacrifice it in order to gain a fleeting advantage over a foreign adversary.

    This essay previously appeared on Vox.com.

    EDITED TO ADD (8/27): The vulnerabilities were seen in the wild within 24 hours, demonstrating how important they were to disclose and patch.

    James Bamford thinks this is the work of an insider. I disagree, but he’s right that the TAO catalog was not a Snowden document.

    People are looking at the quality of the code. It’s not that good.