Initial thoughts on MongoDB’s new Server Side Public License

Post Syndicated from Matthew Garrett original https://mjg59.dreamwidth.org/51230.html

MongoDB just announced that they were relicensing under their new Server Side Public License. This is basically the Affero GPL except with section 13 largely replaced with new text, as follows:

If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Software or modified version.

“Service Source Code” means the Corresponding Source for the Program or the modified version, and the Corresponding Source for all programs that you use to make the Program or modified version available as a service, including, without limitation, management software, user interfaces, application program interfaces, automation software, monitoring software, backup software, storage software and hosting software, all such that a user could run an instance of the service using the Service Source Code you make available.

MongoDB admit that this license is not currently open source in the sense of being approved by the Open Source Initiative, but say:We believe that the SSPL meets the standards for an open source license and are working to have it approved by the OSI.

At the broadest level, AGPL requires you to distribute the source code to the AGPLed work[1] while the SSPL requires you to distribute the source code to everything involved in providing the service. Having a license place requirements around things that aren’t derived works of the covered code is unusual but not entirely unheard of – the GPL requires you to provide build scripts even if they’re not strictly derived works, and you could probably make an argument that the anti-Tivoisation provisions of GPL3 fall into this category.

A stranger point is that you’re required to provide all of this under the terms of the SSPL. If you have any code in your stack that can’t be released under those terms then it’s literally impossible for you to comply with this license. I’m not a lawyer, so I’ll leave it up to them to figure out whether this means you’re now only allowed to deploy MongoDB on BSD because the license would require you to relicense Linux away from the GPL. This feels sloppy rather than deliberate, but if it is deliberate then it’s a massively greater reach than any existing copyleft license.

You can definitely make arguments that this is just a maximalist copyleft license, the AGPL taken to extreme, and therefore it fits the open source criteria. But there’s a point where something is so far from the previously accepted scenarios that it’s actually something different, and should be examined as a new category rather than already approved categories. I suspect that this license has been written to conform to a strict reading of the Open Source Definition, and that any attempt by OSI to declare it as not being open source will receive pushback. But definitions don’t exist to be weaponised against the communities that they seek to protect, and a license that has overly onerous terms should be rejected even if that means changing the definition.

In general I am strongly in favour of licenses ensuring that users have the freedom to take advantage of modifications that people have made to free software, and I’m a fan of the AGPL. But my initial feeling is that this license is a deliberate attempt to make it practically impossible to take advantage of the freedoms that the license nominally grants, and this impression is strengthened by it being something that’s been announced with immediate effect rather than something that’s been developed with community input. I think there’s a bunch of worthwhile discussion to have about whether the AGPL is strong and clear enough to achieve its goals, but I don’t think that this SSPL is the answer to that – and I lean towards thinking that it’s not a good faith attempt to produce a usable open source license.

(It should go without saying that this is my personal opinion as a member of the free software community, and not that of my employer)

[1] There’s some complexities around GPL3 code that’s incorporated into the AGPLed work, but if it’s not part of the AGPLed work then it’s not covered

comment count unavailable comments

[$] A farewell to email

Post Syndicated from corbet original https://lwn.net/Articles/768483/rss

The free-software community was built on email, a distributed technology
that allows people worldwide to communicate regardless of their particular
software environment. While email remains at the core of many projects’
workflow, others are increasingly trying to move away from it. A couple of
recent examples show what is driving this move and where it may be headed.

Notes on the UK IoT cybersec "Code of Practice"

Post Syndicated from Robert Graham original https://blog.erratasec.com/2018/10/notes-on-uk-iot-cybersec-code-of.html

The British government has released a voluntary “Code of Practice” for securing IoT devices. I thought I’d write some notes on it.

First, the good parts

Before I criticize the individual points, I want to praise if for having a clue. So many of these sorts of things are written by the clueless, those who want to be involved in telling people what to do, but who don’t really understand the problem.
The first part of the clue is restricting the scope. Consumer IoT is so vastly different from things like cars, medical devices, industrial control systems, or mobile phones that they should never really be talked about in the same guide.
The next part of the clue is understanding the players. It’s not just the device that’s a problem, but also the cloud and mobile app part that relates to the device. Though they do go too far and include the “retailer”, which is a bit nonsensical.
Lastly, while I’m critical of most all the points on the list and how they are described, it’s probably a complete list. There’s not much missing, and the same time, it includes little that isn’t necessary. In contrast, a lot of other IoT security guides lack important things, or take the “kitchen sink” approach and try to include everything conceivable.

1) No default passwords

Since the Mirai botnet of 2016 famously exploited default passwords, this has been at the top of everyone’s list. It’s the most prominent feature of the recent California IoT law. It’s the major feature of federal proposals.
But this is only a superficial understanding of what really happened. The issue wasn’t default passwords so much as Internet-exposed Telnet.
IoT devices are generally based on Linux which maintains operating-system passwords in the /etc/passwd file. However, devices almost never use that. Instead, the web-based management interface maintains its own password database. The underlying Linux system is vestigial like an appendix and not really used.
But these devices exposed Telnet, providing a path to this otherwise unused functionality. I bought several of the Mirai-vulnerable devices, and none of them used /etc/passwd for anything other than Telnet.
Another way default passwords get exposed in IoT devices is through debugging interfaces. Manufacturers configure the system one way for easy development, and then ship a separate “release” version. Sometimes they make a mistake and ship the development backdoors as well. Programmers often insert secret backdoor accounts into products for development purposes without realizing how easy it is for hackers to discover those passwords.
The point is that this focus on backdoor passwords is misunderstanding the problem. Device makers can easily believe they are compliant with this directive while still having backdoor passwords.
As for the web management interface, saying “no default passwords” is useless. Users have to be able to setup the device the first time, so there has to be some means to connect to the device without passwords initially. Device makers don’t know how to do this without default passwords. Instead of mindless guidance of what not to do, a document needs to be written that explains how devices can do this both securely as well as easy enough for users to use.
Humorously, the footnotes in this section do reference external documents that might explain this, but they are the wrong documents, appropriate for things like website password policies, but inappropriate for IoT web interfaces. This again demonstrates how they have only a superficial understanding of the problem.

2) Implement a vulnerability disclosure policy

This is a clueful item, and it should be the #1 item on every list.
Though they do add garbage on top of this, but demanding companies respond in a “timely manner”, but overall this isn’t a bad section.

3) Keep software updated

This is another superficial understanding of the problem.
Software patching works for desktop and mobile phones because they have interfaces the user interacts with, ones that can both notify the user of a patch as well as the functionality to apply it. IoT devices are usually stuck in a closet somewhere without such interfaces.
Software patching works for normal computers because they sell for hundreds of dollars and thus have sufficient memory and storage to reliably do updates. IoT devices sell for cut-throat margins and have barely enough storage to run. This either precludes updates altogether, or at least means the update isn’t reliable, that upon every update, a small percentage of customer devices will be “bricked”, rendered unusable. Adding $1 for flash memory to a $30 device is not a reasonable solution to the problem.
Software patching works for software because of its enormous margins and longevity. A software product is basically all profit. The same doesn’t apply to hardware, where devices are sold with slim margins. Device makers have a hard time selling them for more because there are always no-named makers of almost identical devices in Shenzen willing to undercut them. (Indeed, looking at Mirai, it appears that was the majority of infected devices, not major brands, but no-named knock-offs). 
The document says that device makers need to publish how long the device will be supported. This ignores the economics of this. Devices makers cannot know how long they will support a device. As long as they are selling new ones, they’ve got incentive and profits to keep supplying updates. After that, they don’t. There’s really no way for them to predict the long term market success of their devices.
Guarantees cost money. If they guarantee security fixes for 10 years, then that’s a liability they have to account for on their balance sheet. It’s a huge risk: if the product fails to sell lots of units, then they are on the hook for a large cost without the necessary income to match it.
Lastly, the entire thing is a canard. Users rarely update firmware for devices. Blaming vendors for not providing security patches/updates means nothing without blaming users for not applying them.

4) Securely store credentials and security-sensitive data

Like many guides, this section makes the superficial statement “Hard-coded credentials in device software are not acceptable”. The reason this is silly is because public-keys are a “credential”, and you indeed want “hard-coded” public-keys. Hard-coded public-key credentials is how you do other security functions, like encrypted and signature verification.
This section tells device makers to use the trusted-enclave features like those found on phones, but this is rather silly. For one thing, that’s a feature of only high-end CPUs, not the low-end CPUs found in such devices. For another thing, IoT devices don’t really contain anything that needs that level of protection.
Storing passwords in clear-text on the device is almost certain adequate security, and this section can be ignored.

5) Communicate securely

In other words, use SSL everywhere, such as on the web-based management interface.
But this is only a superficial understanding of how SSL works. You (generally) can’t use SSL for devices because there’s no secure certificate on the device. It forces users to bypass nasty warnings in the browser, which hurts the entire web ecosystem. Some IoT devices do indeed try to use SSL this way, and it’s bad, very bad.
On the other hand, IoT devices can and should use SSL when connecting outbound to the cloud.

6) Minimise exposed attack surfaces

This is certainly a good suggestion, but it’s a platitude rather than an action item. IoT devices already minimize as much as they can in order to reduce memory/storage requires. Where this is actionable requires subtler understanding. A lot of exposed attack services come from accidents. 
A lot of other exposed attack surfaces come about because device makers know no better way. Actual helpful, meaning advice would consist of telling them what to do in order to solve problems, rather than telling them what not to do.
The reason Mirai-devices exposed Telnet was for things like “remote factory reset”. Mirai infected mostly security cameras which don’t have factory reset buttons. That’s because they are located high up out of reach, or if they are in reach, they don’t want to allow the public to press the factory reset button. Thus, doing a factory reset meant doing it remotely. That appears to be the major reason for Telnet and “hardcoded passwords”, to allow remote factory reset. Instead of telling them not to expose Telnet, you need a guide explaining how to securely do remote factory resets.
This guide discussed “ports”, but the reality is that the attack surface in the web-based management interface on port 80 is usually more than all other ports put together. Focusing on “ports” reflects a superficial understanding of the problem.

7) Ensure software integrity

The guide says “Software on IoT devices should be verified using secure boot
mechanisms”. No, they shouldn’t be. In the name of security, they should do the opposite.
First of all, getting “secure boot” done right is extraordinarily difficult. Apple does it the best with their iPhone and still they get it wrong. For another thing, it’s expensive. Like trusted enclaves in processors, most of the cheap low-end processors used in IoT don’t support it.
But the biggest issue is that you don’t want it. “Secure boot” means the only operating system the device can boot comes from the vendor, which will eventually stop supporting the product, making it impossible to fix any security problem. Not having secure boot means that customers can still be able to patch bugs without the manufacturer’s help.
Instead of secure boot, device makers should do the opposite and make it easy for customers to build their own software. They are required to do so under the GNU Public License anyway. That doesn’t mean open-sourcing everything, they can still provide their private code as binaries. But they should allow users to fix any bug in open-source and repackage a new firmware update.

8) Ensure that personal data is protected

I suppose giving the GDPR, this section is required, but GDPR is a pox on the Internet.

9) Make systems resilient to outages

Given the recent story of Yale locks locking people out of their houses due to a system outage, this seems like an obviously good idea.
But it should be noted that this is hard. Obviously such a lock should be resilient if the network connection is down, or their servers have crashed. But what happens when such a lock can contact their servers, but some other component within their organization has crashed, such that the servers give unexpected responses, neither completely down, but neither completely up and running, either?
We saw that in the Mirai attacks against Dyn. It left a lot servers up and running, but took down on some other component that those servers relied upon, leaving things in an intermediate state that was neither unfunctional nor completely functional.
It’s easy to stand on a soapbox and proclaim devices need to be resilient, but this is unhelpful. What would instead be helpful is a catalog of failures that IoT will typically experience.

10) Monitor system telemetry data

Security telemetry is a desirable feature in general. When a hack happens, you want to review logfiles to see how it happened. This item reflects various efforts to come up with such useful information
But again we see something so devoid of technical details as to be useless. Worse, it’s going to be exploited by others, such as McAffee wanting you to have anti-virus on TV sets, which is an extraordinarily bad idea.

11) Make it easy for consumers to delete personal data

This is kinda silly in that the it’s simply a matter of doing a “factory reset”. Having methods to delete personal details other than factory resets is bad.
The useful bit of advise is that factory resets don’t always “wipe” information, they just “forget” it in a way that can be recovered. Thus, we get printers containing old documents and voting machines with old votes.
On the other hand, this is a guide for “consumer IoT”, so just the normal factory reset is probably sufficient, even if private details can be gleaned.

12) Make installation and maintenance of devices easy

Of course things should be easy, everyone agrees on this. The problem is they don’t know how. Companies like Microsoft and Apple spend billions on this problem and still haven’t cracked it.
My home network WiFi password uses quotes as punctuation to improve security. The Amazon Echo app uses Bluetooth to pair with the device and set which password to use for WiFi. This is well done from a security point of view.
However, their app uses an input field that changes quotes to curly-quotes making it impossible to type in the password. I instead had to go to browser, type the password in the URL field, copy it, then go back to the Alexa app and paste it into the field. Then I could get things to work.
Amazon is better at making devices easy and secure with Echo and they still get things spectacularly wrong.

13) Validate input data

Most security vulnerabilities are due to improper validation of input data. However, “validate input data” is stupid advice. It’s like how most phishing attacks come from strangers, but how telling people to not open emails from strangers is stupid advice. In both cases, it’s a superficial answer that doesn’t really understand how the problem came about.
Let’s take PHP and session cookies, for example. A lot of programmers think the session identifier in PHP is some internal feature of PHP. They therefore trust it, because it isn’t input. They don’t perceive how it’s not internal to PHP, but external, part of HTTP, and something totally hackable by hackers.
Or take the famous Jeep hacker where hackers were able to remotely take control of the car and do mischievous things like turn it off on the highway. The designers didn’t understand how the private connection to the phone network was in fact “input” coming from the Internet. And then there was data from the car’s internal network, which wasn’t seen as “input” from an external source.
Then there is the question of what “validation” means. A lot of programmers try to solve SQL injection by “blacklisting” known bad characters. Hackers are adept at bypassing this, using other bad characters, especially using Unicode. Whitelisting known good characters is a better solution. But even that is still problematic. The proper solution to SQL injection isn’t “input validation” at all, but using “parameterized queries” that don’t care about input.

Conclusion

Like virtually every other guide, this one is based upon platitudes and only a superficial understanding of the problem. It’s got more clue than most, but is still far from something that could actually be useful. The concept here is virtue signaling, declaring what would be virtuous and moral for an IoT device, rather than something that could be useful to device makers in practice.

New – Managed Databases for Amazon Lightsail

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-managed-databases-for-amazon-lightsail/

Amazon Lightsail makes it easy for you to get started with AWS. You choose the operating system (and optional application) that you want to run, pick an instance plan, and create an instance, all in a matter of minutes. Lightsail offers low, predictable pricing, with instance plans that include compute power, storage, and data transfer:

Managed Databases
Today we are making Lightsail even more useful by giving you the ability to create a managed database with a couple of clicks. This has been one of our top customer requests and I am happy to be able to share this news.

This feature is going to be of interest to a very wide range of current and future Lightsail users, including students, independent developers, entrepreneurs, and IT managers. We’ve addressed the most common and complex issues that arise when setting up and running a database. As you will soon see, we have simplified and fine-tuned the process of choosing, launching, securing, accessing, monitoring, and maintaining a database!

Each Lightsail database bundle has a fixed, monthly price that includes the database instance, a generous amount of SSD-backed storage, a terabyte or more of data transfer to the Internet and other AWS regions, and automatic backups that give you point-in-time recovery for a 7-day period. You can also create manual database snapshots that are billed separately.

Creating a Managed Database
Let’s walk through the process of creating a managed database and loading an existing MySQL backup into it. I log in to the Lightsail Console and click Databases to get started. Then I click Create database to move forward:

I can see and edit all of the options at a glance. I choose a location, a database engine and version, and a plan, enter a name, and click Create database (all of these options have good defaults; a single click often suffices):

We are launching with support for MySQL 5.6 and 5.7, and will add support for PostgreSQL 9.6 and 10 very soon. The Standard database plan creates a database in one Availability Zone with no redundancy; the High Availability plan also creates a presence in a second AZ, and is recommended for production use.

Database creation takes just a few minutes, the status turns to Available, and my database is ready to use:

I click on Database-Oregon-1, and I can see the connection details, and have access to other management information & tools:

I’m ready to connect! I create an SSH connection to my Lightsail instance, ensure that the mysql package is installed, and connect using the information above (read Connecting to Your MySQL Database to learn more):

Now I want to import some existing data into my database. Lightsail lets me enable Data import mode in order to defer any backup or maintenance operations:

Enabling data import mode deletes any existing automatic snapshots; you may want to take a manual snapshot before starting your import if you are importing fresh data into an existing database.

I have a large (13 GB) , ancient (2013-era) MySQL backup from a long-dead personal project; I download it from S3, uncompress it, and import it:

I can watch the metrics while the import is underway:

After the import is complete I disable data import mode, and I can run queries against my tables:

To learn more, read Importing Data into Your Database.

Lightsail manages all routine database operations. If I make a mistake and mess up my data, I can use the Emergency Restore to create a fresh database instance from an earlier point in time:

I can rewind by up to 7 days, limited to when I last disabled data import mode.

I can also take snapshots, and use them later to create a fresh database instance:

Things to Know
Here are a couple of things to keep in mind when you use this new feature:

Engine Versions – We plan to support the two latest versions of MySQL, and will do the same for other database engines as we make them available.

High Availability – As is always the case for production AWS systems, you should use the High Availability option in order to maintain a database footprint that spans two Zones. You can switch between Standard and High Availability using snapshots.

Scaling Storage – You can scale to a larger database instance by creating and then restoring a snapshot.

Data Transfer – Data transfer to and from Lightsail instances in the same AWS Region does not count against the usage that is included in your plan.

Amazon RDS – This feature shares core technology with Amazon RDS, and benefits from our operational experience with that family of services.

Available Now
Managed databases are available today in all AWS Regions where Lightsail is available:

Jeff;

Bro becomes Zeek

Post Syndicated from corbet original https://lwn.net/Articles/768486/rss

The Bro network security monitoring project has announced
a name change to “Zeek”. “On the Leadership Team of the Bro Project,
we heard clear concerns from the Bro community that the name ‘Bro’ has
taken on strongly negative connotations, such as ‘Bro culture’. These send
a sharp, anti-inclusive – and wholly unintended and undesirable – message
to those who might use Bro. The problems were significant enough that
during BroCon community sessions, several people have mentioned substantial
difficulties in getting their upper management to even consider using
open-source software with such a seemingly ill-chosen, off-putting
name.

SFLC: Automotive Software Governance and Copyleft

Post Syndicated from corbet original https://lwn.net/Articles/768479/rss

The Software Freedom Law Center has announced
the availability of a
whitepaper [PDF]
about automotive software and copyleft, written by
Mark Shuttleworth and Eben Moglen. At its core, it’s an advertisement for
Ubuntu and Snap, but it does look at some of the issues involved.

The fine grain of interface access rights provided by the snapd
governance agent can thus provide further isolation and security when it
is running user-modified code, guaranteed under the snap packaging
paradigm to cause no other program code to be modified, to break, or to
perform differently because of the presence of the user-modified
program. Such a structure of modification permission can be operated by the
OEM consistent with the requirements of GPLv3. The OEM can publish an
authenticated record of the installation permission issued, indexed by the
Vehicle Identification Number—without publishing the car owner’s
personal information—so that public and private parties can be assured that
no surreptitious modification of vehicle software occurs.

Security updates for Tuesday

Post Syndicated from ris original https://lwn.net/Articles/768478/rss

Security updates have been issued by CentOS (ghostscript and spamassassin), Debian (moin, spice, and tomcat8), Fedora (kernel-headers, kernel-tools, and libgit2), Oracle (ghostscript and tomcat), Red Hat (ghostscript and tomcat), Scientific Linux (ghostscript and tomcat), SUSE (git, kernel, python, and samba), and Ubuntu (net-snmp and thunderbird).

Hard Drive Stats for Q3 2018: Less is More

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/2018-hard-drive-failure-rates/

Backblaze Drive Stats Q3 2018

As of September 30, 2018 Backblaze had 99,636 spinning hard drives. Of that number, there were 1,866 boot drives and 97,770 data drives. This review looks at the quarterly and lifetime statistics for the data drive models in operation in our data centers. In addition, we’ll say goodbye to the last of our 3TB drives, hello to our new 12TB HGST drives, and we’ll explain how we have 584 fewer drives than last quarter, but have added over 40 petabytes of storage. Along the way, we’ll share observations and insights on the data presented and we look forward to you doing the same in the comments.

Hard Drive Reliability Statistics for Q3 2018

At the end of Q3 2018, Backblaze was monitoring 97,770 hard drives used to store data. For our evaluation, we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 45 drives (see why below). This leaves us with 97,600 hard drives. The table below covers what happened in Q3 2018.

Backblaze Q3 2018 Hard Drive Failure Rates chart

Notes and Observations

  • If a drive model has a failure rate of 0%, it only means there were no drive failures of that model during Q3 2018.
  • Quarterly failure rates can be volatile, especially for models that have a small number of drives and/or a small number of Drive Days.
  • There were 170 drives (97,770 minus 97,600) that were not included in the list above because we did not have at least 45 of a given drive model. We use 45 drives of the same model as the minimum number when we report quarterly, yearly, and lifetime drive statistics.

When to Replace a Hard Drive

As noted, at the end of Q3 that we had 584 fewer drives, but over 40 petabytes more storage space. We replaced 3TB, 4TB, and even a handful of 6TB drives with 3,600 new 12TB drives using the very same data center infrastructure, i.e. racks of Storage Pods. The drives we are replacing are about 4 years old. That’s plus or minus a few months depending on how much we paid for the drive and a number of other factors. Keeping lower density drives in service when higher density drives are both available and efficiently priced does not make economic sense.

Why Drive Migration Will Continue

Over the next several years, data growth is expected to explode. Hard drives are still expected to store the bulk of that data, meaning cloud storage companies like Backblaze will have to increase capacity by either increasing existing storage density and/or building, or building out, more data centers. Drive manufacturers, like Seagate and Western Digital, are looking at HDD storage densities of 40TB as early as 2023, just 5 years away. It is significantly less expensive to replace lower density operational drives in a data center versus building a new facility or even building out an existing facility to house the higher density drives.

Goodbye 3TB WD Drives

For the last couple of quarters, we had 180 Western Digital 3TB drives (model: WD30EFRX) remaining — the last of our 3TB drives. In early Q3, they were removed and replaced with 12TB drives. These 3TB drives were purchased in the aftermath of the Thailand drive crisis and installed in mid-2014 and were still hard at work when we replaced them. Sometime over the next couple of years we expect to say goodbye to all of our 4TB drives and upgrade them to 14, 16, or even 20TB drives. After that it will be time to “up-density” our 6TB systems, then our 8TB systems, and so on.

Hello 12TB HGST Drives

In Q3 we added 79 HGST 12TB drives (model: HUH721212ALN604) to the farm. While 79 may seem like an unusual number of drives to add, it represents “stage 2” of our drive testing process. Stage 1 uses 20 drives, the number of hard drives in one Backblaze Vault tome. That is, there are are 20 Storage Pods in a Backblaze Vault, and there is one “test” drive in each Storage Pod. This allows us to compare the performance, etc., of the test tome to the remaining 59 production tomes (which are running already-qualified drives). There are 60 tomes in each Backblaze Vault. In stage 2, we fill an entire Storage Pod with the test drives, adding 59 test drives to the one currently being tested in one of the 20 Storage Pods in a Backblaze Vault.

To date, none of the 79 HGST drives have failed, but as of September 30th, they were installed only 9 days. Let’s see how they perform over the next few months.

A New Drive Count Leader

For the last 4 years, the drive model we’ve deployed the most has been the 4TB Seagate drive, model ST4000DM000. In Q3 we had 24,208 of this drive model, which is now only good enough for second place. The 12TB Seagate drive, model ST12000NM0007, became our new drive count leader with 25,101 drives in Q3.

Lifetime Hard Drive Reliability Statistics

While the quarterly chart presented earlier gets a lot of interest, the real test of any drive model is over time. Below is the lifetime failure rate chart for all the hard drive models in operation as of September 30th, 2018. For each model, we compute their reliability starting from when they were first installed.

Backblaze Lifetime Hard Drive Failure Rates Chart

Notes and Observations

  • The failure rates of all of the larger drives (8, 10, and 12 TB) are very good: 1.21% AFR (Annualized Failure Rate) or less. In particular, the Seagate 10TB drives, which have been in operation for over 1 year now, are performing very nicely with a failure rate of 0.48%.
  • The overall failure rate of 1.71% is the lowest we have ever achieved, besting the previous low of 1.82% from Q2 of 2018.

The Hard Drive Stats Data

The complete data set used to create the information used in this review is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone. It is free.

If you just want the summarized data used to create the tables and charts in this blog post you can download the ZIP file containing the MS Excel spreadsheet.

Good luck and let us know if you find anything interesting.

The post Hard Drive Stats for Q3 2018: Less is More appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Electronics 101.1: Electricity basics

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/electronics-101-1-electricity-basics/

In HackSpace issue 9, Dave Astels helps us get familiar with what electricity is, with some key terms and rules, and with a few basic components. Get your copy of HackSpace magazine in stores now, or download it as a free PDF here.

An animated GIF of Pickachu the Pokemon

tl;dr There’s more to electricity than Pikachu.

Electricity basics

Electricity is fascinating. Most of our technology relies on it: computers, lights, appliances, and even cars, as more and more are hybrid or electric. It follows some well-defined rules, which is what makes it so very useful.

According to Wikipedia, electricity is ‘the set of physical phenomena associated with the presence and motion of electric charge’. And what’s electric charge? That’s the shortage or excess of electrons.

Let’s go back (or forward, depending on where you are in life) to high school science and the atom. An atom is, at a very simplified level, a nucleus surrounded by a number of electrons. The nucleus is (again, viewing it simply) made up of neutrons and protons. Neutrons have no charge, but protons have a positive charge. Electrons have a negative charge. The negative charge on a single electron is the exact opposite of the positive charge on a single proton. The simplest atom, hydrogen, is made from a single proton and a single electron. The net charge of the atom is zero: the positive charge of the proton and the negative charge of the electron cancel – or balance – each other. An atom’s electrons aren’t just in an amorphous cloud around the nucleus: you can think of them as being arranged in layers around the nucleus…rather like an onion. Or perhaps an ogre. This is a very simplified visualisation of it, but it suffices for our purposes.

A diagram of a copper atom and the text '29 Electrons'

Figure 1: A very stylised representation of a copper atom with its electron shell

In a more complex atom, say copper, there are more protons, neutrons, and electrons, and the electrons are in more layers. By default, a copper atom has 29 protons and 35 neutrons in its nucleus, which is surrounded by 29 electrons. The way the electrons are distributed in their layers leaves the copper atom with a single electron in the outermost layer. This is represented in Figure 1 (above). Without getting further into subatomic physics, let’s just say that having that single electron in the outermost layer makes it easier to manipulate. When we put a bunch of copper atoms together to make copper metal (e.g. a wire), it’s easy to move those outermost electrons around inside the metal. Those electrons moving around is electricity. The amount of electrons moving over a period of time is called ‘current’.

A multimeter showing the figure 9.99 with a resistor connected via crocodile clips

A single 10 kΩ resistor reads almost 10 000 ohms (no electrical component is perfect).

We started by talking about electrons and charge. Look back at the Wikipedia definition: ‘presence and motion of electric charge’. Charge is measured in coulombs: 1 coulomb is approximately 6.242 × 1018 electrons. That’s 6 242 000 000 000 000 000 electrons. They’re very small. Actually, this would be -1 coulomb. +1 coulomb would be that many protons (or really, the net lack of that many electrons).

That’s charge. Now let’s consider moving charge, which is far more useful in general (unless your goal is to stick balloons to the wall). Consider some amount of charge moving through a wire. The amount of charge that moves past a specific point (and thus through the wire) over a period of time is called ‘current’ (just like the current in a river) and is measured in amperes, generally just called amps. Specifically, 1 amp is equal to 1 coulomb flowing past a point in 1 second.

Another common term is voltage. You can think of voltage like water pressure; it’s the pressure pushing the electrons (i.e. charge) through a material. The higher the voltage (measured in volts), the faster charge is pushed through, i.e. the higher the current.

The final term is resistance, measured in ohms. Resistance is just what it sounds like. It’s a measure of how much a material resists the movement of electrons. We said that copper allows electrons to move freely. That’s what makes it so common for wires, PCB traces, etc. We say that it is a good conductor. Glass, on the other hand, locks its electrons in place, not letting them move. It’s an example of a good insulator. There are materials that are in between: they let electrons move, but not too freely. These are crucial to making electronics work.

There’s an interesting (and useful) relationship between voltage, current, and resistance called Ohm’s Law (Georg Ohm was the fellow who explored and documented this relationship): the current (denoted I, in amps) flowing through a material is equal to the voltage across the material (denoted V, in volts) divided by the material’s resistance (denoted R, in ohms): I = V/R. This equation is foundational and, as such, very handy.

Lighting up

There aren’t many electronic devices that don’t have at least one LED on them somewhere, especially not gadgety ones. If you look at a simple Arduino Uno, it has LEDs for power, Tx, Rx, and pin 13. The first program using electronic components that most people try is one to blink an LED.

A colour spectrum from red to purple

Figure 2: The colour spectrum

LED stands for light-emitting diode. We’ll come back to diodes in a later instalment; all we need to know right now is that a diode has to go the right way around. So that leaves ‘light-emitting’. That simply means that it gives off light: it lights up. Specifically, it lights up when enough current flows through it. Be careful, though. Put too much current through it and it’ll likely crack in two. Seriously, we’ve done it. Best case scenario, you’ll get a bright pulse of light as it burns out. How much current do they like? 20 milliamps (20mA) is typical. Because an LED is a diode, i.e. a semiconductor (we’ll look at these in more detail in a future instalment too), it defies Ohm’s Law. How? It always has the same voltage across it, regardless of the current flowing through it.

An LED will have a specific Vf (f is for forward, as in ‘forward voltage’), which will be defined in its data sheet.

The voltage varies with the colour of light that the LED emits, but usually between 1.8V and 3.3V. Vf for red LEDs will typically be 1.8V, and for blue LEDs 3V–3.3V. As a rule, LEDs with a higher frequency colour will have a larger Vf. Figure 2 (above) shows the colour spectrum. Colours on the right end are lower in frequency and LEDs emitting those colours will have a lower Vf, while those on the left end have a higher frequency and a higher Vf.

A screenshot of resistor-calculator website

Resistor colour bands show the resistance. Online calculators can help you learn the values.

So an LED will have a fixed Vf, and a typical LED that we’ll use likes about 20mA of current. An LED won’t do anything to limit how much current is flowing through it. That’s what we meant when we said it defies Ohm’s Law.

If we take a blue LED and hooked it to a 3.3V power supply, it will shine happily. Do the same thing with a red LED, and it will blink and burn out. So how do we deal with that? How do we use 3.3V or 5V to make an LED light up without burning out? We simply limit the current flowing through it. And for that, we need a resistor and Ohm’s Law.

Getting protection

Figure 3: An LED with a current-limiting resistor

If we want to power a red LED from a 5V source, we know the following information: current has to be 20mA, Vcc will be 5V, and the voltage across the LED will be 1.8V. Consider the circuit in Figure 3. The voltage across the resistor will be Vcc – Vf, i.e. 5 – 1.8 = 3.2V. We said the current through the LED should be 20mA. Since there is only one path through the circuit that goes through the resistor as well as the LED, all current has to flow through both: whatever amount of current flows through the resistor has to flow through the LED, no more, no less. This is the crucial thing to realise. We can calculate the value of the resistance needed using Ohm’s Law: R = V/I = 3.2V/20mA = 3.2V/0.020A = 160 ohms.

The resistor should have a value of 160 ohms to allow 20mA of current to flow through the LED. Knowing that the 20mA and 1.8V values are approximate and that resistors are not exact (+/- 5 or 10 percent are the most common), we chose a slightly higher-value resistor. Considering common resistor values, go with 180 ohm or 220ohm. A higher-value resistor will allow slightly less current through, which might result in a slightly dimmer light. Try it and see. For practical purposes, simply using a 220 ohm resistor usually works fine.

Parallel lines

In the previous section we connected a resistor and an LED end to end. That’s called a series circuit. If we connected them side by side, it would be a parallel circuit. Consider the circuits in Figure 4.

Figure 4: A – series circuit; B – parallel circuit

We’ll use 5V for Vcc. What is the total resistance between Vcc and GND in each circuit? How much current is flowing through each circuit? What is the voltage across each resistor?

When resistors are connected in series, as in circuit A, the resistances are added. So the two 100 ohm resistors in series have a total resistance of 200 ohms.

When resistors are connected in parallel, as in circuit B, it’s more complex. Each resistor provides a path for current to flow through. So we could use an indirect method to calculate the total resistance. Each resistor is 100 ohms, and has one end connected to 5V and the other to 0V (GND), so the voltage across each one is 5V. The current flowing through each one is 5V/100 ohms = 0.05A, or 50mA. That flows through each resistor, so the total current is 100mA, or 0.1A. The total resistance is then R = V/I = 5V/0.1A = 50 ohms. A more direct way is to use the equation 1/Rt = 1/R1 + 1/R2 + … + 1/Rn, where Rt is the total resistance, and R1, R2, etc. are the values of the individual resistors that are in parallel. Using this, 1/Rt = 1/100 + 1/100 = 2/100 = 1/50. So Rt = 50. This is a quicker way to do it, and only involves the resistor values.

An image of a multimeter

A multimeter can read voltage, ampage, and resistance

Now for current. We know that the series circuit has a total resistance of 200 ohms, so the current will be I = V/R = 5V/200 ohm = 0.025A = 25mA. For one 100 ohm resistor the current is 5V/100 ohm = 0.05A = 50mA. This is expected: if the resistance is lower, there is less ‘resistance’ to current flowing, so with the same voltage, more current will flow. We already computed the current for the parallel circuit: 100mA. This is higher because we know that each resistor has 50mA flowing through it. In a parallel circuit, the currents are added.

A multimeter showing the figure 19.88 with a resistor connected via crocodile clips

Two 10kΩ (kiloohm) resistors in series read (almost) 20kΩ

The final question is what voltage is across each resistor. Let’s look at the parallel circuit first. One end of each resistor is connected to 5V, and the other end of each is connected to 0V (GND). So clearly, the voltage across each one is 5V. In a series circuit it’s different. We can use Ohm’s Law because we’ve calculated the current flowing through each one (0.025A), and that current flows through both resistors. Each resistor is 100 ohm, so the voltage across each one will be V = I×R = 0.025A × 100 ohm = 2.5 V. This makes sense intuitively, since the resistors have the same value and the same current is flowing through both. It makes sense that the voltage across each would be equal, and half of the total. Remember that it’s unlikely to be exactly half, due to the slop in the resistor values.

Let’s do this one more time with unequal resistors. See Figure 5.

Figure 5: A – series circuit; B – parallel circuit

For the series circuit, we simply add the resistances: 100ohm + 82ohm = 182ohm. The current is 5V / 182ohm = 0.0274725A = 27.4725 mA. Because resistors are inexact, it’s safe to call this 27.5mA. The voltages are 100ohm × 0.0275A = 2.75V across the 100 ohm resistor, and 82ohm × 0.275 = 2.25V across the 82 ohm one. The voltages always have to add up, accepting rounding errors. Relative to ground, the voltage at the point between the resistors is 2.75V. What will happen if we make the top resistor smaller (i.e. have a lower resistance)? The total resistance goes down, the current goes up, so the voltage across the 100ohm resistor goes up. This is what’s generally called a voltage divider.

For the parallel circuit we can use 1/Rt = 1/100 + 1/82 = 82/8200 + 100/8200 = 182/8200 = 1/45, so Rt = 45ohm. The total current is 5V / 45ohm = 0.111A = 111mA. For the individual resistors, the currents are 5V / 100ohm = 50mA and 5V / 82ohm = 61mA. Add these up and we have the total current of 111mA. Parallel resistors act as a current divider.

A multimeter showing the figure 4.96 with a resistor connected via crocodile clips

Two 10kΩ resistors in parallel read (almost) 5kΩ.

I encourage you to create these little circuits on a breadboard and measure the resistances, voltages, and currents for yourself.

Resistors in series for a voltage divider, resisters in parallel for a current divider

Consider what happens if we replace the resistor connected to Vcc in a series circuit with a variable resistor. The voltage between the resistors will vary as the value of the resistor does. As the resistance goes down, the voltage goes up. The reverse is true as well: as the resistance goes up, the voltage goes down. One use of this is to replace the variable resistor with a photoresistor. A photoresistor’s value depends on how much light is shining on it (i.e. how many photons are hitting it, to be precise). More light = lower resistance. Now the voltage divider can be used to measure the strength of light. All you need to do is connect the point between the resistors to an analogue input and read it.

Figure 6 Combined parallel and series circuits

We’ve had a brief look at the basic concepts of electricity: charge, current, voltage, and resistance. We’ve also had a closer look at resistors and ways of combining them. We finished with a practical example of a series resistor circuit being used to measure light.

The post Electronics 101.1: Electricity basics appeared first on Raspberry Pi.

Privacy for Tigers

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/10/privacy_for_tig.html

Ross Anderson has some new work:

As mobile phone masts went up across the world’s jungles, savannas and mountains, so did poaching. Wildlife crime syndicates can not only coordinate better but can mine growing public data sets, often of geotagged images. Privacy matters for tigers, for snow leopards, for elephants and rhinos ­ and even for tortoises and sharks. Animal data protection laws, where they exist at all, are oblivious to these new threats, and no-one seems to have started to think seriously about information security.

Video here.

[$] Fighting Spectre with cache flushes

Post Syndicated from corbet original https://lwn.net/Articles/768418/rss

One of the more difficult aspects of the Spectre hardware vulnerability is
finding all of the locations in the code that might be exploitable. There
are many locations that look vulnerable that aren’t, and others that are
exploitable without being obvious. It has long been clear that finding all
of the exploitable spots is a long-term task, and keeping new ones from
being introduced will not be easy. But there may be a simple technique that
can block a large subset of the possible exploits with a minimal cost.

Restrict access to your AWS Glue Data Catalog with resource-level IAM permissions and resource-based policies

Post Syndicated from Ben Snively original https://aws.amazon.com/blogs/big-data/restrict-access-to-your-aws-glue-data-catalog-with-resource-level-iam-permissions-and-resource-based-policies/

A data lake provides a centralized repository that you can use to store all your structured and unstructured data at any scale. A data lake can include both raw datasets and curated, query-optimized datasets. Raw datasets can be quickly ingested, in their original form, without having to force-fit them into a predefined schema. Using data lakes, you can run different types of analytics on both raw and curated datasets. By using Amazon S3 as the storage layer of your data lakes, you can have a set of rich controls at both the bucket and object level. You can use these to define access control policies for the datasets in your lake.

The AWS Glue Data Catalog is a persistent, fully managed metadata store for your data lake on AWS. Using the Glue Data Catalog, you can store, annotate, and share metadata in the AWS Cloud in the same way you do in an Apache Hive Metastore. The Glue Data Catalog also has seamless out-of-box integration with Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

Using AWS Glue, you can also create policies to restrict access to different portions of the catalog based on users, roles, or applied at a resource level. With these policies, you can provide granular control over which users can access the various metadata definitions in your data lake.

Important: The S3 and the AWS Glue Data Catalog policies define the access permissions to the data and metadata definitions respectively. In other words, the AWS Glue Data Catalog policies define the access to the metadata, and the S3 policies define the access to the content itself.

You can restrict which metadata operations can be performed, such as GetDatabases, GetTables, and CreateTable, and others using identiy-based policies (IAM). You can also restrict which data catalog objects those operations are performed on. Additionally, you can limit which catalog objects get returned in the resulting call. A Glue Data Catalog “object” here refers to a database, a table, a user-defined function, or a connection stored in the Glue Data Catalog.

Suppose that you have users that require read access to your production databases and tables in your data lake, and others have additional permissions to dev resources. Suppose also that you have a data lake storing both raw data feeds and curated datasets used by business intelligence, analytics, and machine learning applications. You can set these configurations easily, and many others, using the access control mechanisms in the AWS Glue Data Catalog.

Note: The following example shows how to set up a policy on the AWS Glue Data Catalog. It doesn’t set up the related S3 bucket or object level policies. This means the metadata isn’t discoverable when using Athena, EMR, and tools integrating with the AWS Glue Data Catalog. At the point when someone tries to access an S3 object directly, S3 policy enforcement is important. You should use Data Catalog and S3 bucket or object level policies together.

Fine-grained access control

You can define the access to the metadata using both resource-based and identity-based policies, depending on your organization’s needs. Resource-based policies list the principals that are allowed or denied access to your resources, allowing you to set up policies such as cross-account access. Identity policies are specifically attached to users, groups, and roles within IAM.

The fine-grained access portion of the policy is defined within the Resource clause. This portion defines both the AWS Glue Data Catalog object that the action can be performed on, and what resulting objects get returned by that operation.

Let’s run through an example. Suppose that you want to define a policy that allows a set of users to access only the finegrainacces database. The policy also allows users to return all the tables listed within the database. For the GetTables actions, the resource definitions include these resource statements:

"arn:aws:glue:us-east-1:123456789012:catalog",
"arn:aws:glue:us-east-1:123456789012:database/finegrainaccess",
"arn:aws:glue:us-east-1:123456789012:table/finegrainaccess/*"

The first resource statement at the database Amazon Resource Name (ARN) allows the user to call the operation on the finegrainaccess database. The second ARN allows all the tables within that database to be returned.

Now, what if we want to return only the tables that started with “dev_” from the “finegrainaccess” database? If so, this is how the policy changes:

"arn:aws:glue:us-east-1:123456789012:catalog",
"arn:aws:glue:us-east-1:123456789012:db/finegrainaccess",           
"arn:aws:glue:us-east-1:123456789012:tables/finegrainaccess/dev_*"  

Now, we are specifying dev_ as part of the table’s ARN in the second resource definition. This approach also works with actions for getting the list of databases, partitions for a table, connections, and other operations in the catalog.

Taking it for a spin

Note: This post focuses on the policies for AWS Glue Data Catalog. If you look closely, all of these datasets are pointing to the same S3 locations, which are world-readable. In a full example, you should also set the necessary S3 bucket or object level permissions, or both.

Next, we show an example you can do yourself. The next example creates the following in a Data Catalog.

We set up two users in the example, as shown following.

In the AWS Management Console, launch the AWS CloudFormation template.

Choose Next.

Important: Enter a password for the IAM users to be created. These users will have permissions to run Athena queries, access to your Athena S3 results bucket, and see the AWS Glue databases and tables that the CloudFormation script creates. These permissions must match the minimum requirements of the IAM password policy on the account that you run this example from.

Choose Next, and then on the next page choose Next again.

Lastly, acknowledge that the template will create IAM users and policies.

Then choose Create.

When you refresh your CloudFormation page, you can see the script creating the example resources.

Wait until it’s complete.

This script creates the necessary IAM users and policies attached to them, along with the necessarily databases and tables listed preceding.

After the CloudFormation script completes, you should see these tables if using an administrator user.

If you look on the Outputs tab, you can see the two IAM users that were created along with your IAM sign-in URL.

Note:

If you click the sign-in link in the same browser, the system logs you out. A nice trick is to right-click and open a private or incognito window.

If the provided IAM password doesn’t meet the minimum requirements, you see this message in the CloudFormation script event log:

The specified password is invalid … <why it was invalid>

Looking in the AWS Glue Data Catalog, you can see the tables that just got created by the script.

We can see the script created the structure that we outlined preceding.

Let’s check the two user profiles. If you go into IAM and users, they are set as inline policies. You should see the following for each user.

For the AWS Glue dev user, this section gives us full access to anything in the dev databases:

This section gives us the ability to query and see the prod database:

Lastly, this section gives us access to get tables and partitions from the prod database. You can structure this section so that it explicitly lists the blog_prod database in the resource and only allow that. The following lets someone query for database/* and return only the blog_prod tables. This, in fact, is the default behavior of the console.

Without this, you could still query those two databases explicitly, but the policy would not allow a wildcard query such as the following.

In contrast, the QA user doesn’t have access to the dev database and can only see the tables that start with prod_in the prod database. So the following is what the QA user’s policy looks like.

The query for the prod database is as follows:

Only GetTables and GetPartitions are available for the tables starting with prod_.

Notice the “prod_*” in the resource definition following.

Querying based on the different users

Logging in as the two different IAM users created by the AWS CloudFormation output tab and the password you provided, you can see some differences.

Notice that the QA user can’t see any metadata definitions for the blog_dev database, or the staging_yellow table in the blog_prod database.

Next, sign in as blog_dev_user and go to the Athena console. Notice that the blog user only sees the databases and tables listed that this user is permitted to.

The dev user can create a table under blog_dev, but not the blog_prod database.

Now let’s look under dev_qa_user. Notice that we only see the blog_prod and prod_* tables in Athena.

The QA user can query the datasets that user can see, but the policy doesn’t let that user create a database or tables.

If the QA user tries to query through Athena, and manually pull the metadata outside the console, that user can’t see any of the information. You can test this by running the following.

select * from blog_dev.yellow limit 10;

Conclusion

Data cataloging is an important part of many analytical systems. The AWS Glue Data Catalog provides integration with a wide number of tools. Using the Data Catalog, you also can specify a policy that grants permissions to objects in the Data Catalog. Data lakes require detailed access control at both the content level and the level of the metadata describing the content. In this example, we show how you can define the access policies for the metadata in the catalog.


Additional Reading

Learn how to harmonize, search, and analyze loosely coupled datasets on AWS.

 


About the Author

Ben Snively is a Public Sector Specialist Solutions Architect. He works with government, non-profit and education customers on big data and analytical projects, helping them build solutions using AWS. In his spare time, he adds IoT sensors throughout his house and runs analytics on it.

 

 

 

Celebrating 10 years of Microsoft Windows Server and SQL Server on AWS! Happy Birthday!

Post Syndicated from Betsy Chernoff original https://aws.amazon.com/blogs/compute/celebrating-10-years-of-microsoft-windows-server-and-sql-server-on-aws-happy-birthday/

Contributed by Sandy Carter, Vice President of Windows on AWS and Enterprise Workloads

Happy birthday to all of our AWS customers! In particular, I want to call out Autodesk, RightScale (now part of Flexera), and Suunto (Movescount) – just a few of our customers who have been running Microsoft Windows Server and Microsoft SQL Server on AWS for 10 years! Thank you for your business and seeing the value of Windows on AWS!

So many customers trust their Windows workloads on AWS because of our experience, reliability, security, and performance. IDC (a leading IT Analyst) estimates that AWS accounted for approximately 57.7% of total Windows instances in public cloud IaaS during 2017 – nearly 2x the nearest cloud provider.

Our Windows on AWS customers benefit from the millions of active customers per month across the AWS Cloud. Ancestry, the global leader in family history and consumer genomics, has over 6000 instances on AWS. Nat Natarajan, Executive Vice President of Product and Technology at Ancestry just spoke with us in Seattle. I loved hearing how they are using Windows on AWS.

“AWS provides us with the flexibility we need to stay at the forefront of consumer genomics, as the science and technology in the space continues to rapidly evolve. We’re confident that AWS provides us with unmatched scalability, security, and privacy.”

Reliability is one the reasons why NextGen Healthcare, provider of tailored healthcare solutions for ambulatory practices and healthcare providers around the world, trusts AWS to run their SQL Server databases. One of the foundations of our reliability is how we design our Regions. AWS has 18 Regions around the globe, each of which are made up of two or more Availability Zones. Availability Zones are physically separate locations with independent infrastructure engineered to be insulated from failures in other Availability Zones. Today we have 55 Availability Zones across these 18 Regions, and we’ve announced plans for 12 more Availability Zones and four more Regions.

I talk to so many of our customers every week who tell me that their Windows and SQL Server workloads run better on AWS. For example, eMarketer enables thousands of companies around the world to better understand markets and consumer behavior. This helps them get the data they need to succeed in a competitive and fast changing digital economy. They recently told me how they started their digital transformation initiative on another public cloud.

“We chose to move our Microsoft workloads to AWS because of your extensive migration experience, higher availability, and better performance. We are seeing 35% cost savings and thrilled to see 4x faster launch times now.” – Ryan Hoffman, Senior Vice President of Engineering

One of the things I get asked about more and more is, can you modernize those Windows apps as well? Using serverless compute on AWS Lambda, Windows containers, and Amazon Machine Learning (Amazon ML), you can really take those Windows apps into the 21st century! For example, Mitek, the global leader in mobile capture and identity verification software solutions, wanted to modernize their Mobile Verify application to accelerate integration across multiple regions and environments. They leveraged Windows containers using Amazon ECS so they could focus their resources on developing more features instead of servers, VMs, and patching. They reduced their deployment time from hours to minutes!

We know .NET developers love using their existing tools. We created tools such as AWS Toolkit for Visual Studio and AWS Tools for Visual Studio Team Services (VSTS) to provide integration into many popular AWS services. Agero tells us how easy it is for their .NET developers to get started with AWS. Agero provides connected vehicle data, roadside assistance, and claims management services to over 115 million drivers and leading insurers.

“We experimented with AWS Elastic Beanstalk and found it was the simplest, fastest way to get .NET code running in AWS.” Bernie Gracy, Chief Digital Officer

Of course, most of our customers use Microsoft Active Directory on-premises for directory-based identity-related services, and some also use Azure AD to manage users with Office365. Customers use AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) to easily integrate AWS resources with on-premises AD and Azure AD. That way, there’s no data to be synchronized or replicated from on-premises to AWS. AWS Managed AD lets you use the same administration tools and built-in features such as single sign-on (SSO) or Group Policy as you use on-premises. And we now enable our customers to share a single directory with multiple AWS accounts within an AWS Region!

This birthday is significant to us here at Amazon Web Services as we obsess over our customers. Over 90% of our roadmap items are driven directly from you! With hundreds of thousands of active customers running Windows on AWS, that’s a lot of great ideas. In fact, did you know that our premier serverless engine, AWS Lambda, which lets you run .NET Core without provisioning or managing servers, came directly from you, our customers?

Some of you wanted an easier way to jumpstart Windows Server projects on AWS, which led us to build Amazon Lightsail, giving you compute, storage and networking with a low, predictable price. Based on feedback from machine learning practitioners and researchers, we launched our AWS Deep Learning AMI for Windows so you can quickly launch Amazon EC2 instances pre-installed with popular deep learning frameworks including Apache MXNet, Caffe and Tensorflow.

Licensing Options

Our customer obsession means that we are committed to helping you lower your total cost of ownership (TCO). When I talk to customers, they tell me they appreciate that AWS does not approach their cloud migration journey as a way to lock-in additional software license subscriptions. For example, TSO Logic, one of our AWS Partner Network (APN) partners, described in a blog the work we did with one of our joint customers, a privately-held U.S. company with more than 70,000 employees, operating in 50 countries. We helped this customer save 22 percent on their SQL Server workloads by optimizing core counts and reducing licensing costs.

Delaware North, a global leader in hospitality management and food service management, uses our pay-as-you-go licenses to scale-up their SQL Server instances during peak periods, without having to pay for those licenses for multiple years. Many customers also use License Mobility benefits to bring their Microsoft application licenses to AWS, and other customers, such as Xero, the accounting software platform for small and medium-sized businesses, reduce costs by bringing their own Windows Server Datacenter Edition and SQL Server Enterprise Edition licenses to AWS on our Amazon EC2 Dedicated Hosts. And, we also have investment programs to help qualified customers offset Microsoft licensing costs when migrating to the AWS Cloud.

We know that many of you are thinking about what to do with legacy applications still using the 2008 versions of SQL Server and Windows Server. I hear from many leaders who don’t want to base their cloud strategy on software end-of-support. AWS provides flexibility to easily upgrade and modernize your legacy workloads. ClickSoftware Technologies, the SaaS provider of field service management solutions, found how easy it was to upgrade to a current version of SQL Server on AWS.

“After migrating to AWS, we upgraded to SQL Server 2016 using SQL Server 2008 in compatibility mode, which meant we did not have to make any application changes, and now have a fully supported version of SQL Server.” – Udi Keidar, VP of Cloud Services, ClickSoftware

Did you know you can also bring your Microsoft licenses to VMware Cloud on AWS? VMware Cloud on AWS is a great solution when you need to execute a fast migration – whether that’s due to running out of data center space, an upcoming lease expiration, or a natural disaster such as the recent hurricanes. Massachusetts Institute of Technology (MIT) started with a proof of concept (POC) and moved their initial 300 VMs in less than 96 hours, with just one employee. Over the next three months they migrated of all of their 2,800 production VMs to VMware Cloud on AWS.

Looking Forward

Next month, I hope you’ll join me at AWS re:Invent to learn “What’s New with Microsoft and .NET on AWS” as well as the dozens of other sessions we have for Windows on AWS for IT leaders, DevOps engineers, system administrators, DBAs and .NET developers. We have so many new innovations to share with you!

I want to thank each you for trusting us these last ten years with your most critical business applications and allowing us to continue to help you innovate and transform your business. If you’d like to learn more about how we can help you bring your applications built on Windows Server and SQL Server to the cloud, please check out the following resources and events or contact us!

  1. https://aws.amazon.com/windows/
  2. https://aws.amazon.com/sql/
  3. https://aws.amazon.com/directoryservice/
  4. https://aws.amazon.com/windows/windows-study-guide/
  5. AWS .NET Developer Center

Upcoming Events

  1. October 23, 2018 Webinar: Migrating Microsoft SQL Server 2008 Databases to AWS
  2. Live with AM & Nicki – A fun new twitch.tv series to show you how to build a modern web application on AWS!
  3. November 26-30 2018 re:Invent: Check out the complete list of sessions for Windows, SQL Server, Active Directory and .NET on AWS!

 

 

 

New – How to better monitor your custom application metrics using Amazon CloudWatch Agent

Post Syndicated from Helen Lin original https://aws.amazon.com/blogs/devops/new-how-to-better-monitor-your-custom-application-metrics-using-amazon-cloudwatch-agent/

This blog was contributed by Zhou Fang, Sr. Software Development Engineer for Amazon CloudWatch and Helen Lin, Sr. Product Manager for Amazon CloudWatch

Amazon CloudWatch collects monitoring and operational data from both your AWS resources and on-premises servers, providing you with a unified view of your infrastructure and application health. By default, CloudWatch automatically collects and stores many of your AWS services’ metrics and enables you to monitor and alert on metrics such as high CPU utilization of your Amazon EC2 instances. With the CloudWatch Agent that launched last year, you can also deploy the agent to collect system metrics and application logs from both your Windows and Linux environments. Using this data collected by CloudWatch, you can build operational dashboards to monitor your service and application health, set high-resolution alarms to alert and take automated actions, and troubleshoot issues using Amazon CloudWatch Logs.

We recently introduced CloudWatch Agent support for collecting custom metrics using StatsD and collectd. It’s important to collect system metrics like available memory, and you might also want to monitor custom application metrics. You can use these custom application metrics, such as request count to understand the traffic going through your application or understand latency so you can be alerted when requests take too long to process. StatsD and collectd are popular, open-source solutions that gather system statistics for a wide variety of applications. By combining the system metrics the agent already collects, with the StatsD protocol for instrumenting your own metrics and collectd’s numerous plugins, you can better monitor, analyze, alert, and troubleshoot the performance of your systems and applications.

Let’s dive into an example that demonstrates how to monitor your applications using the CloudWatch Agent.  I am operating a RESTful service that performs simple text encoding. I want to use CloudWatch to help monitor a few key metrics:

  • How many requests are coming into my service?
  • How many of these requests are unique?
  • What is the typical size of a request?
  • How long does it take to process a job?

These metrics help me understand my application performance and throughput, in addition to setting alarms on critical metrics that could indicate service degradation, such as request latency.

Step 1. Collecting StatsD metrics

My service is running on an EC2 instance, using Amazon Linux AMI 2018.03.0. Make sure to attach the CloudWatchAgentServerPolicy AWS managed policy so that the CloudWatch agent can collect and publish metrics from this instance:

Here is the service structure:

 

The “/encode” handler simply returns the base64 encoded string of an input text.  To monitor key metrics, such as total and unique request count as well as request size and method response time, I used StatsD to define these custom metrics.

@RestController

public class EncodeController {

    @RequestMapping("/encode")
    public String encode(@RequestParam(value = "text") String text) {
        long startTime = System.currentTimeMillis();
        statsd.incrementCounter("totalRequest.count", new String[]{"path:/encode"});
        statsd.recordSetValue("uniqueRequest.count", text, new String[]{"path:/encode"});
        statsd.recordHistogramValue("request.size", text.length(), new String[]{"path:/encode"});
        String encodedString = Base64.getEncoder().encodeToString(text.getBytes());
        statsd.recordExecutionTime("latency", System.currentTimeMillis() - startTime, new String[]{"path:/encode"});
        return encodedString;
    }
}

Note that I need to first choose a StatsD client from here.

The “/status” handler responds with a health check ping.  Here I am monitoring my available JVM memory:

@RestController
public class StatusController {

    @RequestMapping("/status")
    public int status() {
        statsd.recordGaugeValue("memory.free", Runtime.getRuntime().freeMemory(), new String[]{"path:/status"});
        return 0;
    }
}

 

Step 2. Emit custom metrics using collectd (optional)

collectd is another popular, open-source daemon for collecting application metrics. If I want to use the hundreds of available collectd plugins to gather application metrics, I can also use the CloudWatch Agent to publish collectd metrics to CloudWatch for 15-months retention. In practice, I might choose to use either StatsD or collectd to collect custom metrics, or I have the option to use both. All of these use cases  are supported by the CloudWatch agent.

Using the same demo RESTful service, I’ll show you how to monitor my service health using the collectd cURL plugin, which passes the collectd metrics to CloudWatch Agent via the network plugin.

For my RESTful service, the “/status” handler returns HTTP code 200 to signify that it’s up and running. This is important to monitor the health of my service and trigger an alert when the application does not respond with a HTTP 200 success code. Additionally, I want to monitor the lapsed time for each health check request.

To collect these metrics using collectd, I have a collectd daemon installed on the EC2 instance, running version 5.8.0. Here is my collectd config:

LoadPlugin logfile
LoadPlugin curl
LoadPlugin network

<Plugin logfile>
  LogLevel "debug"
  File "/var/log/collectd.log"
  Timestamp true
</Plugin>

<Plugin curl>
    <Page "status">
        URL "http://localhost:8080/status";
        MeasureResponseTime true
        MeasureResponseCode true
    </Page>
</Plugin>

<Plugin network>
    <Server "127.0.0.1" "25826">
        SecurityLevel Encrypt
        Username "user"
        Password "secret"
    </Server>
</Plugin>

 

For the cURL plugin, I configured it to measure response time (latency) and response code (HTTP status code) from the RESTful service.

Note that for the network plugin, I used Encrypt mode which requires an authentication file for the CloudWatch Agent to authenticate incoming collectd requests.  Click here for full details on the collectd installation script.

 

Step 3. Configure the CloudWatch agent

So far, I have shown you how to:

A.  Use StatsD to emit custom metrics to monitor my service health
B.  Optionally use collectd to collect metrics using plugins

Next, I will install and configure the CloudWatch agent to accept metrics from both the StatsD client and collectd plugins.

I installed the CloudWatch Agent following the instructions in the user guide, but here are the detailed steps:

Install CloudWatch Agent:

wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip -O AmazonCloudWatchAgent.zip && unzip -o AmazonCloudWatchAgent.zip && sudo ./install.sh

Configure CloudWatch Agent to receive metrics from StatsD and collectd:

{
  "metrics": {
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "collectd": {},
      "statsd": {}
    }
  }
}

Pass the above config (config.json) to the CloudWatch Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:config.json -s

In case you want to skip these steps and just execute my sample agent install script, you can find it here.

 

Step 4. Generate and monitor application traffic in CloudWatch

Now that I have the CloudWatch agent installed and configured to receive StatsD and collect metrics, I’m going to generate traffic through the service:

echo "send 100 requests"
for i in {1..100}
do
   curl "localhost:8080/encode?text=TextToEncode_${i}[email protected]#%"
   echo ""
   sleep 1
done

 

Next, I log in to the CloudWatch console and check that the service is up and running. Here’s a graph of the StatsD metrics:

 

Here is a graph of the collectd metrics:

 

Conclusion

With StatsD and collectd support, you can now use the CloudWatch Agent to collect and monitor your custom applications in addition to the system metrics and application logs it already collects. Furthermore, you can create operational dashboards with these metrics, set alarms to take automated actions when free memory is low, and troubleshoot issues by diving into the application logs.  Note that StatsD supports both Windows and Linux operating systems while collectd is Linux only.  For Windows, you can also continue to use Windows Performance Counters to collect custom metrics instead.

The CloudWatch Agent with custom metrics support (version 1.203420.0 or later) is available in all public AWS Regions, AWS GovCloud (US), with AWS China (Beijing) and AWS China (Ningxia) coming soon.

The agent is free to use; you pay the usual CloudWatch prices for logs and custom metrics.

For more details, head over to the CloudWatch user guide for StatsD and collectd.

Using Cromwell with AWS Batch

Post Syndicated from Josh Rad original https://aws.amazon.com/blogs/compute/using-cromwell-with-aws-batch/

Contributed by W. Lee Pang and Emil Lerch, WWPS Professional Services

DNA is often referred to as the “source code of life.” All living cells contain long chains of deoxyribonucleic acid that encode instructions on how they are constructed and behave in their surroundings. Genomics is the study of the structure and function of DNA at the molecular level. It has recently shown immense potential to provide improved detection, diagnosis, and treatment of human diseases.

Continuous improvements in genome sequencing technologies have accelerated genomics research by providing unprecedented speed, accuracy, and quantity of DNA sequence data. In fact, the rate of sequencing efficiency has been shown to outpace Moore’s law. Processing this influx of genomic data is ideally aligned with the power and scalability of cloud computing.

Genomic data processing typically uses a wide assortment of specialized bioinformatics tools, like sequence alignment algorithms, variant callers, and statistical analysis methods. These tools are run in sequence as workflow pipelines that can range from a couple of steps to many long toolchains executing in parallel.

Traditionally, bioinformaticians and genomics scientists relied on Bash, Perl, or Python scripts to orchestrate their pipelines. As pipelines have gotten more complex, and maintainability and reproducibility have become standard requirements in science, the need for specialized orchestration tooling and portable workflow definitions has grown significantly.

What is Cromwell?

The Broad Institute’s Cromwell is purpose-built for this need. It is a workflow execution engine for orchestrating command line and containerized tools. Most importantly, it is the engine that drives the GATK Best Practices genome analysis pipeline.

Workflows for Cromwell are defined using the Workflow Definition Language (WDL – pronounced “widdle”), a flexible meta-scripting language that allows researchers to focus on the pieces of their workflow that matter. That’s the tools for each step and their respective inputs and outputs, and not the plumbing in between.

Genomics data is not small (on the order of TBs-PBs for one experiment!), so processing it usually requires significant computing scale, like HPC clusters and cloud computing. Cromwell has previously enabled this with support for many backends such as Spark, and HPC frameworks like Sun GridEngine and SLURM.

AWS and Cromwell

We are excited to announce that Cromwell now supports AWS! In this post, we go over how to configure Cromwell on AWS and get started running genomics pipelines in the cloud.

In a nutshell, the AWS backend for Cromwell is a layer that communicates with AWS Batch. Why AWS Batch? As stated before, genomics analysis pipelines are composed of many different tools. Each of these tools can have specific computing requirements. Operations like genome alignment can be memory-intensive, whereas joint genotyping may be compute-heavy.

AWS Batch dynamically provisions the optimal quantity and type of compute resources (for example, CPU or memory-optimized instances). Provisioning is based on the volume and specific resource requirements of the batch jobs submitted. This means that each step of a genomics workflow gets the most ideal instance to run on.

The AWS backend translates Cromwell task definitions into batch job definitions and submits them via API calls to a user-specified batch queue. Runtime parameters such as the container image to use, and resources like desired vCPUs and memory are also translated from the WDL task and transmitted to the batch job. A number of environment variables are automatically set on the job to support data localization and de-localization to the job instance. Ultimately, scientists and genomics researchers should be familiar with the backend method to submit jobs to AWS Batch because it uses their existing WDL files and research processes.

Getting started

To get started using Cromwell with AWS, create a custom AMI. This is necessary to ensure that the AMI is private to the account, encrypted, and has tooling specific to genomics workloads and Cromwell.

One feature of this tooling is the automatic creation and attachment of additional Amazon Elastic Block Store (Amazon EBS) capacity as additional data is copied onto the EC2 instance for processing. It also contains an ECS agent that has been customized to the needs of Cromwell, and a Cromwell Docker image responsible for interfacing the Cromwell task with Amazon S3.

After the custom AMI is created, install Cromwell on your workstation or EC2 instance. Configure an S3 bucket to hold Cromwell execution directories. For the purposes of this post, we refer to the bucket as s3-bucket-name. Lastly, go to the AWS Batch console, and create a job queue. Save the ARN of the queue, as this is needed later.

To get up these resources with a single click, this link provides a set of AWS CloudFormation templates that gets all the needed infrastructure running in minutes.

The next step is to configure Cromwell to work with AWS Batch and your newly created S3 bucket. Use the sample hello.wdl and hello.inputs files from the Cromwell AWS backend tutorial. You also need a custom configuration file so that Cromwell can interact with AWS Batch.

The following sample file can be used on an EC2 instance with the appropriate IAM role attached, or on a developer workstation with the AWS CLI configured. Keep in mind that you must replace <s3-bucket-name> in the configuration file with the appropriate bucket name. Also, replace “your ARN here” with the ARN of the job queue that you created earlier.

// aws.conf

include required(classpath("application"))

aws {

    application-name = "cromwell"
    
    auths = [
        {
         name = "default"
         scheme = "default"
        }
    ]
    
    region = "default"
    // uses region from ~/.aws/config set by aws configure command,
    // or us-east-1 by default
}

engine {
     filesystems {
         s3 {
            auth = "default"
         }
    }
}

backend {
     default = "AWSBATCH"
     providers {
         AWSBATCH {
             actor-factory = "cromwell.backend.impl.aws.AwsBatchBackendLifecycleActorFactory"
             config {
                 // Base bucket for workflow executions
                 root = "s3://<s3-bucket-name>/cromwell-execution"
                
                 // A reference to an auth defined in the `aws` stanza at the top. This auth is used to create
                 // Jobs and manipulate auth JSONs.
                 auth = "default"

                 numSubmitAttempts = 3
                 numCreateDefinitionAttempts = 3

                 concurrent-job-limit = 16
                
                 default-runtime-attributes {
                    queueArn: "<your ARN here>"
                 }
                
                 filesystems {
                     s3 {
                         // A reference to a potentially different auth for manipulating files via engine functions.
                         auth = "default"
                     }
                 }
             }
         }
     }
}

Now, you can run your workflow. The following command runs Hello World, and ensures that everything is connected properly:

$ java -Dconfig.file=aws.conf -jar cromwell-34.jar run hello.wdl -i hello.inputs

After the workflow has run, your workflow logs should report the workflow outputs.

[info] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.
{
 "outputs": {
    "wf_hello.hello.message": "Hello World! Welcome to Cromwell . . . on AWS!"
 },
 "id": "08213b40-bcf5-470d-b8b7-1d1a9dccb10e"
}

You also see your job in the “succeeded” section of the AWS Batch Jobs console.

After the environment is configured properly, other Cromwell WDL files can be used as usual.

Conclusion
With AWS Batch, a customized AMI instance, and Cromwell workflow definitions, AWS provides a simple solution to process genomics data easily. We invite you to incorporate this into your automated pipeline.

Security updates for Monday

Post Syndicated from ris original https://lwn.net/Articles/768406/rss

Security updates have been issued by Arch Linux (wireshark-cli), Debian (imagemagick, otrs2, tomcat7, and wireshark), Fedora (ca-certificates, dislocker, dolphin-emu, kernel-headers, kernel-tools, libgit2, mbedtls, mingw-openjpeg2, nekovm, openjpeg2, patch, strongswan, and thunderbird), Mageia (firefox, git, nextcloud, and texlive), Oracle (kernel and openssl), Scientific Linux (spamassassin), SUSE (libtirpc), and Ubuntu (requests).

How DNA Databases Violate Everyone’s Privacy

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/10/how_dna_databas.html

If you’re an American of European descent, there’s a 60% chance you can be uniquely identified by public information in DNA databases. This is not information that you have made public; this is information your relatives have made public.

Research paper:

“Identity inference of genomic data using long-range familial searches.”

Abstract: Consumer genomics databases have reached the scale of millions of individuals. Recently, law enforcement authorities have exploited some of these databases to identify suspects via distant familial relatives. Using genomic data of 1.28 million individuals tested with consumer genomics, we investigated the power of this technique. We project that about 60% of the searches for individuals of European-descent will result in a third cousin or closer match, which can allow their identification using demographic identifiers. Moreover, the technique could implicate nearly any US-individual of European-descent in the near future. We demonstrate that the technique can also identify research participants of a public sequencing project. Based on these results, we propose a potential mitigation strategy and policy implications to human subject research.

A good news article.

CHIPSEC – Platform Security Assessment Framework For Firmware Hacking

Post Syndicated from Darknet original https://www.darknet.org.uk/2018/10/chipsec-platform-security-assessment-framework-for-firmware-hacking/?utm_source=rss&utm_medium=social&utm_campaign=darknetfeed

CHIPSEC – Platform Security Assessment Framework For Firmware Hacking

CHIPSEC is a platform security assessment framework for PCs including hardware, system firmware (BIOS/UEFI), and platform components for firmware hacking.

It includes a security test suite, tools for accessing various low-level interfaces, and forensic capabilities. It can be run on Windows, Linux, Mac OS X and UEFI shell.

You can use CHIPSEC to find vulnerabilities in firmware, hypervisors and hardware configuration, explore low-level system assets and even detect firmware implants.

Read the rest of CHIPSEC – Platform Security Assessment Framework For Firmware Hacking now! Only available at Darknet.

How AWS SideTrail verifies key AWS cryptography code

Post Syndicated from Daniel Schwartz-Narbonne original https://aws.amazon.com/blogs/security/how-aws-sidetrail-verifies-key-aws-cryptography-code/

We know you want to spend your time learning valuable new skills, building innovative software, and scaling up applications — not worrying about managing infrastructure. That’s why we’re always looking for ways to help you automate the management of AWS services, particularly when it comes to cloud security. With that in mind, we recently developed SideTrail, an open source, program analysis tool that helps AWS developers verify key security properties of cryptographic implementations. In other words, the tool gives you assurances that secret information is kept secret. For example, developers at AWS have used this tool to verify that an AWS open source library, s2n, is protected against cryptographic side-channels. Let’s dive a little deeper into what this tool is and how it works.

SideTrail’s secret sauce is its automated reasoning capability, a mathematical proof-based technology, that verifies the correctness of critical code and helps detect errors that could lead to data leaks. Automated reasoning mathematically checks every possible input and reports any input that could cause an error. Full detail about SideTrail can be found in a scientific publication from VSTTE, a prominent software conference.

Automated reasoning is also useful because it can help prevent timing side-channels, a process by which data may be inadvertently revealed. Automated reasoning is what powers SideTrail to be able to detect the presence of these issues. If a code change causes a timing side-channel to be introduced, the mathematical proof will fail, and you’ll be notified about the flaw before the code is released.

Timing side-channels potentially allows attackers to learn about sensitive information, such as encryption keys, by observing the timing of messages on a network. For example, let’s say you have some code that checks a password as follows:


for (i = 0; i < length; ++i) {
	if (password[i] != input[i]) {
		send("bad password");
  	}
}

The amount of time until the reply message is received depends on which byte in the password is incorrect. As a result, an attacker can start guessing what the first character of the password is:

  • a*******
  • b*******
  • c*******

When the time to receive the error message changes, the first letter in the password is revealed. Repeating this process for each of the remaining characters turns a complex, exponential guessing challenge into a comparatively simple, linear one. SideTrail identifies these timing gaps, alerting developers to potential data leaks that could impact customers.

Here’s how to get started with SideTrail:

  1. Download SideTrail from GitHub.
  2. Follow the instructions provided with the download to setup your environment.
  3. Annotate your code with a threat model. The threat model is used to determine:
    • Which values should be considered secret. For example, a cryptographic key would be considered a secret, whereas the IP address of a public server wouldn’t.
    • The minimum timing difference an attacker can observe.
  4. Compile your code using SideTrail and analyze the report (for full guidance on how to analyze your report, read the VSTTE paper).

How SideTrail proves code correctness

The prior example had a data leak detected by SideTrail. However, SideTrail can do more than just detect leaks caused timing side-channels: it can prove that correct code won’t result in such leaks. One technique that developers can use to prevent data loss is to ensure that all paths in a program take the same amount of time to run no matter what a secret value may be. SideTrail helps to prove that such code is correct. Consider the following code that implements this technique:


int example(int pub, int secret) { 
	if (secret > pub) { 
  		pub -= secret; 
  	} else {
  		pub += secret; 
	}
	return pub;
}

Following the steps outlined in the prior section, a developer creates a threat model for this code by determining which values are public and which are secret. After annotating their code accordingly, they then compile it using SideTrail, which verifies that all paths in the code take the same amount of time. SideTrail verifies the code by leveraging the industry-standard Clang compiler to compile the code to LLVM bitcode. Working at the LLVM bitcode level allows SideTrail to work on the same optimized representation the compiler does. This is important because compiler optimizations can potentially affect the timing behavior of code, and hence the effectiveness of countermeasures used in that code. To maximize the effectiveness of SideTrail, developers can compile code for SideTrail using the same compiler configuration that’s used for released code.

The result of this process is a modified program that tracks the time required to execute each instruction. Here is how the result looks, inclusive of the annotations SideTrail includes (in red) to indicate the runtime of each operation. For more detail on the LLVM annotations, please read the VSTTE paper referenced earlier in this post.


int example(int pub, int secret, int *cost) { 
	*cost += GT_OP_COST;  
	if (secret > pub) { 
		*cost += SUBTRACTION_OP_COST;
		pub -= secret; 
  	} else {
		*cost += ADDITION_OP_COST;
  		pub += secret; 
  	} 
	return pub;
} 

If you’re wondering how this works in practice, you can inspect the SideTrail model by examining the annotated LLVM bitcode. Guidance on inspecting the SideTrail model can be found in the VSTTE paper.

To check whether the above code has a timing side-channel, SideTrail makes two copies of the code, symbolically executes both copies, and then checks whether the runtime differs. SideTrail does this by generating a wrapper and then using an automated reasoning engine to mathematically verify the correctness of this test for all possible inputs. If automated reasoning engine does this successfully, you can have confidence that the code does not contain a timing side-channel.

Conclusion

By using SideTrail, developers will be able to prove that there are no timing side channels present in their code, providing increased assurance against data loss. For example, Amazon s2n uses code-balancing countermeasures to mitigate this kind of attack. Through the use of SideTrail, we’ve proven the correctness of these countermeasures by running regression tests on every check-in to s2n, allowing AWS developers to build on top of this library with even more confidence, while delivering product and technology updates to customers with the highest security standards available.

More information about SideTrail and our other research can be found on our Provable Security hub. SideTrail is also available for public use as an open-source project.

If you have feedback about this blog post, submit comments in the Comments section below.

Want more AWS Security news? Follow us on Twitter.

Daniel Schwartz-Narbonne

Daniel is a Software Development Engineer in the AWS Automated Reasoning Group. Prior to joining Amazon, he earned a PhD at Princeton, where he developed a software framework to debug parallel programs. Then, at New York University, he designed a tool that automatically isolates and explains the cause of crashes in C programs. When he’s not working, you might find Daniel in the kitchen making dinner for his family, in a tent camping, or volunteering as an EMT with a local ambulance squad.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close