Tag Archives: about us

Backblaze’s Upgrade Guide for macOS High Sierra

2017-09-14 Roderick Bauer

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/macos-high-sierra-upgrade-guide/

Apple introduced macOS 10.13 “High Sierra” at its 2017 Worldwide Developers Conference in June. On Tuesday, we learned we don’t have long to wait — the new OS will be available on September 25. It’s a free upgrade, and millions of Mac users around the world will rush to install it.

We understand. A new OS from Apple is exciting, But please, before you upgrade, we want to remind you to back up your Mac. You want your data to be safe from unexpected problems that could happen in the upgrade. We do, too. To make that easier, Backblaze offers this macOS High Sierra upgrade guide.

Why Upgrade to macOS 10.13 High Sierra?

High Sierra, as the name suggests, is a follow-on to the previous macOS, Sierra. Its major focus is on improving the base OS with significant improvements that will support new capabilities in the future in the file system, video, graphics, and virtual/augmented reality.

But don’t despair; there also are outward improvements that will be readily apparent to everyone when they boot the OS for the first time. We’ll cover both the inner and outer improvements coming in this new OS.

Under the Hood of High Sierra

APFS (Apple File System)

Apple has been rolling out its first file system upgrade for a while now. It’s already in iOS: now High Sierra brings APFS to the Mac. Apple touts APFS as a new file system optimized for Flash/SSD storage and featuring strong encryption, better and faster file handling, safer copying and moving of files, and other improved file system fundamentals.

We went into detail about the enhancements and improvements that APFS has over the previous file system, HFS+, in an earlier post. Many of these improvements, including enhanced performance, security and reliability of data, will provide immediate benefits to users, while others provide a foundation for future storage innovations and will require work by Apple and third parties to support in their products and services.

Most of us won’t notice these improvements, but we’ll benefit from better, faster, and safer file handling, which I think all of us can appreciate.

Video

High Sierra includes High Efficiency Video Encoding (HEVC, aka H.265), which preserves better detail and color while also introducing improved compression over H.264 (MPEG-4 AVC). Even existing Macs will benefit from the HEVC software encoding in High Sierra, but newer Mac models include HEVC hardware acceleration for even better performance.

Metal 2

macOS High Sierra introduces Metal 2, the next-generation of Apple’s Metal graphics API that was launched three years ago. Apple claims that Metal 2 provides up to 10x better performance in key areas. It provides near-direct access to the graphics processor (GPU), enabling the GPU to take control over key aspects of the rendering pipeline. Metal 2 will enhance the Mac’s capability for machine learning, and is the technology driving the new virtual reality platform on Macs.

Virtual Reality

We’re about to see an explosion of virtual reality experiences on both the Mac and iOS thanks to High Sierra and iOS 11. Content creators will be able to use apps like Final Cut Pro X, Epic Unreal 4 Editor, and Unity Editor to create fully immersive worlds that will revolutionize entertainment and education and have many professional uses, as well.

Users will want the new iMac with Retina 5K display or the upcoming iMac Pro to enjoy them, or any supported Mac paired with the latest external GPU and VR headset.

Outward Improvements

Siri

Expect a more nature voice from Siri in High Sierra. She or he will be less robotic, with greater expression and use of intonation in speech. Siri will also learn more about your preferences in things like music, helping you choose music that fits your taste and putting together playlists expressly for you. Expect Siri to be able to answer your questions about music-related trivia, as well.

Siri: what does “scaramouche” refer to in the song Bohemian Rhapsody?

Photos

Photos has been redesigned with a new layout and new tools. A redesigned Edit view includes new tools for fine-tuning color and contrast and making adjustments within a defined color range. Some fun elements for creating special effects and memories also have been added. Photos now works with external apps such as Photoshop and Pixelmator. Compatibility with third-party extension adds printing and publishing services to help get your photos out into the world.

Safari

Apple claims that Safari in High Sierra is the world’s fastest desktop browser, outperforming Chrome and other browsers in a range of benchmark tests. They’ve also added autoplay blocking for those pesky videos that play without your permission and tracking blocking to help protect your privacy.

Can My Mac Run macOS High Sierra 10.13?

All Macs introduced in mid 2010 or later are compatible. MacBook and iMac computers introduced in late 2009 are also compatible. You’ll need OS X 10.7.5 “Lion” or later installed, along with at least 2 GB RAM and 8.8 GB of available storage to manage the upgrade.
Some features of High Sierra require an internet connection or an Apple ID. You can check to see if your Mac is compatible with High Sierra on Apple’s website.

Conquering High Sierra — What Do I Do Before I Upgrade?

Back Up That Mac!

It’s always smart to back up before you upgrade the operating system or make any other crucial changes to your computer. Upgrading your OS is a major change to your computer, and if anything goes wrong…well, you don’t want that to happen.

We recommend the 3-2-1 Backup Strategy to make sure your data is safe. What does that mean? Have three copies of your data. There’s the “live” version on your Mac, a local backup (Time Machine, another copy on a local drive or other computer), and an offsite backup like Backblaze. No matter what happens to your computer, you’ll have a way to restore the files if anything goes wrong. Need help understanding how to back up your Mac? We have you covered with a handy Mac backup guide.

Check for App and Driver Updates

This is when it helps to do your homework. Check with app developers or device manufacturers to find if their apps and devices have updates to work with High Sierra. Visit their websites or use the Check for Updates feature built into most apps (often found in the File or Help menus).

If you’ve downloaded apps through the Mac App Store, make sure to open them and click on the Updates button to download the latest updates.

Updating can be hit or miss when you’ve installed apps that didn’t come from the Mac App Store. To make it easier, visit the MacUpdate website. MacUpdate tracks changes to thousands of Mac apps.

Will Backblaze work with macOS High Sierra?

Yes. We’ve taken care to ensure that Backblaze works with High Sierra. We’ve already enhanced our Macintosh client to report the space available on an APFS container and we plan to add additional support for APFS capabilities that enhance Backblaze’s capabilities in the future.

Of course, we’ll watch Apple’s release carefully for any last minute surprises. We’ll officially offer support for High Sierra once we’ve had a chance to thoroughly test the release version.

Set Aside Time for the Upgrade

Depending on the speed of your Internet connection and your computer, upgrading to High Sierra will take some time. You’ll be able to use your Mac straightaway after answering a few questions at the end of the upgrade process.

If you’re going to install High Sierra on multiple Macs, a time-and-bandwidth-saving tip came from a Backblaze customer who suggested copying the installer from your Mac’s Applications folder to a USB Flash drive (or an external drive) before you run it. The installer routinely deletes itself once the upgrade process is completed, but if you grab it before that happens you can use it on other computers.

Where Do I get High Sierra?

Apple says that High Sierra will be available on September 25. Like other Mac operating system releases, Apple offers macOS 10.13 High Sierra for download from the Mac App Store, which is included on the Mac. As long as your Mac is supported and running OS X 10.7.5 “Lion” (released in 2012) or later, you can download and run the installer. It’s free. Thank you, Apple.

Better to be Safe than Sorry

Back up your Mac before doing anything to it, and make Backblaze part of your 3-2-1 backup strategy. That way your data is secure. Even if you have to roll back after an upgrade, or if you run into other problems, your data will be safe and sound in your backup.

Tell us How it Went

Are you getting ready to install High Sierra? Still have questions? Let us know in the comments. Tell us how your update went and what you like about the new release of macOS.

And While You’re Waiting for High Sierra…

While you’re waiting for Apple to release High Sierra on September 25, you might want to check out these other posts about using your Mac and Backblaze.

The post Backblaze’s Upgrade Guide for macOS High Sierra appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

How to Migrate All of Your Data from CrashPlan

2017-08-25 Andy Klein

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/how-to-migrate-your-data-from-crashplan/

With CrashPlan deciding to leave the consumer backup space, ex-customers are faced with having to migrate their data to a new cloud backup service. Uploading your data from your computer to a new service is onerous enough, but one thing that seems to be getting overlooked is the potential for the files that reside in CrashPlan Central, but not on your computer, to be lost during the migration to a new provider. Here’s an overview of the migration process to make sure you don’t lose data you wish to keep.

Why would you lose files?

By default CrashPlan for Home does not delete files from CrashPlan Central (their cloud storage servers) after they are uploaded from your computer. Unless you changed your CrashPlan “Frequency and versions” settings, all of the files you uploaded are still there. This includes all the files you deleted from your computer. For example, you may have a folder of old videos that you uploaded to CrashPlan and then deleted from your computer because of space concerns. This folder of old video files is still in your CrashPlan archive. It is very likely you have files stored in CrashPlan Central that are not on your computer. Such files are now in migration limbo, and we’ll get to those files in a minute, but first…

Get Started Now

CrashPlan was kind enough to make sure that everyone will have at least 60 days from August 22nd, 2017 to transfer their data. Most people will have more time, but everyone must be migrated by the end of October 2018.

Regardless, it’s better to get started now as it can take some time to upload your data to another backup provider. The first step in migrating your files is to choose a new cloud backup provider. Let’s assume you choose Backblaze Personal Backup.

Crashplan Migration Steps

The first step is to migrate all the data that is currently on your computer to Backblaze. Once you install Backblaze on your computer, it will automatically scan your system to locate the data to upload to Backblaze. The upload will continue automatically. You can speed up or slow down how quickly Backblaze will upload files by adjusting your performance settings for your Mac or for your Windows PC. In addition, any changes and new files are automatically uploaded as well. Backblaze keeps up to 30 days’ worth of file versions and always keeps the most recent version of every data file currently on your computer.

Question — Should you remove CrashPlan from your computer before migrating to Backblaze?
Answer — No.

If your computer fails during the upload to Backblaze, you’ll still have a full backup with CrashPlan. During the upload period you may want to decrease the resources (CPU and Network) used by CrashPlan and increase the resources available to Backblaze. You can “pause” CrashPlan for up to 24 hours, but that is a manual operation and may not be practical. In any case, you’ll also need to have CrashPlan around to recover those files in migration limbo.

Saving the Files in Migration Limbo

Let’s divide this process into two major parts: recovering the files and getting them stored somewhere else.

Recovering Files in Limbo

Choose a recovery device — Right now you don’t know how many files you will need to recover, but once you know that information, you’ll need a device to hold them. We recommend that you use an external USB hard drive as your recovery device. If you believe you will only have a small number of limbo files, then a thumb drive will work.
Locate the Limbo files — Open the CrashPlan App on your computer and select the “Restore” menu item on the left. As an example, you can navigate to a given folder and see the files in that folder as shown below:

Click on the “Show deleted files” box as shown below to display all the files, including those that are deleted. As an example, the same files listed above are shown below, and the list now includes the deleted file IMG_6533.JPG.

Deleted files can be visually identified via the different icon and the text shown grayed out. Navigate through your folder/directory structure and select the files you wish to recover. Yes, this can take a while. You only need to click on the deleted files as the other files are currently still on your computer and being backed up directly to Backblaze.
Make sure you change the restore location. By default this is set to “Desktop.” Click on the word “Desktop” to toggle through your options. Click on the option, and you’ll be able to change your backup destination to any mounted device connected to your system. As an example, we’ve chosen to restore the deleted files to the USB external drive named “Backblaze.”
Click “Restore” to restore the files you have selected.

Storing the Restored Limbo Files

Now that you have an external USB hard drive with the recovered Limbo files, let’s get them saved to the cloud. With Backblaze you have two options. The first option is to make the Limbo files part of your Backblaze backup. You can do this in two ways.

Copy the Limbo files to your computer and they will be automatically backed up to Backblaze with the rest of your files.

– or –

Connect the external USB Hard Drive to your computer and configure Backblaze to back up that device. This device should remain connected to the computer while the backup occurs, and then once every couple of weeks to make sure that nothing has changed on the hard drive.

If neither of the above solutions works for you, the other option is to use the Backblaze B2 Cloud Storage service.

What is Backblaze B2 Cloud Storage?

B2 Cloud Storage is a service for storing files in the cloud. Files are available for download at any time, either through the API or through a browser-compatible URL. Files stored in the B2 cloud are not deleted unless you explicitly delete them. In that way it is very similar to CrashPlan. Here’s some help, if you are unsure about the difference between Backblaze Personal Backup and Backblaze B2.

There are four ways to access B2: 1) a Web GUI, 2) a Command-line interface (CLI), 3) an API, and 4) via partner integrations, such a CloudBerry, Synology, Arq, QNAP, GoodSync and many more you can find on our B2 integrations page. Most CrashPlan users will find either the Web GUI or a partner integration to be the way to go. Note: There is an additional cost to use the B2 service, and we’ll get to that shortly.

Since you already have a Backblaze account, you just have to log in to your account. Click on “My Settings” on the left hand navigation and enable B2 Cloud Storage. If you haven’t already done so you will be asked to provide a Mobile number for contact and authentication purposes.
To use the B2 Web GUI, you create a B2 “bucket” and then drag-and-drop the files into the B2 bucket.
You can also choose to use a B2 partner integration to store your data into B2.

If you use B2 to store your Limbo files rescued from CrashPlan and you use Backblaze to back up your computer, you will be able to access and manage all of your data from your one Backblaze account.

What does all this cost?

If you are only going to use Backblaze Personal Backup to back up your computer, then you will pay $50/year per computer.

If you decide to combine the use of Backblaze Personal Backup and Backblaze B2, let’s assume you have 500 GB of data to back up from your computer to Backblaze. Let’s also assume you have to store 100 GB of data in Backblaze B2 that you rescued from CrashPlan limbo. Your annual cost would be:

To back up 500 GB:

— Backblaze Personal Backup — 1 year/1 computer — $50.00

To archive 100 GB:

— Backblaze B2 — 100 GB @ $0.005/GB/month for 12 months — $6.00

The Total Annual Cost to store your CrashPlan data in Backblaze, including your recovered deleted files, is $56.00.

Migrating from CrashPlan to Carbonite

If you are considering migrating your CrashPlan for Home account to Carbonite, you will still have to upload your data to Carbonite. There is no automatic process to copy the files from CrashPlan to Carbonite. You will also have to recover the Limbo files we’ve been speaking about using the process we’ve outlined above. In summary, when moving from CrashPlan for Home to any other vendor you will have to reupload your data to the new vendor.

One More Option

There is one more option you can use when you move your data from CrashPlan to another cloud service. You can download all of your data from CrashPlan, including the active and deleted files, to a local computer or device such an external USB Hard Drive. Then you can upload all that data to the new cloud backup provider. Of course this will mean all that data makes two trips through your local network — down and then back up. This will take time and could be very taxing on any bandwidth limits you may have in place from your network provider.

If you have the bandwidth and the time, this can be a good option, as all your files stored in CrashPlan Central are included in your backup. But, if you have a lot of data and/or a slow internet connection, this can take a really, really long time.

Join Our Webinar for More Information

You can sign up for our upcoming webinar, “Migrating from CrashPlan for Home to Backblaze” on September 7th at 10:00 am PDT if you’d like to learn more about the migration methods we covered today. Please note, you will need to register for this webinar by either signing up for a Backblaze BrightTALK channel account or using your existing BrightTALK account.

CrashPlan Replacement

Now that you are faced with replacing your CrashPlan for Home account, don’t wait until your contract is about to run out. Give yourself at least a couple of months to make sure all the data, including the Limbo data, is safely migrated somewhere else.

Also, regardless of which option you chose for migrating your data from CrashPlan to a new cloud backup service, once everything is moved and you’ve checked to make sure you got everything, then and only then should you turn off your CrashPlan account and uninstall CrashPlan.

An Invitation

If you are a CrashPlan for Home user going through the migration to a new cloud backup service, and have ideas to help other users through the migration process, let us know in the comments. We’ll update this post with any relevant ideas from the community.

The post How to Migrate All of Your Data from CrashPlan appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The Pronunciation Training Machine

2017-08-25 Alex Bate

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/pronunciation-training-machine/

Using a Raspberry Pi, an Arduino, an Adafruit NeoPixel Ring and a servomotor, Japanese makers HomeMadeGarbage produced this Pronunciation Training Machine to help their parents distinguish ‘L’s and ‘R’s when speaking English.

L R 発音矯正ギブスお母ちゃん編 Pronunciation training machine #right #light #raspberrypi #arduino #neopixel

23 Likes, 1 Comments – Home Made Garbage (@homemadegarbage) on Instagram: “L R 発音矯正ギブスお母ちゃん編 Pronunciation training machine #right #light #raspberrypi #arduino #neopixel”

How does an Pronunciation Training Machine work?

As you can see in the video above, the machine utilises the Google Cloud Speech API to recognise their parents’ pronunciation of the words ‘right’ and ‘light’. Correctly pronounce the former, and the servo-mounted arrow points to the right. Pronounce the later and the NeoPixel Ring illuminates because, well, you just said “light”.

You can find the full code for the project on its hackster page here.

Variations on the idea

It’s a super-cute project with great potential, and the concept could easily be amended for other training purposes. How about using motion sensors to help someone learn their left from their right?

A photo of hands with left and right written on them - English Pronunciation Training

Wait…your left or my left?
image c/o tattly

Or use random.choice to switch on LEDs over certain images, and speech recognition to reward a correct answer? Light up a picture of a cat, for example, and when the player says “cat”, they receive a ‘purr’ or a treat?

A photo of a kitten - English Pronunciation Training

Obligatory kitten picture
image c/o somewhere on the internet!

Raspberry Pi-based educational aids do not have to be elaborate builds. They can use components as simple as a servo and an LED, and still have the potential to make great improvements in people’s day-to-day lives.

Your own projects

If you’ve created an educational tool using a Raspberry Pi, we’d love to see it. The Raspberry Pi itself is an educational tool, so you’re helping it to fulfil its destiny! Make sure you share your projects with us on social media, or pop a link in the comments below. We’d also love to see people using the Pronunciation Training Machine (or similar projects), so make sure you share those too!

A massive shout out to Artie at hackster.io for this heads-up, and for all the other Raspberry Pi projects he sends my way. What a star!

The post The Pronunciation Training Machine appeared first on Raspberry Pi.

Announcing Dedicated IP Pools

2017-08-17 Brent Meyer

Post Syndicated from Brent Meyer original https://aws.amazon.com/blogs/ses/announcing-dedicated-ip-pools/

The Amazon SES team is pleased to announce that you can now create groups of dedicated IP addresses, called dedicated IP pools, for your email sending activities.

Prior to the availability of this feature, if you leased several dedicated IP addresses to use with Amazon SES, there was no way to specify which dedicated IP address to use for a specific email. Dedicated IP pools solve this problem by allowing you to send emails from specific IP addresses.

This post includes information and procedures related to dedicated IP pools.

What are dedicated IP pools?

In order to understand dedicated IP pools, you should first be familiar with the concept of dedicated IP addresses. Customers who send large volumes of email will typically lease one or more dedicated IP addresses to use when sending mail from Amazon SES. To learn more, see our blog post about dedicated IP addresses.

If you lease several dedicated IP addresses for use with Amazon SES, you can organize these addresses into groups, called pools. You can then associate each pool with a configuration set. When you send an email that specifies a configuration set, that email will be sent from the IP addresses in the associated pool.

When should I use dedicated IP pools?

Dedicated IP pools are especially useful for customers who send several different types of email using Amazon SES. For example, if you use Amazon SES to send both marketing emails and transactional emails, you can create a pool for marketing emails and another for transactional emails.

By using dedicated IP pools, you can isolate the sender reputations for each of these types of communications. Using dedicated IP pools gives you complete control over the sender reputations of the dedicated IP addresses you lease from Amazon SES.

How do I create and use dedicated IP pools?

There are two basic steps for creating and using dedicated IP pools. First, create a dedicated IP pool in the Amazon SES console and associate it with a configuration set. Next, when you send email, be sure to specify the configuration set associated with the IP pool you want to use.

For step-by-step procedures, see Creating Dedicated IP Pools in the Amazon SES Developer Guide.

Will my email sending process change?

If you do not use dedicated IP addresses with Amazon SES, then your email sending process will not change.

If you use dedicated IP pools, your email sending process may change slightly. In most cases, you will need to specify a configuration set in the emails you send. To learn more about using configuration sets, see Specifying a Configuration Set When You Send Email in the Amazon SES Developer Guide.

Any dedicated IP addresses that you lease that are not part of a dedicated IP pool will automatically be added to a default pool. If you send email without specifying a configuration set that is associated with a pool, then that email will be sent from one of the addresses in the default pool.

Dedicated IP pools are now available in the following AWS Regions: us-west-2 (Oregon), us-east-1 (Virginia), and eu-west-1 (Ireland).

We hope you enjoy this feature. If you have any questions or comments, please leave a comment on this post, or let us know in the Amazon SES Forum.

Turbocharge your Apache Hive queries on Amazon EMR using LLAP

2017-08-04 Jigar Mistry

Post Syndicated from Jigar Mistry original https://aws.amazon.com/blogs/big-data/turbocharge-your-apache-hive-queries-on-amazon-emr-using-llap/

Apache Hive is one of the most popular tools for analyzing large datasets stored in a Hadoop cluster using SQL. Data analysts and scientists use Hive to query, summarize, explore, and analyze big data.

With the introduction of Hive LLAP (Low Latency Analytical Processing), the notion of Hive being just a batch processing tool has changed. LLAP uses long-lived daemons with intelligent in-memory caching to circumvent batch-oriented latency and provide sub-second query response times.

This post provides an overview of Hive LLAP, including its architecture and common use cases for boosting query performance. You will learn how to install and configure Hive LLAP on an Amazon EMR cluster and run queries on LLAP daemons.

What is Hive LLAP?

Hive LLAP was introduced in Apache Hive 2.0, which provides very fast processing of queries. It uses persistent daemons that are deployed on a Hadoop YARN cluster using Apache Slider. These daemons are long-running and provide functionality such as I/O with DataNode, in-memory caching, query processing, and fine-grained access control. And since the daemons are always running in the cluster, it saves substantial overhead of launching new YARN containers for every new Hive session, thereby avoiding long startup times.

When Hive is configured in hybrid execution mode, small and short queries execute directly on LLAP daemons. Heavy lifting (like large shuffles in the reduce stage) is performed in YARN containers that belong to the application. Resources (CPU, memory, etc.) are obtained in a traditional fashion using YARN. After the resources are obtained, the execution engine can decide which resources are to be allocated to LLAP, or it can launch Apache Tez processors in separate YARN containers. You can also configure Hive to run all the processing workloads on LLAP daemons for querying small datasets at lightning fast speeds.

LLAP daemons are launched under YARN management to ensure that the nodes don’t get overloaded with the compute resources of these daemons. You can use scheduling queues to make sure that there is enough compute capacity for other YARN applications to run.

Why use Hive LLAP?

With many options available in the market (Presto, Spark SQL, etc.) for doing interactive SQL over data that is stored in Amazon S3 and HDFS, there are several reasons why using Hive and LLAP might be a good choice:

For those who are heavily invested in the Hive ecosystem and have external BI tools that connect to Hive over JDBC/ODBC connections, LLAP plugs in to their existing architecture without a steep learning curve.
It’s compatible with existing Hive SQL and other Hive tools, like HiveServer2, and JDBC drivers for Hive.
It has native support for security features with authentication and authorization (SQL standards-based authorization) using HiveServer2.
LLAP daemons are aware about of the columns and records that are being processed which enables you to enforce fine-grained access control.
It can use Hive’s vectorization capabilities to speed up queries, and Hive has better support for Parquet file format when vectorization is enabled.
It can take advantage of a number of Hive optimizations like merging multiple small files for query results, automatically determining the number of reducers for joins and groupbys, etc.
It’s optional and modular so it can be turned on or off depending on the compute and resource requirements of the cluster. This lets you to run other YARN applications concurrently without reserving a cluster specifically for LLAP.

How do you install Hive LLAP in Amazon EMR?

To install and configure LLAP on an EMR cluster, use the following bootstrap action (BA):

s3://aws-bigdata-blog/artifacts/Turbocharge_Apache_Hive_on_EMR/configure-Hive-LLAP.sh

This BA downloads and installs Apache Slider on the cluster and configures LLAP so that it works with EMR Hive. For LLAP to work, the EMR cluster must have Hive, Tez, and Apache Zookeeper installed.

You can pass the following arguments to the BA.

Argument	Definition	Default value
`--instances`	Number of instances of LLAP daemon	Number of core/task nodes of the cluster
`--cache`	Cache size per instance	20% of physical memory of the node
`--executors`	Number of executors per instance	Number of CPU cores of the node
`--iothreads`	Number of IO threads per instance	Number of CPU cores of the node
`--size`	Container size per instance	50% of physical memory of the node
`--xmx`	Working memory size	50% of container size
`--log-level`	Log levels for the LLAP instance	INFO

LLAP example

This section describes how you can try the faster Hive queries with LLAP using the TPC-DS testbench for Hive on Amazon EMR.

Use the following AWS command line interface (AWS CLI) command to launch a 1+3 nodes m4.xlarge EMR 5.6.0 cluster with the bootstrap action to install LLAP:

aws emr create-cluster --release-label emr-5.6.0 \
--applications Name=Hadoop Name=Hive Name=Hue Name=ZooKeeper Name=Tez \
--bootstrap-actions '[{"Path":"s3://aws-bigdata-blog/artifacts/Turbocharge_Apache_Hive_on_EMR/configure-Hive-LLAP.sh","Name":"Custom action"}]' \ 
--ec2-attributes '{"KeyName":"<YOUR-KEY-PAIR>","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-xxxxxxxx","EmrManagedSlaveSecurityGroup":"sg-xxxxxxxx","EmrManagedMasterSecurityGroup":"sg-xxxxxxxx"}' 
--service-role EMR_DefaultRole \
--enable-debugging \
--log-uri 's3n://<YOUR-BUCKET/' --name 'test-hive-llap' \
--instance-groups '[{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":1}],"EbsOptimized":true},"InstanceGroupType":"MASTER","InstanceType":"m4.xlarge","Name":"Master - 1"},{"InstanceCount":3,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":32,"VolumeType":"gp2"},"VolumesPerInstance":1}],"EbsOptimized":true},"InstanceGroupType":"CORE","InstanceType":"m4.xlarge","Name":"Core - 2"}]' 
--region us-east-1

After the cluster is launched, log in to the master node using SSH, and do the following:

Open the hive-tpcds folder:
cd /home/hadoop/hive-tpcds/
Start Hive CLI using the testbench configuration, create the required tables, and run the sample query:
hive –i testbench.settings hive> source create_tables.sql; hive> source query55.sql;
This sample query runs on a 40 GB dataset that is stored on Amazon S3. The dataset is generated using the data generation tool in the TPC-DS testbench for Hive.It results in output like the following:
This screenshot shows that the query finished in about 47 seconds for LLAP mode. Now, to compare this to the execution time without LLAP, you can run the same workload using only Tez containers:
hive> set hive.llap.execution.mode=none; hive> source query55.sql;

This query finished in about 80 seconds.

The difference in query execution time is almost 1.7 times when using just YARN containers in contrast to running the query on LLAP daemons. And with every rerun of the query, you notice that the execution time substantially decreases by the virtue of in-memory caching by LLAP daemons.

Conclusion

In this post, I introduced Hive LLAP as a way to boost Hive query performance. I discussed its architecture and described several use cases for the component. I showed how you can install and configure Hive LLAP on an Amazon EMR cluster and how you can run queries on LLAP daemons.

If you have questions about using Hive LLAP on Amazon EMR or would like to share your use cases, please leave a comment below.

Additional Reading

Learn how to to automatically partition Hive external tables with AWS.

About the Author

Jigar Mistry is a Hadoop Systems Engineer with Amazon Web Services. He works with customers to provide them architectural guidance and technical support for processing large datasets in the cloud using open-source applications. In his spare time, he enjoys going for camping and exploring different restaurants in the Seattle area.

timeShift(GrafanaBuzz, 1w) Issue 7

2017-08-04 Blogs on Grafana Labs Blog

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2017/08/04/timeshiftgrafanabuzz-1w-issue-7/

Hard to believe it’s already August! This week there were a ton of articles to highlight. It’s really exciting to see how many data aficionados there are out there coming up with new ways to connect Grafana to their data, wherever it may live. In this issue we cover crypto currency visualization, home automation setups and breakdown the installation in a number of environments. Enjoy!

Grafana 4.4.2 Released

Grafana v4.4.2 is now Available for download

This week’s release includes a few updates and some bug fixes.

To see the full changelog, head over to our community site.

From the Blogosphere

Monitoring CouchDB with Prometheus, Grafana and Docker: Geoff walks us through all of the steps to get Prometheus, Alertmanager and Grafana installed in Docker to monitor and alert on a CouchDB cluster. These six steps will have you up and running in no time.
Try InfluxDB and Grafana by Docker: Continuing with our Docker theme, Xiao breaks down all of the pieces, explores the configuration options, and explains the Docker commands to setup a simple monitoring stack by using collectd, InfluxDB and Grafana.
Installation of Collectd, Graphite and Grafana – Part 2: Last week we covered the first article in a series focused on setting up a complete Graphite stack. This week we tackle installing Graphite, its components, and Grafana on the server.
Grafana and Home Automation: More and more pieces of our homes are becoming “smart”, so why not monitor them? This article walks you through collecting data from home automation software Jeedom, sending metrics to InfluxDB, and visualizing and alerting in Grafana – so you can know how your smart-toaster is performing.
Making an Awesome Dashboard for your Crypto Currencies in 3 Steps: Christian lays out three steps that will help you keep an eye on your Bitcoin and Ethereum investments. His PHP script fetches things like current price, current balances, earnings, and sends the data to InfluxDB via UDP. He’s also created a dashboard that’s ready to import so you can get back to mining.
FHEM #6 – Grafana and InfluxDB: We’re seeing more and more articles about using Grafana to monitor home automation. This is the sixth article in a series getting data from FHEM into Grafana using InfdluxDB. It also touches on connecting Grafana to MariaDB, taking advantage of Grafana’s alpha native MySQL support.
Installation Overview of Node Exporter, Prometheus and Grafana: Looking to get started with Prometheus? Frits walks us through installing Node Exporter, Prometheus, and Grafana and importing our first dashboard.
Collect Metrics from Liberty Apps and Display in Grafana: This in-depth article covers adding custom metrics to your Liberty application and how to monitor these metrics using collectd, Graphite and Grafana.
Gatling, Graphite, Grafana: Your Application Under High Surveillance!: David explores Gatling, for load testing which can write the data to Graphite and over to Grafana for visualization and alerting.

Plugins and Dashboards

Last week’s timeShift was packed full of plugin updates, as well as a couple of new ones. This week was a little quieter on the plugin front, but we still have a new data source plugin to announce. It’s easy to install this new plugin via the grafana-cli for an on-prem Grafana instance, or a 1-click install on Hosted Grafana.

NEW PLUGIN

PRTG Data Source – This data source visualizes data from the Paessler PRTG monitoring system. The easy to use query editor included with this plugin gives access to an array of PRTG metadata properties including Status, Message, Active, Tags, Priority, and more. Annotation support to show sensor status messages on graphs.

Install Now

This week’s MVC (Most Valuable Contributor)

This week we highlight a contributor who is going to make everyone waiting for Elasticsearch alerting in Grafana jump for joy!

lvheyang (Cui Wenzheng)
lvheyang submitted a PR for Elasticsearch Alerting that must have been a huge amount of work and is of impressive quality. Thank you very much for your hard work and we look forward to releasing Alerting for Elasticsearch in Grafana 5.0.0.

Tweet of the Week

We scour Twitter each week to find an interesting/beautiful dashboard and show it off! #monitoringLove

Nice @jorgedlcruz! – Making data more beautiful, one boring number at a time 🙂 – Also, I spy one of my favorite panels in there, the Worldmap Panel!

Having fun with pflogsumm and mailq, I'm addicted! I turn boring numbers into beautiful dashboards. How to monitor Zimbra with Grafana, soon pic.twitter.com/WFNtg7vHNk

— Jorge de la Cruz (@jorgedlcruz) August 2, 2017

Upcoming Events

We love when people talk about Grafana at meetups and conferences.

Wednesday, August 16, 2017 – 7:30pm | Apprenda HQ

433 River Street, 4th Floor, Troy, NY

Kubernetes focused event! demo from Apprenda and how Kubernetes is used @ GitHub:
Steve Wade, is a Principal Kubernetes Consultant from London and will be providing some fundamental information about the Kubernetes ecosystem as well as overview of its core components. He’ll also talk about some monitoring and alerting best practices learned from working with Kubernetes customers and demo how Prometheus, Grafana and Slack can be used to monitor, visualize and alert on both the Kubernetes platform as well as application workloads.

Aaron Brown, a Site Reliability Engineer at Github, will dive into the ways in which Kubernetes is used within Github to make software development and deployment more efficient.

RSVP

What do you think?

Please tell us how we’re doing. We want to make sure this continues to be a valuable resource for the Grafana community. Submit a comment on this article below, or post something at our community forum. Help us make this better!

New – Amazon Connect and Amazon Lex Integration

2017-08-03 Randall Hunt

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/aws/new-amazon-connect-and-amazon-lex-integration/

I’m really excited to share some recent enhancements to two of my favorite services: Amazon Connect and Amazon Lex. Amazon Connect is a self-service, cloud-based contact center service that makes it easy for any business to deliver better customer service at lower cost. Amazon Lex is a service for building conversational interfaces using voice and text. By integrating these two services you can take advantage of Lex‘s automatic speech recognition (ASR) and natural language processing/understading (NLU) capabilities to create great self-service experiences for your customers. To enable this integration the Lex team added support for 8kHz speech input – more on that later. Why should you care about this? Well, if the a bot can solve the majority of your customer’s requests your customers spend less time waiting on hold and more time using your products.

If you need some more background on Amazon Connect or Lex I strongly recommend Jeff’s previous posts[1][2] on these services – especially if you like LEGOs.

Let’s dive in and learn to use this new integration. We’ll take an application that we built on our Twitch channel and modify it for this blog. At the application’s core a user calls an Amazon Connect number which connects them to an Lex bot which invokes an AWS Lambda function based on an intent from Lex. So what does our little application do?

I want to finally settle the question of what the best code editor is: I like vim, it’s a spectacular editor that does one job exceptionally well – editing code (it’s the best). My colleague Jeff likes emacs, a great ~~operating system~~ editor… if you were born with extra joints in your fingers. My colleague Tara loves Visual Studio and sublime. Rather than fighting over what the best editor is I thought we might let you, dear reader, vote. Don’t worry you can even vote for butterflies.

Interested in voting? Call +1 614-569-4019 and tell us which editor you’re voting for! We don’t store your number or record your voice so feel free to vote more than once for vim. Want to see the votes live? http://best-editor-ever.s3-website-us-east-1.amazonaws.com/.

Now, how do we build this little contraption? We’ll cover each component but since we’ve talked about Lex and Lambda before we’ll focus mostly on the Amazon Connect component. I’m going to assume you already have a connect instance running.

Amazon Lex

Let’s start with the Lex side of things. We’ll create a bot named VoteEditor with two intents: VoteEditor with a single slot called editor and ConnectToAgent with no slots. We’ll populate our editor slot full of different code editor names (maybe we’ll leave out emacs).

AWS Lambda

Our Lambda function will also be fairly simple. First we’ll create a Amazon DynamoDB table to store our votes. Then we’ll make a helper method to respond to Lex (build_response) – it will just wrap our message in a Lex friendly response format. Now we just have to figure out our flow logic.


def lambda_handler(event, context):
    if 'ConnectToAgent' == event['currentIntent']['name']:
        return build_response("Ok, connecting you to an agent.")
    elif 'VoteEditor' == event['currentIntent']['name']:
        editor = event['currentIntent']['slots']['editor']
        resp = ddb.update_item(
            Key={"name": editor.lower()},
            UpdateExpression="SET votes = :incr + if_not_exists(votes, :default)",
            ExpressionAttributeValues={":incr": 1, ":default": 0},
            ReturnValues="ALL_NEW"
        )
        msg = "Awesome, now {} has {} votes!".format(
            resp['Attributes']['name'],
            resp['Attributes']['votes'])
        return build_response(msg)

Let’s make sure we understand the code. So, if we got a vote for an editor and it doesn’t exist yet then we add that editor with 1 vote. Otherwise we increase the number of votes on that editor by 1. If we get a request for an agent, we terminate the flow with a nice message. Easy. Now we just tell our Lex bot to use our Lambda function to fulfill our intents. We can test that everything is working over text in the Lex console before moving on.

Amazon Connect

Before we can use our Lex bot in a Contact Flow we have to make sure our Amazon Connect instance has access to it. We can do this by hopping over to the Amazon Connect service console, selecting our instance, and navigating to “Contact Flows”. There should be a section called Lex where you can add your bots!

Now that our Amazon Connect instance can invoke our Lex bot we can create a new Contact Flow that contains our Lex bot. We add the bot to our flow through the “Get customer input” widget from the “Interact” category.

Once we’re on the widget we have a “DTMF” tab for taking input from number keys on a phone or the “Amazon Lex” tab for taking voiceinput and passing it to the Lex service. We’ll use the Lex tab and put in some configuration.

Lots of options, but in short we add the bot we want to use (including the version of the bot), the intents we want to use from our bot, and a short prompt to introduce the bot (and mayb prompt the customer for input).

Our final contact flow looks like this:

A real world example might allow a customer to perform many transactions through a Lex bot. Then on an error or ConnectToAgent intent put the customer into a queue where they could talk to a real person. It could collect and store information about users and populate a rich interface for an agent to use so they could jump right into the conversation with all the context they need.

I want to especially highlight the advantage of 8kHz audio support in Lex. Lex originally only supported speech input that was sampled at a higher rate than the 8 kHz input from the phone. Modern digital communication appliations typically use audio signals sampled at a minimum of 16 kHz. This higher fidelity recroding makes it easier differentiate between sounds like “ess” (/s/) and “eff” (/f/) – or so the audio experts tell me. Phones, however, use a much lower quality recording. Humans, and their ears, are pretty good at using surrounding words to figure out what a voice is saying from a lower quality recording (just check the NASA apollo recordings for proof of this). Most digital phone systems are setup to use 8 kHz sampling by default – it’s a nice tradeoff in bandwidth and fidelity. That’s why your voice sometimes sounds different on the phone. On top of this fundmental sampling rate issue you also have to deal with the fact that a lot of phone call data is already lossy (can you hear me now?). There are thousands of different devices from hundreds of different manufacturers, and tons of different software implentations. So… how do you solve this recognition issue?

The Lex team decided that the best way to address this was to expand the set of models they were using for speech recognition to include an 8kHz model. Support for an 8 kHz telephony audio sampling rate provides increased speech recognition accuracy and fidelity for your contact center interactions. This was a great effort by the team that enables a lot of customers to do more with Amazon Connect.

One final note is that Amazon Connect uses the exact same PostContent endpoint that you can use as an external developer so you don’t have to be a Amazon Connect user to take advantage of this 8kHz feature in Lex.

I hope you guys enjoyed this post and as always the real details are in the docs and API Reference.

– Randall

Wanted: Automation Systems Administrator

2017-07-27 Yev

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-automation-systems-administrator/

Are you an Automation Systems Administrator who is looking for a challenging and fast-paced working environment? Want to a join our dynamic team and help Backblaze grow to new heights? Our Operations team is a distributed and collaborative group of individual contributors. We work closely together to build and maintain our home grown cloud storage farm, carefully controlling costs by utilizing open source and various brands of technology, as well as designing our own cloud storage servers. Members of Operations participate in the prioritization and decision making process, and make a difference everyday. The environment is challenging, but we balance the challenges with rewards, and we are looking for clever and innovative people to join us.

Responsibilities:

Develop and deploy automated provisioning & updating of systems
Lead projects across a range of IT disciplines
Understand environment thoroughly enough to administer/debug any system
Participate in the 24×7 on-call rotation and respond to alerts as needed

Requirements:

Expert knowledge of automated provisioning
Expert knowledge of Linux administration (Debian preferred)
Scripting skills
Experience in automation/configuration management
Position based in the San Mateo, California Corporate Office

Required for all Backblaze Employees

Good attitude and willingness to do whatever it takes to get the job done.
Desire to learn and adapt to rapidly changing technologies and work environment.
Relentless attention to detail.
Excellent communication and problem solving skills.
Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

Company Description:
Founded in 2007, Backblaze started with a mission to make backup software elegant and provide complete peace of mind. Over the course of almost a decade, we have become a pioneer in robust, scalable low cost cloud backup. Recently, we launched B2 – robust and reliable object storage at just $0.005/gb/mo. Part of our differentiation is being able to offer the lowest price of any of the big players while still being profitable.

We’ve managed to nurture a team oriented culture with amazingly low turnover. We value our people and their families. Don’t forget to check out our “About Us” page to learn more about the people and some of our perks.

We have built a profitable, high growth business. While we love our investors, we have maintained control over the business. That means our corporate goals are simple – grow sustainably and profitably.

Some Backblaze Perks:

Competitive healthcare plans
Competitive compensation and 401k
All employees receive Option grants
Unlimited vacation days
Strong coffee
Fully stocked Micro kitchen
Catered breakfast and lunches
Awesome people who work on awesome projects
Childcare bonus
Normal work hours
Get to bring your pets into the office
San Mateo Office – located near Caltrain and Highways 101 & 280.

If this sounds like you — follow these steps:

Send an email to [email protected] with the position in the subject line.
Include your resume.
Tell us a bit about your experience and why you’re excited to work with Backblaze.

The post Wanted: Automation Systems Administrator appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Wanted: Site Reliability Engineer

2017-07-27 Yev

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-site-reliability-engineer/

Are you a Site Reliability Engineer who is looking for a challenging and fast-paced working environment? Want to a join our dynamic team and help Backblaze grow to new heights? Our Operations team is a distributed and collaborative group of individual contributors. We work closely together to build and maintain our home grown cloud storage farm, carefully controlling costs by utilizing open source and various brands of technology, as well as designing our own cloud storage servers. Members of Operations participate in the prioritization and decision making process, and make a difference everyday. The environment is challenging, but we balance the challenges with rewards, and we are looking for clever and innovative people to join us.

Responsibilities:

Lead projects across a range of IT disciplines
Understand environment thoroughly enough to administer/debug any system
Collaborate on automated provisioning & updating of systems
Collaborate on network administration and security
Collaborate on database administration
Participate in the 24×7 on-call rotation and respond to alerts
as needed

Requirements:

Expert knowledge of Linux administration (Debian preferred)
Scripting skills
Experience in automation/configuration management (Ansible preferred)
Position based in the San Mateo, California Corporate Office

Required for all Backblaze Employees

Good attitude and willingness to do whatever it takes to get the job done.
Desire to learn and adapt to rapidly changing technologies and work environment.
Relentless attention to detail.
Excellent communication and problem solving skills.
Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

Some Backblaze Perks:

Competitive healthcare plans
Competitive compensation and 401k
All employees receive Option grants
Unlimited vacation days
Strong coffee
Fully stocked Micro kitchen
Catered breakfast and lunches
Awesome people who work on awesome projects
Childcare bonus
Normal work hours
Get to bring your pets into the office
San Mateo Office – located near Caltrain and Highways 101 & 280.

If this sounds like you — follow these steps:

Send an email to [email protected] with the position in the subject line.
Include your resume.
Tell us a bit about your experience and why you’re excited to work with Backblaze.

The post Wanted: Site Reliability Engineer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Wanted: Network Systems Administrator

2017-07-27 Yev

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-network-systems-administrator/

Are you a Network Systems Administrator who is looking for a challenging and fast-paced working environment? Want to a join our dynamic team and help Backblaze grow to new heights? Our Operations team is a distributed and collaborative group of individual contributors. We work closely together to build and maintain our home grown cloud storage farm, carefully controlling costs by utilizing open source and various brands of technology, as well as designing our own cloud storage servers. Members of Operations participate in the prioritization and decision making process, and make a difference everyday. The environment is challenging, but we balance the challenges with rewards, and we are looking for clever and innovative people to join us.

Responsibilities:

Own the network administration and security
Lead projects across a range of IT disciplines
Understand environment thoroughly enough to administer/debug any system
Participate in the 24×7 on-call rotation and respond to alerts as needed

Requirements:

Expert knowledge of network administration and security
Expert knowledge of Linux administration (Debian preferred)
Scripting skills
Position based in the San Mateo, California Corporate Office

Required for all Backblaze Employees

Good attitude and willingness to do whatever it takes to get the job done.
Desire to learn and adapt to rapidly changing technologies and work environment.
Relentless attention to detail.
Excellent communication and problem solving skills.
Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

Some Backblaze Perks:

Competitive healthcare plans
Competitive compensation and 401k
All employees receive Option grants
Unlimited vacation days
Strong coffee
Fully stocked Micro kitchen
Catered breakfast and lunches
Awesome people who work on awesome projects
Childcare bonus
Normal work hours
Get to bring your pets into the office
San Mateo Office – located near Caltrain and Highways 101 & 280.

If this sounds like you — follow these steps:

Send an email to [email protected] with the position in the subject line.
Include your resume.
Tell us a bit about your experience and why you’re excited to work with Backblaze.

The post Wanted: Network Systems Administrator appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Wanted: Database Systems Administrator

2017-07-27 Yev

Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-database-systems-administrator/

Are you a Database Systems Administrator who is looking for a challenging and fast-paced working environment? Want to a join our dynamic team and help Backblaze grow to new heights? Our Operations team is a distributed and collaborative group of individual contributors. We work closely together to build and maintain our home grown cloud storage farm, carefully controlling costs by utilizing open source and various brands of technology, as well as designing our own cloud storage servers. Members of Operations participate in the prioritization and decision making process, and make a difference everyday. The environment is challenging, but we balance the challenges with rewards, and we are looking for clever and innovative people to join us.

Responsibilities:

Own the administration of Cassandra and MySQL
Lead projects across a range of IT disciplines
Understand environment thoroughly enough to administer/debug the system
Participate in the 24×7 on-call rotation and respond to alerts as needed

Requirements:

Expert knowledge of Cassandra & MySQL
Expert knowledge of Linux administration (Debian preferred)
Scripting skills
Experience in automation/configuration management
Position is based in the San Mateo, California corporate office

Required for all Backblaze Employees

Good attitude and willingness to do whatever it takes to get the job done.
Desire to learn and adapt to rapidly changing technologies and work environment.
Relentless attention to detail.
Excellent communication and problem solving skills.
Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy.

Some Backblaze Perks:

Competitive healthcare plans
Competitive compensation and 401k
All employees receive Option grants
Unlimited vacation days
Strong coffee
Fully stocked Micro kitchen
Catered breakfast and lunches
Awesome people who work on awesome projects
Childcare bonus
Normal work hours
Get to bring your pets into the office
San Mateo Office – located near Caltrain and Highways 101 & 280.

If this sounds like you — follow these steps:

Send an email to [email protected] with the position in the subject line.
Include your resume.
Tell us a bit about your experience and why you’re excited to work with Backblaze.

The post Wanted: Database Systems Administrator appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Setting up Read Replica Clusters with HBase on Amazon S3

2017-07-20 Tony Nguyen

Post Syndicated from Tony Nguyen original https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/

Many customers have taken advantage of the numerous benefits of running Apache HBase on Amazon S3 for data storage, including lower costs, data durability, and easier scalability. Customers such as FINRA have lowered their costs by 60% by moving to an HBase on S3 architecture along with the numerous operational benefits that come with decoupling storage from compute and using S3 as the storage layer. HBase on S3 allows you to turn on a cluster and immediately start querying against data within S3 rather than having to go through a lengthy snapshot restore process.

With the launch of Amazon EMR 5.7.0, you can now take the high availability and durability of HBase on S3 one step further to the cluster level, where you can now start multiple HBase read-only clusters that can connect to the same HBase root directory in S3. This allows you to ensure that your data is always reachable through read replica clusters and run their clusters across multiple Availability Zones.

In this post, I guide you through setting up read replica clusters with HBase on S3.

HBase Overview

Apache HBase is a massively scalable, distributed big data store in the Apache Hadoop ecosystem. It is an open-source, non-relational, versioned database that runs on top of the Hadoop Distributed Filesystem (HDFS). It is built for random, strictly consistent, real time access for tables with billions of rows and millions of columns. It has tight integration with Apache Hadoop, Apache Hive, and Apache Pig, so you can easily combine massively parallel analytics with fast data access. HBase’s data model, throughput, and fault tolerance are a good match for workloads in ad tech, web analytics, financial services, applications using time-series data, and many more.

Table structure in HBase, like many NoSQL technologies, should be directly influenced by the queries and access patterns of the data. Query performance varies drastically based on the way the cluster has to process and return the data.

HBase on S3

To use HBase on S3 read replicas, you must first be using HBase on S3. For those unfamiliar with HBase on S3 architecture, this section conveys some of the basics.

By using S3 as a data store for HBase, you can separate your cluster’s storage and compute nodes. This enables you to cut costs by sizing your cluster for your compute requirements. You don’t have to pay to store your entire dataset with 3x replication in the on-cluster Hadoop Distributed File System (HDFS).

EMR configures HBase on Amazon S3 to cache data in-memory and on-disk in your cluster to improve read performance from S3. You can quickly and easily scale up or scale down compute nodes without impacting your underlying storage. Or you can terminate your cluster to cut costs and quickly restore it in another Availability Zone.

HBase with support for S3 is available on EMR releases from 5.2.0 onward. To use S3 as a data store, configure the storage mode and specify a root directory in your HBase configuration. Also, it’s recommended to enable EMRFS consistent view. For more information, see Apache HBase on Amazon S3.

Use cases for HBase on S3 read replica clusters

Using HBase on S3 allows your data to be stored safely and durably. It persists the data off-cluster, which eliminates the dangers of data loss for persisted writes when the cluster is terminated. However, there can situations where you want to make sure that your data on HBase is highly available, even in the rare event of the cluster or Availability Zone failure. Another case could be when you want the ability to have multiple clusters access the same root directory in S3. If you have a primary cluster that goes under heavy load during bulk loads, writes, and compactions, this feature allows you to create secondary clusters that off-load and separate the read load from the write load, ensuring that you meet your read SLAs while optimizing around cost and performance.

The following diagram shows HBase on S3 without read replicas. In this scenario, events such as cluster failure or Availability Zone failure render users unable to access data on HBase.

The HBase root directory, including HFiles and metadata, resides in S3:

Prior to EMR 5.7.0, multiple clusters could not be pointed to the same root directory. For architectures requiring high availability, you needed to create duplicate data on S3.

With the launch of EMR 5.7.0, you can now start multiple HBase read-only clusters that can connect to the same HBase root directory in S3. This allows you to ensure that your data is always reachable through read replica clusters and run your clusters across multiple Availability Zones.

Here are a few example architectures that use HBase on S3 with read replicas, showing before and after for possible downtime events.

HBase read replica in the same Availability Zone – Resilient to primary cluster failure.

HBase read replica in different Availability Zones – Resilient to Availability Zone failure.

Another great way to think about using HBase on S3 read replicas is being able to right-size your clusters for the appropriate load. For example, if you anticipate light read load but still want to ensure high availability, you can size your read replicas to be small and of smaller instance types.

An additional example would be a bulk load scenario, where you have your write cluster scale up to peak load during a bulk load. You then scale the cluster back down to a minimal size after bulk load completion. During scaling, your read replicas can maintain a read-specific sizing or even have a heterogeneous sizing mixture.

Walkthrough

Use the following steps to set up your HBase on S3 read replica environment. This functionality is only available with EMR 5.7.0 or later.

Create an EMR cluster with HBase on S3

aws emr create-cluster --termination-protected --applications Name=Hadoop Name=Hive Name=HBase Name=Spark Name=Phoenix --ec2-attributes '{"KeyName":""}' --release-label emr-5.7.0 --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m3.xlarge","Name":"Master - 1"},{"InstanceCount":20,"BidPrice":"0.15","InstanceGroupType":"CORE","InstanceType":"m3.2xlarge","Name":"Core - 2"}]' --configurations '[{"Classification":"emrfs-site","Properties":{"fs.s3.consistent.retryPeriodSeconds":"1","fs.s3.consistent":"true","fs.s3.consistent.retryCount":"5","fs.s3.consistent.metadata.tableName":"YOUR_CONSISTENT_VIEW_TABLE_NAME"},"Configurations":[]},{"Classification":"hbase","Properties":{"hbase.emr.storageMode":"s3","hbase.emr.readreplica.enabled":"true"},"Configurations":[]},{"Classification":"hbase-site","Properties":{"hbase.rootdir":"s3:///"},"Configurations":[]}]' --service-role EMR_DefaultRole --name 'HBase Read Replica'

Sample Configuration JSON:

[  
   {  
      "Classification":"hbase-site",
      "Properties":{  
         "hbase.rootdir":"s3://{S3_LOCATION}",
      }
   },
   {  
      "Classification":"hbase",
      "Properties":{  
         "hbase.emr.storageMode":"s3",
         "hbase.emr.readreplica.enabled":"true"
      }
   }
]

Add data to the primary

When loading a system with HBase read replicas, you must make sure all writes from the primary cluster are flushed as HFiles to S3. The read-replica clusters read from these HFiles, and are unable to read any new writes that haven’t been flushed from the Memstore yet. To ensure that the read replica clusters are reading the most recent data, follow the steps below:

Insert data as usual into the HBase primary cluster (BulkLoading is preferred for large amounts of data).
Ensure that data is flushed (using the flush command) to S3. Note: This is only relevant if not using BulkLoading.
Wait for any region splits or merges to complete to ensure that the hbase:meta table is in a consistent state.
If any regions have changed (splits, merges) or any table metadata has been modified (tables added/removed), run the refresh_meta command on the read replica cluster.
On the read replica cluster, run the refresh_hfiles command for the updated table(s).

Read data from a replica

You can now read data as you would normally from your clusters.

Screenshot of reading from the primary cluster

Screenshot of reading from the read replica cluster

As you can see, both clusters return the same values.

Keeping your read replicas consistent

To keep your read replica cluster consistent, follow these general guidelines:

On the read replica:

Run ‘refresh_hfiles’ when:

Records are added/modified for a table.

Run ‘refresh_meta’ when:

Regions have changed (splits, compacts) or any table metadata has been modified (tables added/removed)

On the primary cluster:

If compactions are enabled, run a compaction to avoid inconsistencies when major compactions are triggered (minor compactions are handled).

Related properties and commands

HBase properties:

Config	Default	Explanation
hbase.meta.table.suffix	“”	Adds a suffix to the meta table name: value=’test’ -> ‘hbase:meta_test’
hbase.global.readonly.enabled	false	Puts the entire cluster into read-only mode
Hbase.meta.startup.refresh	false	Syncs the meta table with the backing storage. Used to pick up new tables or regions.

Reminder: These commands are automatically set if hbase.emr.readreplica.enabled is set to true.

HBase commands:

Command	Description
refresh_hfiles <Tablename	Refreshes HFiles from disk. Used to pick up new edits on a read replica.
clear_block_cache <tablename>	Clears the cache for the specified table.
refresh_meta	Syncs the meta table with the backing storage. Used to pick up new tables/regions.

Summary

You are now able to create highly available read-replica clusters for your existing HBase clusters backed by S3. You can retain read access to your HBase data in case of cluster failure, I/O intensive compactions, and even the unlikely event of an Availability Zone failure. For more information, see using HBase on S3 Read Replica clusters.

I encourage you to give the HBase on S3 read replica feature a try!

Next Steps

Take your skills to the next level. Learn tips for migrating to Apache HBase on S3 from HDFS.

How to Configure Even Stronger Password Policies to Help Meet Your Security Standards by Using AWS Directory Service for Microsoft Active Directory

2017-07-12 Ravi Turlapati

Post Syndicated from Ravi Turlapati original https://aws.amazon.com/blogs/security/how-to-configure-even-stronger-password-policies-to-help-meet-your-security-standards-by-using-aws-directory-service-for-microsoft-active-directory/

With AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also known as AWS Microsoft AD, you can now create and enforce custom password policies for your Microsoft Windows users. AWS Microsoft AD now includes five empty password policies that you can edit and apply with standard Microsoft password policy tools such as Active Directory Administrative Center (ADAC). With this capability, you are no longer limited to the default Windows password policy. Now, you can configure even stronger password policies and define lockout policies that specify when to lock out an account after login failures.

In this blog post, I demonstrate how to edit these new password policies to help you meet your security standards by using AWS Microsoft AD. I also introduce the password attributes you can modify and demonstrate how to apply password policies to user groups in your domain.

Prerequisites

The instructions in this post assume that you already have the following components running:

An active AWS Microsoft AD directory.
An Amazon EC2 for Windows Server instance that is domain joined to your AWS Microsoft AD directory and on which you have installed ADAC.

If you still need to meet these prerequisites before proceeding:

For information about how to deploy AWS Microsoft AD, see How to Create a Microsoft AD directory.
To join an EC2 for Windows Server instance to your AWS Microsoft AD domain on which you run ADAC, see Manually Add a Windows Instance.
Follow the steps in Installing the Active Directory Administration Tools to install Active Directory Administration Tools on your EC2 for Windows Server instance.

Scenario overview

Let’s say I am the Active Directory (AD) administrator of Example Corp. At Example Corp., we have a group of technical administrators, several groups of senior managers, and general, nontechnical employees. I need to create password policies for these groups that match our security standards.

Our general employees have access only to low-sensitivity information. However, our senior managers regularly access confidential information and we want to enforce password complexity (a mix of upper and lower case letters, numbers, and special characters) to reduce the risk of data theft. For our administrators, we want to enforce password complexity policies to prevent unauthorized access to our system administration tools.

Our security standards call for the following enforced password and account lockout policies:

General employees – To make it easier for nontechnical general employees to remember their passwords, we do not enforce password complexity. However, we want to enforce a minimum password length of 8 characters and a lockout policy after 6 failed login attempts as a minimum bar to protect against unwanted access to our low-sensitivity information. If a general employee forgets their password and becomes locked out, we let them try again in 5 minutes, rather than require escalated password resets. We also want general employees to rotate their passwords every 60 days with no duplicated passwords in the past 10 password changes.
Senior managers – For senior managers, we enforce a minimum password length of 10 characters and require password complexity. An account lockout is enforced after 6 failed attempts with an account lockout duration of 15 minutes. Senior managers must rotate their passwords every 45 days, and they cannot duplicate passwords from the past 20 changes.
Administrators – For administrators, we enforce password complexity with a minimum password length of 15 characters. We also want to lock out accounts after 6 failed attempts, have password rotation every 30 days, and disallow duplicate passwords in the past 30 changes. When a lockout occurs, we require a special administrator to intervene and unlock the account so that we can be aware of any potential hacking.
Fine-Grained Password Policy administrators – To ensure that only trusted administrators unlock accounts, we have two special administrator accounts (admin and midas) that can unlock accounts. These two accounts have the same policy as the other administrators except they have an account lockout duration of 15 minutes, rather than requiring a password reset. These two accounts are also the accounts used to manage Example Corp.’s password policies.

The following table summarizes how I edit each of the four policies I intend to use.

Policy name	EXAMPLE-PSO-01	EXAMPLE-PSO-02	EXAMPLE-PSO-03	EXAMPLE-PSO-05
Precedence	10	20	30	50
User group	Fine-Grained Password Policy Administrators	Other Administrators	Senior Managers	General Employees
Minimum password length	15	15	10	8
Password complexity	Enable	Enable	Enable	Disable
Maximum password age	30 days	30 days	45 days	60 days
Account complexity	Enable	Enable	Enable	Disable
Number of failed logon attempts allowed	6	6	6	6
Duration	15 minutes	Not applicable	15 minutes	5 minutes
Password history	24	30	20	10
Until admin manually unlocks account	Not applicable	Selected	Not applicable	Not applicable

To implement these password policies, I use 4 of the 5 new password policies available in AWS Microsoft AD:

I first explain how to configure the password policies.
I then demonstrate how to apply the four password policies that match Example Corp.’s security standards for these user groups.

1. Configure password policies in AWS Microsoft AD

To help you get started with password policies, AWS has added the Fine-Grained Pwd Policy Admins AD security group to your AWS Microsoft AD directory. Any user or other security group that is part of the Fine-Grained Pwd Policy Admins group has permissions to edit and apply the five new password policies. By default, your directory Admin is part of the new group and can add other users or groups to this group.

Adding users to the Fine-Grained Pwd Policy Admins user group

Follow these steps to add more users or AD security groups to the Fine-Grained Pwd Policy Admins security group so that they can administer fine-grained password policies:

Launch ADAC from your managed instance.
Switch to the Tree View and navigate to CORP > Users.
Find the Fine Grained Pwd Policy Admins user group. Add any users or groups in your domain to this group.

Edit password policies

To edit fine-grained password policies, open ADAC from any management instance joined to your domain. Switch to the Tree View and navigate to System > Password Settings Container. You will see the five policies containing the string -PSO- that AWS added to your directory, as shown in the following screenshot. Select a policy to edit it.

After editing the password policy, apply the policy by adding users or AD security groups to these policies by choosing Add. The default domain GPO applies if you do not configure any of the five password policies. For additional details about using Password Settings Container, go to Step-by-Step: Enabling and Using Fine-Grained Password Policies in AD on the Microsoft TechNet Blog.

The password attributes you can edit

AWS allows you to edit all of the password attributes except Precedence (I explain more about Precedence in the next section). These attributes include:

Password history
Minimum password length
Minimum password age
Maximum password age
Store password using reversible encryption
Password must meet complexity requirements

You also can enforce the following attributes for account lockout settings:

The number of failed login attempts allowed
Account lockout duration
Reset failed login attempts after a specified duration

For more details about how these attributes affect password enforcement, see AD DS: Fine-Grained Password Policies on Microsoft TechNet.

Understanding password policy precedence

AD password policies have a precedence (a numerical attribute that AD uses to determine the resultant policy) associated with them. Policies with a lower value for Precedence have higher priority than other policies. A user inherits all policies that you apply directly to the user or to any groups to which the user belongs. For example, suppose jsmith is a member of the HR group and also a member of the MANAGERS group. If I apply a policy with a Precedence of 50 to the HR group and a policy with a Precedence of 40 to MANAGERS, the policy with the Precedence value of 40 ranks higher and AD applies that policy to jsmith.

If you apply multiple policies to a user or group, the resultant policy is determined as follows by AD:

If you apply a policy directly to a user, AD enforces the lowest directly applied password policy.
If you did not apply a policy directly to the user, AD enforces the policy with the lowest Precedence value of all policies inherited by the user through the user’s group membership.

For more information about AD fine-grained policies, see AD DS: Fine-Grained Password Policies on Microsoft TechNet.

2. Apply password policies to user groups

In this section, I demonstrate how to apply Example Corp.’s password policies. Except in rare cases, I only apply policies by group membership, which ensures that AD does not enforce a lower priority policy on an individual user if have I added them to a group with a higher priority policy.

Because my directory is new, I use a Remote Desktop Protocol (RDP) connection to sign in to the Windows Server instance I domain joined to my AWS Microsoft AD directory. Signing in with the admin account, I launch ADAC to perform the following tasks:

First, I set up my groups so that I can apply password policies to them. Later, I can create user accounts and add them to my groups and AD applies the right policy by using the policy precedence and resultant policy algorithms I discussed previously. I start by adding the two special administrative accounts (admin and midas) that I described previously to the Fine-Grained Pwd Policy Admins. Because AWS Microsoft AD adds my default admin account to Fine-Grained Pwd Policy Admins, I only need to create midas and then add midas to the Fine-Grained Pwd Policy Admins group.
Next, I create the Other Administrators, Senior Managers, and General Employees groups that I described previously, as shown in the following screenshot.

For this post’s example, I use these four policies:

EXAMPLE-PSO-01 (highest priority policy) – For the administrators who manage Example Corp.’s password policies. Applying this highest priority policy to the Fine-Grained Pwd Policy Admins group prevents these users from being locked out if they also are assigned to a different policy.
EXAMPLE-PSO-02 (the second highest priority policy) – For Example Corp.’s other administrators.
EXAMPLE-PSO-03 (the third highest priority policy) – For Example Corp.’s senior managers.
EXAMPLE-PSO-05 (the lowest priority policy) – For Example Corp.’s general employees.

This leaves me one password policy (EXAMPLE-PSO-04) that I can use for in the future if needed.

I start by editing the policy, EXAMPLE-PSO-01. To edit the policy, I follow the Edit password policies section from earlier in this post. When finished, I add the Fine-Grained Pwd Policy Admins group to that policy, as shown in the following screenshot. I then repeat the process for each of the remaining policies, as described in the Scenario overview section earlier in this post.

Though AD enforces new password policies, the timing related to how password policies replicate in the directory, the types of attributes that are changed, and the timing of user password changes can cause variability in the immediacy of policy enforcement. In general, after the policies are replicated throughout the directory, attributes that affect account lockout and password age take effect. Attributes that affect the quality of a password, such as password length, take effect when the password is changed. If the password age for a user is in compliance, but their password strength is out of compliance, the user is not forced to change their password. For more information password policy impact, see this Microsoft TechNet article.

Summary

In this post, I have demonstrated how you can configure strong password policies to meet your security standards by using AWS Microsoft AD. To learn more about AWS Microsoft AD, see the AWS Directory Service home page.

If you have comments about this post, submit them in the “Comments” section below. If you have questions about this blog post, start a new thread on the Directory Service forum.

– Ravi

Blue/Green Deployments with Amazon EC2 Container Service

2017-07-01 Nathan Taber

Post Syndicated from Nathan Taber original https://aws.amazon.com/blogs/compute/bluegreen-deployments-with-amazon-ecs/

This post and accompanying code was generously contributed by:


Jeremy Cowan Solutions Architect	Anuj Sharma DevOps Cloud Architect	Peter Dalbhanjan Solutions Architect

Deploying software updates in traditional non-containerized environments is hard and fraught with risk. When you write your deployment package or script, you have to assume that the target machine is in a particular state. If your staging environment is not an exact mirror image of your production environment, your deployment could fail. These failures frequently cause outages that persist until you re-deploy the last known good version of your application. If you are an Operations Manager, this is what keeps you up at night.

Increasingly, customers want to do testing in production environments without exposing customers to the new version until the release has been vetted. Others want to expose a small percentage of their customers to the new release to gather feedback about a feature before it’s released to the broader population. This is often referred to as canary analysis or canary testing. In this post, I introduce patterns to implement blue/green and canary deployments using Application Load Balancers and target groups.

If you’d like to try this approach to blue/green deployments, we have open sourced the code and AWS CloudFormation templates in the ecs-blue-green-deployment GitHub repo. The workflow builds an automated CI/CD pipeline that deploys your service onto an ECS cluster and offers a controlled process to swap target groups when you’re ready to promote the latest version of your code to production. You can quickly set up the environment in three steps and see the blue/green swap in action. We’d love for you to try it and send us your feedback!

Benefits of blue/green

Blue/green deployments are a type of immutable deployment that help you deploy software updates with less risk. The risk is reduced by creating separate environments for the current running or “blue” version of your application, and the new or “green” version of your application.

This type of deployment gives you an opportunity to test features in the green environment without impacting the current running version of your application. When you’re satisfied that the green version is working properly, you can gradually reroute the traffic from the old blue environment to the new green environment by modifying DNS. By following this method, you can update and roll back features with near zero downtime.

A typical blue/green deployment involves shifting traffic between 2 distinct environments.

This ability to quickly roll traffic back to the still-operating blue environment is one of the key benefits of blue/green deployments. With blue/green, you should be able to roll back to the blue environment at any time during the deployment process. This limits downtime to the time it takes to realize there’s an issue in the green environment and shift the traffic back to the blue environment. Furthermore, the impact of the outage is limited to the portion of traffic going to the green environment, not all traffic. If the blast radius of deployment errors is reduced, so is the overall deployment risk.

Containers make it simpler

Historically, blue/green deployments were not often used to deploy software on-premises because of the cost and complexity associated with provisioning and managing multiple environments. Instead, applications were upgraded in place.

Although this approach worked, it had several flaws, including the ability to roll back quickly from failures. Rollbacks typically involved re-deploying a previous version of the application, which could affect the length of an outage caused by a bad release. Fixing the issue took precedence over the need to debug, so there were fewer opportunities to learn from your mistakes.

Containers can ease the adoption of blue/green deployments because they’re easily packaged and behave consistently as they’re moved between environments. This consistency comes partly from their immutability. To change the configuration of a container, update its Dockerfile and rebuild and re-deploy the container rather than updating the software in place.

Containers also provide process and namespace isolation for your applications, which allows you to run multiple versions of them side by side on the same Docker host without conflicts. Given their small sizes relative to virtual machines, you can binpack more containers per host than VMs. This lets you make more efficient use of your computing resources, reducing the cost of blue/green deployments.

Fully Managed Updates with Amazon ECS

Amazon EC2 Container Service (ECS) performs rolling updates when you update an existing Amazon ECS service. A rolling update involves replacing the current running version of the container with the latest version. The number of containers Amazon ECS adds or removes from service during a rolling update is controlled by adjusting the minimum and maximum number of healthy tasks allowed during service deployments.

When you update your service’s task definition with the latest version of your container image, Amazon ECS automatically starts replacing the old version of your container with the latest version. During a deployment, Amazon ECS drains connections from the current running version and registers your new containers with the Application Load Balancer as they come online.

Target groups

A target group is a logical construct that allows you to run multiple services behind the same Application Load Balancer. This is possible because each target group has its own listener.

When you create an Amazon ECS service that’s fronted by an Application Load Balancer, you have to designate a target group for your service. Ordinarily, you would create a target group for each of your Amazon ECS services. However, the approach we’re going to explore here involves creating two target groups: one for the blue version of your service, and one for the green version of your service. We’re also using a different listener port for each target group so that you can test the green version of your service using the same path as the blue service.

With this configuration, you can run both environments in parallel until you’re ready to cut over to the green version of your service. You can also do things such as restricting access to the green version to testers on your internal network, using security group rules and placement constraints. For example, you can target the green version of your service to only run on instances that are accessible from your corporate network.

Swapping Over

When you’re ready to replace the old blue service with the new green service, call the ModifyListener API operation to swap the listener’s rules for the target group rules. The change happens instantaneously. Afterward, the green service is running in the target group with the port 80 listener and the blue service is running in the target group with the port 8080 listener. The diagram below is an illustration of the approach described.

Scenario

Two services are defined, each with their own target group registered to the same Application Load Balancer but listening on different ports. Deployment is completed by swapping the listener rules between the two target groups.

The second service is deployed with a new target group listening on a different port but registered to the same Application Load Balancer.

By using 2 listeners, requests to blue services are directed to the target group with the port 80 listener, while requests to the green services are directed to target group with the port 8080 listener.

After automated or manual testing, the deployment can be completed by swapping the listener rules on the Application Load Balancer and sending traffic to the green service.

Caveats

There are a few caveats to be mindful of when using this approach. This method:

Assumes that your application code is completely stateless. Store state outside of the container.
Doesn’t gracefully drain connections. The swapping of target groups is sudden and abrupt. Therefore, be cautious about using this approach if your service has long-running transactions.
Doesn’t allow you to perform canary deployments. While the method gives you the ability to quickly switch between different versions of your service, it does not allow you to divert a portion of the production traffic to a canary or control the rate at which your service is deployed across the cluster.

Canary testing

While this type of deployment automates much of the heavy lifting associated with rolling deployments, it doesn’t allow you to interrupt the deployment if you discover an issue midstream. Rollbacks using the standard Amazon ECS deployment require updating the service’s task definition with the last known good version of the container. Then, you wait for Amazon ECS to schedule and deploy it across the cluster. If the latest version introduces a breaking change that went undiscovered during testing, this might be too slow.

With canary testing, if you discover the green environment is not operating as expected, there is no impact on the blue environment. You can route traffic back to it, minimizing impaired operation or downtime, and limiting the blast radius of impact.

This type of deployment is particularly useful for A/B testing where you want to expose a new feature to a subset of users to get their feedback before making it broadly available.

For canary style deployments, you can use a variation of the blue/green swap that involves deploying the blue and the green service to the same target group. Although this method is not as fast as the swap, it allows you to control the rate at which your containers are replaced by adjusting the task count for each service. Furthermore, it gives you the ability to roll back by adjusting the number of tasks for the blue and green services respectively. Unlike the swap approach described above, connections to your containers are drained gracefully. We plan to address canary style deployments for Amazon ECS in a future post.

Conclusion

With AWS, you can operationalize your blue/green deployments using Amazon ECS, an Application Load Balancer, and target groups. I encourage you to adapt the code published to the ecs-blue-green-deployment GitHub repo for your use cases and look forward to reading your feedback.

If you’re interested in learning more, I encourage you to read the Blue/Green Deployments on AWS and Practicing Continuous Integration and Continuous Delivery on AWS whitepapers.

If you have questions or suggestions, please comment below.

[$] Guarding personally identifiable information

2017-06-08 jake

Post Syndicated from jake original https://lwn.net/Articles/724672/rss

There is no viable way to prevent data from being collected about us in the
current age of computing. But if institutions insist on knowing our
financial status, purchasing habits, health information,
political preferences, and so on, they have a responsibility to keep this
data—known as personally identifiable information (PII)—from leaking to
unauthorized recipients. At the 2017 Strata data
conference in London, Steve Touw presented a session
on privacy-enhancing technologies. In a fast-paced 40 minutes he
covered the EU regulations about privacy, the most popular technical
measures used to protect PII, and some pointed opinions about what works
and what should be thrown into the dustbin.

Ransomware and the Internet of Things

2017-05-25 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/05/ransomware_and_.html

As devastating as the latest widespread ransomware attacks have been, it’s a problem with a solution. If your copy of Windows is relatively current and you’ve kept it updated, your laptop is immune. It’s only older unpatched systems on your computer that are vulnerable.

Patching is how the computer industry maintains security in the face of rampant Internet insecurity. Microsoft, Apple and Google have teams of engineers who quickly write, test and distribute these patches, updates to the codes that fix vulnerabilities in software. Most people have set up their computers and phones to automatically apply these patches, and the whole thing works seamlessly. It isn’t a perfect system, but it’s the best we have.

But it is a system that’s going to fail in the “Internet of things”: everyday devices like smart speakers, household appliances, toys, lighting systems, even cars, that are connected to the web. Many of the embedded networked systems in these devices that will pervade our lives don’t have engineering teams on hand to write patches and may well last far longer than the companies that are supposed to keep the software safe from criminals. Some of them don’t even have the ability to be patched.

Fast forward five to 10 years, and the world is going to be filled with literally tens of billions of devices that hackers can attack. We’re going to see ransomware against our cars. Our digital video recorders and web cameras will be taken over by botnets. The data that these devices collect about us will be stolen and used to commit fraud. And we’re not going to be able to secure these devices.

Like every other instance of product safety, this problem will never be solved without considerable government involvement.

For years, I have been calling for more regulation to improve security in the face of this market failure. In the short term, the government can mandate that these devices have more secure default configurations and the ability to be patched. It can issue best-practice regulations for critical software and make software manufacturers liable for vulnerabilities. It’ll be expensive, but it will go a long way toward improved security.

But it won’t be enough to focus only on the devices, because these things are going to be around and on the Internet much longer than the two to three years we use our phones and computers before we upgrade them. I expect to keep my car for 15 years, and my refrigerator for at least 20 years. Cities will expect the networks they’re putting in place to last at least that long. I don’t want to replace my digital thermostat ever again. Nor, if I ever need one, do I want a surgeon to ever have to go back in to replace my computerized heart defibrillator in order to fix a software bug.

No amount of regulation can force companies to maintain old products, and it certainly can’t prevent companies from going out of business. The future will contain billions of orphaned devices connected to the web that simply have no engineers able to patch them.

Imagine this: The company that made your Internet-enabled door lock is long out of business. You have no way to secure yourself against the ransomware attack on that lock. Your only option, other than paying, and paying again when it’s reinfected, is to throw it away and buy a new one.

Ultimately, we will also need the network to block these attacks before they get to the devices, but there again the market will not fix the problem on its own. We need additional government intervention to mandate these sorts of solutions.

None of this is welcome news to a government that prides itself on minimal intervention and maximal market forces, but national security is often an exception to this rule. Last week’s cyberattacks have laid bare some fundamental vulnerabilities in our computer infrastructure and serve as a harbinger. There’s a lot of good research into robust solutions, but the economic incentives are all misaligned. As politically untenable as it is, we need government to step in to create the market forces that will get us out of this mess.

This essay previously appeared in the New York Times. Yes, I know I’m repeating myself.

The AWS EU (London) Region Achieves Public Services Network (PSN) Assurance

2017-05-04 Oliver Bell

Post Syndicated from Oliver Bell original https://aws.amazon.com/blogs/security/aws-uk-region-achieves-public-services-network-psn-assurance/

AWS is excited to announce that the AWS EU (London) Region has achieved Public Services Network (PSN) assurance. This means that the EU (London) Region can now be connected to the PSN (or PSN customers) by PSN-certified AWS Direct Connect partners. PSN assurance demonstrates to our UK Public Sector customers that the EU (London) Region has met the stringent requirements of PSN and provides an assured platform on which to build UK Public Sector services. Customers are required to ensure that applications and configurations applied to their AWS instances meet the PSN standards, and they must undertake PSN certification for the content, platform, applications, systems, and networks they run on AWS (but no longer need to include AWS infrastructure and products in their certification).

In conjunction with our Standardized Architecture for UK-OFFICIAL, PSN assurance enables UK Public Sector organizations to move their UK-OFFICIAL classified data to the EU (London) Region in a controlled and risk-managed manner. AWS has also created a UK-OFFICIAL on AWS Quick Start, which provisions an environment suitable for UK-OFFICIAL classified data. This Quick Start includes guidance and controls that help public sector organizations manage risks and ensure security when handling UK-OFFICIAL information assets.

You can download the EU (London) Region PSN Code of Connection and Service Compliance certificates through AWS Artifact. For further information about using AWS in the context of the National Cyber Security Centre (NCSC) UK’s Cloud Security Principles, see Using AWS in the Context of NCSC UK’s Cloud Security Principles.

– Oliver

Pioneers: the second challenge is…

2017-04-24 Olympia Brown

Post Syndicated from Olympia Brown original https://www.raspberrypi.org/blog/pioneers-second-challenge/

Pioneers, your next challenge is here!

Do you like making things? Do you fancy trying something new? Are you aged 11 to 16? The Pioneers programme is ready to challenge you to create something new using technology.

As you’ll know if you took part last time, Pioneers challenges are themed. So here’s the lovely Ana from ZSL London Zoo to reveal the theme of the next challenge:

Your next challenge, if you choose to accept it, is…

MakeYourIdeas The second Pioneers challenge is here! Wahoo! Have you registered your team yet? Make sure you do. Head to the Pioneers website for more details: http://www.raspberrypi.org/pioneers

Make it Outdoors

You have until the beginning of July to make something related to the outdoors. As Ana said, the outdoors is pretty big, so here are some ideas:

You could use a Raspberry Pi to track and look after your pet, or a motion time-lapse camera rig to monitor your school grounds.
Or how about using an Arduino to photograph bursting bubbles or make a rain-sensing light up umbrella, or building a speedometer for your bike?
You could charge your phone while walking or while you’re cycling. And if you’re on your bike, why not stay safe at night with a light-up jacket?
There are some beautiful free satellite images out there, why not use them to create your own landscapes?

Resources and discounted kit

If you’re looking at all of these projects and thinking that you don’t know where to start, never fear! Our free resources offer a great starting point for any new project, and can help you to build on your existing skills and widen your scope for creating greatness.

We really want to see your creativity and ingenuity though, so we’d recommend using these projects as starting points rather than just working through the instructions. To help us out, the wonderful Pimoroni are offering 15 percent off kit for our Getting started with wearables and Getting started with picamera resources. You should also check out our new Poo near you resource for an example of a completely code-based project.

For this cycle of Pioneers, thanks to our friends at the Shell Centenary Scholarship Fund, we are making bursaries available to teams to cover the cost of these basic kits (one per team). This is for teens who haven’t taken part in digital making activities before, and for whom the financial commitment would be a barrier to taking part. Details about the bursaries and the discount will be sent to you when you register.

Your Pioneers team

We’ve introduced a few new things for this round of Pioneers, so pay special attention if you took part last time round!

We’re looking for UK-based teams of between two and five people, aged between 11 and 16, to work together to create something related to the outdoors. We’ve found that in our experience there are three main ways to run a Pioneers team. It’s up to you to decide how you’ll proceed when it comes to your participation in Pioneers.

You could organise a Group that meets once or twice a week. We find this method works well for school-based teams that can meet at the end of a school day for an hour or two every week.
You could mentor a Squad that is largely informal, where the members probably already have a good idea of what they’re doing. A Squad tends to be more independent, and meetings may be sporadic, informal or online only. This option isn’t recommended if it’s your first competition like this, or if you’re not a techie yourself.
You could join a local Event at a technology hub near you. We’re hoping to run more and more of these events around the country as Pioneers evolves and grows. If you think you’d like to help us run a Pioneers Event, get in touch! We love to hear from people who want to spread their love of making, and we’ll support you as much as we possibly can to get your event rocking along. If you want to run a Pioneers Event, you will need to preregister on the Pioneers website so that we can get you all the support you need well before you open your doors.

#MakeYourIdeas

As always, we’re excited to watch the progress of your projects via social media channels such as Twitter, Instagram, and Snapchat. As you work on your build, make sure to share the ‘making of…’ stages with us using #MakeYourIdeas.

For inspiration from previous entries, here’s the winner announcement video for the last Pioneers challenge:

Winners of the first Pioneers challenge are…

After months of planning and making, the first round of Pioneers is over! We laid down the epic challenge of making us laugh. And boy, did the teams deliver. We can honestly say that my face hurt from all the laughing on judging day. Congratulations to everyone who took part.

Once you’ve picked a project, the first step is to register. What are you waiting for? Head to the Pioneers website to get started!

The post Pioneers: the second challenge is… appeared first on Raspberry Pi.

Amazon Lex – Now Generally Available

2017-04-19 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-lex-now-generally-available/

During AWS re:Invent I showed you how you could use Amazon Lex to build conversational voice & text interfaces. At that time we launched Amazon Lex in preview form and invited developers to sign up for access. Powered by the same deep learning technologies that drive Amazon Alexa, Amazon Lex allows you to build web & mobile applications that support engaging, lifelike interactions.

Today I am happy to announce that we are making Amazon Lex generally available, and that you can start using it today! Here are some of the features that we added during the preview:

Slack Integration – You can now create Amazon Lex bots that respond to messages and events sent to a Slack channel. Click on the Channels tab of your bot, select Slack, and fill in the form to get a callback URL for use with Slack:

Follow the tutorial (Integrating an Amazon Lex Bot with Slack) to see how to do this yourself.

Twilio Integration – You can now create Amazon Lex bots that respond to SMS messages sent to a Twilio SMS number. Again, you simply click on Channels, select Twilio, and fill in the form:

To learn more, read Integrating an Amazon Lex Bot with Twilio SMS.

SDK Support – You can now use the AWS SDKs to build iOS, Android, Java, JavaScript, Python, .Net, Ruby, PHP, Go, and C++ bots that span mobile, web, desktop, and IoT platforms and interact using either text or speech. The SDKs also support the build process for bots; you can programmatically add sample utterances, create slots, add slot values, and so forth. You can also manage the entire build, test, and deployment process programmatically.

Voice Input on Test Console – The Amazon Lex test console now supports voice input when used on the Chrome browser. Simply click on the microphone:

Utterance Monitoring – Amazon Lex now records utterances that were not recognized by your bot, otherwise known as missed utterances. You can review the list and add the relevant ones to your bot:

You can also watch the following CloudWatch metrics to get a better sense of how your users are interacting with your bot. Over time, as you add additional utterances and improve your bot in other ways, the metrics should be on the decline.

Text Missed Utterances (PostText)
Text Missed Utterances (PostContent)
Speech Missed Utterances

Easy Association of Slots with Utterances – You can now highlight text in the sample utterances in order to identify slots and add values to slot types:

Improved IAM Support – Amazon Lex permissions are now configured automatically from the console; you can now create bots without having to create your own policies.

Preview Response Cards – You can now view a preview of the response cards in the console:

To learn more, read about Using a Response Card.

Go For It
Pricing is based on the number of text and voice responses processed by your application; see the Amazon Lex Pricing page for more info.

I am really looking forward to seeing some awesome bots in action! Build something cool and let me know what you come up with.

— Jeff;

New- Introducing AWS CodeStar – Quickly Develop, Build, and Deploy Applications on AWS

2017-04-19 Tara Walker

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/new-aws-codestar/

It wasn’t too long ago that I was on a development team working toward completing a software project by a release deadline and facing the challenges most software teams face today in developing applications. Challenges such as new project environment setup, team member collaboration, and the day-to-day task of keeping track of the moving pieces of code, configuration, and libraries for each development build. Today, with companies’ need to innovate and get to market faster, it has become essential to make it easier and more efficient for development teams to create, build, and deploy software.

Unfortunately, many organizations face some key challenges in their quest for a more agile, dynamic software development process. The first challenge most new software projects face is the lengthy setup process that developers have to complete before they can start coding. This process may include setting up of IDEs, getting access to the appropriate code repositories, and/or identifying infrastructure needed for builds, tests, and production.

Collaboration is another challenge that most development teams may face. In order to provide a secure environment for all members of the project, teams have to frequently set up separate projects and tools for various team roles and needs. In addition, providing information to all stakeholders about updates on assignments, the progression of development, and reporting software issues can be time-consuming.

Finally, most companies desire to increase the speed of their software development and reduce the time to market by adopting best practices around continuous integration and continuous delivery. Implementing these agile development strategies may require companies to spend time in educating teams on methodologies and setting up resources for these new processes.

Now Presenting: AWS CodeStar

To help development teams ease the challenges of building software while helping to increase the pace of releasing applications and solutions, I am excited to introduce AWS CodeStar.

AWS CodeStar is a cloud service designed to make it easier to develop, build, and deploy applications on AWS by simplifying the setup of your entire development project. AWS CodeStar includes project templates for common development platforms to enable provisioning of projects and resources for coding, building, testing, deploying, and running your software project.

The key benefits of the AWS CodeStar service are:

Easily create new projects using templates for Amazon EC2, AWS Elastic Beanstalk, or AWS Lambda using five different programming languages; JavaScript, Java, Python, Ruby, and PHP. By selecting a template, the service will provision the underlying AWS services needed for your project and application.
Unified experience for access and security policies management for your entire software team. Projects are automatically configured with appropriate IAM access policies to ensure a secure application environment.
Pre-configured project management dashboard for tracking various activities, such as code commits, build results, deployment activity and more.
Running sample code to help you get up and running quickly enabling you to use your favorite IDEs, like Visual Studio, Eclipse, or any code editor that supports Git.
Automated configuration of a continuous delivery pipeline for each project using AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy.
Integration with Atlassian JIRA Software for issue management and tracking directly from the AWS CodeStar console

With AWS CodeStar, development teams can build an agile software development workflow that now only increases the speed in which teams and deploy software and bug fixes, but also enables developers to build software that is more inline with customers’ requests and needs.

An example of a responsive development workflow using AWS CodeStar is shown below:

Journey Into AWS CodeStar

Now that you know a little more about the AWS CodeStar service, let’s jump into using the service to set up a web application project. First, I’ll go to into the AWS CodeStar console and click the Start a project button.

If you have not setup the appropriate IAM permissions, AWS CodeStar will show a dialog box requesting permission to administer AWS resources on your behalf. I will click the Yes, grant permissions button to grant AWS CodeStar the appropriate permissions to other AWS resources.

However, I received a warning that I do not have administrative permissions to AWS CodeStar as I have not applied the correct policies to my IAM user. If you want to create projects in AWS CodeStar, you must apply the AWSCodeStarFullAccess managed policy to your IAM user or have an IAM administrative user with full permissions for all AWS services.

Now that I have added the aforementioned permissions in IAM, I can now use the service to create a project. To start, I simply click on the Create a new project button and I am taken to the hub of the AWS CodeStar service.

At this point, I am presented with over twenty different AWS CodeStar project templates to choose from in order to provision various environments for my software development needs. Each project template specifies the AWS Service used to deploy the project, the supported programming language, and a description of the type of development solution implemented. AWS CodeStar currently supports the following AWS Services: Amazon EC2, AWS Lambda, and AWS Elastic Beanstalk. Using preconfigured AWS CloudFormation templates, these project templates can create software development projects like microservices, Alexa skills, web applications, and more with a simple click of a button.

For my first AWS CodeStar project, I am going to build a serverless web application using Node.js and AWS Lambda using the Node.js/AWS Lambda project template.

You will notice for this template AWS CodeStar sets up all of the tools and services you need for a development project including an AWS CodePipeline connected with the services; AWS CodeBuild, AWS CloudFormation, and Amazon CloudWatch. I’ll name my new AWS CodeStar project, TaraWebProject, and click Create Project.

Since this is my first time creating an AWS CodeStar, I will see a dialog that asks about the setup of my AWS CodeStar user settings. I’ll type Tara in the textbox for the Display Name and add my email address in the Email textbox. This information is how I’ll appear to others in the project.

The next step is to select how I want to edit my project code. I have decided to edit my TaraWebProject project code using the Visual Studio IDE. With Visual Studio, it will be essential for me to configure it to use the AWS Toolkit for Visual Studio 2015 to access AWS resources while editing my project code. On this screen, I am also presented with the link to the AWS CodeCommit Git repository that AWS CodeStar configured for my project.

The provisioning and tool setup for my software development project is now complete. I’m presented with the AWS CodeStar dashboard for my software project, TaraWebProject, which allows me to manage the resources for the project. This includes the management of resources, such as code commits, team membership and wiki, continuous delivery pipeline, Jira issue tracking, project status and other applicable project resources.

What is really cool about AWS CodeStar for me is that it provides a working sample project from which I can start the development of my serverless web application. To view the sample of my new web application, I will go to the Application endpoints section of the dashboard and click the link provided.

A new browser window will open and will display the sample web application AWS CodeStar generated to help jumpstart my development. A cool feature of the sample application is that the background of the sample app changes colors based on the time of day.

Let’s now take a look at the code used to build the sample website. In order to view the code, I will back to my TaraWebProject dashboard in the AWS CodeStar console and select the Code option from the sidebar menu.

This takes me to the tarawebproject Git repository in the AWS CodeCommit console. From here, I can manually view the code for my web application, the commits made in the repo, the comparison of commits or branches, as well as, create triggers in response to my repo events.

This provides a great start for me to start developing my AWS hosted web application. Since I opted to integrate AWS CodeStar with Visual Studio, I can update my web application by using the IDE to make code changes that will be automatically included in the TaraWebProject every time I commit to the provisioned code repository.

You will notice that on the AWS CodeStar TaraWebProject dashboard, there is a message about connecting the tools to my project repository in order to work on the code. Even though I have already selected Visual Studio as my IDE of choice, let’s click on the Connect Tools button to review the steps to connecting to this IDE.

Again, I will see a screen that will allow me to choose which IDE: Visual Studio, Eclipse, or Command Line tool that I wish to use to edit my project code. It is important for me to note that I have the option to change my IDE choice at any time while working on my development project. Additionally, I can connect to my Git AWS CodeCommit repo via HTTPS and SSH. To retrieve the appropriate repository URL for each protocol, I only need to select the Code repository URL dropdown and select HTTPS or SSH and copy the resulting URL from the text field.

After selecting Visual Studio, CodeStar takes me to the steps needed in order to integrate with Visual Studio. This includes downloading the AWS Toolkit for Visual Studio, connecting the Team Explorer to AWS CodeStar via AWS CodeCommit, as well as, how to push changes to the repo.

After successfully connecting Visual Studio to my AWS CodeStar project, I return to the AWS CodeStar TaraWebProject dashboard to start managing the team members working on the web application with me. First, I will select the Setup your team tile so that I can go to the Project Team page.

On my TaraWebProject Project Team page, I’ll add a team member, Jeff, by selecting the Add team member button and clicking on the Select user dropdown. Team members must be IAM users in my account, so I’ll click on the Create new IAM user link to create an IAM accounts for Jeff.

When the Create IAM user dialog box comes up, I will enter an IAM user name, Display name, and Email Address for the team member, in this case, Jeff Barr. There are three types of project roles that Jeff can be granted, Owner, Contributor, or Viewer. For the TaraWebProject application, I will grant him the Contributor project role and allow him to have remote access by select the Remote access checkbox. Now I will create Jeff’s IAM user account by clicking the Create button.

This brings me to the IAM console to confirm the creation of the new IAM user. After reviewing the IAM user information and the permissions granted, I will click the Create user button to complete the creation of Jeff’s IAM user account for TaraWebProject.

After successfully creating Jeff’s account, it is important that I either send Jeff’s login credentials to him in email or download the credentials .csv file, as I will not be able to retrieve these credentials again. I would need to generate new credentials for Jeff if I leave this page without obtaining his current login credentials. Clicking the Close button returns me to the AWS CodeStar console.

Now I can complete adding Jeff as a team member in the TaraWebProject by selecting the JeffBarr-WebDev IAM role and clicking the Add button.

I’ve successfully added Jeff as a team member to my AWS CodeStar project, TaraWebProject enabling team collaboration in building the web application.

Another thing that I really enjoy about using the AWS CodeStar service is I can monitor all of my project activity right from my TaraWebProject dashboard. I can see the application activity, any recent code commits, and track the status of any project actions, such as the results of my build, any code changes, and the deployments from in one comprehensive dashboard. AWS CodeStar ties the dashboard into Amazon CloudWatch with the Application activity section, provides data about the build and deployment status in the Continuous Deployment section with AWS CodePipeline, and shows the latest Git code commit with AWS CodeCommit in the Commit history section.

Summary

In my journey of the AWS CodeStar service, I created a serverless web application that provisioned my entire development toolchain for coding, building, testing, and deployment for my TaraWebProject software project using AWS services. Amazingly, I have yet to scratch the surface of the benefits of using AWS CodeStar to manage day-to-day software development activities involved in releasing applications.

AWS CodeStar makes it easy for you to quickly develop, build, and deploy applications on AWS. AWS CodeStar provides a unified user interface, enabling you to easily manage your software development activities in one place. AWS CodeStar allows you to choose from various templates to setting up projects using AWS Lambda, Amazon EC2, or AWS Elastic Beanstalk. It comes pre-configured with a project management dashboard, an automated continuous delivery pipeline, and a Git code repository using AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy allowing developers to implement modern agile software development best practices. Each AWS CodeStar project gives developers a head start in development by providing working code samples that can be used with popular IDEs that support Git. Additionally, AWS CodeStar provides out of the box integration with Atlassian JIRA Software providing a project management and issue tracking system for your software team directly from the AWS CodeStar console.

You can get started using the AWS CodeStar service for developing new software projects on AWS today. Learn more by reviewing the AWS CodeStar product page and the AWS CodeStar user guide documentation.

– Tara

Noise

Tag Archives: about us

Announcing Dedicated IP Pools

What are dedicated IP pools?

When should I use dedicated IP pools?

How do I create and use dedicated IP pools?

Will my email sending process change?

timeShift(GrafanaBuzz, 1w) Issue 7

Grafana 4.4.2 Released

From the Blogosphere

Plugins and Dashboards

This week’s MVC (Most Valuable Contributor)

Tweet of the Week

Upcoming Events

What do you think?

Setting up Read Replica Clusters with HBase on Amazon S3

HBase Overview

HBase on S3

Use cases for HBase on S3 read replica clusters

Walkthrough

Create an EMR cluster with HBase on S3

Add data to the primary

Read data from a replica

Keeping your read replicas consistent

Related properties and commands

Summary

Next Steps

How to Configure Even Stronger Password Policies to Help Meet Your Security Standards by Using AWS Directory Service for Microsoft Active Directory

Prerequisites

Scenario overview

1. Configure password policies in AWS Microsoft AD

Adding users to the Fine-Grained Pwd Policy Admins user group

Edit password policies

The password attributes you can edit

Understanding password policy precedence

2. Apply password policies to user groups

Summary

[$] Guarding personally identifiable information

Ransomware and the Internet of Things

The collective thoughts of the interwebz