Tag Archives: Sidebar

Getting Rid of Your Mac? Here’s How to Securely Erase a Hard Drive or SSD

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-to-wipe-a-mac-hard-drive/

erasing a hard drive and a solid state drive

What do I do with a Mac that still has personal data on it? Do I take out the disk drive and smash it? Do I sweep it with a really strong magnet? Is there a difference in how I handle a hard drive (HDD) versus a solid-state drive (SSD)? Well, taking a sledgehammer or projectile weapon to your old machine is certainly one way to make the data irretrievable, and it can be enormously cathartic as long as you follow appropriate safety and disposal protocols. But there are far less destructive ways to make sure your data is gone for good. Let me introduce you to secure erasing.

Which Type of Drive Do You Have?

Before we start, you need to know whether you have a HDD or a SSD. To find out, or at least to make sure, you click on the Apple menu and select “About this Mac.” Once there, select the “Storage” tab to see which type of drive is in your system.

The first example, below, shows a SATA Disk (HDD) in the system.

SATA HDD

In the next case, we see we have a Solid State SATA Drive (SSD), plus a Mac SuperDrive.

Mac storage dialog showing SSD

The third screen shot shows an SSD, as well. In this case it’s called “Flash Storage.”

Flash Storage

Make Sure You Have a Backup

Before you get started, you’ll want to make sure that any important data on your hard drive has moved somewhere else. OS X’s built-in Time Machine backup software is a good start, especially when paired with Backblaze. You can learn more about using Time Machine in our Mac Backup Guide.

With a local backup copy in hand and secure cloud storage, you know your data is always safe no matter what happens.

Once you’ve verified your data is backed up, roll up your sleeves and get to work. The key is OS X Recovery — a special part of the Mac operating system since OS X 10.7 “Lion.”

How to Wipe a Mac Hard Disk Drive (HDD)

NOTE: If you’re interested in wiping an SSD, see below.

    1. Make sure your Mac is turned off.
    2. Press the power button.
    3. Immediately hold down the command and R keys.
    4. Wait until the Apple logo appears.
    5. Select “Disk Utility” from the OS X Utilities list. Click Continue.
    6. Select the disk you’d like to erase by clicking on it in the sidebar.
    7. Click the Erase button.
    8. Click the Security Options button.
    9. The Security Options window includes a slider that enables you to determine how thoroughly you want to erase your hard drive.

There are four notches to that Security Options slider. “Fastest” is quick but insecure — data could potentially be rebuilt using a file recovery app. Moving that slider to the right introduces progressively more secure erasing. Disk Utility’s most secure level erases the information used to access the files on your disk, then writes zeroes across the disk surface seven times to help remove any trace of what was there. This setting conforms to the DoD 5220.22-M specification.

  1. Once you’ve selected the level of secure erasing you’re comfortable with, click the OK button.
  2. Click the Erase button to begin. Bear in mind that the more secure method you select, the longer it will take. The most secure methods can add hours to the process.

Once it’s done, the Mac’s hard drive will be clean as a whistle and ready for its next adventure: a fresh installation of OS X, being donated to a relative or a local charity, or just sent to an e-waste facility. Of course you can still drill a hole in your disk or smash it with a sledgehammer if it makes you happy, but now you know how to wipe the data from your old computer with much less ruckus.

The above instructions apply to older Macintoshes with HDDs. What do you do if you have an SSD?

Securely Erasing SSDs, and Why Not To

Most new Macs ship with solid state drives (SSDs). Only the iMac and Mac mini ship with regular hard drives anymore, and even those are available in pure SSD variants if you want.

If your Mac comes equipped with an SSD, Apple’s Disk Utility software won’t actually let you zero the hard drive.

Wait, what?

In a tech note posted to Apple’s own online knowledgebase, Apple explains that you don’t need to securely erase your Mac’s SSD:

With an SSD drive, Secure Erase and Erasing Free Space are not available in Disk Utility. These options are not needed for an SSD drive because a standard erase makes it difficult to recover data from an SSD.

In fact, some folks will tell you not to zero out the data on an SSD, since it can cause wear and tear on the memory cells that, over time, can affect its reliability. I don’t think that’s nearly as big an issue as it used to be — SSD reliability and longevity has improved.

If “Standard Erase” doesn’t quite make you feel comfortable that your data can’t be recovered, there are a couple of options.

FileVault Keeps Your Data Safe

One way to make sure that your SSD’s data remains secure is to use FileVault. FileVault is whole-disk encryption for the Mac. With FileVault engaged, you need a password to access the information on your hard drive. Without it, that data is encrypted.

There’s one potential downside of FileVault — if you lose your password or the encryption key, you’re screwed: You’re not getting your data back any time soon. Based on my experience working at a Mac repair shop, losing a FileVault key happens more frequently than it should.

When you first set up a new Mac, you’re given the option of turning FileVault on. If you don’t do it then, you can turn on FileVault at any time by clicking on your Mac’s System Preferences, clicking on Security & Privacy, and clicking on the FileVault tab. Be warned, however, that the initial encryption process can take hours, as will decryption if you ever need to turn FileVault off.

With FileVault turned on, you can restart your Mac into its Recovery System (by restarting the Mac while holding down the command and R keys) and erase the hard drive using Disk Utility, once you’ve unlocked it (by selecting the disk, clicking the File menu, and clicking Unlock). That deletes the FileVault key, which means any data on the drive is useless.

FileVault doesn’t impact the performance of most modern Macs, though I’d suggest only using it if your Mac has an SSD, not a conventional hard disk drive.

Securely Erasing Free Space on Your SSD

If you don’t want to take Apple’s word for it, if you’re not using FileVault, or if you just want to, there is a way to securely erase free space on your SSD. It’s a little more involved but it works.

Before we get into the nitty-gritty, let me state for the record that this really isn’t necessary to do, which is why Apple’s made it so hard to do. But if you’re set on it, you’ll need to use Apple’s Terminal app. Terminal provides you with command line interface access to the OS X operating system. Terminal lives in the Utilities folder, but you can access Terminal from the Mac’s Recovery System, as well. Once your Mac has booted into the Recovery partition, click the Utilities menu and select Terminal to launch it.

From a Terminal command line, type:

diskutil secureErase freespace VALUE /Volumes/DRIVE

That tells your Mac to securely erase the free space on your SSD. You’ll need to change VALUE to a number between 0 and 4. 0 is a single-pass run of zeroes; 1 is a single-pass run of random numbers; 2 is a 7-pass erase; 3 is a 35-pass erase; and 4 is a 3-pass erase. DRIVE should be changed to the name of your hard drive. To run a 7-pass erase of your SSD drive in “JohnB-Macbook”, you would enter the following:

diskutil secureErase freespace 2 /Volumes/JohnB-Macbook

And remember, if you used a space in the name of your Mac’s hard drive, you need to insert a leading backslash before the space. For example, to run a 35-pass erase on a hard drive called “Macintosh HD” you enter the following:

diskutil secureErase freespace 3 /Volumes/Macintosh\ HD

Something to remember is that the more extensive the erase procedure, the longer it will take.

When Erasing is Not Enough — How to Destroy a Drive

If you absolutely, positively need to be sure that all the data on a drive is irretrievable, see this Scientific American article (with contributions by Gleb Budman, Backblaze CEO), How to Destroy a Hard Drive — Permanently.

The post Getting Rid of Your Mac? Here’s How to Securely Erase a Hard Drive or SSD appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

AWS Certificate Manager Launches Private Certificate Authority

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/aws/aws-certificate-manager-launches-private-certificate-authority/

Today we’re launching a new feature for AWS Certificate Manager (ACM), Private Certificate Authority (CA). This new service allows ACM to act as a private subordinate CA. Previously, if a customer wanted to use private certificates, they needed specialized infrastructure and security expertise that could be expensive to maintain and operate. ACM Private CA builds on ACM’s existing certificate capabilities to help you easily and securely manage the lifecycle of your private certificates with pay as you go pricing. This enables developers to provision certificates in just a few simple API calls while administrators have a central CA management console and fine grained access control through granular IAM policies. ACM Private CA keys are stored securely in AWS managed hardware security modules (HSMs) that adhere to FIPS 140-2 Level 3 security standards. ACM Private CA automatically maintains certificate revocation lists (CRLs) in Amazon Simple Storage Service (S3) and lets administrators generate audit reports of certificate creation with the API or console. This service is packed full of features so let’s jump in and provision a CA.

Provisioning a Private Certificate Authority (CA)

First, I’ll navigate to the ACM console in my region and select the new Private CAs section in the sidebar. From there I’ll click Get Started to start the CA wizard. For now, I only have the option to provision a subordinate CA so we’ll select that and use my super secure desktop as the root CA and click Next. This isn’t what I would do in a production setting but it will work for testing out our private CA.

Now, I’ll configure the CA with some common details. The most important thing here is the Common Name which I’ll set as secure.internal to represent my internal domain.

Now I need to choose my key algorithm. You should choose the best algorithm for your needs but know that ACM has a limitation today that it can only manage certificates that chain up to to RSA CAs. For now, I’ll go with RSA 2048 bit and click Next.

In this next screen, I’m able to configure my certificate revocation list (CRL). CRLs are essential for notifying clients in the case that a certificate has been compromised before certificate expiration. ACM will maintain the revocation list for me and I have the option of routing my S3 bucket to a custome domain. In this case I’ll create a new S3 bucket to store my CRL in and click Next.

Finally, I’ll review all the details to make sure I didn’t make any typos and click Confirm and create.

A few seconds later and I’m greeted with a fancy screen saying I successfully provisioned a certificate authority. Hooray! I’m not done yet though. I still need to activate my CA by creating a certificate signing request (CSR) and signing that with my root CA. I’ll click Get started to begin that process.

Now I’ll copy the CSR or download it to a server or desktop that has access to my root CA (or potentially another subordinate – so long as it chains to a trusted root for my clients).

Now I can use a tool like openssl to sign my cert and generate the certificate chain.


$openssl ca -config openssl_root.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in csr/CSR.pem -out certs/subordinate_cert.pem
Using configuration from openssl_root.cnf
Enter pass phrase for /Users/randhunt/dev/amzn/ca/private/root_private_key.pem:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
stateOrProvinceName   :ASN.1 12:'Washington'
localityName          :ASN.1 12:'Seattle'
organizationName      :ASN.1 12:'Amazon'
organizationalUnitName:ASN.1 12:'Engineering'
commonName            :ASN.1 12:'secure.internal'
Certificate is to be certified until Mar 31 06:05:30 2028 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

After that I’ll copy my subordinate_cert.pem and certificate chain back into the console. and click Next.

Finally, I’ll review all the information and click Confirm and import. I should see a screen like the one below that shows my CA has been activated successfully.

Now that I have a private CA we can provision private certificates by hopping back to the ACM console and creating a new certificate. After clicking create a new certificate I’ll select the radio button Request a private certificate then I’ll click Request a certificate.

From there it’s just similar to provisioning a normal certificate in ACM.

Now I have a private certificate that I can bind to my ELBs, CloudFront Distributions, API Gateways, and more. I can also export the certificate for use on embedded devices or outside of ACM managed environments.

Available Now
ACM Private CA is a service in and of itself and it is packed full of features that won’t fit into a blog post. I strongly encourage the interested readers to go through the developer guide and familiarize themselves with certificate based security. ACM Private CA is available in in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), EU (Frankfurt) and EU (Ireland). Private CAs cost $400 per month (prorated) for each private CA. You are not charged for certificates created and maintained in ACM but you are charged for certificates where you have access to the private key (exported or created outside of ACM). The pricing per certificate is tiered starting at $0.75 per certificate for the first 1000 certificates and going down to $0.001 per certificate after 10,000 certificates.

I’m excited to see administrators and developers take advantage of this new service. As always please let us know what you think of this service on Twitter or in the comments below.

Randall

Stretch for PCs and Macs, and a Raspbian update

Post Syndicated from Simon Long original https://www.raspberrypi.org/blog/stretch-pcs-macs-raspbian-update/

Today, we are launching the first Debian Stretch release of the Raspberry Pi Desktop for PCs and Macs, and we’re also releasing the latest version of Raspbian Stretch for your Pi.

Raspberry Pi Desktop Stretch splash screen

For PCs and Macs

When we released our custom desktop environment on Debian for PCs and Macs last year, we were slightly taken aback by how popular it turned out to be. We really only created it as a result of one of those “Wouldn’t it be cool if…” conversations we sometimes have in the office, so we were delighted by the Pi community’s reaction.

Seeing how keen people were on the x86 version, we decided that we were going to try to keep releasing it alongside Raspbian, with the ultimate aim being to make simultaneous releases of both. This proved to be tricky, particularly with the move from the Jessie version of Debian to the Stretch version this year. However, we have now finished the job of porting all the custom code in Raspbian Stretch to Debian, and so the first Debian Stretch release of the Raspberry Pi Desktop for your PC or Mac is available from today.

The new Stretch releases

As with the Jessie release, you can either run this as a live image from a DVD, USB stick, or SD card or install it as the native operating system on the hard drive of an old laptop or desktop computer. Please note that installing this software will erase anything else on the hard drive — do not install this over a machine running Windows or macOS that you still need to use for its original purpose! It is, however, safe to boot a live image on such a machine, since your hard drive will not be touched by this.

We’re also pleased to announce that we are releasing the latest version of Raspbian Stretch for your Pi today. The Pi and PC versions are largely identical: as before, there are a few applications (such as Mathematica) which are exclusive to the Pi, but the user interface, desktop, and most applications will be exactly the same.

For Raspbian, this new release is mostly bug fixes and tweaks over the previous Stretch release, but there are one or two changes you might notice.

File manager

The file manager included as part of the LXDE desktop (on which our desktop is based) is a program called PCManFM, and it’s very feature-rich; there’s not much you can’t do in it. However, having used it for a few years, we felt that it was perhaps more complex than it needed to be — the sheer number of menu options and choices made some common operations more awkward than they needed to be. So to try to make file management easier, we have implemented a cut-down mode for the file manager.

Raspberry Pi Desktop Stretch - file manager

Most of the changes are to do with the menus. We’ve removed a lot of options that most people are unlikely to change, and moved some other options into the Preferences screen rather than the menus. The two most common settings people tend to change — how icons are displayed and sorted — are now options on the toolbar and in a top-level menu rather than hidden away in submenus.

The sidebar now only shows a single hierarchical view of the file system, and we’ve tidied the toolbar and updated the icons to make them match our house style. We’ve removed the option for a tabbed interface, and we’ve stomped a few bugs as well.

One final change was to make it possible to rename a file just by clicking on its icon to highlight it, and then clicking on its name. This is the way renaming works on both Windows and macOS, and it’s always seemed slightly awkward that Unix desktop environments tend not to support it.

As with most of the other changes we’ve made to the desktop over the last few years, the intention is to make it simpler to use, and to ease the transition from non-Unix environments. But if you really don’t like what we’ve done and long for the old file manager, just untick the box for Display simplified user interface and menus in the Layout page of Preferences, and everything will be back the way it was!

Raspberry Pi Desktop Stretch - preferences GUI

Battery indicator for laptops

One important feature missing from the previous release was an indication of the amount of battery life. Eben runs our desktop on his Mac, and he was becoming slightly irritated by having to keep rebooting into macOS just to check whether his battery was about to die — so fixing this was a priority!

We’ve added a battery status icon to the taskbar; this shows current percentage charge, along with whether the battery is charging, discharging, or connected to the mains. When you hover over the icon with the mouse pointer, a tooltip with more details appears, including the time remaining if the battery can provide this information.

Raspberry Pi Desktop Stretch - battery indicator

While this battery monitor is mainly intended for the PC version, it also supports the first-generation pi-top — to see it, you’ll only need to make sure that I2C is enabled in Configuration. A future release will support the new second-generation pi-top.

New PC applications

We have included a couple of new applications in the PC version. One is called PiServer — this allows you to set up an operating system, such as Raspbian, on the PC which can then be shared by a number of Pi clients networked to it. It is intended to make it easy for classrooms to have multiple Pis all running exactly the same software, and for the teacher to have control over how the software is installed and used. PiServer is quite a clever piece of software, and it’ll be covered in more detail in another blog post in December.

We’ve also added an application which allows you to easily use the GPIO pins of a Pi Zero connected via USB to a PC in applications using Scratch or Python. This makes it possible to run the same physical computing projects on the PC as you do on a Pi! Again, we’ll tell you more in a separate blog post this month.

Both of these applications are included as standard on the PC image, but not on the Raspbian image. You can run them on a Pi if you want — both can be installed from apt.

How to get the new versions

New images for both Raspbian and Debian versions are available from the Downloads page.

It is possible to update existing installations of both Raspbian and Debian versions. For Raspbian, this is easy: just open a terminal window and enter

sudo apt-get update
sudo apt-get dist-upgrade

Updating Raspbian on your Raspberry Pi

How to update to the latest version of Raspbian on your Raspberry Pi. Download Raspbian here: More information on the latest version of Raspbian: Buy a Raspberry Pi:

It is slightly more complex for the PC version, as the previous release was based around Debian Jessie. You will need to edit the files /etc/apt/sources.list and /etc/apt/sources.list.d/raspi.list, using sudo to do so. In both files, change every occurrence of the word “jessie” to “stretch”. When that’s done, do the following:

sudo apt-get update 
sudo dpkg --force-depends -r libwebkitgtk-3.0-common
sudo apt-get -f install
sudo apt-get dist-upgrade
sudo apt-get install python3-thonny
sudo apt-get install sonic-pi=2.10.0~repack-rpt1+2
sudo apt-get install piserver
sudo apt-get install usbbootgui

At several points during the upgrade process, you will be asked if you want to keep the current version of a configuration file or to install the package maintainer’s version. In every case, keep the existing version, which is the default option. The update may take an hour or so, depending on your network connection.

As with all software updates, there is the possibility that something may go wrong during the process, which could lead to your operating system becoming corrupted. Therefore, we always recommend making a backup first.

Enjoy the new versions, and do let us know any feedback you have in the comments or on the forums!

The post Stretch for PCs and Macs, and a Raspbian update appeared first on Raspberry Pi.

New – VPC Endpoints for DynamoDB

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/

Starting today Amazon Virtual Private Cloud (VPC) Endpoints for Amazon DynamoDB are available in all public AWS regions. You can provision an endpoint right away using the AWS Management Console or the AWS Command Line Interface (CLI). There are no additional costs for a VPC Endpoint for DynamoDB.

Many AWS customers run their applications within a Amazon Virtual Private Cloud (VPC) for security or isolation reasons. Previously, if you wanted your EC2 instances in your VPC to be able to access DynamoDB, you had two options. You could use an Internet Gateway (with a NAT Gateway or assigning your instances public IPs) or you could route all of your traffic to your local infrastructure via VPN or AWS Direct Connect and then back to DynamoDB. Both of these solutions had security and throughput implications and it could be difficult to configure NACLs or security groups to restrict access to just DynamoDB. Here is a picture of the old infrastructure.

Creating an Endpoint

Let’s create a VPC Endpoint for DynamoDB. We can make sure our region supports the endpoint with the DescribeVpcEndpointServices API call.


aws ec2 describe-vpc-endpoint-services --region us-east-1
{
    "ServiceNames": [
        "com.amazonaws.us-east-1.dynamodb",
        "com.amazonaws.us-east-1.s3"
    ]
}

Great, so I know my region supports these endpoints and I know what my regional endpoint is. I can grab one of my VPCs and provision an endpoint with a quick call to the CLI or through the console. Let me show you how to use the console.

First I’ll navigate to the VPC console and select “Endpoints” in the sidebar. From there I’ll click “Create Endpoint” which brings me to this handy console.

You’ll notice the AWS Identity and Access Management (IAM) policy section for the endpoint. This supports all of the fine grained access control that DynamoDB supports in regular IAM policies and you can restrict access based on IAM policy conditions.

For now I’ll give full access to my instances within this VPC and click “Next Step”.

This brings me to a list of route tables in my VPC and asks me which of these route tables I want to assign my endpoint to. I’ll select one of them and click “Create Endpoint”.

Keep in mind the note of warning in the console: if you have source restrictions to DynamoDB based on public IP addresses the source IP of your instances accessing DynamoDB will now be their private IP addresses.

After adding the VPC Endpoint for DynamoDB to our VPC our infrastructure looks like this.

That’s it folks! It’s that easy. It’s provided at no cost. Go ahead and start using it today. If you need more details you can read the docs here.

Weekly roundup: Potluck jam

Post Syndicated from Eevee original https://eev.ee/dev/2017/05/09/weekly-roundup-potluck-jam/

Oh hey I did a bunch of stuff.

  • patreon: I wrote the usual monthly update and posted my progress so far on my book for $4 patrons. Also I meticulously tagged all my old posts for some reason, so I guess now you can find photos of Pearl if you want to?

  • blog: I finally fixed the archives page to not be completely flooded by roundups, and added a link to it, which has been missing since the last theme refresh ages ago. Also updated the sidebar to match the front page.

  • flora: I hastily cobbled together a species sheet VN, the first new one in a year or so.

  • music: I made a song. It’s okay. It feels incomplete, probably because I got tired of working on it partway through and gave it a hasty ending. Maybe I’ll iterate on it sometime. I also started mucking around with an attempt at contributing to the Flora glitch album, but I have absolutely no idea what I’m doing.

  • potluck: I came up with a weird idea: ask the general public for sprites, then make a game out of whatever I get. I made a tiny Flask thing for submitting tiles and the results so far are… weird. Nine days to go until I close tile submissions, if you want to give this a shot.

  • dev: I fixed some mGBA bugs with multiple gamepads, woohoo.

  • art: I did some doodling that wasn’t too bad. Mostly I started trying to flesh out the fox flux tileset, which is proving more difficult than expected.

I have some Patreon obligations I oughta get to sooner rather than later; I should probably get a skeleton engine ready for the potluck game, whatever it ends up being; and I’d like to get some more book work done. But at the moment I’m mostly practicing art and trying to make tiles, since the art is going to be the major blocker for expanding fox flux into a somewhat bigger game.

New- Introducing AWS CodeStar – Quickly Develop, Build, and Deploy Applications on AWS

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/new-aws-codestar/

It wasn’t too long ago that I was on a development team working toward completing a software project by a release deadline and facing the challenges most software teams face today in developing applications. Challenges such as new project environment setup, team member collaboration, and the day-to-day task of keeping track of the moving pieces of code, configuration, and libraries for each development build. Today, with companies’ need to innovate and get to market faster, it has become essential to make it easier and more efficient for development teams to create, build, and deploy software.

Unfortunately, many organizations face some key challenges in their quest for a more agile, dynamic software development process. The first challenge most new software projects face is the lengthy setup process that developers have to complete before they can start coding. This process may include setting up of IDEs, getting access to the appropriate code repositories, and/or identifying infrastructure needed for builds, tests, and production.

Collaboration is another challenge that most development teams may face. In order to provide a secure environment for all members of the project, teams have to frequently set up separate projects and tools for various team roles and needs. In addition, providing information to all stakeholders about updates on assignments, the progression of development, and reporting software issues can be time-consuming.

Finally, most companies desire to increase the speed of their software development and reduce the time to market by adopting best practices around continuous integration and continuous delivery. Implementing these agile development strategies may require companies to spend time in educating teams on methodologies and setting up resources for these new processes.

Now Presenting: AWS CodeStar

To help development teams ease the challenges of building software while helping to increase the pace of releasing applications and solutions, I am excited to introduce AWS CodeStar.

AWS CodeStar is a cloud service designed to make it easier to develop, build, and deploy applications on AWS by simplifying the setup of your entire development project. AWS CodeStar includes project templates for common development platforms to enable provisioning of projects and resources for coding, building, testing, deploying, and running your software project.

The key benefits of the AWS CodeStar service are:

  • Easily create new projects using templates for Amazon EC2, AWS Elastic Beanstalk, or AWS Lambda using five different programming languages; JavaScript, Java, Python, Ruby, and PHP. By selecting a template, the service will provision the underlying AWS services needed for your project and application.
  • Unified experience for access and security policies management for your entire software team. Projects are automatically configured with appropriate IAM access policies to ensure a secure application environment.
  • Pre-configured project management dashboard for tracking various activities, such as code commits, build results, deployment activity and more.
  • Running sample code to help you get up and running quickly enabling you to use your favorite IDEs, like Visual Studio, Eclipse, or any code editor that supports Git.
  • Automated configuration of a continuous delivery pipeline for each project using AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy.
  • Integration with Atlassian JIRA Software for issue management and tracking directly from the AWS CodeStar console

With AWS CodeStar, development teams can build an agile software development workflow that now only increases the speed in which teams and deploy software and bug fixes, but also enables developers to build software that is more inline with customers’ requests and needs.

An example of a responsive development workflow using AWS CodeStar is shown below:

Journey Into AWS CodeStar

Now that you know a little more about the AWS CodeStar service, let’s jump into using the service to set up a web application project. First, I’ll go to into the AWS CodeStar console and click the Start a project button.

If you have not setup the appropriate IAM permissions, AWS CodeStar will show a dialog box requesting permission to administer AWS resources on your behalf. I will click the Yes, grant permissions button to grant AWS CodeStar the appropriate permissions to other AWS resources.

However, I received a warning that I do not have administrative permissions to AWS CodeStar as I have not applied the correct policies to my IAM user. If you want to create projects in AWS CodeStar, you must apply the AWSCodeStarFullAccess managed policy to your IAM user or have an IAM administrative user with full permissions for all AWS services.

Now that I have added the aforementioned permissions in IAM, I can now use the service to create a project. To start, I simply click on the Create a new project button and I am taken to the hub of the AWS CodeStar service.

At this point, I am presented with over twenty different AWS CodeStar project templates to choose from in order to provision various environments for my software development needs. Each project template specifies the AWS Service used to deploy the project, the supported programming language, and a description of the type of development solution implemented. AWS CodeStar currently supports the following AWS Services: Amazon EC2, AWS Lambda, and AWS Elastic Beanstalk. Using preconfigured AWS CloudFormation templates, these project templates can create software development projects like microservices, Alexa skills, web applications, and more with a simple click of a button.

For my first AWS CodeStar project, I am going to build a serverless web application using Node.js and AWS Lambda using the Node.js/AWS Lambda project template.

You will notice for this template AWS CodeStar sets up all of the tools and services you need for a development project including an AWS CodePipeline connected with the services; AWS CodeBuild, AWS CloudFormation, and Amazon CloudWatch. I’ll name my new AWS CodeStar project, TaraWebProject, and click Create Project.

Since this is my first time creating an AWS CodeStar, I will see a dialog that asks about the setup of my AWS CodeStar user settings. I’ll type Tara in the textbox for the Display Name and add my email address in the Email textbox. This information is how I’ll appear to others in the project.

The next step is to select how I want to edit my project code. I have decided to edit my TaraWebProject project code using the Visual Studio IDE. With Visual Studio, it will be essential for me to configure it to use the AWS Toolkit for Visual Studio 2015 to access AWS resources while editing my project code. On this screen, I am also presented with the link to the AWS CodeCommit Git repository that AWS CodeStar configured for my project.

The provisioning and tool setup for my software development project is now complete. I’m presented with the AWS CodeStar dashboard for my software project, TaraWebProject, which allows me to manage the resources for the project. This includes the management of resources, such as code commits, team membership and wiki, continuous delivery pipeline, Jira issue tracking, project status and other applicable project resources.

What is really cool about AWS CodeStar for me is that it provides a working sample project from which I can start the development of my serverless web application. To view the sample of my new web application, I will go to the Application endpoints section of the dashboard and click the link provided.

A new browser window will open and will display the sample web application AWS CodeStar generated to help jumpstart my development. A cool feature of the sample application is that the background of the sample app changes colors based on the time of day.

Let’s now take a look at the code used to build the sample website. In order to view the code, I will back to my TaraWebProject dashboard in the AWS CodeStar console and select the Code option from the sidebar menu.

This takes me to the tarawebproject Git repository in the AWS CodeCommit console. From here, I can manually view the code for my web application, the commits made in the repo, the comparison of commits or branches, as well as, create triggers in response to my repo events.

This provides a great start for me to start developing my AWS hosted web application. Since I opted to integrate AWS CodeStar with Visual Studio, I can update my web application by using the IDE to make code changes that will be automatically included in the TaraWebProject every time I commit to the provisioned code repository.

You will notice that on the AWS CodeStar TaraWebProject dashboard, there is a message about connecting the tools to my project repository in order to work on the code. Even though I have already selected Visual Studio as my IDE of choice, let’s click on the Connect Tools button to review the steps to connecting to this IDE.

Again, I will see a screen that will allow me to choose which IDE: Visual Studio, Eclipse, or Command Line tool that I wish to use to edit my project code. It is important for me to note that I have the option to change my IDE choice at any time while working on my development project. Additionally, I can connect to my Git AWS CodeCommit repo via HTTPS and SSH. To retrieve the appropriate repository URL for each protocol, I only need to select the Code repository URL dropdown and select HTTPS or SSH and copy the resulting URL from the text field.

After selecting Visual Studio, CodeStar takes me to the steps needed in order to integrate with Visual Studio. This includes downloading the AWS Toolkit for Visual Studio, connecting the Team Explorer to AWS CodeStar via AWS CodeCommit, as well as, how to push changes to the repo.

After successfully connecting Visual Studio to my AWS CodeStar project, I return to the AWS CodeStar TaraWebProject dashboard to start managing the team members working on the web application with me. First, I will select the Setup your team tile so that I can go to the Project Team page.

On my TaraWebProject Project Team page, I’ll add a team member, Jeff, by selecting the Add team member button and clicking on the Select user dropdown. Team members must be IAM users in my account, so I’ll click on the Create new IAM user link to create an IAM accounts for Jeff.

When the Create IAM user dialog box comes up, I will enter an IAM user name, Display name, and Email Address for the team member, in this case, Jeff Barr. There are three types of project roles that Jeff can be granted, Owner, Contributor, or Viewer. For the TaraWebProject application, I will grant him the Contributor project role and allow him to have remote access by select the Remote access checkbox. Now I will create Jeff’s IAM user account by clicking the Create button.

This brings me to the IAM console to confirm the creation of the new IAM user. After reviewing the IAM user information and the permissions granted, I will click the Create user button to complete the creation of Jeff’s IAM user account for TaraWebProject.

After successfully creating Jeff’s account, it is important that I either send Jeff’s login credentials to him in email or download the credentials .csv file, as I will not be able to retrieve these credentials again. I would need to generate new credentials for Jeff if I leave this page without obtaining his current login credentials. Clicking the Close button returns me to the AWS CodeStar console.

Now I can complete adding Jeff as a team member in the TaraWebProject by selecting the JeffBarr-WebDev IAM role and clicking the Add button.

I’ve successfully added Jeff as a team member to my AWS CodeStar project, TaraWebProject enabling team collaboration in building the web application.

Another thing that I really enjoy about using the AWS CodeStar service is I can monitor all of my project activity right from my TaraWebProject dashboard. I can see the application activity, any recent code commits, and track the status of any project actions, such as the results of my build, any code changes, and the deployments from in one comprehensive dashboard. AWS CodeStar ties the dashboard into Amazon CloudWatch with the Application activity section, provides data about the build and deployment status in the Continuous Deployment section with AWS CodePipeline, and shows the latest Git code commit with AWS CodeCommit in the Commit history section.

Summary

In my journey of the AWS CodeStar service, I created a serverless web application that provisioned my entire development toolchain for coding, building, testing, and deployment for my TaraWebProject software project using AWS services. Amazingly, I have yet to scratch the surface of the benefits of using AWS CodeStar to manage day-to-day software development activities involved in releasing applications.

AWS CodeStar makes it easy for you to quickly develop, build, and deploy applications on AWS. AWS CodeStar provides a unified user interface, enabling you to easily manage your software development activities in one place. AWS CodeStar allows you to choose from various templates to setting up projects using AWS Lambda, Amazon EC2, or AWS Elastic Beanstalk. It comes pre-configured with a project management dashboard, an automated continuous delivery pipeline, and a Git code repository using AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy allowing developers to implement modern agile software development best practices. Each AWS CodeStar project gives developers a head start in development by providing working code samples that can be used with popular IDEs that support Git. Additionally, AWS CodeStar provides out of the box integration with Atlassian JIRA Software providing a project management and issue tracking system for your software team directly from the AWS CodeStar console.

You can get started using the AWS CodeStar service for developing new software projects on AWS today. Learn more by reviewing the AWS CodeStar product page and the AWS CodeStar user guide documentation.

Tara

B2 for Beginners: Inside the B2 Web interface

Post Syndicated from Peter Cohen original https://www.backblaze.com/blog/b2-for-beginners-inside-the-b2-web-interface/

B2 for Beginners

B2 for Beginners: Inside the B2 Web interface

B2 Cloud Storage enables you to store data in the cloud at a fraction of what you’ll pay other services. For instance, we’re one-fourth of the price of Amazon’s S3. We’ve made it easy to access thanks to a web interface, API and command line interface. Let’s get to know the web interface a bit better, because it’s the easiest way to get around B2 and it’s a good way to get a handle on the fundamentals of B2 use.

Anyone with a Backblaze account can set up B2 access by visiting My Settings. Look for Enabled Products and check B2 Cloud Storage.

B2 is accessed the same way as your Backblaze Computer Backup. The sidebar on the left side of your My Account window shows you all the Backblaze services you use, including B2. Let’s go through the individual links under B2 Cloud Storage to get a sense of what they are and what they do.

Buckets

Data in B2 is stored in buckets. Think of a bucket as a top-level folder or directory. You can create as many buckets as you want. What’s more, you can put in as many files as you want. Buckets can contain files of any type or size.

Buckets

Third-party applications and services can integrate with B2, and many already do. The Buckets screen is where you can get your Account ID information and create an application key – a unique identifier your apps will use to securely connect to B2. If you’re using a third-party app that needs access to your bucket, such as a NAS backup app or a file sync tool, this is where you’ll find the info you need to connect. (We’ll have more info about how to backup your NAS to B2 very soon!)

The Buckets window lists the buckets you’ve created and provides basic information including creation date, ID, public or private type, lifecycle information, number of files, size and snapshots.

Click the Bucket Settings link to adjust each bucket’s individual settings. You can specify if files in the bucket are public or private. Private files can’t be shared, while public ones can be.

You can also tag your bucket with customized information encoded in JSON format. Custom info can contain letters, numbers, “-” and “_”.

Browse Files

Click the Upload/Download button to see a directory of each bucket. Alternately, click the Browse Files link on the left side of the B2 interface.

Browse Files

You can create a new subdirectory by clicking the New Folder button, or begin to upload files by clicking the Upload button. You can drag and drop files you’d like to upload and Backblaze will handle that for you. Alternately, clicking on the dialog box that appears will enable you to select the files on your computer you’d like to upload.

Info Button

Next to each individual file is an information button. Click it for details about the file, including name, location, kind, size and other details. You’ll also see a “Friendly URL” link. If the bucket is public and you’d like to share this file with others, you may copy that Friendly URL and paste it into an email or message to let people know where to find it.

Download

You can download the contents of your buckets by clicking the checkbox next to the filename and clicking the Download button. You can also delete files and create snapshots. Snapshots are helpful if you want to preserve copies of your files in their present state for some future download or recovery. You can also create a snapshot of the full bucket. If you have a large snapshot, you can order it as a hard drive instead of downloading it. We’ll get more into snapshots in a future blog post.

Lifecycle Settings

We recently introduced Lifecycle Settings to keep your buckets from getting cluttered with too many versions of files. Our web interface lets you manage these settings for each individual bucket.

Lifecycle Rules

By default, the bucket’s lifecycle setting is to keep all versions of files you upload. The web interface lets you adjust that so B2 only keeps the last file version, keeps the last file for a specific number of days, or keeps files based on your own custom rule. You can determine the file path, the number of days until the file is hidden, and the number of days until the file is deleted.

Lifecycle Rules

Reports

Backblaze updates your account daily with details on what’s happening with your B2 files. These reports are accessible through the B2 interface under the Reports tab. Clicking on reports will reveal an easy to understand visual charge showing you the average number of GB stored, total GB downloaded and total number of transactions for the month.

Reports

Look further down the page for a breakdown of monthly transactions by type, along with charts that help you track average GB stored, GB downloaded and count of average stored files for the month.

Caps and Alerts

One of our goals with B2 was to take the surprise out cloud storage fees. The B2 web GUI sports a Caps & Alerts section to help you control how much you spend on B2.

Caps & Alerts

This is where you can see – and limit – daily storage caps, daily downloads, and daily transactions. “Transactions” are interactions with your account like creating a new bucket, listing the contents of a bucket, or downloading a file.

You can make sure to send those alerts to your cell phone and email, so you’ll never be hit with an unwelcome surprise in the form of an unexpected bill. The first 10 GB of storage is free, with unlimited free uploads and 1 GB of free downloads each day.

Edit Caps

Click the Edit Caps button to enter dollar amount limits for storage, download bandwidth, Class B and Class C transactions separately (or specify No Cap if you don’t want to be encumbered). This way, you maintain control over how much you spend with B2.

And There’s More

That’s an overview of the B2 web GUI to help you get started using B2 Cloud Storage. If you’re more technical and are interested in connecting to B2 using our API instead, make sure to check out our B2 Starter Guide for a comprehensive overview of what’s under the hood.

Still have questions about the B2 web GUI, or ideas for how we can make it better? Fire away in the comments, we want to hear from you!

The post B2 for Beginners: Inside the B2 Web interface appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Firefox 47

Post Syndicated from ris original http://lwn.net/Articles/690154/rss

Firefox 47 has been released. This version enables the VP9 video codec for
users with fast machines, plays embedded YouTube videos with HTML5 video if
Flash is not installed, and more. There is a blog
post
about these and other improvements. “Now, we are making it
even easier to access synced tabs directly in your desktop Firefox
browser. If you’re logged into your Firefox Account, you will see all open
tabs from your smartphone or other computers within the sidebar. In the
sidebar you can also search for specific tabs quickly and easily.

See the release
notes
for more information.

Twitter’s missing manual

Post Syndicated from Eevee original https://eev.ee/blog/2016/02/20/twitters-missing-manual/

I mentioned recently, buried in a post about UI changes, that Twitter’s latest earnings report included this bombshell:

We are going to fix the broken windows and confusing parts, like the [email protected] syntax and @reply rules, that we know inhibit usage and drive people away

There’s an interesting problem here. UI is hard. You can’t just slap a button on the screen for every feature that could conceivably be used at any given time. Some features are only of interest to so-called “power users”, so they’re left subtle, spread by word-of-mouth. Some features you try to make invisible and heuristic. Some features are added just to solve one influential user’s problem. Some features are, ah, accidental.

A sufficiently mature, popular, and interesting product thus tends to accumulate a small pile of hidden features, sometimes not documented or even officially acknowledged. I’d say this is actually a good thing! Using something for a while should absolutely reward you with a new trick every so often — that below-the-surface knowledge makes you feel involved with the thing you’re using and makes it feel deeper overall.

The hard part is striking a balance. On one end of the spectrum you have tools like Notepad, where the only easter egg is that pressing F5 inserts the current time. On the other end you have tools like vim, which consist exclusively of easter eggs.

One of Twitter’s problems is that it’s tilted a little too far towards the vim end of the scale. It looks like a dead-simple service, but those humble 140 characters have been crammed full of features over the years, and the ways they interact aren’t always obvious. There are rules, and the rules generally make sense once you know them, but it’s also really easy to overlook them.

Here, then, is a list of all the non-obvious things about Twitter that I know. Consider it both a reference for people who aren’t up to their eyeballs in Twitter, and an example of how these hidden features can pile up. I’m also throwing in a couple notes on etiquette, because I think that’s strongly informed by the shape of the platform.

Text

  • Tweets are limited to 140 Unicode characters, meaning that even astral plane characters (such as emoji) only count as one.

  • Leading and trailing whitespace is stripped from tweets.

  • Tweets may contain newlines, and there doesn’t seem to be any limit to how many.

  • In the middle of a tweet, strings of whitespace (e.g. multiple spaces) are preserved. However, more than two consecutive newlines will be reduced to only two.

  • Anything remotely resembling a link will be mangled into some http://t.co/asdf link-shortened garbage. In some cases, such as when talking about a domain name, this can make the tweet longer. You can defeat this by sticking an invisible character, such as U+200D ZERO WIDTH JOINER, around the final dot so it no longer looks like a domain name.

    In official clients, links are shown unmangled, but without the protocol and truncated to about 20 characters. The link to this article, for example, shows as eev.ee/blog/2016/02/2…. However, at least on Web Twitter, copy-pasting preserves the link in full, including protocol.

    Note that Twitter’s knowledge of domains is not exhaustive — it will link “google.com” but not “eev.ee”.

  • For the sake of its SMS-based roots, Twitter supports performing several commands by typing them in a tweet. In particular, if you start a tweet with the word d or m or dm, the second word will be treated as a username, and the rest of the tweet will be DM’d to that user.

  • Accounts managed by multiple people, such as support accounts or politicians’ accounts, sometimes sign tweets with a ^ followed by the author’s initials. This has no special significance to Twitter.

  • You cannot use astral plane characters (which includes most emoji) in your display name or bio; they will be silently stripped. However, you can use anything from the Miscellaneous Symbols or Dingbats blocks, and many of these characters are rendered with color glyphs on Web Twitter. Results may vary on phones and in other clients.

Replies and mentions

A tweet can “mention” other users, which just means including their @handle somewhere in the tweet. This will notify every mentioned user of the tweet.

You can reply to tweets, which threads them together. A tweet can only have one parent (or no parent), but any number of replies. Everything on Twitter is thus arranged into a number of trees, where the root of the tree is a new tweet not responding to anything, and replies branch out from there.

  • A tweet that begins with a mention — that is, the very first character is @ and it’s immediately followed by an extant username — won’t appear on your profile on Web Twitter. It’ll still appear on the “with replies” page. It’ll also appear on your profile on Android Twitter, which doesn’t separate replies from not.

  • A “mention” can only be an existing username. If “foo” is not the name of a user, then @foo is not a mention, and none of the rules for mentions apply.

  • A tweet that begins with a mention won’t appear on the timelines of anyone who follows you, unless they also follow the first person you mention. That is, if you tweet @foo @bar heya, it’ll only appear on the timelines of people who follow both you and @foo.

  • If you put some other character before the first @, the previous rule no longer applies, and your tweet will appear to all your followers. So [email protected] @bar heya will be visible to everyone (and show on your Web profile). This is called “dot-replying”. The dot isn’t actually special; it’s just an easy-to-type and unobtrusive character. I like to use or . Some people prefer to put the mentions at the end instead, producing heya @foo @bar.

    Dot-replying in the middle of a tweet is not strictly necessary, but sometimes it’s useful for disambiguation. If you’re replying to @foo, and want to say something about @bar, it would come out as @foo @bar is pretty great, which is a little difficult to read. Adding a dot to make @foo [email protected] doesn’t do anything as far as Twitter is concerned, but can make it clear that @bar is the subject of a sentence rather than a person being talked to.

  • One last visibility wrinkle: if a tweet appears in your timeline because it begins with a mention of someone you follow, the tweet it replies to will also appear, even if it wouldn’t have on its own.

    Consider three users: A, B, and C. You follow B and C, but not A. User A makes a tweet, and B replies to it (@A cool!). Neither tweet will appear on your timeline — A’s won’t because you don’t follow A, and B’s won’t because it begins with a mention of someone you don’t follow. Then C replies to B’s tweet, which puts the mention of B first (see below). Both B’s and C’s tweets will now appear on your timeline — C’s appears because it begins with a mention of someone you do follow, and B’s appears for context.

    In other words, even if a tweet doesn’t appear on your timeline initially, it may show up later due to the actions of a third party.

  • A reply must, somewhere, mention the author of the tweet it’s replying to. If you reply to a tweet and delete the author’s @handle, it’ll become a new top-level tweet rather than a reply. You can see this in some clients, like Android Twitter: there’s “replying to (display name)” text indicating it’s a reply, and that text disappears if you delete the @handle.

  • There is one exception to the previous rule: if you’re replying to yourself, you don’t have to include your own @handle, even though clients include it by default. So if you want to say something that spans multiple tweets, you can just keep replying to yourself and deleting the @handle. (This is sometimes called a “tweetstorm”.)

    It’s a really good idea to do this whenever you’re making multiple tweets about something. Otherwise, someone who stumbles upon one of the tweets later will have no idea what the context was, and won’t be able to find it without scrolling back however long on your profile.

    If you reply to yourself but leave your @handle at the beginning, the first tweet will appear on your profile, but the others won’t, because they start with a mention. On the other hand, letting the entire tweetstorm appear on your profile can be slightly confusing — the individual tweets will appear in reverse order, because tweets don’t appear threaded on profiles. On Web Twitter, at least, the followups will have a “view conversation” link that hints that they’re replies.

    Either way, the replies will still appear on your followers’ timelines. Even if the replies begin with your @handle, they still begin with a mention of someone your followers all follow: you!

    I’m told that many third-party clients don’t support replying to yourself without your handle included, and the API documentation doesn’t mention that it’s a feature. But I’m also told that only first-party clients require you to mention someone in a reply in order to thread, and that third-party clients will merrily thread anything to anything. (I remember when Web Twitter allowed that, so I totally believe the API still does.) If you don’t use the official clients, I guess give it a whirl and see what happens.

  • The previous rule also applies when making longer replies to someone else. Reply to them once, then reply to yourself with the next tweet (and remove your own @handle). You’ll end up with three tweets all threaded together.

    This is even more important, because Twitter shows the replies to a single tweet in a somewhat arbitrary order, bubbling “important” ones to the top. If you write a very long response and break it across three tweets, all replying to the same original tweet, they’ll probably show as an incoherent jumble to anyone reading the thread. If you make each tweet a reply to the previous one, they’re guaranteed to stay in order.

  • Replying to a tweet will also prefill the @handle of anyone mentioned in the tweet. Replying to a retweet will additionally prefill the @handle of the person who retweeted it. The original author’s @handle always appears first. In some cases, it’s polite to remove some of these; you only need the original author’s @handle to make a reply. (It’s not uncommon to accumulate multiple mentions, then end up in an extended conversation with only one other person, while constantly notifying several third parties. Or you may want to remove the @handle of a locked account that retweeted a public account, to protect their privacy.)

  • Prefilling @handles is done client-side, so some clients have slightly different behavior. In particular, I’ve seen a few people who reply to their own reply to someone else (in order to thread a longer thought), and end up with their own @handle at the beginning of the second reply! You probably don’t want that, because now the second reply begins with a mention of someone all of your followers follow — yourself! — and so both tweets will appear on your followers’ timelines.

  • In official clients (Web and Android, at least), long threads of tweets are collapsed on your timeline. Only the first tweet and the last two tweets are visible. If you have a lot to say about something, it’s a good idea to put the important bits in one of those three tweets where your followers will actually see them. This is another reason it’s polite to thread your tweets together — it saves people from having their timelines flooded by your tweetstorm.

    Sometimes, it’s possible to see multiple “branches” of the same conversation on your timeline. For example, if A makes a few tweets, and B and C both reply, and you follow all three of them, then you’ll see B’s replies and C’s replies separately. Clients don’t handle this particularly well and it can become a bit of a clusterfuck, with the same root tweet appearing multiple times.

  • Because official clients treat a thread as a single unit, you can effectively “bump” your own tweet by replying to it. Your reply is new, so it’ll appear on your followers’ timelines; but the client will also include the first tweet in the thread as context, regardless of its age.

  • When viewing a single tweet, official clients may not show the replies in chronological order. Usually the “best” replies are bumped to the top. “Best” is entirely determined by Twitter, but it seems to work fairly well.

    If you reply to yourself, your own replies will generally appear first, but this is not guaranteed. If you want to link to someone else’s long chain of tweets, it’s safest to link to the last tweet in the thread, since there can only be one unambiguous trail of parent tweets leading back to the beginning. This also saves readers from digging through terrible replies by third parties.

  • If reply to a tweet with @foo heya, and @foo later renames their account to @quux, the tweet will retain its threading even though it no longer mentions the author of the parent tweet. However, your reply will now appear on your profile, because it doesn’t begin with the handle of an existing user. Note that this means it’s fairly easy for a non-follower to figure out what you renamed your account to, by searching for replies to your old name.

  • Threads are preserved even if some of the tweets are hidden (either because you’ve blocked some participants, or because they have their accounts set to private). Those tweets won’t appear for you, but any visible replies to them will.

  • If a tweet in the middle of a thread is deleted (or the author’s account is deleted), the thread will break at that point. Replies to the deleted tweet won’t be visible when looking at the parent, and the parent won’t be visible when looking at the replies.

  • You can quote tweets by including a link to them in your tweet, which will cause the quoted tweet to appear in a small box below yours. This does not create a reply and will not be part of the quoted tweet’s thread. If you want to do that, you can’t use the retweet/quote button. You have to reply to the tweet, manually include a link to it, and be sure to mention the author.

  • When you quote a tweet, the author is notified; however, unlike a retweet, they won’t be notified when people like or retweet your quote (unless you also mention them). If you don’t want to notify the author, you can take a screenshot (though this doesn’t let them delete the tweet) or use a URL shortener (though this doesn’t let them obviously disable a quote by blocking you).

  • Due to the nature of Twitter, it’s common for a tweet to end up on many people’s timelines simultaneously and attract many similar replies within a short span of time. It’s polite to check the existing replies to a popular tweet, or a tweet from a popular person, before giving your two cents.

  • It’s generally considered rude to barge into the middle of a conversation between two other people, especially if they seem to know each other much better than you know them, and especially if you’re being antagonistic. There are myriad cases where this may be more or less appropriate, and no hard and fast rules. You’re a passerby overhearing two people talking on the street; act accordingly.

  • Someone unrecognized who replies to you — especially someone who doesn’t follow you, or who is replying to the middle of a conversation, or who is notably arrogant or obnoxious — is often referred to as a “rando”.

  • When you quote or publicly mention someone for the sake of criticizing them, be aware that you’re exposing them to all of your followers, some of whom may be eager for an argument. If you have a lot of followers, you might inadvertently invite a dogpile.

Hashtags

Hashtags are a # character followed by some number of non-whitespace characters. Anecdotally, they seem to be limited server-side to 100 characters, but I haven’t found any documentation on this.

  • Exactly which characters may appear in a hashtag is somewhat inconsistent, and has quietly changed at least once.

  • The only real point to hashtags is that you can click on them in clients to jump directly to search results. Note that searching for #foo will only find #foo, but searching for foo will find both foo and #foo.

  • Hashtags can appear in the “trending” widget, but so can any other regular text.

  • There is no reason to tag a bunch of random words in your tweets. No one is searching Twitter for #funny. Doing this makes you look like an out-of-touch marketer who’s trying much too hard.

  • People do sometimes use hashtags as “asides” or “moods”, but in this case the tag isn’t intended to be searched for, and the real point of using a hashtag is that the link color offsets it from the main text.

  • Twitter also supports “cashtags”, which are prefixed with a $ instead and are generally stock symbols. I only even know this because it makes shell and Perl code look goofy.

Media

A tweet may have one embedded attachment.

  • You may explicitly include a set of up to four images or a video or a poll. You cannot combine this within a single tweet. Brands™ have access to a handful of other embedded gizmos.

  • If you include images or a video, you will lose 24 characters of writing space, because a direct link to the images/video will be silently added to the end of your tweet. This is for the sake of text-only clients, e.g. people using Twitter over SMS, so they can see that there’s an attachment and possibly view it in a browser.

  • Including a poll will not append a link, but curiously, you’ll still lose 24 characters. It’s possible this is a client bug, but it happens in both Web and Android Twitter.

  • Alternative clients may not support new media types at first. In particular, people who used TweetDeck were frequently confused right after polls were launched, because TweetDeck showed only the tweet text and no indication that a poll had ever been there. Some third-party clients still don’t support polls. Consider mentioning when you’re using a new attachment type. Might I suggest prefixing your tweet with 📊?

  • If you don’t include an explicit attachment, Twitter will examine the links in your tweet, in reverse order. If you link to a tweet, that tweet will be quoted in yours. If you link to a website that supports Twitter “cards” (small brief descriptions of a site, possibly with images), that card will be attached. There can only be one attachment, so as soon as Twitter finds something it can use, it stops looking.

  • You can embed someone else‘s media in your own tweet by ending it with a link to the media URL — that is, the one that ends with /photo/1. This is different from a quoted tweet, and won’t notify the original tweeter.

  • Quoted tweets are always just tweets that include links to other tweets. Even if the tweet is deleted, an embed box will still appear, though it’ll be grayed out and say the tweet is unavailable.

    If the link is the last thing to appear in the tweet text, official clients will not show the link. This can be extremely confusing if you try to link to two tweets — the first one will be left as a regular link, and the second one will be replaced by a quoted tweet, so at a glance it looks like you linked to a tweet and it was also embedded. A workaround for this is just to add text after the final link, so it’s not the last thing in the tweet and thus isn’t hidden.

  • Twitter cards may be associated with a Twitter account. On Android Twitter (not Web Twitter!), replying to a tweet with a card will also include the @handle for the associated account. For example, replying to a tweet that links to a YouTube video will prefill @YouTube. This is pretty goofy, since YouTube itself didn’t make the video, and it causes replies to notify the person even though the original link doesn’t.

  • Uploaded media may be flagged as “sensitive”, which generally means “pornographic”. This will require viewers to click through a warning to see the media, unless they’re logged in and have their account set to skip the warning. Flagged media also won’t appear in the sidebar on profile pages for people who have the warning enabled.

  • The API supports marking individual tweets as containing sensitive media, but official clients do not — instead, there’s an account setting that applies to everything you upload from that point forward. Media may also be flagged by other users as sensitive. Twitter also has some sort of auto-detection for sensitive media, which I only know about because it sometimes thinks photos of my hairless cats are pornographic.

  • If your own tweets have “sensitive” media attached, you will have to click through the warning, even if you have the warning disabled. A Twitter employee tells me this is so you’re aware when your own tweets are flagged, but the message still tells you to disable the warning in account settings, so this is mostly just confusing.

    Curiously, if you see your own tweet via a retweet, the warning doesn’t appear.

Blocking and muting

  • A blocked user cannot view your profile. They can, of course, use a different account, or merely log out. This is entirely client-side, too, so it’s possible that some clients don’t even support this “feature”.

  • A blocked user cannot like or retweet your tweets.

  • A blocked user cannot follow you. If you block someone who’s already following you, they’ll be forced to immediately unfollow. Likewise, you cannot follow a blocked user.

  • A blocked user’s tweets won’t appear on your timeline, or in any thread. As of fairly recently, their tweets won’t appear in search results, either. However, if you view the profile of someone who’s retweeted a blocked user, you will still see that retweet.

  • A blocked user can see your tweets, if someone they follow retweets you.

  • A blocked user can mention or reply to you, though you won’t be notified either by the tweet itself or by any retweets/likes. However, if someone else replies to them, your @handle will be prefilled, and you’ll be notified. Also, other people viewing your tweets will still see their replies threaded.

  • A blocked user can link to your tweets — however, rather than an embedded quote, their tweet will have a gray “this tweet is unavailable” box attached. This effect is retroactive. However (I think?), if a quoted tweet can’t be shown, the link to the tweet is left visible, so people can still click it to view the tweet manually.

  • Muting has two different effects. If you mute someone you’re following, their tweets won’t appear in your timeline, but you’ll still get notifications from them. This can be useful if you set your phone to only buzz on notifications from people you follow. If you mute someone you’re not following, nothing they do will send you notifications. Either way, their tweets will still be visible in threads and search results.

  • Relatedly, if you follow someone who’s a little eager with the retweeting, you can turn off just their retweets. It’s in the menu on their profile.

  • It’s trivial to tell whether someone’s blocked you, since their profile will tell you. However, it’s impossible to know for sure if someone has muted you or is just manually ignoring you, since being muted doesn’t actually prevent you from doing anything.

  • You can block and mute someone at the same time, though this has no special effect. If you unblock them, they’ll just still be muted.

  • The API strips out tweets from blocked and muted users server-side for streaming requests (such as your timeline), but leaves it up to the client for other requests (such as viewing a single tweet). So it’s possible that a client will neglect to apply the usual rule of “you never see a blocked user’s tweets in threads”. In particular, I’ve heard several reports that this is the case in the official iOS Twitter (!).

  • Tweeting screenshots of “you have been blocked” is getting pretty old and we can probably stop doing it.

  • Almost all of Twitter’s advanced search options are exposed on the advanced search page. All of them are shorthand for using a prefix in your search query; for example, “from these accounts” just becomes something like from:username.

  • The one that isn’t listed there is filter:, which is only mentioned in the API documentation. It can appear as filter:safe, filter:media, filter:images, or filter:links. It’s possible there are other undocumented forms.

  • Search applies to unshortened links, so you can find links to a website just by searching for its URL. However, because Twitter displays links without a protocol (http://), you have to leave it off when searching. Be aware that people who mention your work without mentioning you might be saying unkind things about it.

    That said, I’ve also run into cases where searching for a partial URL doesn’t find tweets that I already know exist, and I’m not sure why.

  • As a side effect, you can search for quotes of a given user’s tweets by searching for twitter.com/username/status, because all tweet URLs begin with that prefix. This will also include any tweets from that user that have photos or video attached, because attaching media appends a photo URL, but you can fix that by adding -from:username.

  • Searching for to:foo will only find tweets that begin with @foo; dot-replies and other mentions are not included. Searching for @foo will find mentions as well as tweets from that person. To find only someone’s mentions, you can search for @foo -from:foo. You can combine this with the above trick to find quotes as well.

  • I’ve been told that from: only applies to the handle a user had when the tweet was made (i.e. doesn’t take renames into account), but this doesn’t match my own experience. It’s possible the behavior is different depending on whether the old handle has been reclaimed by someone else.

  • Some clients, such as TweetDeck, support showing live feeds of search results right alongside your timeline and notifications. It’s therefore possible for people to keep an eye on a live stream of everyone who’s talking about them, even when their @handle isn’t mentioned. Bear this in mind when grumbling, especially about people whose attention you’d prefer to avoid.

  • Namesearch — that is, look for mentions of you or your work that don’t actually @-mention you — with caution. Liking or replying amicably to tweets that compliment you is probably okay. Starting arguments with people who dislike your work is rude and kind of creepy, and certainly not likely to improve anyone’s impression of you.

Locked accounts

  • You may set your account to private, which will hide your tweets from the general public. Only people who follow you will be able to see your tweets. Twitter calls this “protected”, but since it shows a lock icon next to your handle, everyone calls it “locked”.

  • Specifically: your banner, avatar, display name, and bio (including location, website, etc.) are still public. The number of tweets, follows, followers, likes, and lists you have are also public. Your actual tweets, media, follows, followers, lists, etc. are all hidden.

  • iOS Twitter hides the bio and numbers, as well, which is sort of inconvenient if you were using it to explain who you are and who you’re cool with having follow you.

  • When you lock your account, any existing followers will remain. Anyone else will only be able to send a follow request, which you can then approve or deny. You can force anyone to unfollow you at any time (whether locked or not) by blocking and then unblocking them. Or just blocking them.

  • Follow requests are easy to miss; only a few places in the UI make a point of telling you when you have new ones.

  • Approving or denying a follow request doesn’t directly notify the requester. If you approve, obviously they’ll start seeing your tweets in their timeline. If you deny, the only difference is that if they look at your profile again, the follow button will no longer say “pending”.

  • If you unlock your account, any pending follow requests are automatically accepted.

  • The only way to see your pending follows (accounts you have tried to follow that haven’t yet accepted) is via the API, or a client that makes use of the API. The official clients don’t show them anywhere.

  • No one can retweet a locked account, not even followers.

  • Quoting doesn’t work with locked account; the quoted tweet will only show the “unavailable” message, even if a locked account quotes itself. Clicking the tweet link will still work, of course, as long as you follow the quoted account.

  • Locked accounts never create notifications for people who aren’t following them. A locked account can like, retweet, quote, follow, etc. as usual, and the various numbers will go up, but only their followers will be notified.

  • A locked account can reply to another account that doesn’t follow them, but that account won’t have any way to tell. However, an unlocked third party that follows both accounts could then make another reply, which would prefill both @handles, and (if left unchanged) alert the other account to the locked account’s presence.

  • Similarly, if a locked account retweets a public account, anyone who tries to reply to the retweet will get the locked account’s @handle prefilled.

  • If a locked account likes some of your tweets (or retweets you, or replies, etc.), and then you follow them, you won’t see retroactive notifications for that activity. Notifications from accounts you can’t see are never created in the first place, not merely hidden from your view. Of course, you’re free to look through their tweets and likes manually once you follow them.

  • Locked accounts do not appear in the lists of who liked or retweeted a tweet (except, of course, when viewed by someone following them). Web Twitter will hint at this by saying something akin to “X users have asked not to be shown in this view.” at the bottom of such a list.

  • While a locked account’s own follows and followers are hidden, a locked account will still appear publicly in the following/follower lists of other unlocked accounts. There is no blessed way to automatically cross-reference this, but be aware that the existence of a locked account is still public. In particular, if you follow someone who keeps an eye on their follower count, they can just look at their own list of followers to find you.

  • Anyone can still mention a locked account, whether or not they follow it, and it’ll receive notifications.

  • Open DMs (“receive direct messages from anyone”) work as normal for locked accounts. A locked account can send DMs to anyone with open DMs, and a locked account may turn on open DMs to receive DMs from anyone.

  • Replies to a locked account are not protected in any way. If a locked account participates in a thread, its own tweets will be hidden from non-followers, but any public tweets will be left in. Also, anyone can search for mentions of a locked account to find conversations it’s participated in, and may be able to infer what the locked account was saying from context.

API, other clients, etc.

I’ve mentioned issues with non-primary clients throughout, but a couple more things to be aware of:

  • Web Twitter has some keyboard shortcuts, which you can view by pressing ?.

  • When I say Web Twitter throughout this document, I mean desktop Web Twitter; there’s also a mobile Web Twitter, which is much simpler.

  • The official API doesn’t support a number of Twitter features, including polls, ads, and DMs with multiple participants. Clients that use the API (i.e. clients not made by Twitter) thus cannot support these features.

  • Even TweetDeck, which is maintained by Twitter, frequently lags behind in feature support. TweetDeck had the original (client-side-only) implementation of muting, but even after Twitter added it as a real feature, TweetDeck was never changed to make use of it. So TweetDeck’s muting is separate from Twitter’s muting.

  • Tweets know what client they were sent from. Official Twitter apps don’t show this any more, but it’s still available in the API, and some alternative clients show it.

  • By default, Twitter allows people to find your account by searching for your email address or phone number. You may wish to turn this off.

  • Twitter has a “collections” feature, which lets you put any public tweets you like (even other people’s) in a group for other people to look over. However, no primary client lets you create one; you have to do it via the API, the second-party client TweetDeck, the somewhat convoluted Curator that seems more aimed at news media and business, or a third-party client. Collections aren’t listed anywhere public (you have to link to them directly) — the only place to see even a list of your own collections via primary means is the “Collection” tab when creating a new widget on the web. Tweets in a collection are by default shown in the order you added them, newest first; the API allows reordering them, and Curator supports dragging to reorder, but TweetDeck doesn’t support reordering at all.

  • Lists are a thing. I’ve never really used them. They don’t support a lot of the features the regular timeline does; for example, threaded tweets aren’t shown together, and lists don’t provide access to locked accounts. You can create a private list and add people to it to follow them without their knowledge, though.

  • You can “promote” a tweet, i.e. turn it into an ad, which is generally only of interest to advertisers. However, promoted tweets have the curious property that they don’t appear on your profile or in your likes or in search results for anyone. It’s possible to target a promoted tweet at a specific list of users (or no one!), which allows for a couple creative hacks that you’ll have to imagine yourself.

  • And then there’s the verified checkmark (given out arbitrarily), the power tools given to verified users (mysterious), the limits on duplicate tweets and follows and whatnot (pretty high), the analytics tools (pretty but pointless), the secret API-only notifications (Twitter tells you when your tweet is unfavorited!), the Web Twitter metadata that let me write a hack to hide mentions from non-followers… you get the idea.

How to Govern Your Application Deployments by Using Amazon EC2 Container Service and Docker

Post Syndicated from Michael Capicotto original https://blogs.aws.amazon.com/security/post/Tx3UTL7PQ6796V5/How-to-Govern-Your-Application-Deployments-by-Using-Amazon-EC2-Container-Service

Governance among IT teams has become increasingly challenging, especially when dealing with application deployments that involve many different technologies. For example, consider the case of trying to collocate multiple applications on a shared operating system. Accidental conflicts can stem from the applications themselves, or the underlying libraries and network ports they rely on. The likelihood of conflicts is heightened even further when security functionality is involved, such as intrusion prevention or access logging. Such concerns have typically resulted in security functions being relegated to their own independent operating systems via physical or virtual hardware isolation (for example, an inline firewall device).

In this blog post, I will show you how to eliminate these potential conflicts while also deploying your applications in a continuous and secure manner at scale. We will do this by collocating different applications on the same operating system, through the use of Amazon EC2 Container Service (ECS) and Docker.

Let’s start with a brief overview of Docker.

Docker explained

Simply put, Docker allows you to create containers, which wrap your applications into a complete file system and contain everything that this software needs to run. This means you can transport that container onto any environment running Docker, and it will run the same, while staying isolated from other containers and the host operating system. This isolation between containers eliminates any potential conflicts the applications may have with each other, because they are each running in their own separate run-time environments.

The hands-on portion of this post will focus on creating two Docker containers: one containing a simple web application (“application container”), and the other containing a reverse proxy with throttling enabled (“proxy container”), which is used to protect the web application. These containers will be collocated on the same underlying Amazon EC2 instance using ECS; however, all network traffic between the web application and the outside world will be forced through the proxy container, as shown in the following diagram. This tiered network access can be the basis of a security overlay solution in which a web application is not directly reachable from the network to which its underlying instance is connected. All inbound application requests are forced through a proxy container that throttles requests. In practice, this container could also perform activities such as filtering, logging, and intrusion detection.

Figure 1. Network isolation using Docker containers

Create your Docker containers

To start let’s create a Docker container that contains a simple PHP web application. This Docker container will represent the application container in the previous diagram. For a more detailed guide to the following steps, see Docker Basics.

Install Docker

Launch an instance with the Amazon Linux AMI. For more information, see Launching an Instance in the Amazon EC2 User Guide for Linux Instances.

Connect to your instance. For more information, see Connect to Your Linux Instance.

Update the installed packages and package cache on your instance:

[ec2-user ~]$ sudo yum update -y

Install Docker:

[ec2-user ~]$ sudo yum install -y docker

Start the Docker service:

[ec2-user ~]$ sudo service docker start

Add the ec2-user to the Docker group so that you can execute Docker commands without using sudo:

[ec2-user ~]$ sudo usermod -a -G docker ec2-user

Log out and log back in again to pick up the new Docker group permissions.
 

Verify that the ec2-user can run Docker commands without sudo:

[ec2-user ~]$ docker info

Sign up for a Docker Hub account

Docker uses images to launch containers, and these images are stored in repositories. The most common Docker image repository (and the default repository for the Docker daemon) is Docker Hub. Although you don’t need a Docker Hub account to use ECS or Docker, having a Docker Hub account gives you the freedom to store your modified Docker images so that you can use them in your ECS task definitions. Docker Hub offers public and private registries. You can create a private registry on Docker Hub and configure private registry authentication on your ECS container instances to use your private images in task definitions.

For more information about Docker Hub and to sign up for an account, go to https://hub.docker.com.

Create a Docker image containing a simple PHP application

Install git and use it to clone the simple PHP application from our GitHub repository onto your system:

[ec2-user ~]$ sudo yum install -y git

[ec2-user ~]$ git clone https://github.com/awslabs/ecs-demo-php-simple-app

Change directories to the ecs-demo-php-simple-app folder:

[ec2-user ~]$ cd ecs-demo-php-simple-app

Examine the Dockerfile in this folder:

[ec2-user ecs-demo-php-simple-app]$ cat Dockerfile

​A Dockerfile is a manifest that contains instructions about how to build your Docker image. For more information about Dockerfiles, go to the Dockerfile Reference.
 

Build the Docker image from our Dockerfile. Replace the placeholder user name with your Docker Hub user name (be sure to include the blank space and period at the end of the command):

[ec2-user ecs-demo-php-simple-app]$ docker build -t my-dockerhub-username/amazon-ecs-sample .

Run docker images to verify that the image was created correctly and that the image name contains a repository that you can push to (in this example, your Docker Hub user name):

[ec2-user ecs-demo-php-simple-app]$ docker images

Upload the Docker image to your Docker Hub account.

Log in to Docker Hub:

[ec2-user ecs-demo-php-simple-app]$ docker login

Check to ensure your login worked:

[ec2-user ecs-demo-php-simple-app]$ docker info

Push your image to Docker Hub:

[ec2-user ecs-demo-php-simple-app]$ docker push my-dockerhub-username/amazon-ecs-sample

Now that you’ve created this first Docker image, you can move on to create your second Docker image, which will be deployed into the proxy container.

Create a reverse proxy Docker image

For our second Docker image, you will build a reverse proxy using NGINX and enable throttling. This will allow you to simulate security functionality for the purpose of this blog post. In practice, this proxy container could contain any security-related software you desire, and could be produced by a security team and delivered to the team responsible for deployments as a standalone artifact.

Using SSH, connect to the Amazon Linux instance you used in the last section.

Ensure that the Docker service is running and you are logged in to your Docker Hub account (instructions in previous section).

Create a local directory called proxy-image, and switch into it.

In this directory, you will create two files. You can copy and paste the contents for each as follows.

First, create a file called Dockerfile. This is a file used to build a Docker image according to your specifications. Copy and paste the following contents into the file. You are using a base Ubuntu image, running an update command, installing NGINX (your reverse proxy), telling Docker to copy the other file from your local machine to the Docker image, and then finally exposing port 80 for HTTP traffic.

FROM ubuntu
RUN apt-get update && apt-get install -y nginx
COPY nginx.conf /etc/nginx/nginx.conf
RUN echo "daemon off;" >> /etc/nginx/nginx.conf
EXPOSE 80
CMD service nginx start

Next, create a supporting file called nginx.conf. You want to overwrite the standard NGINX configuration file to ensure it is configured as a reverse proxy for all HTTP traffic. Throttling has been left out for the time being.

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
 
events {
     worker_connections 768;
     # multi_accept on;
}
 
http {
  server {
    listen               80;

# Proxy pass to servlet container

    location / {
      proxy_pass                http://application-container:80;
    }

  }
}

Now you are ready to build your proxy image. Run the following command with your specific Docker Hub information to instruct Docker to do so (be sure to include the blank space and period at the end of the command).

docker build -t my-dockerhub-username/proxy-image .

When this process completes, push your built image to Docker Hub.

docker push my-dockerhub-username/proxy-image

You have now successfully built both of your Docker images and pushed them to Docker Hub. You can now move on to deploying Docker images with Amazon ECS.

Deploy your Docker images with Amazon ECS

Amazon ECS is a container management service that allows you to manage and deploy Docker containers at scale. In this section, we will use ECS to deploy your two containers on a single instance. All inbound and outbound traffic from your application will be funneled through the proxy container, allowing you to enforce security measures on your application without modifying the application container in which it lives.

In the following diagram, you can see a visual representation of the ECS architecture we will be using. An ECS cluster is simply a logical grouping of container instances (we are just using one) that you will deploy your containers onto. A task definition specifies one or more container definitions, including which Docker image to use, port mappings, and more. This task definition allows you to model your application by having different containers work together. An instantiation of this task definition is called a task.

Figure 2. The ECS architecture described in this blog post

Use ECS to deploy your containers

Now that we have both Docker images built and stored in the Docker Hub repository, we can use ECS to deploy these containers.

Create your ECS task definition

Navigate to the AWS Management Console, and then to the EC2 Container Service page.

If you haven’t used ECS before, you should see a page with a Get started button. Click this button, and then click Cancel at the bottom of the page. If you have used ECS before, go straight to Step b.

Click Task Definitions in the left sidebar, and then click Create a New Task Definition. 

Give your task definition a name, such as SecurityExampleTask.

Click Add Container, and set up the first container definition with the following parameters, inserting the path to both your proxy image stored in Docker Hub (in other words, username/proxy-image), and the path to your web application image in Docker Hub (in the Links box). Don’t forget to click Advanced container configuration and complete all the fields.

Container Name: proxy-container
Image: username/proxy-image
Memory: 256
Port Mappings
Host port: 80
Container port: 80
Protocol: tcp
CPU: 256
Links: application-container

After you have populated the fields, click Add. Then, repeat the same process for the application container according to the following specifications. Note that the application container does not need a link back to the proxy container—doing this one way will suffice for this example.

Container Name: application-container
Image: username/amazon-ecs-sample
Memory: 256
CPU: 256

After you have populated the fields, click Add. Now, click the Configure via JSON tab to see the task definition that you have created. When you are done viewing this, click Create.

Now that you have created your task definition, you can move on to the next step.

Deploy an ECS container instance

In the ECS console, click Clusters in the left sidebar. If a cluster called default does not already exist, click Create Cluster and create a cluster called default (case sensitive).

Launch an instance with an ECS-optimized Amazon Machine Image (AMI), ensuring it has a public IP address and a path to the Internet. For more information, see Launching an Amazon ECS Container Instance. This is the instance onto which you’ll deploy your Docker images.

When your instance is up and running, navigate to the ECS section of the AWS Management Console, and click Clusters.

Click the cluster called default. You should see your instance under the ECS Instances tab. After you have verified this, you can move on to the next step.

Run your ECS task

Navigate to the Task Definitions tab on the left of the AWS Management Console, and select the check box next to the task definition you created. Click Actions, and then select Run Task.

On the next page, ensure the cluster is set to Default and the number of tasks is 1, and then click Run Task.

After the process completes, click the Clusters tab on the left of the AWS Management Console, select the default cluster, and then click the Tasks tab. Here, you can see your running task. It should have a green Running status. After you have verified this, you can proceed to the next step. If you see a Pending status, the task is still being deployed.

Click the ECS Instances tab, where you should see the container instance that you created earlier. Click the container instance to get more information, including its public IP address. If you copy and paste this public IP address into your browser’s address bar, you should see your sample PHP website!

If you do not see your PHP website, first ensure you have built your web application correctly by following the steps above in “Creating a Docker image containing a simple PHP application,” including pushing this image to the Docker Hub. Then, ensure your task is in the green Running state.

Try to refresh the page a couple times, and you will notice that no throttling is currently taking place. To fix this, make a slight modification. First, sign back in to the Amazon Linux instance where you built the two Docker images, and navigate to the proxy-image directory. Change the nginx.conf file to look like the following example. (Notice that a couple lines [highlighted] have been added to throttle requests to 3 per minute. This is an extremely low rate and is used only to show a working solution during this example.)

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;
 
events {
     worker_connections 768;
     # multi_accept on;
}
 
http {
  limit_req_zone  $binary_remote_addr  zone=one:10m   rate=3r/m;
  server {
    listen               80;
 
    # Proxy pass to servlet container
 
    location / {
      proxy_pass                http://application-container:80;
      limit_req zone=one burst=5 nodelay;
    }
  }
}

Following the same steps you followed earlier in “Create a reverse proxy Docker image” (specifically steps 5 and 6), rebuild the proxy image and push it to your Docker Hub. Now, stop the task that is currently running in the ECS console, and deploy it again by selecting Run New Task and choosing the same task as before. This will pick up the new image that you pushed to Docker Hub.

When you see a status of Running next to the task in the ECS console, paste the container instance’s public IP into your browser’s address bar again. You should see the sample PHP website. Refresh the page and wait for it to load, repeating this process a few times. On the fourth refresh, an error should be shown, and the page is not displayed. This means your throttling is functioning correctly.

Congratulations on completing this walkthrough!

Closing Notes

In this example, we performed Docker image creation and manual task definition creation. (Also see Set up a build pipeline with Jenkins and Amazon ECS on the AWS Application Management Blog for a walkthrough of how to automate Docker task definition creation using Jenkins.) This separation between image definition and deployment configuration can be leveraged for governance purposes. The web and security tiers can be owned by different teams that produce Docker images as artifacts, and a third team can ensure that the two tiers are deployed alongside one another and in this specific tiered-network configuration.

We hope that you find this example of leveraging a deployment service for security and governance purposes useful. You can find additional examples of application configuration options in the AWS Application Management Blog.

We look forward to hearing about your use of this strategy in your organization’s deployment process. If you have questions or comments, post them below or on the EC2 forum.

– Michael

Building a Near Real-Time Discovery Platform with AWS

Post Syndicated from Assaf Mentzer original https://blogs.aws.amazon.com/bigdata/post/Tx1Z6IF7NA8ELQ9/Building-a-Near-Real-Time-Discovery-Platform-with-AWS

Assaf Mentzer is a Senior Consultant for AWS Professional Services

In the spirit of the U.S presidential election of 2016, in this post I use Twitter public streams to analyze the candidates’ performance, both Republican and Democrat, in a near real-time fashion. I show you how to integrate AWS managed services—Amazon Kinesis Firehose, AWS Lambda (Python function), and Amazon Elasticsearch Service—to create an end-to-end, near real-time discovery platform.

The following screenshot is an example of a Kibana dashboard on top of geo-tagged tweet data. This screenshot was taken during the fourth republican presidential debate (November 10th, 2015).

Kibana dashboard on top of geotagged tweet data

The dashboard shows tweet data related to the presidential candidates (only tweets that contain a candidate’s name):

Top 10 Twitter mentions (@username) – you can see that Donald Trump is the most mentioned candidate

Sentiment analysis

Map visualization – Washington DC is the most active area

The dashboard has drill-down capabilities; choosing one of the sentiments in the pie chart or one of the @mentions in the bar chart changes the view of the entire dashboard accordingly. For example, you can see the sentiment analysis and geographic distribution for a specific candidate. The dashboard shows data from the last hour, and is configured to refresh the data every 30 seconds.

Because the platform built in this post collects all geo-tagged public Twitter data and filters data only in the dashboard layer, you can use the same solution for other use cases by just changing the filter search terms.

Use same solution for other use cases by changing filter search terms

Architecture

This platform has the following architecture:

A producer device (in this case, the Twitter feed) puts data into Amazon Kinesis Firehose.

Firehose automatically buffers the data (in this case, 5MB size or 5 minutes interval, whichever condition is satisfied first) and delivers the data to Amazon S3.

A Python Lambda function is triggered when a new file is created on S3 and indexes the S3 file content to Amazon Elasticsearch Service.

The Kibana application runs on top of the Elasticsearch index to provide a visual display of the data.

Platform architecture

Important: Streaming data can be pushed directly to Amazon Elasticsearch Service. The architecture described in this post is recommended when data has to be persisted on S3 for further batch/advanced analysis (lambda architecture,not related to AWS Lambda)  in addition to the near-real-time analysis on top of Elasticsearch Service, which might retain only “hot data” (last x hours).

Prerequisites

To create this platform, you’ll need an AWS account and a Twitter application. Sign in with your Twitter account and create a new application at https://apps.twitter.com/. Make sure your application is set for ‘read-only’ access and then choose Create My Access Token at the bottom of the Keys and Access Tokens tab. By this point, you should have four Twitter application keys: consumer key (API key), consumer secret (API secret), access token, and access token secret. Write down these keys.

Create Amazon Elasticsearch Service cluster

Start by creating an Amazon Elasticsearch Service cluster that will hold your data for near real-time analysis. Elasticsearch Service includes built-in support for Kibana, which is used for visualization on top of Elasticsearch Service.

Sign in to the Amazon Elasticsearch Service console.

Choose Create a new domain (or Get Started, if this is your first time in the console).

Name your domain “es-twitter-demo” and choose Next.

Keep the default selections and choose Next.

Choose the Allow open access to the domain template for the access policy and click Next.

Note: This is not a recommended approach, and should only be used for this demo. Please read the documentation for how to setup the proper permissions.

Choose Confirm and create.

Within ~10 minutes, your domain is ready. When the creation process has reached a status of Active, the domain should be associated with both an Elasticsearch Service endpoint and a Kibana URL, which you need to store for later steps.

Your domain is ready

Create an IAM role for Firehose

Use a Firehose delivery stream to ingest the Twitter streaming data and put it to Amazon S3. Before you can ingest the data into Firehose, you need to set up an IAM role to allow Firehose to call AWS services on your behalf. In this example, the Twitter feed which is your producer application creates the Firehose delivery stream based on the IAM role.

Create a new IAM role named “firehose_delivery_role” based on the following policy (please replace A_BUCKET_YOU_SETUP_FOR_THIS_DEMO with your S3 bucket):

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::A_BUCKET_YOU_SETUP_FOR_THIS_DEMO/*"
]
}
]
}

Create a Lambda function

For this example, use a Python function (lambda_function.py) that is triggered when a new file is created on S3. The function does the following:

Reads the file content

Parses the content to JSON format (Elasticsearch Service stores documents in JSON format).

Analyzes Twitter data (tweet_utils.py):

Extracts Twitter mentions (@username) from the tweet text.

Extracts sentiment based on emoticons. If there’s no emoticon in the text the function uses textblob sentiment analysis.

Loads the data to Elasticsearch Service (twitter_to_es.py) using the elasticsearch-py library.

The Python code is available in aws-big-data-blog repository.

Download the deployment package and unzip to the s3-twitter-to-es-python folder.

Modify the s3-twitter-to-es-python/config.py file by changing the value of es_host to the Elasticsearch Service endpoint of your domain.

Zip the folder content on your local environment as my-s3-twitter-to-es-python.zip (important: zip the folder content, not the folder itself).

Sign in to the Lambda console.

Choose Create a Lambda function (or Get started now if this is your first time using the service).

Choose Skip in the blueprints screen.

Name your function (e.g., s3-twitter-to-es-python).

Choose Python 2.7 runtime and upload the zip file my-s3-twitter-to-es-python.zip.

Make sure the Handler field value is lambda_function.lambda_handler.

Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role).

Keep memory at 128MB and choose a 2min timeout.

Choose Next and Create function, then wait until the function is created.

On the Event sources tab, choose Add event source.

Choose the event source type S3, select the bucket, and choose the event type Object Created (All).

Enter a value for S3 Prefix (e.g., twitter/raw-data/) to ensure the function doesn’t trigger when data is uploaded elsewhere in the bucket.

Make sure that the event source is enabled and click Submit.

Feed the producer with Twitter streaming data

Your producer is a Node.js application that connects to the Twitter feed via the Twitter stream API and puts the streaming data into Firehose. The code is available aws-big-data-blog repository.

To use the producer application, you have to install Node.js (go to https://nodejs.org to install it on your local machine). Alternatively, you can launch a t2.micro EC2 instance based on the Amazon Linux AMI and run the following command:

sudo yum -y install nodejs npm –enablerepo=epel

Download the application, unzip the file, and run npm install from the twitter-streaming-firehose-nodejs  folder.

Modify the Config.js file with your settings (change <YOUR PARAMETERS> as follows:

firehose

DeliveryStreamName – Name your stream. The app creates the delivery stream if it does not exist.

BucketARN: Use the bucket matched to the Lambda function.

RoleARN: Get your account ID from the IAM dashboard users sign-in link https://Your_AWS_Account_ID.signin.aws.amazon.com/console/. Use the Firehose role you created earlier (“firehose_delivery_role”).

Prefix: Use the same s3 prefix that you used in your Lambda function event source (e.g., twitter/raw-data/).

twitter – Enter your twitter application keys.

region – Enter your Firehose region (e.g., us-east-1, us-west-2, eu-west-1).

Make sure your aws credentials are configured under <HOME FOLDER>/.aws/credentials as follows:

[default]
aws_access_key_id=
aws_secret_access_key=

Now that your Config.js file is modified, you can open a console window and initiate execution of your program by running the following command:

node twitter_stream_producer_app

Wait a few seconds until the delivery stream is active, and then you should see Twitter data on your screen. The app collect tweets from the US but you can modify the locations in Config.js file. For more information, go to twitter geolocation.

Discover and analyze data

Wait a few minutes until Firehose has time to deliver enough files to Amazon S3 to make it interesting to review. The files should appear under the following bucket:

s3://<bucket>/<prefix>/<year>/<month>/<day>/<hour>/

Open Kibana in your browser using your Kibana URL. To start discovering the data stored in Elasticsearch Service, you need to create an index pattern pointing to your Elasticsearch index, which is like a ‘database’ in a relational database. For more information, go to What is an Elasticsearch Index?.

Create an index pattern as follows:

Create an index pattern

On the Discover tab, choose Add near the text field on the left sidebar. You should get the following result:

Start exploring the data by choosing any field in the left sidebar and filter. You can search for a specific term by replacing the asterisk (*) in the search field with your terms. You can also filter by time by choosing the Time Filter icon at the top right.

For example, you can search for the term “Trump” to discover and understand the data related to one of the candidates.

search for the term Trump to discover and understand the data related to one of the candidates

In this 2016 election discovery platform, you can analyze the performance of the presidential candidates: How many tweets they got, the sentiment of those tweets (positive/negative/neutral/confused), and how the tweets are geographically distributed (identifying politically active areas).

Because this is a near real-time discovery platform, you can measure the immediate impact of political events on the candidates’ popularity (for example, during a political debate).

Create a dashboard

To visualize candidates’ popularity in Twitter (in how many tweets a candidate was mentioned), create a top mentions bar chart.

On the Discover tab, choose the mentions field on the left sidebar.

Choose Visualize (ignore the warning).

Choose Visualize

On the X-Axis tab, change the size from 20 to 10 and choose Apply.

Choose the Save Visualization icon at the top right.

Enter a name and choose Save.

To analyze how tweets related to the 2016 election are geographically distributed in order to identify politically active areas, create a tile map.

On the Discover tab, choose the coordinates.coordinates field.

Choose Visualize.

Note: By default, in the Node.js app, tweets are collected only from the U.S.

To center the map, choose the crop  icon.

Choose Save Visualization.

To identify candidates’ popularity (or unpopularity), visualize the sentiments field. Because there are only 4 potential values (positive/negative/neutral/confused), you can use a pie chart visualization.

On the Visualize tab, choose the New Visualization icon ().

Choose Pie chart.

Choose new search, Split Slices, Terms aggregation, and the sentiments field.

Choose Apply and Save Visualization.

Combine all the visualizations into a single dashboard.

On the Dashboard tab, choose Add Visualization () at the top right corner, and select a visualization.

Repeat the previous step for all other visualizations.

Choose Save Dashboard, enter a name for your dashboard, and choose Save.

Now you can search for the presidential candidates in the data. Put the following search terms in the search filter field:

realDonaldTrump,realBenCarson,JebBush,tedcruz,ChrisChristie,JohnKasich,
GovMikeHuckabee,RandPaul,MarcoRubio,CarlyFiorina,JebBush,HillaryClinton,
MartinOMalley,BernieSanders

Search for candidates in the data

You’ve got yourself a dashboard! Select your preferred candidate in the bar chart to drill down to performance.

Conclusion

AWS managed services, like Amazon Kinesis Firehose, AWS Lambda, and Amazon Elasticsearch Service, take care of provisioning and maintaining the infrastructure components when building near real time applications and enable you to focus on your business logic.

You can quickly and easily tie these services together to create a near real-time discovery platform. For this post, we analyzed the performance of the 2016 presidential candidates, but this type of platform can be used for a variety of other use cases.

If you have questions or suggestions, please leave a comment below.

————————–

Related

Getting Started with Elasticsearch and Kibana on Amazon EMR

 

Building a Near Real-Time Discovery Platform with AWS

Post Syndicated from Assaf Mentzer original https://blogs.aws.amazon.com/bigdata/post/Tx1Z6IF7NA8ELQ9/Building-a-Near-Real-Time-Discovery-Platform-with-AWS

Assaf Mentzer is a Senior Consultant for AWS Professional Services

In the spirit of the U.S presidential election of 2016, in this post I use Twitter public streams to analyze the candidates’ performance, both Republican and Democrat, in a near real-time fashion. I show you how to integrate AWS managed services—Amazon Kinesis Firehose, AWS Lambda (Python function), and Amazon Elasticsearch Service—to create an end-to-end, near real-time discovery platform.

The following screenshot is an example of a Kibana dashboard on top of geo-tagged tweet data. This screenshot was taken during the fourth republican presidential debate (November 10th, 2015).

Kibana dashboard on top of geotagged tweet data

The dashboard shows tweet data related to the presidential candidates (only tweets that contain a candidate’s name):

Top 10 Twitter mentions (@username) – you can see that Donald Trump is the most mentioned candidate

Sentiment analysis

Map visualization – Washington DC is the most active area

The dashboard has drill-down capabilities; choosing one of the sentiments in the pie chart or one of the @mentions in the bar chart changes the view of the entire dashboard accordingly. For example, you can see the sentiment analysis and geographic distribution for a specific candidate. The dashboard shows data from the last hour, and is configured to refresh the data every 30 seconds.

Because the platform built in this post collects all geo-tagged public Twitter data and filters data only in the dashboard layer, you can use the same solution for other use cases by just changing the filter search terms.

Use same solution for other use cases by changing filter search terms

Architecture

This platform has the following architecture:

A producer device (in this case, the Twitter feed) puts data into Amazon Kinesis Firehose.

Firehose automatically buffers the data (in this case, 5MB size or 5 minutes interval, whichever condition is satisfied first) and delivers the data to Amazon S3.

A Python Lambda function is triggered when a new file is created on S3 and indexes the S3 file content to Amazon Elasticsearch Service.

The Kibana application runs on top of the Elasticsearch index to provide a visual display of the data.

Platform architecture

Important: Streaming data can be pushed directly to Amazon Elasticsearch Service. The architecture described in this post is recommended when data has to be persisted on S3 for further batch/advanced analysis (lambda architecture,not related to AWS Lambda)  in addition to the near-real-time analysis on top of Elasticsearch Service, which might retain only “hot data” (last x hours).

Prerequisites

To create this platform, you’ll need an AWS account and a Twitter application. Sign in with your Twitter account and create a new application at https://apps.twitter.com/. Make sure your application is set for ‘read-only’ access and then choose Create My Access Token at the bottom of the Keys and Access Tokens tab. By this point, you should have four Twitter application keys: consumer key (API key), consumer secret (API secret), access token, and access token secret. Write down these keys.

Create Amazon Elasticsearch Service cluster

Start by creating an Amazon Elasticsearch Service cluster that will hold your data for near real-time analysis. Elasticsearch Service includes built-in support for Kibana, which is used for visualization on top of Elasticsearch Service.

Sign in to the Amazon Elasticsearch Service console.

Choose Create a new domain (or Get Started, if this is your first time in the console).

Name your domain “es-twitter-demo” and choose Next.

Keep the default selections and choose Next.

Choose the Allow open access to the domain template for the access policy and click Next.

Note: This is not a recommended approach, and should only be used for this demo. Please read the documentation for how to setup the proper permissions.

Choose Confirm and create.

Within ~10 minutes, your domain is ready. When the creation process has reached a status of Active, the domain should be associated with both an Elasticsearch Service endpoint and a Kibana URL, which you need to store for later steps.

Your domain is ready

Create an IAM role for Firehose

Use a Firehose delivery stream to ingest the Twitter streaming data and put it to Amazon S3. Before you can ingest the data into Firehose, you need to set up an IAM role to allow Firehose to call AWS services on your behalf. In this example, the Twitter feed which is your producer application creates the Firehose delivery stream based on the IAM role.

Create a new IAM role named “firehose_delivery_role” based on the following policy (please replace A_BUCKET_YOU_SETUP_FOR_THIS_DEMO with your S3 bucket):

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::A_BUCKET_YOU_SETUP_FOR_THIS_DEMO/*"
]
}
]
}

Create a Lambda function

For this example, use a Python function (lambda_function.py) that is triggered when a new file is created on S3. The function does the following:

Reads the file content

Parses the content to JSON format (Elasticsearch Service stores documents in JSON format).

Analyzes Twitter data (tweet_utils.py):

Extracts Twitter mentions (@username) from the tweet text.

Extracts sentiment based on emoticons. If there’s no emoticon in the text the function uses textblob sentiment analysis.

Loads the data to Elasticsearch Service (twitter_to_es.py) using the elasticsearch-py library.

The Python code is available in aws-big-data-blog repository.

Download the deployment package and unzip to the s3-twitter-to-es-python folder.

Modify the s3-twitter-to-es-python/config.py file by changing the value of es_host to the Elasticsearch Service endpoint of your domain.

Zip the folder content on your local environment as my-s3-twitter-to-es-python.zip (important: zip the folder content, not the folder itself).

Sign in to the Lambda console.

Choose Create a Lambda function (or Get started now if this is your first time using the service).

Choose Skip in the blueprints screen.

Name your function (e.g., s3-twitter-to-es-python).

Choose Python 2.7 runtime and upload the zip file my-s3-twitter-to-es-python.zip.

Make sure the Handler field value is lambda_function.lambda_handler.

Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role).

Keep memory at 128MB and choose a 2min timeout.

Choose Next and Create function, then wait until the function is created.

On the Event sources tab, choose Add event source.

Choose the event source type S3, select the bucket, and choose the event type Object Created (All).

Enter a value for S3 Prefix (e.g., twitter/raw-data/) to ensure the function doesn’t trigger when data is uploaded elsewhere in the bucket.

Make sure that the event source is enabled and click Submit.

Feed the producer with Twitter streaming data

Your producer is a Node.js application that connects to the Twitter feed via the Twitter stream API and puts the streaming data into Firehose. The code is available aws-big-data-blog repository.

To use the producer application, you have to install Node.js (go to https://nodejs.org to install it on your local machine). Alternatively, you can launch a t2.micro EC2 instance based on the Amazon Linux AMI and run the following command:

sudo yum -y install nodejs npm –enablerepo=epel

Download the application, unzip the file, and run npm install from the twitter-streaming-firehose-nodejs  folder.

Modify the Config.js file with your settings (change <YOUR PARAMETERS> as follows:

firehose

DeliveryStreamName – Name your stream. The app creates the delivery stream if it does not exist.

BucketARN: Use the bucket matched to the Lambda function.

RoleARN: Get your account ID from the IAM dashboard users sign-in link https://Your_AWS_Account_ID.signin.aws.amazon.com/console/. Use the Firehose role you created earlier (“firehose_delivery_role”).

Prefix: Use the same s3 prefix that you used in your Lambda function event source (e.g., twitter/raw-data/).

twitter – Enter your twitter application keys.

region – Enter your Firehose region (e.g., us-east-1, us-west-2, eu-west-1).

Make sure your aws credentials are configured under <HOME FOLDER>/.aws/credentials as follows:

[default]
aws_access_key_id=
aws_secret_access_key=

Now that your Config.js file is modified, you can open a console window and initiate execution of your program by running the following command:

node twitter_stream_producer_app

Wait a few seconds until the delivery stream is active, and then you should see Twitter data on your screen. The app collect tweets from the US but you can modify the locations in Config.js file. For more information, go to twitter geolocation.

Discover and analyze data

Wait a few minutes until Firehose has time to deliver enough files to Amazon S3 to make it interesting to review. The files should appear under the following bucket:

s3://<bucket>/<prefix>/<year>/<month>/<day>/<hour>/

Open Kibana in your browser using your Kibana URL. To start discovering the data stored in Elasticsearch Service, you need to create an index pattern pointing to your Elasticsearch index, which is like a ‘database’ in a relational database. For more information, go to What is an Elasticsearch Index?.

Create an index pattern as follows:

Create an index pattern

On the Discover tab, choose Add near the text field on the left sidebar. You should get the following result:

Start exploring the data by choosing any field in the left sidebar and filter. You can search for a specific term by replacing the asterisk (*) in the search field with your terms. You can also filter by time by choosing the Time Filter icon at the top right.

For example, you can search for the term “Trump” to discover and understand the data related to one of the candidates.

search for the term Trump to discover and understand the data related to one of the candidates

In this 2016 election discovery platform, you can analyze the performance of the presidential candidates: How many tweets they got, the sentiment of those tweets (positive/negative/neutral/confused), and how the tweets are geographically distributed (identifying politically active areas).

Because this is a near real-time discovery platform, you can measure the immediate impact of political events on the candidates’ popularity (for example, during a political debate).

Create a dashboard

To visualize candidates’ popularity in Twitter (in how many tweets a candidate was mentioned), create a top mentions bar chart.

On the Discover tab, choose the mentions field on the left sidebar.

Choose Visualize (ignore the warning).

Choose Visualize

On the X-Axis tab, change the size from 20 to 10 and choose Apply.

Choose the Save Visualization icon at the top right.

Enter a name and choose Save.

To analyze how tweets related to the 2016 election are geographically distributed in order to identify politically active areas, create a tile map.

On the Discover tab, choose the coordinates.coordinates field.

Choose Visualize.

Note: By default, in the Node.js app, tweets are collected only from the U.S.

To center the map, choose the crop  icon.

Choose Save Visualization.

To identify candidates’ popularity (or unpopularity), visualize the sentiments field. Because there are only 4 potential values (positive/negative/neutral/confused), you can use a pie chart visualization.

On the Visualize tab, choose the New Visualization icon ().

Choose Pie chart.

Choose new search, Split Slices, Terms aggregation, and the sentiments field.

Choose Apply and Save Visualization.

Combine all the visualizations into a single dashboard.

On the Dashboard tab, choose Add Visualization () at the top right corner, and select a visualization.

Repeat the previous step for all other visualizations.

Choose Save Dashboard, enter a name for your dashboard, and choose Save.

Now you can search for the presidential candidates in the data. Put the following search terms in the search filter field:

realDonaldTrump,realBenCarson,JebBush,tedcruz,ChrisChristie,JohnKasich,
GovMikeHuckabee,RandPaul,MarcoRubio,CarlyFiorina,JebBush,HillaryClinton,
MartinOMalley,BernieSanders

Search for candidates in the data

You’ve got yourself a dashboard! Select your preferred candidate in the bar chart to drill down to performance.

Conclusion

AWS managed services, like Amazon Kinesis Firehose, AWS Lambda, and Amazon Elasticsearch Service, take care of provisioning and maintaining the infrastructure components when building near real time applications and enable you to focus on your business logic.

You can quickly and easily tie these services together to create a near real-time discovery platform. For this post, we analyzed the performance of the 2016 presidential candidates, but this type of platform can be used for a variety of other use cases.

If you have questions or suggestions, please leave a comment below.

————————–

Related

Getting Started with Elasticsearch and Kibana on Amazon EMR