Tag Archives: pc

MPAA Revenue Drops 20% as Movie Studios Cut Back

2018-05-27 Ernesto

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-revenue-drops-20-as-movie-studios-cut-back-180527/

As a united front for Hollywood, the MPAA has booked many anti-piracy successes in recent years.

Through its involvement in the shutdowns of Popcorn Time, YIFY, isoHunt, Hotfile, Megaupload and several other platforms, the organization has worked hard to get results.

Less visibly but at least as important, the MPAA uses its influence to lobby lawmakers, while orchestrating and managing anti-piracy campaigns both in the United States and abroad.

All this work doesn’t come for free, obviously. To pay the bills the MPAA relies on six major movie studios for financial support. Over the past several years, these revenues had stabilized, but according to its latest tax filing, they are dropping now.

The IRS filing, covering the fiscal year 2016, puts total revenue at $57 million, down from $73 million. The Hollywood studios paid the bulk of this sum through membership fees which total $50 million. That’s a 22% reduction compared to a year earlier.

At the end of the year, this resulted in a significant loss of $8 million. While that’s a lot of money, the MPAA is not in imminent danger, as the organization still has over $10 million in net assets and funds.

We haven’t seen any explanation for the lower membership fees. It could be more permanent, but it may also be an agreed decision, as there’s enough money in the bank.

Going over the numbers, we see that salaries make up a large chunk of the expenses. Chris Dodd, the former MPAA Chairman and CEO, was the highest paid employee with a total income of more than $3.4 million, including a $275,000 bonus.

This compensation was for Dodd’s last full year as CEO. He was replaced by Charles Rivkin last year, another political heavyweight, who previously served as Assistant Secretary of State for Economic and Business Affairs in the Obama administration.

Dodd’s compensation took up nearly 10% of the entire salary budget. The rest is divided over the MPAA’s other 196 employees. This brings the total workforce to 197, which is down as well, from 224 employees a year earlier.

Moving on, it’s always interesting to see where the MPAA’s grants and other types of funding go to.

As reported previously, the group donates handsomely to various research initiatives. This includes a recurring million dollar grant for Carnegie Mellon’s ‘Initiative for Digital Entertainment Analytics’ (IDEA), which focuses on piracy related topics.

Another major beneficiary is the Copyright Alliance, a non-profit that represents copyright holders large and small on a variety of issues. The group was co-founded by the MPAA and received $750,000 in support according to the latest filing.

The total grants budget is $3.1 million and includes many smaller payments, which is not that different from previous years. Other large cost items are the lobbying budget, which totaled $3.6 million, and $5.3 million in legal fees.

Aside from the big dip in revenues, there are no real outliers in the filing.

—

A copy of the MPAA’s latest form 990 is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Orchestrate Apache Spark applications using AWS Step Functions and Apache Livy

2018-05-25 Tanzir Musabbir

Post Syndicated from Tanzir Musabbir original https://aws.amazon.com/blogs/big-data/orchestrate-apache-spark-applications-using-aws-step-functions-and-apache-livy/

The adoption of Apache Spark has increased significantly over the past few years, and running Spark-based application pipelines is the new normal. Spark jobs that are in an ETL (extract, transform, and load) pipeline have different requirements—you must handle dependencies in the jobs, maintain order during executions, and run multiple jobs in parallel. In most of these cases, you can use workflow scheduler tools like Apache Oozie, Apache Airflow, and even Cron to fulfill these requirements.

Apache Oozie is a widely used workflow scheduler system for Hadoop-based jobs. However, its limited UI capabilities, lack of integration with other services, and heavy XML dependency might not be suitable for some users. On the other hand, Apache Airflow comes with a lot of neat features, along with powerful UI and monitoring capabilities and integration with several AWS and third-party services. However, with Airflow, you do need to provision and manage the Airflow server. The Cron utility is a powerful job scheduler. But it doesn’t give you much visibility into the job details, and creating a workflow using Cron jobs can be challenging.

What if you have a simple use case, in which you want to run a few Spark jobs in a specific order, but you don’t want to spend time orchestrating those jobs or maintaining a separate application? You can do that today in a serverless fashion using AWS Step Functions. You can create the entire workflow in AWS Step Functions and interact with Spark on Amazon EMR through Apache Livy.

In this post, I walk you through a list of steps to orchestrate a serverless Spark-based ETL pipeline using AWS Step Functions and Apache Livy.

Input data

For the source data for this post, I use the New York City Taxi and Limousine Commission (TLC) trip record data. For a description of the data, see this detailed dictionary of the taxi data. In this example, we’ll work mainly with the following three columns for the Spark jobs.

Column name	Column description
RateCodeID	Represents the rate code in effect at the end of the trip (for example, 1 for standard rate, 2 for JFK airport, 3 for Newark airport, and so on).
FareAmount	Represents the time-and-distance fare calculated by the meter.
TripDistance	Represents the elapsed trip distance in miles reported by the taxi meter.

The trip data is in comma-separated values (CSV) format with the first row as a header. To shorten the Spark execution time, I trimmed the large input data to only 20,000 rows. During the deployment phase, the input file tripdata.csv is stored in Amazon S3 in the <<your-bucket>>/emr-step-functions/input/ folder.

The following image shows a sample of the trip data:

Solution overview

The next few sections describe how Spark jobs are created for this solution, how you can interact with Spark using Apache Livy, and how you can use AWS Step Functions to create orchestrations for these Spark applications.

At a high level, the solution includes the following steps:

Trigger the AWS Step Function state machine by passing the input file path.
The first stage in the state machine triggers an AWS Lambda
The Lambda function interacts with Apache Spark running on Amazon EMR using Apache Livy, and submits a Spark job.
The state machine waits a few seconds before checking the Spark job status.
Based on the job status, the state machine moves to the success or failure state.
Subsequent Spark jobs are submitted using the same approach.
The state machine waits a few seconds for the job to finish.
The job finishes, and the state machine updates with its final status.

Let’s take a look at the Spark application that is used for this solution.

Spark jobs

For this example, I built a Spark jar named spark-taxi.jar. It has two different Spark applications:

MilesPerRateCode – The first job that runs on the Amazon EMR cluster. This job reads the trip data from an input source and computes the total trip distance for each rate code. The output of this job consists of two columns and is stored in Apache Parquet format in the output path.

The following are the expected output columns:

rate_code – Represents the rate code for the trip.
total_distance – Represents the total trip distance for that rate code (for example, sum(trip_distance)).

RateCodeStatus – The second job that runs on the EMR cluster, but only if the first job finishes successfully. This job depends on two different input sets:

csv – The same trip data that is used for the first Spark job.
miles-per-rate – The output of the first job.

This job first reads the tripdata.csv file and aggregates the fare_amount by the rate_code. After this point, you have two different datasets, both aggregated by rate_code. Finally, the job uses the rate_code field to join two datasets and output the entire rate code status in a single CSV file.

The output columns are as follows:

rate_code_id – Represents the rate code type.
total_distance – Derived from first Spark job and represents the total trip distance.
total_fare_amount – A new field that is generated during the second Spark application, representing the total fare amount by the rate code type.

Note that in this case, you don’t need to run two different Spark jobs to generate that output. The goal of setting up the jobs in this way is just to create a dependency between the two jobs and use them within AWS Step Functions.

Both Spark applications take one input argument called rootPath. It’s the S3 location where the Spark job is stored along with input and output data. Here is a sample of the final output:

The next section discusses how you can use Apache Livy to interact with Spark applications that are running on Amazon EMR.

Using Apache Livy to interact with Apache Spark

Apache Livy provides a REST interface to interact with Spark running on an EMR cluster. Livy is included in Amazon EMR release version 5.9.0 and later. In this post, I use Livy to submit Spark jobs and retrieve job status. When Amazon EMR is launched with Livy installed, the EMR master node becomes the endpoint for Livy, and it starts listening on port 8998 by default. Livy provides APIs to interact with Spark.

Let’s look at a couple of examples how you can interact with Spark running on Amazon EMR using Livy.

To list active running jobs, you can execute the following from the EMR master node:

curl localhost:8998/sessions

If you want to do the same from a remote instance, just change localhost to the EMR hostname, as in the following (port 8998 must be open to that remote instance through the security group):

curl http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:8998/sessions

To submit a Spark job through Livy pointing to the application jar in S3, you can do the following:

curl -X POST --data '{"file": "s3://<<bucket-location>/spark.jar", "className": "com.example.SparkApp "]}' -H "Content-Type: application/json" http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:8998/batches

Spark submit through Livy returns a session ID that starts from 0. Using that session ID, you can retrieve the status of the job as shown following:

curl http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:8998/batches/0 | python -m json.tool

Through Spark submit, you can pass multiple arguments for the Spark job and Spark configuration settings. You can also do that using Livy, by passing the S3 path through the args parameter, as shown following:

curl -X POST --data '{"file": "s3://<<bucket-location>>/spark.jar", "className": "com.example.SparkApp", “args”: [“s3://bucket-path”]}' -H "Content-Type: application/json" http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:8998/batches

All Apache Livy REST calls return a response as JSON, as shown in the following image:

If you want to pretty-print that JSON response, you can pipe command with Python’s JSON tool as follows:

curl http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:8998/batches/3 | python -m json.tool

For a detailed list of Livy APIs, see the Apache Livy REST API page. This post uses GET /batches and POST /batches.

In the next section, you create a state machine and orchestrate Spark applications using AWS Step Functions.

Using AWS Step Functions to create a Spark job workflow

AWS Step Functions automatically triggers and tracks each step and retries when it encounters errors. So your application executes in order and as expected every time. To create a Spark job workflow using AWS Step Functions, you first create a Lambda state machine using different types of states to create the entire workflow.

First, you use the Task state—a simple state in AWS Step Functions that performs a single unit of work. You also use the Wait state to delay the state machine from continuing for a specified time. Later, you use the Choice state to add branching logic to a state machine.

The following is a quick summary of how to use different states in the state machine to create the Spark ETL pipeline:

Task state – Invokes a Lambda function. The first Task state submits the Spark job on Amazon EMR, and the next Task state is used to retrieve the previous Spark job status.
Wait state – Pauses the state machine until a job completes execution.
Choice state – Each Spark job execution can return a failure, an error, or a success state So, in the state machine, you use the Choice state to create a rule that specifies the next action or step based on the success or failure of the previous step.

Here is one of my Task states, MilesPerRateCode, which simply submits a Spark job:

"MilesPerRate Job": {
      "Type": "Task",
      "Resource":"arn:aws:lambda:us-east-1:xxxxxx:function:blog-miles-per-rate-job-submit-function",
      "ResultPath": "$.jobId",
      "Next": "Wait for MilesPerRate job to complete"
 }

This Task state configuration specifies the Lambda function to execute. Inside the Lambda function, it submits a Spark job through Livy using Livy’s POST API. Using ResultPath, it tells the state machine where to place the result of the executing task. As discussed in the previous section, Spark submit returns the session ID, which is captured with $.jobId and used in a later state.

The following code section shows the Lambda function, which is used to submit the MilesPerRateCode job. It uses the Python request library to submit a POST against the Livy endpoint hosted on Amazon EMR and passes the required parameters in JSON format through payload. It then parses the response, grabs id from the response, and returns it. The Next field tells the state machine which state to go to next.

from botocore.vendored import requests
import json

def lambda_handler(event, context):
  headers = { "content-type": "application/json" }
  url = 'http://xxxxxx.compute-1.amazonaws.com:8998/batches'
  payload = {
    'file' : 's3://<<s3-location>>/emr-step-functions/spark-taxi.jar',
    'className' : 'com.example.MilesPerRateCode',
    'args' : [event.get('rootPath')]
  }
  res = requests.post(url, data = json.dumps(payload), headers = headers, verify = False)
  json_data = json.loads(res.text)
  return json_data.get('id')

Just like in the MilesPerRate job, another state submits the RateCodeStatus job, but it executes only when all previous jobs have completed successfully.

Here is the Task state in the state machine that checks the Spark job status:

"Query RateCodeStatus job status": {
   "Type": "Task",
   "Resource": "arn:aws:lambda:us-east-1:xxxxx:function:blog-spark-job-status-function",
   "Next": "RateCodeStatus job complete?",
   "ResultPath": "$.jobStatus"
}

Just like other states, the preceding Task executes a Lambda function, captures the result (represented by jobStatus), and passes it to the next state. The following is the Lambda function that checks the Spark job status based on a given session ID:

from botocore.vendored import requests
import json

def lambda_handler(event, context):
    jobid = event.get('jobId')
    url     = 'http://xxxx.compute-1.amazonaws.com:8998/batches/' + str(jobid)
    res = requests.get(url)
    json_data = json.loads(res.text)
    return json_data.get('state')

In the Choice state, it checks the Spark job status value, compares it with a predefined state status, and transitions the state based on the result. For example, if the status is success, move to the next state (RateCodeJobStatus job), and if it is dead, move to the MilesPerRate job failed state.

"MilesPerRate job complete?": {
   "Type": "Choice",
   "Choices": [{
       "Variable": "$.jobStatus",
       "StringEquals": "success",
       "Next": "RateCodeStatus Job"
   },{
       "Variable": "$.jobStatus",
       "StringEquals": "dead",
       "Next": "MilesPerRate job failed"
   }],
   "Default": "Wait for MilesPerRate job to complete"
}

Walkthrough using AWS CloudFormation

To set up this entire solution, you need to create a few AWS resources. To make it easier, I have created an AWS CloudFormation template. This template creates all the required AWS resources and configures all the resources that are needed to create a Spark-based ETL pipeline on AWS Step Functions.

This CloudFormation template requires you to pass the following four parameters during initiation.

Parameter	Description
`ClusterSubnetID`	The subnet where the Amazon EMR cluster is deployed and Lambda is configured to talk to this subnet.
`KeyName`	The name of the existing EC2 key pair to access the Amazon EMR cluster.
`VPCID`	The ID of the virtual private cloud (VPC) where the EMR cluster is deployed and Lambda is configured to talk to this VPC.
`S3RootPath`	The Amazon S3 path where all required files (input file, Spark job, and so on) are stored and the resulting data is written.

IMPORTANT: These templates are designed only to show how you can create a Spark-based ETL pipeline on AWS Step Functions using Apache Livy. They are not intended for production use without modification. And if you try this solution outside of the us-east-1 Region, download the necessary files from s3://aws-data-analytics-blog/emr-step-functions, upload the files to the buckets in your Region, edit the script as appropriate, and then run it.

To launch the CloudFormation stack, choose Launch Stack:

Launching this stack creates the following list of AWS resources.

Logical ID	Resource Type	Description
`StepFunctionsStateExecutionRole`	IAM role	IAM role to execute the state machine and have a trust relationship with the states service.
`SparkETLStateMachine`	AWS Step Functions state machine	State machine in AWS Step Functions for the Spark ETL workflow.
`LambdaSecurityGroup`	Amazon EC2 security group	Security group that is used for the Lambda function to call the Livy API.
`RateCodeStatusJobSubmitFunction`	AWS Lambda function	Lambda function to submit the RateCodeStatus job.
`MilesPerRateJobSubmitFunction`	AWS Lambda function	Lambda function to submit the MilesPerRate job.
`SparkJobStatusFunction`	AWS Lambda function	Lambda function to check the Spark job status.
`LambdaStateMachineRole`	IAM role	IAM role for all Lambda functions to use the lambda trust relationship.
`EMRCluster`	Amazon EMR cluster	EMR cluster where Livy is running and where the job is placed.

During the AWS CloudFormation deployment phase, it sets up S3 paths for input and output. Input files are stored in the <<s3-root-path>>/emr-step-functions/input/ path, whereas spark-taxi.jar is copied under <<s3-root-path>>/emr-step-functions/.

The following screenshot shows how the S3 paths are configured after deployment. In this example, I passed a bucket that I created in the AWS account s3://tm-app-demos for the S3 root path.

If the CloudFormation template completed successfully, you will see Spark-ETL-State-Machine in the AWS Step Functions dashboard, as follows:

Choose the Spark-ETL-State-Machine state machine to take a look at this implementation. The AWS CloudFormation template built the entire state machine along with its dependent Lambda functions, which are now ready to be executed.

On the dashboard, choose the newly created state machine, and then choose New execution to initiate the state machine. It asks you to pass input in JSON format. This input goes to the first state MilesPerRate Job, which eventually executes the Lambda function blog-miles-per-rate-job-submit-function.

Pass the S3 root path as input:

{

“rootPath”: “s3://tm-app-demos”

}

Then choose Start Execution:

The rootPath value is the same value that was passed when creating the CloudFormation stack. It can be an S3 bucket location or a bucket with prefixes, but it should be the same value that is used for AWS CloudFormation. This value tells the state machine where it can find the Spark jar and input file, and where it will write output files. After the state machine starts, each state/task is executed based on its definition in the state machine.

At a high level, the following represents the flow of events:

Execute the first Spark job, MilesPerRate.
The Spark job reads the input file from the location <<rootPath>>/emr-step-functions/input/tripdata.csv. If the job finishes successfully, it writes the output data to <<rootPath>>/emr-step-functions/miles-per-rate.
If the Spark job fails, it transitions to the error state MilesPerRate job failed, and the state machine stops. If the Spark job finishes successfully, it transitions to the RateCodeStatus Job state, and the second Spark job is executed.
If the second Spark job fails, it transitions to the error state RateCodeStatus job failed, and the state machine stops with the Failed status.
If this Spark job completes successfully, it writes the final output data to the <<rootPath>>/emr-step-functions/rate-code-status/ It also transitions the RateCodeStatus job finished state, and the state machine ends its execution with the Success status.

This following screenshot shows a successfully completed Spark ETL state machine:

The right side of the state machine diagram shows the details of individual states with their input and output.

When you execute the state machine for the second time, it fails because the S3 path already exists. The state machine turns red and stops at MilePerRate job failed. The following image represents that failed execution of the state machine:

You can also check your Spark application status and logs by going to the Amazon EMR console and viewing the Application history tab:

I hope this walkthrough paints a picture of how you can create a serverless solution for orchestrating Spark jobs on Amazon EMR using AWS Step Functions and Apache Livy. In the next section, I share some ideas for making this solution even more elegant.

Next steps

The goal of this post is to show a simple example that uses AWS Step Functions to create an orchestration for Spark-based jobs in a serverless fashion. To make this solution robust and production ready, you can explore the following options:

In this example, I manually initiated the state machine by passing the rootPath as input. You can instead trigger the state machine automatically. To run the ETL pipeline as soon as the files arrive in your S3 bucket, you can pass the new file path to the state machine. Because CloudWatch Events supports AWS Step Functions as a target, you can create a CloudWatch rule for an S3 event. You can then set AWS Step Functions as a target and pass the new file path to your state machine. You’re all set!
You can also improve this solution by adding an alerting mechanism in case of failures. To do this, create a Lambda function that sends an alert email and assigns that Lambda function to a Fail That way, when any part of your state fails, it triggers an email and notifies the user.
If you want to submit multiple Spark jobs in parallel, you can use the Parallel state type in AWS Step Functions. The Parallel state is used to create parallel branches of execution in your state machine.

With Lambda and AWS Step Functions, you can create a very robust serverless orchestration for your big data workload.

Cleaning up

When you’ve finished testing this solution, remember to clean up all those AWS resources that you created using AWS CloudFormation. Use the AWS CloudFormation console or AWS CLI to delete the stack named Blog-Spark-ETL-Step-Functions.

Summary

In this post, I showed you how to use AWS Step Functions to orchestrate your Spark jobs that are running on Amazon EMR. You used Apache Livy to submit jobs to Spark from a Lambda function and created a workflow for your Spark jobs, maintaining a specific order for job execution and triggering different AWS events based on your job’s outcome.
Go ahead—give this solution a try, and share your experience with us!

Additional Reading

If you found this post useful, be sure to check out Building a Real World Evidence Platform on AWS and Building High-Throughput Genomics Batch Workflows on AWS: Workflow Layer (Part 4 of 4).

About the Author

Tanzir Musabbir is an EMR Specialist Solutions Architect with AWS. He is an early adopter of open source Big Data technologies. At AWS, he works with our customers to provide them architectural guidance for running analytics solutions on Amazon EMR, Amazon Athena & AWS Glue. Tanzir is a big Real Madrid fan and he loves to travel in his free time.

Integrating JFrog Artifactory with AWS CodePipeline

2018-05-24 Erin McGill

Post Syndicated from Erin McGill original https://aws.amazon.com/blogs/devops/integrating-jfrog-artifactory-with-aws-codepipeline/

When I talk with customers and partners, I find that they are in different stages in the adoption of DevOps methodologies. They are automating the creation of application artifacts and the deployment of their applications to different infrastructure environments. In many cases, they are creating and supporting multiple applications using a variety of coding languages and artifacts.

The management of these processes and artifacts can be challenging, but using the right tools and methodologies can simplify the process.

In this post, I will show you how you can automate the creation and storage of application artifacts through the implementation of a pipeline and custom deploy action in AWS CodePipeline. The example includes a Node.js code base stored in an AWS CodeCommit repository. A Node Package Manager (npm) artifact is built from the code base, and the build artifact is published to a JFrog Artifactory npm repository.

I frequently recommend AWS CodePipeline, the AWS continuous integration and continuous delivery tool. You can use it to quickly innovate through integration and deployment of new features and bug fixes by building a workflow that automates the build, test, and deployment of new versions of your application. And, because AWS CodePipeline is extensible, it allows you to create a custom action that performs customized, automated actions on your behalf.

JFrog’s Artifactory is a universal binary repository manager where you can manage multiple applications, their dependencies, and versions in one place. Artifactory also enables you to standardize the way you manage your package types across all applications developed in your company, no matter the code base or artifact type.

If you already have a Node.js CodeCommit repository, a JFrog Artifactory host, and would like to automate the creation of the pipeline, including the custom action and CodeBuild project, you can use this AWS CloudFormation template to create your AWS CloudFormation stack.

The project code can be found in this GitHub repository: https://github.com/aws-samples/aws-codepipeline-custom-job-worker-for-jfrog-artifactory.

The AWS CodePipeline workflow

This figure shows the path defined in the pipeline for this project. It starts with a change to Node.js source code committed to a private code repository in AWS CodeCommit. With this change, CodePipeline triggers AWS CodeBuild to create the npm package from the node.js source code. After the build, CodePipeline triggers the custom action job worker to commit the build artifact to the designated artifact repository in Artifactory.

This blog post assumes you have already:

· Created a CodeCommit repository that contains a Node.js project.

· Configured a two-stage pipeline in AWS CodePipeline.

The Source stage of the pipeline is configured to poll the Node.js CodeCommit repository. The Build stage is configured to use a CodeBuild project to build the npm package using a buildspec.yml file located in the code repository.

If you do not have a Node.js repository, you can create a CodeCommit repository that contains this simple ‘Hello World’ project. This project also includes a buildspec.yml file that is used when you define your CodeBuild project. It defines the steps to be taken by CodeBuild to create the npm artifact.

If you do not already have a pipeline set up in CodePipeline, you can use this template to create a pipeline with a CodeCommit source action and a CodeBuild build action through the AWS Command Line Interface (AWS CLI). If you do not want to install the AWS CLI on your local machine, you can use AWS Cloud9, our managed integrated development environment (IDE), to interact with AWS APIs.

In your development environment, open your favorite editor and fill out the template with values appropriate to your project. For information, see the readme in the GitHub repository.

Use this CLI command to create the pipeline from the template:

aws codepipeline create-pipeline --cli-input-json file://source-build-actions-codepipeline.json --region 'us-west-2'

It creates a pipeline that has a CodeCommit source action and a CodeBuild build action.

Integrating JFrog Artifactory

JFrog Artifactory provides default repositories for your project needs. For my NPM package repository, I am using the default virtual npm repository (named npm) that is available in Artifactory Pro. You might want to consider creating a repository per project but for the example used in this post, using the default lets me get started without having to configure a new repository.

I can use the steps in the Set Me Up -> npm section on the landing page to configure my worker to interact with the default NPM repository.

Custom actions in AWS CodePipeline

A custom action in AWS CodePipeline contains:

· custom action definition

Describes the required values to run the custom action. I will define my custom action in the ‘Deploy’ category, identify the provider as ‘Artifactory’, of version ‘1’, and specify a variety of configurationProperties whose values will be defined when this stage is added to my pipeline.

· custom action job worker

Polls CodePipeline for a job, scanning for its action-definition properties. In this blog post, after a job has been found, the job worker does the work required to publish the npm artifact to the Artifactory repository.

My custom action definition in JSON:

{
    "category": "Deploy",
    "configurationProperties": [{
        "name": "TypeOfArtifact",
        "required": true,
        "key": true,
        "secret": false,
        "description": "Package type, ex. npm for node packages",
        "type": "String"
    },
    {   "name": "RepoKey",
        "required": true,
        "key": true,
	"secret": false,
	"type": "String",
	"description": "Name of the repository in which this artifact should be stored"
    },
    {   "name": "UserName",
        "required": true,
        "key": true,
	"secret": false,
	"type": "String",
	"description": "Username for authenticating with the repository"
    },
    {   "name": "Password",
        "required": true,
        "key": true,
	"secret": true,
	"type": "String",
	"description": "Password for authenticating with the repository"
    },
    {   "name": "EmailAddress",
        "required": true,
        "key": true,
	"secret": false,
	"type": "String",
	"description": "Email address used to authenticate with the repository"
    },
    {   "name": "ArtifactoryHost",
        "required": true,
        "key": true,
	"secret": false,
	"type": "String",
	"description": "Public address of Artifactory host, ex: https://myexamplehost.com or http://myexamplehost.com:8080"
    }],
    "provider": "Artifactory",
    "version": "1",
    "settings": {
        "entityUrlTemplate": "{Config:ArtifactoryHost}/artifactory/webapp/#/artifacts/browse/tree/General/{Config:RepoKey}"
    },
    "inputArtifactDetails": {
        "maximumCount": 5,
        "minimumCount": 1
    },
    "outputArtifactDetails": {
        "maximumCount": 5,
        "minimumCount": 0
    }
}

There are seven sections to the custom action definition:

category: This is the stage in which you will be creating this action. It can be Source, Build, Deploy, Test, Invoke, Approval. Except for source actions, the category section simply allows us to organize our actions. I am setting the category for my action as ‘Deploy’ because I’m using it to publish my node artifact to my Artifactory instance.
configurationProperties: These are the parameters or variables required for your project to authenticate and commit your artifact. In the case of my custom worker, I need:
- TypeOfArtifact: In this case, npm, because it’s for the Node Package Manager.
- RepoKey: The name of the repository. In this case, it’s the default npm.
- UserName and Password for the user to authenticate with the Artifactory repository.
- EmailAddress used to authenticate with the repository.
- Artifactory host name or IP address.
provider: The name you define for your custom action stage. I have named the provider Artifactory.
version: Version number for the custom action. Because this is the first version, I set the version number to 1.
entityUrlTemplate: This URL is presented to your users for the deploy stage along with the title you define in your provider. The link takes the user to their artifact repository page in the Artifactory host.
inputArtifactDetails: The number of artifacts to expect from the previous stage in the pipeline.
outputArtifactDetails: The number of artifacts that should be the result from the custom action stage. Later in this blog post, I define 0 for my output artifacts because I am publishing the artifact to the Artifactory repository as the final action.

After I define the custom action in a JSON file, I use the AWS CLI to create the custom action type in CodePipeline:

aws codepipeline create-custom-action-type --cli-input-json file://artifactory_custom_action_deploy_npm.json --region='us-west-2'

After I create the custom action type in the same region as my pipeline, I edit the pipeline to add a Deploy stage and configure it to use the custom action I created for Artifactory:

I have created a custom worker for the actions required to commit the npm artifact to the Artifactory repository. The worker is in Python and it runs in a loop on an Amazon EC2 instance. My custom worker polls for a deploy job and publishes the NPM artifact to the Artifactory repository.

The EC2 instance is running Amazon Linux and has an IAM instance role attached that gives the worker permission to access CodePipeline. The worker process is as follows:

Take the configuration properties from the custom worker and poll CodePipeline for a custom action job.
After there is a job in the job queue with the appropriate category, provider, and version, acknowledge the job.
Download the zipped artifact created in the previous Build stage from the provided S3 buckets with the provided temporary credentials.
Unzip the artifact into a temporary directory.
A user-defined Artifactory user name and password is used to receive a temporary API key from Artifactory.
To avoid having to write the password to a file, use that temporary API key and user name to authenticate with the NPM repository.
Publish the Node.js package to the specified repository.

Because I am running my custom worker on an Amazon Linux EC2 instance, I installed npm with the following command:

sudo yum install nodejs npm --enablerepo=epel

For my custom worker, I used pip to install the required Python libraries:

pip install boto3 requests

For a full Python package list, see requirements.txt in the GitHub repository.

Let’s take a look at some of the code snippets from the worker.

First, the worker polls for jobs:

def action_type():
    ActionType = {
        'category': 'Deploy',
        'owner': 'Custom',
        'provider': 'Artifactory',
        'version': '1' }
    return(ActionType)

def poll_for_jobs():
    try:
        artifactory_action_type = action_type()
        print(artifactory_action_type)
        jobs = codepipeline.poll_for_jobs(actionTypeId=artifactory_action_type)
        while not jobs['jobs']:
            time.sleep(10)
            jobs = codepipeline.poll_for_jobs(actionTypeId=artifactory_action_type)
            if jobs['jobs']:
                print('Job found')
        return jobs['jobs'][0]
    except ClientError as e:
        print("Received an error: %s" % str(e))
        raise

When there is a job in the queue, the poller returns a number of values from the queue such as jobId, the input and output S3 buckets for artifacts, temporary credentials to access the S3 buckets, and other configuration details from the stage in the pipeline.

Here is an example of the return response:

{
	'jobs': [
		{
			'nonce': '3',
			'data': {
				'inputArtifacts': [
					{
						'name': 'Output',
						'location': {
							'type': 'S3',
							's3Location': {
								'objectKey': 'ArtifactoryNPMwithCo/Output/Key,
							'bucketName': '123456789012-codepipelineartifact-us-west-2'
							}
						}
					}
				],
				'pipelineContext': {
						'action': {
							'name': 'Deploy'
						},
						'pipelineName': 'ArtifactoryNPMwithCodeDeploy',
						'stage': {
							'name': 'Deploy'
						}
				},
				'actionTypeId': {
					'category': 'Deploy',
					'owner': 'Custom',
					'version': '1',
					'provider': 'Artifactory'
				},
				'outputArtifacts': [
					{
						'name': 'ArtifactoryOut',
						'location': {
							'type': 'S3',
							's3Location': {
								'objectKey': 'ArtifactoryNPMwithCo/Artifactor/Key,
								'bucketName': '123456789012-codepipelineartifact-us-west-2'
							}
						}
					}
				],
				'actionConfiguration': {
					'configuration': {
						'UserName': 'admin',
						'ArtifactoryHost': 'https://artifactory.myexamplehost.com',
						'Password': 'xxx',
						'EmailAddress': '[email protected]',
						'TypeOfArtifact': 'npm',
						'RepoKey': 'npm'
					}
				},
				'artifactCredentials': {
					'secretAccessKey': 'SECRET',
			               'sessionToken':‘FQoDYXdz...XdndMF',
					'accessKeyId': 'ACCESSKEY'
					}
				},
				'id': 'a0eb',
				'accountId': '123456789012
			}
		],
		'ResponseMetadata': {
			'RetryAttempts': 0,
			'HTTPStatusCode': 200,
			'RequestId': a88b-cdbd5d08b9de',
			'HTTPHeaders': {
				'x-amzn-requestid': '77343c2d-eff4’,
				'content-length': '2461',
				'content-type': 'application/x-amz-json-1.1'
			}
		}
}

After successfully receiving the job details, the worker sends an acknowledgement to CodePipeline to ensure that the work on the job is not duplicated by other workers watching for the same job:

def job_acknowledge(jobId, nonce):
    try:
        print('Acknowledging job')
        result = codepipeline.acknowledge_job(jobId=jobId, nonce=nonce)
        return result
    except Exception as e:
        print("Received an error when trying to acknowledge the job: %s" % str(e))
        raise

With the job now acknowledged, the worker publishes the source code artifact into the desired repository. The worker gets the value of the artifact S3 bucket and objectKey from the inputArtifacts in the response from the poll_for_jobs API request. Next, the worker creates a new directory in /tmp and downloads the S3 object into this directory:

def get_bucket_location(bucketName, init_client):
    region = init_client.get_bucket_location(Bucket=bucketName)['LocationConstraint']
    if not region:
        region = 'us-east-1'
    return region


def get_s3_artifact(bucketName, objectKey, ak, sk, st):
    init_s3 = boto3.client('s3')
    region = get_bucket_location(bucketName, init_s3)
    session = Session(aws_access_key_id=ak,
                      aws_secret_access_key=sk,
                      aws_session_token=st)

    s3 = session.resource('s3',
                          region_name=region,
                          config=botocore.client.Config(signature_version='s3v4'))
    try:
        tempdirname = tempfile.mkdtemp()
    except OSError as e:
        print('Could not write temp directory %s' % tempdirname)
        raise
    bucket = s3.Bucket(bucketName)
    obj = bucket.Object(objectKey)
    filename = tempdirname + '/' + objectKey
    try:
        if os.path.dirname(objectKey):
            directory = os.path.dirname(filename)
            os.makedirs(directory)
        print('Downloading the %s object and writing it to disk in %s location' % (objectKey, tempdirname))
        with open(filename, 'wb') as data:
            obj.download_fileobj(data)
    except ClientError as e:
        print('Downloading the object and writing the file to disk raised this error: ' + str(e))
        raise
    return(filename, tempdirname)

Because the downloaded artifact from S3 is a zip file, the worker must unzip it first. To have a clean area in which to work, I extract the downloaded zip archive into a new directory:

def unzip_codepipeline_artifact(artifact, origtmpdir):
    # create a new temp directory
    # Unzip artifact into new directory
    try:
        newtempdir = tempfile.mkdtemp()
        print('Extracting artifact %s into temporary directory %s' % (artifact, newtempdir))
        zip_ref = zipfile.ZipFile(artifact, 'r')
        zip_ref.extractall(newtempdir)
        zip_ref.close()
        shutil.rmtree(origtmpdir)
        return(os.listdir(newtempdir), newtempdir)
    except OSError as e:
        if e.errno != errno.EEXIST:
            shutil.rmtree(newtempdir)
            raise

The worker now has the npm package that I want to store in my Artifactory NPM repository.

To authenticate with the NPM repository, the worker requests a temporary token from the Artifactory host. After receiving this temporary token, it creates a .npmrc file in the worker user’s home directory that includes a hash of the user name and temporary token. After it has authenticated, the worker runs npm config set registry <URL OF REPOSITORY> to configure the npm registry value to be the Artifactory host. Next, the worker runs npm publish –registry <URL OF REPOSITORY>, which publishes the node package to the NPM repository in the Artifactory host.

def push_to_npm(configuration, artifact_list, temp_dir, jobId):
    reponame = configuration['RepoKey']
    art_type = configuration['TypeOfArtifact']
    print("Putting artifact into NPM repository " + reponame)
    token, hostname, username = gen_artifactory_auth_token(configuration)
    npmconfigfile = create_npmconfig_file(configuration, username, token)
    url = hostname + '/artifactory/api/' + art_type + '/' + reponame
    print("Changing directory to " + str(temp_dir))
    os.chdir(temp_dir)
    try:
        print("Publishing following files to the repository: %s " % os.listdir(temp_dir))
        print("Sending artifact to Artifactory NPM registry URL: " + url)
        subprocess.call(["npm", "config", "set", "registry", url])
        req = subprocess.call(["npm", "publish", "--registry", url])
        print("Return code from npm publish: " + str(req))
        if req != 0:
            err_msg = "npm ERR! Recieved non OK response while sending response to Artifactory. Return code from npm publish: " + str(req)
            signal_failure(jobId, err_msg)
        else:
            signal_success(jobId)
    except requests.exceptions.RequestException as e:
       print("Received an error when trying to commit artifact %s to repository %s: " % (str(art_type), str(configuration['RepoKey']), str(e)))
       raise
    return(req, npmconfigfile)

If the return value from publishing to the repository is not 0, the worker signals a failure to CodePipeline. If the value is 0, the worker signals success to CodePipeline to indicate that the stage of the pipeline has been completed successfully.

For the custom worker code, see npm_job_worker.py in the GitHub repository.

I run my custom worker on an EC2 instance using the command python npm_job_worker.py, with an optional --version flag that can be used to specify worker versions other than 1. Then I trigger a release change in my pipeline:

From my custom worker output logs, I have just committed a package named node_example at version 1.0.3:

On artifact: index.js
Committing to the repo: https://artifactory.myexamplehost.com/artifactory/api/npm/npm
Sending artifact to Artifactory URL: https:// artifactoryhost.myexamplehost.com/artifactory/api/npm/npm
npm config: 0
npm http PUT https://artifactory.myexamplehost.com/artifactory/api/npm/npm/node_example
npm http 201 https://artifactory.myexamplehost.com/artifactory/api/npm/npm/node_example
+ [email protected]
Return code from npm publish: 0
Signaling success to CodePipeline

After that has been built successfully, I can find my artifact in my Artifactory repository:

To help you automate this process, I have created this AWS CloudFormation template that automates the creation of the CodeBuild project, the custom action, and the CodePipeline pipeline. It also launches the Amazon EC2-based custom job worker in an AWS Auto Scaling group. This template requires you to have a VPC and CodeCommit repository for your Node.js project. If you do not currently have a VPC in which you want to run your custom worker EC2 instances, you can use this AWS QuickStart to create one. If you do not have an existing Node.js project, I’ve provided a sample project in the GitHub repository.

Conclusion

I‘ve shown you the steps to integrate your JFrog Artifactory repository with your CodePipeline workflow. I’ve shown you how to create a custom action in CodePipeline and how to create a custom worker that works in your CI/CD pipeline. To dig deeper into custom actions and see how you can integrate your Artifactory repositories into your AWS CodePipeline projects, check out the full code base on GitHub.

If you have any questions or feedback, feel free to reach out to us through the AWS CodePipeline forum.

Erin McGill is a Solutions Architect in the AWS Partner Program with a focus on DevOps and automation tooling.

HackSpace magazine 7: Internet of Everything

2018-05-24 Andrew Gregory

Post Syndicated from Andrew Gregory original https://www.raspberrypi.org/blog/hackspace-magazine-7-internet-of-everything/

We’re usually averse to buzzwords at HackSpace magazine, but not this month: in issue 7, we’re taking a deep dive into the Internet of Things.

Internet of Things (IoT)

To many people, IoT is a shady term used by companies to sell you something you already own, but this time with WiFi; to us, it’s a way to make our builds smarter, more useful, and more connected. In HackSpace magazine #7, you can join us on a tour of the boards that power IoT projects, marvel at the ways in which other makers are using IoT, and get started with your first IoT project!

Awesome projects

DIY retro computing: this issue, we’re taking our collective hat off to Spencer Owen. He stuck his home-brew computer on Tindie thinking he might make a bit of beer money — now he’s paying the mortgage with his making skills and inviting others to build modules for his machine. And if that tickles your fancy, why not take a crack at our Z80 tutorial? Get out your breadboard, assemble your jumper wires, and prepare to build a real-life computer!

Shameless patriotism: combine Lego, Arduino, and the car of choice for 1960 gold bullion thieves, and you’ve got yourself a groovy weekend project. We proudly present to you one man’s epic quest to add LED lights (controllable via a smartphone!) to his daughter’s LEGO Mini Cooper.

Makerspaces

Patriotism intensifies: for the last 200-odd years, the Black Country has been a hotbed of making. Urban Hax, based in Walsall, is the latest makerspace to show off its riches in the coveted Space of the Month pages. Every space has its own way of doing things, but not every space has a portrait of Rob Halford on the wall. All hail!

Diversity: advice on diversity often boils down to ‘Be nice to people’, which might feel more vague than actionable. This is where we come in to help: it is truly worth making the effort to give people of all backgrounds access to your makerspace, so we take a look at why it’s nice to be nice, and at the ways in which one makerspace has put niceness into practice — with great results.

And there’s more!

We also show you how to easily calculate the size and radius of laser-cut gears, use a bank of LEDs to etch PCBs in your own mini factory, and use chemistry to mess with your lunch menu.

All this plus much, much more waits for you in HackSpace magazine issue 7!

Get your copy of HackSpace magazine

If you like the sound of that, you can find HackSpace magazine in WHSmith, Tesco, Sainsbury’s, and independent newsagents in the UK. If you live in the US, check out your local Barnes & Noble, Fry’s, or Micro Center next week. We’re also shipping to stores in Australia, Hong Kong, Canada, Singapore, Belgium, and Brazil, so be sure to ask your local newsagent whether they’ll be getting HackSpace magazine.

And if you can’t get to the shops, fear not: you can subscribe from £4 an issue from our online shop. And if you’d rather try before you buy, you can always download the free PDF. Happy reading, and happy making!

The post HackSpace magazine 7: Internet of Everything appeared first on Raspberry Pi.

[$] What’s coming in OpenLDAP 2.5

2018-05-23 corbet

Post Syndicated from corbet original https://lwn.net/Articles/755207/rss

If pressed, I will admit to thinking that, if
NIS was good enough for Charles Babbage, it’s
good enough for me. I am therefore not a huge fan of
LDAP; I feel I can detect in it the heavy hand of the ITU,
which seems to
wish to apply X.500 to
everything. Nevertheless, for secure, distributed, multi-platform identity
management it’s quite hard to beat. If you decide to run an LDAP server
on Unix, one of the major free implementations is slapd, the core
engine of the OpenLDAP project.
Howard Chu is the chief architect of the project,
and spoke at FLOSS 2018 about the upcoming 2.5 release. Any rumors
that he might have passed the time while the room filled up by giving
a short but nicely rendered fiddle recital are completely true.

Security updates for Monday

2018-05-21 ris

Post Syndicated from ris original https://lwn.net/Articles/755076/rss

Security updates have been issued by Arch Linux (lib32-curl, lib32-libcurl-compat, lib32-libcurl-gnutls, libcurl-compat, and libcurl-gnutls), CentOS (firefox), Debian (imagemagick), Fedora (exiv2, LibRaw, and love), Gentoo (chromium), Mageia (kernel, librelp, and miniupnpc), openSUSE (curl, enigmail, ghostscript, libvorbis, lilypond, and thunderbird), Red Hat (Red Hat OpenStack Platform director), and Ubuntu (firefox).

All Systems Go! 2018 CfP Open

2018-05-21 Lennart Poettering

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/all-systems-go-2018-cfp-open.html

The All Systems Go! 2018 Call for Participation is Now Open!

The Call for Participation (CFP) for All Systems Go!
2018 is now open. We’d like to invite you
to submit your proposals for consideration to the CFP submission
site.

ASG image

The CFP will close on July 30th. Notification of acceptance and
non-acceptance will go out within 7 days of the closing of the CFP.

All topics relevant to foundational open-source Linux technologies are
welcome. In particular, however, we are looking for proposals
including, but not limited to, the following topics:

Low-level container executors and infrastructure
IoT and embedded OS infrastructure
BPF and eBPF filtering
OS, container, IoT image delivery and updating
Building Linux devices and applications
Low-level desktop technologies
Networking
System and service management
Tracing and performance measuring
IPC and RPC systems
Security and Sandboxing

While our focus is definitely more on the user-space side of things,
talks about kernel projects are welcome, as long as they have a clear
and direct relevance for user-space.

For more information please visit our conference
website!

Connect, collaborate, and learn at AWS Global Summits in 2018

2018-05-18 Tina Kelleher

Post Syndicated from Tina Kelleher original https://aws.amazon.com/blogs/big-data/connect-collaborate-and-learn-at-aws-global-summits-in-2018/

Regardless of your career path, there’s no denying that attending industry events can provide helpful career development opportunities — not only for improving and expanding your skill sets, but for networking as well. According to this article from PayScale.com, experts estimate that somewhere between 70-85% of new positions are landed through networking.

Narrowing our focus to networking opportunities with cloud computing professionals who’re working on tackling some of today’s most innovative and exciting big data solutions, attending big data-focused sessions at an AWS Global Summit is a great place to start.

AWS Global Summits are free events that bring the cloud computing community together to connect, collaborate, and learn about AWS. As the name suggests, these summits are held in major cities around the world, and attract technologists from all industries and skill levels who’re interested in hearing from AWS leaders, experts, partners, and customers.

In addition to networking opportunities with top cloud technology providers, consultants and your peers in our Partner and Solutions Expo, you’ll also hone your AWS skills by attending and participating in a multitude of education and training opportunities.

Here’s a brief sampling of some of the upcoming sessions relevant to big data professionals:

May 31^st : Big Data Architectural Patterns and Best Practices on AWS | AWS Summit – Mexico City

June 6^th-7^th: Various (click on the “Big Data & Analytics” header) | AWS Summit – Berlin

June 20-21^st : [email protected] | Public Sector Summit – Washington DC

June 21^st: Enabling Self Service for Data Scientists with AWS Service Catalog | AWS Summit – Sao Paulo

Be sure to check out the main page for AWS Global Summits, where you can see which cities have AWS Summits planned for 2018, register to attend an upcoming event, or provide your information to be notified when registration opens for a future event.

ExtraTorrent Replacement Displays Warning On Predecessor’s Shutdown Anniversary

2018-05-18 Andy

Post Syndicated from Andy original https://torrentfreak.com/extratorrent-replacement-displays-warning-on-predecessors-shutdown-anniversary-180518/

Exactly one year ago, millions of users in the BitTorrent community went into mourning with the shock depature of one of its major players.

ExtraTorrent was founded in back in November 2006, at a time when classic platforms such as TorrentSpy and Mininova were dominating the torrent site landscape. But with dedication and determination, the site amassed millions of daily visitors, outperforming every other torrent site apart from the mighty Pirate Bay.

Then, on May 17, 2017, everything came crashing down.

“ExtraTorrent has shut down permanently,” a note in the site read. “ExtraTorrent with all mirrors goes offline. We permanently erase all data. Stay away from fake ExtraTorrent websites and clones. Thx to all ET supporters and torrent community. ET was a place to be….”

While ExtraTorrent staff couldn’t be more clear in advising people to stay away from clones, few people listened to their warnings. Within hours, new sites appeared claiming to be official replacements for the much-loved torrent site and people flocked to them in their millions.

One of those was ExtraTorrent.ag, a torrent site connected to the operators of EZTV.ag, which appeared as a replacement in the wake of the official EZTV’s demise. Graphically very similar to the original ExtraTorrent, the .ag ‘replacement’ had none of its namesake’s community or unique content. But that didn’t dent its popularity.

ExtraTorrent.ag

At the start of this week, ExtraTorrent.ag was one of the most popular torrent sites on the Internet. With an Alexa rank of around 2,200, it would’ve clinched ninth position in our Top 10 Torrent Sites report earlier this year. However, after registering the site’s domain a year ago, something seems to have gone wrong.

Yesterday, on the anniversary of ExtraTorrent’s shutdown and exactly a year after the ExtraTorrent.ag domain was registered, ExtraTorrent.ag disappeared only to be replaced by a generic landing page, as shown below.

ExtraTorrent.ag landing page

This morning, however, there appear to be additional complications. Accessing with Firefox produces the page above but attempting to do so with Chrome produces an ominous security warning.

Chrome warning

Indeed, those protected by MalwareBytes won’t be able to access the page at all, since ExtraTorrent.ag redirects to the domain FindBetterResults.com, which the anti-malware app flags as malicious.

The change was reported to TF by the operator of domain unblocking site Unblocked.lol, which offers torrent site proxies as well as access to live TV and sports.

“I noticed when I started receiving emails saying ExtraTorrent was redirecting to some parked domain. When I jumped on the PC and checked myself it was just redirecting to a blank page,” he informs us.

“First I thought they’d blocked our IP address so I used some different ones. But I soon discovered the domain was in fact parked.”

So what has happened to this previously-functioning domain?

Whois records show that ExtraTorrent.ag was created on May 17, 2017 and appears to have been registered for a year. Yesterday, on May 17, 2018, the domain was updated to list what could potentially be a new owner, with an expiry date of May 17, 2019.

Once domains have expired, they usually enter an ‘Auto-Renew Grace Period’ for up to 45 days. This is followed by a 30-day ‘Redemption Grace Period’. At the end of this second period, domains cannot be renewed and are released for third-parties to register. That doesn’t appear to have been the case here.

So, to find out more about the sudden changes we reached out to the email address listed in the WHOIS report but received no response. Should we hear more we’ll update this report but in the meantime the Internet has lost one of its largest torrent sites and gained a rather pointless landing page with potential security risks.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

EC2 Instance Update – C5 Instances with Local NVMe Storage (C5d)

2018-05-18 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/ec2-instance-update-c5-instances-with-local-nvme-storage-c5d/

As you can see from my EC2 Instance History post, we add new instance types on a regular and frequent basis. Driven by increasingly powerful processors and designed to address an ever-widening set of use cases, the size and diversity of this list reflects the equally diverse group of EC2 customers!

Near the bottom of that list you will find the new compute-intensive C5 instances. With a 25% to 50% improvement in price-performance over the C4 instances, the C5 instances are designed for applications like batch and log processing, distributed and or real-time analytics, high-performance computing (HPC), ad serving, highly scalable multiplayer gaming, and video encoding. Some of these applications can benefit from access to high-speed, ultra-low latency local storage. For example, video encoding, image manipulation, and other forms of media processing often necessitates large amounts of I/O to temporary storage. While the input and output files are valuable assets and are typically stored as Amazon Simple Storage Service (S3) objects, the intermediate files are expendable. Similarly, batch and log processing runs in a race-to-idle model, flushing volatile data to disk as fast as possible in order to make full use of compute resources.

New C5d Instances with Local Storage
In order to meet this need, we are introducing C5 instances equipped with local NVMe storage. Available for immediate use in 5 regions, these instances are a great fit for the applications that I described above, as well as others that you will undoubtedly dream up! Here are the specs:

Instance Name	vCPUs	RAM	Local Storage	EBS Bandwidth	Network Bandwidth
c5d.large	2	4 GiB	1 x 50 GB NVMe SSD	Up to 2.25 Gbps	Up to 10 Gbps
c5d.xlarge	4	8 GiB	1 x 100 GB NVMe SSD	Up to 2.25 Gbps	Up to 10 Gbps
c5d.2xlarge	8	16 GiB	1 x 225 GB NVMe SSD	Up to 2.25 Gbps	Up to 10 Gbps
c5d.4xlarge	16	32 GiB	1 x 450 GB NVMe SSD	2.25 Gbps	Up to 10 Gbps
c5d.9xlarge	36	72 GiB	1 x 900 GB NVMe SSD	4.5 Gbps	10 Gbps
c5d.18xlarge	72	144 GiB	2 x 900 GB NVMe SSD	9 Gbps	25 Gbps

Other than the addition of local storage, the C5 and C5d share the same specs. Both are powered by 3.0 GHz Intel Xeon Platinum 8000-series processors, optimized for EC2 and with full control over C-states on the two largest sizes, giving you the ability to run two cores at up to 3.5 GHz using Intel Turbo Boost Technology.

You can use any AMI that includes drivers for the Elastic Network Adapter (ENA) and NVMe; this includes the latest Amazon Linux, Microsoft Windows (Server 2008 R2, Server 2012, Server 2012 R2 and Server 2016), Ubuntu, RHEL, SUSE, and CentOS AMIs.

Here are a couple of things to keep in mind about the local NVMe storage:

Naming – You don’t have to specify a block device mapping in your AMI or during the instance launch; the local storage will show up as one or more devices (/dev/nvme*1 on Linux) after the guest operating system has booted.

Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated.

Lifetime – Local NVMe devices have the same lifetime as the instance they are attached to, and do not stick around after the instance has been stopped or terminated.

Available Now
C5d instances are available in On-Demand, Reserved Instance, and Spot form in the US East (N. Virginia), US West (Oregon), EU (Ireland), US East (Ohio), and Canada (Central) Regions. Prices vary by Region, and are just a bit higher than for the equivalent C5 instances.

— Jeff;

PS – We will be adding local NVMe storage to other EC2 instance types in the months to come, so stay tuned!

timeShift(GrafanaBuzz, 1w) Issue 45

2018-05-18 Blogs on Grafana Labs Blog

Post Syndicated from Blogs on Grafana Labs Blog original https://grafana.com/blog/2018/05/18/timeshiftgrafanabuzz-1w-issue-45/

Welcome to TimeShift In addition to our weekly roundup of Grafana-related articles and upcoming events, we’re pleased to announce Grafana v5.1.3 has been released! Also – We’re hiring! Be sure to check out our open positions. Latest Release: Grafana 5.1.3 This latest point release fixes a scrolling issue that was reported in Firefox. In Graph panels, the legend text shifted with each move of the scrollbar (#11830). See what else is new in Grafana v5.

Extending Amazon Linux 2 with EPEL and Let’s Encrypt

2018-05-15 Chris Munns

Post Syndicated from Chris Munns original https://aws.amazon.com/blogs/compute/extending-amazon-linux-2-with-epel-and-lets-encrypt/

This post courtesy of Jeff Levine Solutions Architect for Amazon Web Services

Amazon Linux 2 is the next generation of Amazon Linux, a Linux server operating system from Amazon Web Services (AWS). Amazon Linux 2 offers a high-performance Linux environment suitable for organizations of all sizes. It supports applications ranging from small websites to enterprise-class, mission-critical platforms.

Amazon Linux 2 includes support for the LAMP (Linux/Apache/MariaDB/PHP) stack, one of the most popular platforms for deploying websites. To secure the transmission of data-in-transit to such websites and prevent eavesdropping, organizations commonly leverage Secure Sockets Layer/Transport Layer Security (SSL/TLS) services which leverage certificates to provide encryption. The LAMP stack provided by Amazon Linux 2 includes a self-signed SSL/TLS certificate. Such certificates may be fine for internal usage but are not acceptable when attestation by a certificate authority is required.

In this post, I discuss how to extend the capabilities of Amazon Linux 2 by installing Let’s Encrypt, a certificate authority provided by the Internet Security Research Group. Let’s Encrypt offers basic SSL/TLS certificates for DNS hosts at no charge that you can use to add encryption-in-transit to a single web server. For commercial or multi-server configurations, you should consider AWS Certificate Manager and Elastic Load Balancing.

Let’s Encrypt also requires the certbot package, which you install from EPEL, the Extra Packaged for Enterprise Linux collection. Although EPEL is not included with Amazon Linux 2, I show how you can install it from the Fedora Project.

Walkthrough

At a high level, you perform the following tasks for this walkthrough:

Provision a VPC, Amazon Linux 2 instance, and LAMP stack.
Install and enable the EPEL repository.
Install and configure Let’s Encrypt.
Validate the installation.
Clean up.

Prerequisites and costs

To follow along with this walkthrough, you need the following:
An AWS account that provides access to Amazon EC2 and Amazon VPC.
An Amazon EC2 key pair.
A program such as PuTTY that allows you to connect to the Amazon Linux 2 instance using the SSH protocol.
Working knowledge of Amazon EC2 and Amazon VPC.
The ability to configure DNS entries for a host domain.

You may incur charges for the resources you use including, but not limited to, the Amazon EC2 instance and the associated network charges.

Step 1: Provision a VPC, Amazon Linux 2 instance, and LAMP stack

Create a VPC with a single public subnet, a routing table, and an internet gateway.
Launch an Amazon Linux 2 instance in the VPC that you just created. Make sure that you do the following:
1. Select the Amazon Linux 2 AMI.
2. Choose t2.micro for the instance type.
3. Accept all other default values including with regard to storage.
4. Create a new security group and accept the default rule that allows TCP port 22 (SSH) from everywhere (0.0.0.0/0 in IPv4). For the purposes of this walkthrough, permitting access from all IP addresses is reasonable. In a production environment, you may restrict access to different addresses.
Allocate and associate an Elastic IP address to the server when it enters the running state.
Install a LAMP stack.
Browse to the Elastic IP address that you just created and confirm that you can see the Apache test page, as illustrated below.

Step 2: Install and enable EPEL

Connect to your Amazon Linux 2 instance at the Elastic IP address that you just created.
Download and install the EPEL repository using the following commands:
```
cd /tmp
wget -O epel.rpm –nv \
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y ./epel.rpm
```
Respond “Y” to all requests for approval to install the software.

Step 3: Install and configure Let’s Encrypt

If you are no longer connected to the Amazon Linux 2 instance, connect to it at the Elastic IP address that you just created.
Install certbot, the Let’s Encrypt client to be used to obtain an SSL/TLS certificate and install it into Apache.
```
sudo yum install python2-certbot-apache.noarch
```
Respond “Y” to all requests for approval to install the software.
If you see a message appear about SELinux, you can safely ignore it. This is a known issue with the latest version of certbot.
Create a DNS “A record” that maps a host name to the Elastic IP address. For this post, assume that the name of the host is lamp.example.com. If you are hosting your DNS in Amazon Route 53, do this by creating the appropriate record set.
After the “A record” has propagated, browse to lamp.example.com. The Apache test page should appear. If the page does not appear, use a tool such as nslookup on your workstation to confirm that the DNS record has been properly configured.
You are now ready to install Let’s Encrypt. Let’s Encrypt does the following:
- Confirms that you have control over the DNS domain being used, by having you create a DNS TXT record using the value that it provides.
- Obtains an SSL/TLS certificate.
- Modifies the Apache-related scripts to use the SSL/TLS certificate and redirects users browsing the site in HTTP mode to HTTPS mode.

Use the following command to install certbot:

sudo certbot -i apache -a manual \
--preferred-challenges dns -d lamp.example.com

The options have the following meanings:


-i apache                                          Use the Apache installer.
-a manual                                          Authenticate domain ownership manually.
--preferred-challenges dns    Use DNS TXT records for authentication challenge.
-d lamp.example.com           Specify the domain for the SSL/TLS certificate.

You are prompted for the following information:
E-mail address for renewals?                Enter an email address for certificate renewals.
Accept the terms of services?               Respond as appropriate.
Send your e-mail address to the EFF?    Respond as appropriate.
Log your current IP address?                Respond as appropriate.
You are prompted to deploy a DNS TXT record with the name “_acme-challenge.lamp.example.com” with the supplied value, as shown below.
After you enter the record, wait until the TXT record propagates. To look up the TXT record to confirm the deployment, use the nslookup command in a separate command window, as shown below. Remember to use the set ty=txt command before entering the TXT record.
You are prompted to select a virtual host. There is only one, so choose 1. The final prompt asks whether to redirect HTTP traffic to HTTPS. To perform the redirection, choose 2. That completes the configuration of Let’s Encrypt.
To enable HTTPS (TCP port 443) traffic, add a rule to the security group for your Amazon Linux 2 instance.

Step: 4: Validate the installation

Browse to the http:// lamp.example.com site. You are redirected to the SSL/TLS page https://lamp.example.com.
To look at the encryption information, use the appropriate actions within your browser. For example, in Firefox, you can open the padlock and traverse the menus.
In the encryption technical details, you can see from the “Connection Encrypted” line that traffic to the website is now encrypted using TLS 1.2.

Security note: As of the time of publication, this website also supports TLS 1.0. I recommend that you disable this protocol because of some known vulnerabilities associated with it. To do this:

Edit the file /etc/letsencrypt/options-ssl-apache.conf.
Look for the line beginning with SSLProtocol and change it to the following:
```
SSLProtocol             all -SSLv2 -SSLv3 -TLSv1
```
Save the file. After you make changes to this file, Let’s Encrypt no longer automatically updates it. Periodically check your log files for recommended updates to this file.
Restart the httpd server with the following command:
```
sudo service httpd restart
```

Step 5: Cleanup

Use the following steps to avoid incurring any further costs.

Terminate the Amazon Linux 2 instance that you created.
Release the Elastic IP address that you allocated.
Revert any DNS changes that you made, including the A and TXT records.

Conclusion

Amazon Linux 2 is an excellent option for hosting websites through the LAMP stack provided by the Amazon-Linux-Extras feature. You can then enhance the security of the Apache web server by installing EPEL and Let’s Encrypt. Let’s Encrypt provisions an SSL/TLS certificate, optionally installs it for you on the Apache server, and enables data-in-transit encryption.
You can get started with Amazon Linux 2 in just a few clicks.

The AWS Shared Responsibility Model and GDPR

2018-05-15 Chad Woolf

Post Syndicated from Chad Woolf original https://aws.amazon.com/blogs/security/the-aws-shared-responsibility-model-and-gdpr/

The EU’s General Data Protection Regulation (GDPR) describes data processor and data controller roles, and some customers and AWS Partner Network (APN) partners are asking how this affects the long-established AWS Shared Responsibility Model. I wanted to take some time to help folks understand shared responsibilities for us and for our customers in context of the GDPR.

How does the AWS Shared Responsibility Model change under GDPR? The short answer – it doesn’t. AWS is responsible for securing the underlying infrastructure that supports the cloud and the services provided; while customers and APN partners, acting either as data controllers or data processors, are responsible for any personal data they put in the cloud. The shared responsibility model illustrates the various responsibilities of AWS and our customers and APN partners, and the same separation of responsibility applies under the GDPR.

AWS responsibilities as a data processor

The GDPR does introduce specific regulation and responsibilities regarding data controllers and processors. When any AWS customer uses our services to process personal data, the controller is usually the AWS customer (and sometimes it is the AWS customer’s customer). However, in all of these cases, AWS is always the data processor in relation to this activity. This is because the customer is directing the processing of data through its interaction with the AWS service controls, and AWS is only executing customer directions. As a data processor, AWS is responsible for protecting the global infrastructure that runs all of our services. Controllers using AWS maintain control over data hosted on this infrastructure, including the security configuration controls for handling end-user content and personal data. Protecting this infrastructure, is our number one priority, and we invest heavily in third-party auditors to test our security controls and make any issues they find available to our customer base through AWS Artifact. Our ISO 27018 report is a good example, as it tests security controls that focus on protection of personal data in particular.

AWS has an increased responsibility for our managed services. Examples of managed services include Amazon DynamoDB, Amazon RDS, Amazon Redshift, Amazon Elastic MapReduce, and Amazon WorkSpaces. These services provide the scalability and flexibility of cloud-based resources with less operational overhead because we handle basic security tasks like guest operating system (OS) and database patching, firewall configuration, and disaster recovery. For most managed services, you only configure logical access controls and protect account credentials, while maintaining control and responsibility of any personal data.

Customer and APN partner responsibilities as data controllers — and how AWS Services can help

Our customers can act as data controllers or data processors within their AWS environment. As a data controller, the services you use may determine how you configure those services to help meet your GDPR compliance needs. For example, AWS Services that are classified as Infrastructure as a Service (IaaS), such as Amazon EC2, Amazon VPC, and Amazon S3, are under your control and require you to perform all routine security configuration and management that would be necessary no matter where the servers were located. With Amazon EC2 instances, you are responsible for managing: guest OS (including updates and security patches), application software or utilities installed on the instances, and the configuration of the AWS-provided firewall (called a security group).

To help you realize data protection by design principles under the GDPR when using our infrastructure, we recommend you protect AWS account credentials and set up individual user accounts with Amazon Identity and Access Management (IAM) so that each user is only given the permissions necessary to fulfill their job duties. We also recommend using multi-factor authentication (MFA) with each account, requiring the use of SSL/TLS to communicate with AWS resources, setting up API/user activity logging with AWS CloudTrail, and using AWS encryption solutions, along with all default security controls within AWS Services. You can also use advanced managed security services, such as Amazon Macie, which assists in discovering and securing personal data stored in Amazon S3.

For more information, you can download the AWS Security Best Practices whitepaper or visit the AWS Security Resources or GDPR Center webpages. In addition to our solutions and services, AWS APN partners can provide hundreds of tools and features to help you meet your security objectives, ranging from network security and configuration management to access control and data encryption.

Brutus 2: the gaming PC case of your dreams

2018-05-15 Janina Ander

Post Syndicated from Janina Ander original https://www.raspberrypi.org/blog/brutus-2-gaming-pc-case/

Attention, case modders: take a look at the Brutus 2, an extremely snazzy computer case with a partly transparent, animated side panel that’s powered by a Pi. Daniel Otto and Carsten Lehman have a current crowdfunder for the case; their video is in German, but the looks of the build speak for themselves. There are some truly gorgeous effects here.

der BRUTUS 2 by 3nb Gaming

Vorbestellungen ab sofort auf https://www.startnext.com/brutus2 Weitere Infos zu uns auf: https://3nb.de https://www.facebook.com/3nb.de https://www.instagram.com/3nb.de Über 3nb: – GbR aus Leipzig, gegründet 2017 – wir kommen aus den Bereichen Elektronik und Informatik – erstes Produkt: der Brutus One ein Gaming PC mit transparentem Display in der Seite Kurzinfo Brutus 2: – Markencomputergehäuse für Gaming- /Casemoddingszene – Besonderheit: animiertes Seitenfenster angesteuert mit einem Raspberry Pi – Vorteile von unserem Case: o Case ist einzeln lieferbar und nicht nur als komplett-PC o kein Leistungsverbrauch der Grafikkarte dank integriertem Raspberry Pi o bessere Darstellung von Texten und Grafiken durch unscharfen Hintergrund

What’s case modding?

Case modding just means modifying your computer or gaming console’s case, and it’s very popular in the gaming community. Some mods are functional, while others improve the way the case looks. Lots of dedicated gamers don’t only want a powerful computer, they also want it to look amazing — at home, or at LAN parties and games tournaments.

The Brutus 2 case

The Brutus 2 case is made by Daniel and Carsten’s startup, 3nb electronics, and it’s a product that is officially Powered by Raspberry Pi. Its standout feature is the semi-transparent TFT screen, which lets you play any video clip you choose while keeping your gaming hardware on display. It looks incredibly cool. All the graphics for the case’s screen are handled by a Raspberry Pi, so it doesn’t use any of your main PC’s GPU power and your gaming won’t suffer.

The software

To use Brutus 2, you just need to run a small desktop application on your PC to choose what you want to display on the case. A number of neat animations are included, and you can upload your own if you want.

So far, the app only runs on Windows, but 3nb electronics are planning to make the code open-source, so you can modify it for other operating systems, or to display other file types. This is true to the spirit of the case modding and Raspberry Pi communities, who love adapting, retrofitting, and overhauling projects and code to fit their needs.

Daniel and Carsten say that one of their campaign’s stretch goals is to implement more functionality in the Brutus 2 app. So in the future, the case could also show things like CPU temperature, gaming stats, and in-game messages. Of course, there’s nothing stopping you from integrating features like that yourself.

If you have any questions about the case, you can post them directly to Daniel and Carsten here.

The crowdfunding campaign

The Brutus 2 campaign on Startnext is currently halfway to its first funding goal of €10000, with over three weeks to go until it closes. If you’re quick, you still be may be able to snatch one of the early-bird offers. And if your whole guild NEEDS this, that’s OK — there are discounts for bulk orders.

The post Brutus 2: the gaming PC case of your dreams appeared first on Raspberry Pi.

Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena, and Amazon Redshift

2018-05-11 Roy Hasson

Post Syndicated from Roy Hasson original https://aws.amazon.com/blogs/big-data/analyzing-apache-parquet-optimized-data-using-amazon-kinesis-data-firehose-amazon-athena-and-amazon-redshift/

Amazon Kinesis Data Firehose is the easiest way to capture and stream data into a data lake built on Amazon S3. This data can be anything—from AWS service logs like AWS CloudTrail log files, Amazon VPC Flow Logs, Application Load Balancer logs, and others. It can also be IoT events, game events, and much more. To efficiently query this data, a time-consuming ETL (extract, transform, and load) process is required to massage and convert the data to an optimal file format, which increases the time to insight. This situation is less than ideal, especially for real-time data that loses its value over time.

To solve this common challenge, Kinesis Data Firehose can now save data to Amazon S3 in Apache Parquet or Apache ORC format. These are optimized columnar formats that are highly recommended for best performance and cost-savings when querying data in S3. This feature directly benefits you if you use Amazon Athena, Amazon Redshift, AWS Glue, Amazon EMR, or any other big data tools that are available from the AWS Partner Network and through the open-source community.

Amazon Connect is a simple-to-use, cloud-based contact center service that makes it easy for any business to provide a great customer experience at a lower cost than common alternatives. Its open platform design enables easy integration with other systems. One of those systems is Amazon Kinesis—in particular, Kinesis Data Streams and Kinesis Data Firehose.

What’s really exciting is that you can now save events from Amazon Connect to S3 in Apache Parquet format. You can then perform analytics using Amazon Athena and Amazon Redshift Spectrum in real time, taking advantage of this key performance and cost optimization. Of course, Amazon Connect is only one example. This new capability opens the door for a great deal of opportunity, especially as organizations continue to build their data lakes.

Amazon Connect includes an array of analytics views in the Administrator dashboard. But you might want to run other types of analysis. In this post, I describe how to set up a data stream from Amazon Connect through Kinesis Data Streams and Kinesis Data Firehose and out to S3, and then perform analytics using Athena and Amazon Redshift Spectrum. I focus primarily on the Kinesis Data Firehose support for Parquet and its integration with the AWS Glue Data Catalog, Amazon Athena, and Amazon Redshift.

Solution overview

Here is how the solution is laid out:

The following sections walk you through each of these steps to set up the pipeline.

1. Define the schema

When Kinesis Data Firehose processes incoming events and converts the data to Parquet, it needs to know which schema to apply. The reason is that many times, incoming events contain all or some of the expected fields based on which values the producers are advertising. A typical process is to normalize the schema during a batch ETL job so that you end up with a consistent schema that can easily be understood and queried. Doing this introduces latency due to the nature of the batch process. To overcome this issue, Kinesis Data Firehose requires the schema to be defined in advance.

To see the available columns and structures, see Amazon Connect Agent Event Streams. For the purpose of simplicity, I opted to make all the columns of type String rather than create the nested structures. But you can definitely do that if you want.

The simplest way to define the schema is to create a table in the Amazon Athena console. Open the Athena console, and paste the following create table statement, substituting your own S3 bucket and prefix for where your event data will be stored. A Data Catalog database is a logical container that holds the different tables that you can create. The default database name shown here should already exist. If it doesn’t, you can create it or use another database that you’ve already created.

CREATE EXTERNAL TABLE default.kfhconnectblog (
  awsaccountid string,
  agentarn string,
  currentagentsnapshot string,
  eventid string,
  eventtimestamp string,
  eventtype string,
  instancearn string,
  previousagentsnapshot string,
  version string
)
STORED AS parquet
LOCATION 's3://your_bucket/kfhconnectblog/'
TBLPROPERTIES ("parquet.compression"="SNAPPY")

That’s all you have to do to prepare the schema for Kinesis Data Firehose.

2. Define the data streams

Next, you need to define the Kinesis data streams that will be used to stream the Amazon Connect events. Open the Kinesis Data Streams console and create two streams. You can configure them with only one shard each because you don’t have a lot of data right now.

3. Define the Kinesis Data Firehose delivery stream for Parquet

Let’s configure the Data Firehose delivery stream using the data stream as the source and Amazon S3 as the output. Start by opening the Kinesis Data Firehose console and creating a new data delivery stream. Give it a name, and associate it with the Kinesis data stream that you created in Step 2.

As shown in the following screenshot, enable Record format conversion (1) and choose Apache Parquet (2). As you can see, Apache ORC is also supported. Scroll down and provide the AWS Glue Data Catalog database name (3) and table names (4) that you created in Step 1. Choose Next.

To make things easier, the output S3 bucket and prefix fields are automatically populated using the values that you defined in the LOCATION parameter of the create table statement from Step 1. Pretty cool. Additionally, you have the option to save the raw events into another location as defined in the Source record S3 backup section. Don’t forget to add a trailing forward slash “ / “ so that Data Firehose creates the date partitions inside that prefix.

On the next page, in the S3 buffer conditions section, there is a note about configuring a large buffer size. The Parquet file format is highly efficient in how it stores and compresses data. Increasing the buffer size allows you to pack more rows into each output file, which is preferred and gives you the most benefit from Parquet.

Compression using Snappy is automatically enabled for both Parquet and ORC. You can modify the compression algorithm by using the Kinesis Data Firehose API and update the OutputFormatConfiguration.

Be sure to also enable Amazon CloudWatch Logs so that you can debug any issues that you might run into.

Lastly, finalize the creation of the Firehose delivery stream, and continue on to the next section.

4. Set up the Amazon Connect contact center

After setting up the Kinesis pipeline, you now need to set up a simple contact center in Amazon Connect. The Getting Started page provides clear instructions on how to set up your environment, acquire a phone number, and create an agent to accept calls.

After setting up the contact center, in the Amazon Connect console, choose your Instance Alias, and then choose Data Streaming. Under Agent Event, choose the Kinesis data stream that you created in Step 2, and then choose Save.

At this point, your pipeline is complete. Agent events from Amazon Connect are generated as agents go about their day. Events are sent via Kinesis Data Streams to Kinesis Data Firehose, which converts the event data from JSON to Parquet and stores it in S3. Athena and Amazon Redshift Spectrum can simply query the data without any additional work.

So let’s generate some data. Go back into the Administrator console for your Amazon Connect contact center, and create an agent to handle incoming calls. In this example, I creatively named mine Agent One. After it is created, Agent One can get to work and log into their console and set their availability to Available so that they are ready to receive calls.

To make the data a bit more interesting, I also created a second agent, Agent Two. I then made some incoming and outgoing calls and caused some failures to occur, so I now have enough data available to analyze.

5. Analyze the data with Athena

Let’s open the Athena console and run some queries. One thing you’ll notice is that when we created the schema for the dataset, we defined some of the fields as Strings even though in the documentation they were complex structures. The reason for doing that was simply to show some of the flexibility of Athena to be able to parse JSON data. However, you can define nested structures in your table schema so that Kinesis Data Firehose applies the appropriate schema to the Parquet file.

Let’s run the first query to see which agents have logged into the system.

The query might look complex, but it’s fairly straightforward:

WITH dataset AS (
  SELECT 
    from_iso8601_timestamp(eventtimestamp) AS event_ts,
    eventtype,
    -- CURRENT STATE
    json_extract_scalar(
      currentagentsnapshot,
      '$.agentstatus.name') AS current_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        currentagentsnapshot,
        '$.agentstatus.starttimestamp')) AS current_starttimestamp,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.firstname') AS current_firstname,
    json_extract_scalar(
      currentagentsnapshot,
      '$.configuration.lastname') AS current_lastname,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.username') AS current_username,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') AS               current_outboundqueue,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as current_inboundqueue,
    -- PREVIOUS STATE
    json_extract_scalar(
      previousagentsnapshot, 
      '$.agentstatus.name') as prev_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        previousagentsnapshot, 
       '$.agentstatus.starttimestamp')) as prev_starttimestamp,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.firstname') as prev_firstname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.lastname') as prev_lastname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.username') as prev_username,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') as current_outboundqueue,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as prev_inboundqueue
  from kfhconnectblog
  where eventtype <> 'HEART_BEAT'
)
SELECT
  current_status as status,
  current_username as username,
  event_ts
FROM dataset
WHERE eventtype = 'LOGIN' AND current_username <> ''
ORDER BY event_ts DESC

The query output looks something like this:

Here is another query that shows the sessions each of the agents engaged with. It tells us where they were incoming or outgoing, if they were completed, and where there were missed or failed calls.

WITH src AS (
  SELECT
     eventid,
     json_extract_scalar(currentagentsnapshot, '$.configuration.username') as username,
     cast(json_extract(currentagentsnapshot, '$.contacts') AS ARRAY(JSON)) as c,
     cast(json_extract(previousagentsnapshot, '$.contacts') AS ARRAY(JSON)) as p
  from kfhconnectblog
),
src2 AS (
  SELECT *
  FROM src CROSS JOIN UNNEST (c, p) AS contacts(c_item, p_item)
),
dataset AS (
SELECT 
  eventid,
  username,
  json_extract_scalar(c_item, '$.contactid') as c_contactid,
  json_extract_scalar(c_item, '$.channel') as c_channel,
  json_extract_scalar(c_item, '$.initiationmethod') as c_direction,
  json_extract_scalar(c_item, '$.queue.name') as c_queue,
  json_extract_scalar(c_item, '$.state') as c_state,
  from_iso8601_timestamp(json_extract_scalar(c_item, '$.statestarttimestamp')) as c_ts,
  
  json_extract_scalar(p_item, '$.contactid') as p_contactid,
  json_extract_scalar(p_item, '$.channel') as p_channel,
  json_extract_scalar(p_item, '$.initiationmethod') as p_direction,
  json_extract_scalar(p_item, '$.queue.name') as p_queue,
  json_extract_scalar(p_item, '$.state') as p_state,
  from_iso8601_timestamp(json_extract_scalar(p_item, '$.statestarttimestamp')) as p_ts
FROM src2
)
SELECT 
  username,
  c_channel as channel,
  c_direction as direction,
  p_state as prev_state,
  c_state as current_state,
  c_ts as current_ts,
  c_contactid as id
FROM dataset
WHERE c_contactid = p_contactid
ORDER BY id DESC, current_ts ASC

The query output looks similar to the following:

6. Analyze the data with Amazon Redshift Spectrum

With Amazon Redshift Spectrum, you can query data directly in S3 using your existing Amazon Redshift data warehouse cluster. Because the data is already in Parquet format, Redshift Spectrum gets the same great benefits that Athena does.

Here is a simple query to show querying the same data from Amazon Redshift. Note that to do this, you need to first create an external schema in Amazon Redshift that points to the AWS Glue Data Catalog.

SELECT 
  eventtype,
  json_extract_path_text(currentagentsnapshot,'agentstatus','name') AS current_status,
  json_extract_path_text(currentagentsnapshot, 'configuration','firstname') AS current_firstname,
  json_extract_path_text(currentagentsnapshot, 'configuration','lastname') AS current_lastname,
  json_extract_path_text(
    currentagentsnapshot,
    'configuration','routingprofile','defaultoutboundqueue','name') AS current_outboundqueue,
FROM default_schema.kfhconnectblog

The following shows the query output:

Summary

In this post, I showed you how to use Kinesis Data Firehose to ingest and convert data to columnar file format, enabling real-time analysis using Athena and Amazon Redshift. This great feature enables a level of optimization in both cost and performance that you need when storing and analyzing large amounts of data. This feature is equally important if you are investing in building data lakes on AWS.

Additional Reading

If you found this post useful, be sure to check out Analyzing VPC Flow Logs with Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight and Work with partitioned data in AWS Glue.

About the Author

Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan cheering his team on and hanging out with his family.

Introducing the AWS Machine Learning Competency for Consulting Partners

2018-05-10 Randall Hunt

Post Syndicated from Randall Hunt original https://aws.amazon.com/blogs/aws/introducing-the-aws-machine-learning-competency-for-consulting-partners/

Today I’m excited to announce a new Machine Learning Competency for Consulting Partners in the Amazon Partner Network (APN). This AWS Competency program allows APN Consulting Partners to demonstrate a deep expertise in machine learning on AWS by providing solutions that enable machine learning and data science workflows for their customers. This new AWS Competency is in addition to the Machine Learning comptency for our APN Technology Partners, that we launched at the re:Invent 2017 partner summit.

These APN Consulting Partners help organizations solve their machine learning and data challenges through:

Providing data services that help data scientists and machine learning practitioners prepare their enterprise data for training.
Platform solutions that provide data scientists and machine learning practitioners with tools to take their data, train models, and make predictions on new data.
SaaS and API solutions to enable predictive capabilities within customer applications.

Why work with an AWS Machine Learning Competency Partner?

The AWS Competency Program helps customers find the most qualified partners with deep expertise. AWS Machine Learning Competency Partners undergo a strict validation of their capabilities to demonstrate technical proficiency and proven customer success with AWS machine learning tools.

If you’re an AWS customer interested in machine learning workloads on AWS, check out our AWS Machine Learning launch partners below:

Interested in becoming an AWS Machine Learning Competency Partner?

APN Partners with experience in Machine Learning can learn more about becoming an AWS Machine Learning Competency Partner here. To learn more about the benefits of joining the AWS Partner Network, see our APN Partner website.

Thanks to the AWS Partner Team for their help with this post!
– Randall

AWS Online Tech Talks – May and Early June 2018

2018-05-10 Devin Watson

Post Syndicated from Devin Watson original https://aws.amazon.com/blogs/aws/aws-online-tech-talks-may-and-early-june-2018/

AWS Online Tech Talks – May and Early June 2018

Join us this month to learn about some of the exciting new services and solution best practices at AWS. We also have our first re:Invent 2018 webinar series, “How to re:Invent”. Sign up now to learn more, we look forward to seeing you.

Note – All sessions are free and in Pacific Time.

Tech talks featured this month:

Analytics & Big Data

May 21, 2018 | 11:00 AM – 11:45 AM PT – Integrating Amazon Elasticsearch with your DevOps Tooling – Learn how you can easily integrate Amazon Elasticsearch Service into your DevOps tooling and gain valuable insight from your log data.

May 23, 2018 | 11:00 AM – 11:45 AM PT – Data Warehousing and Data Lake Analytics, Together – Learn how to query data across your data warehouse and data lake without moving data.

May 24, 2018 | 11:00 AM – 11:45 AM PT – Data Transformation Patterns in AWS – Discover how to perform common data transformations on the AWS Data Lake.

Compute

May 29, 2018 | 01:00 PM – 01:45 PM PT – Creating and Managing a WordPress Website with Amazon Lightsail – Learn about Amazon Lightsail and how you can create, run and manage your WordPress websites with Amazon’s simple compute platform.

May 30, 2018 | 01:00 PM – 01:45 PM PT – Accelerating Life Sciences with HPC on AWS – Learn how you can accelerate your Life Sciences research workloads by harnessing the power of high performance computing on AWS.

Containers

May 24, 2018 | 01:00 PM – 01:45 PM PT – Building Microservices with the 12 Factor App Pattern on AWS – Learn best practices for building containerized microservices on AWS, and how traditional software design patterns evolve in the context of containers.

Databases

May 21, 2018 | 01:00 PM – 01:45 PM PT – How to Migrate from Cassandra to Amazon DynamoDB – Get the benefits, best practices and guides on how to migrate your Cassandra databases to Amazon DynamoDB.

May 23, 2018 | 01:00 PM – 01:45 PM PT – 5 Hacks for Optimizing MySQL in the Cloud – Learn how to optimize your MySQL databases for high availability, performance, and disaster resilience using RDS.

DevOps

May 23, 2018 | 09:00 AM – 09:45 AM PT – .NET Serverless Development on AWS – Learn how to build a modern serverless application in .NET Core 2.0.

Enterprise & Hybrid

May 22, 2018 | 11:00 AM – 11:45 AM PT – Hybrid Cloud Customer Use Cases on AWS – Learn how customers are leveraging AWS hybrid cloud capabilities to easily extend their datacenter capacity, deliver new services and applications, and ensure business continuity and disaster recovery.

IoT

May 31, 2018 | 11:00 AM – 11:45 AM PT – Using AWS IoT for Industrial Applications – Discover how you can quickly onboard your fleet of connected devices, keep them secure, and build predictive analytics with AWS IoT.

Machine Learning

May 22, 2018 | 09:00 AM – 09:45 AM PT – Using Apache Spark with Amazon SageMaker – Discover how to use Apache Spark with Amazon SageMaker for training jobs and application integration.

May 24, 2018 | 09:00 AM – 09:45 AM PT – Introducing AWS DeepLens – Learn how AWS DeepLens provides a new way for developers to learn machine learning by pairing the physical device with a broad set of tutorials, examples, source code, and integration with familiar AWS services.

Management Tools

May 21, 2018 | 09:00 AM – 09:45 AM PT – Gaining Better Observability of Your VMs with Amazon CloudWatch – Learn how CloudWatch Agent makes it easy for customers like Rackspace to monitor their VMs.

Mobile

May 29, 2018 | 11:00 AM – 11:45 AM PT – Deep Dive on Amazon Pinpoint Segmentation and Endpoint Management – See how segmentation and endpoint management with Amazon Pinpoint can help you target the right audience.

Networking

May 31, 2018 | 09:00 AM – 09:45 AM PT – Making Private Connectivity the New Norm via AWS PrivateLink – See how PrivateLink enables service owners to offer private endpoints to customers outside their company.

Security, Identity, & Compliance

May 30, 2018 | 09:00 AM – 09:45 AM PT – Introducing AWS Certificate Manager Private Certificate Authority (CA) – Learn how AWS Certificate Manager (ACM) Private Certificate Authority (CA), a managed private CA service, helps you easily and securely manage the lifecycle of your private certificates.

June 1, 2018 | 09:00 AM – 09:45 AM PT – Introducing AWS Firewall Manager – Centrally configure and manage AWS WAF rules across your accounts and applications.

Serverless

May 22, 2018 | 01:00 PM – 01:45 PM PT – Building API-Driven Microservices with Amazon API Gateway – Learn how to build a secure, scalable API for your application in our tech talk about API-driven microservices.

Storage

May 30, 2018 | 11:00 AM – 11:45 AM PT – Accelerate Productivity by Computing at the Edge – Learn how AWS Snowball Edge support for compute instances helps accelerate data transfers, execute custom applications, and reduce overall storage costs.

June 1, 2018 | 11:00 AM – 11:45 AM PT – Learn to Build a Cloud-Scale Website Powered by Amazon EFS – Technical deep dive where you’ll learn tips and tricks for integrating WordPress, Drupal and Magento with Amazon EFS.

Cloudflare Fails to Exclude Daily Stormer Evidence From Piracy Trial

2018-05-04 Ernesto

Post Syndicated from Ernesto original https://torrentfreak.com/cloudflare-fails-to-exclude-daily-stormer-evidence-from-piracy-trial-180504/

Last summer Cloudflare CEO Matthew Prince decided to terminate the account of controversial neo-Nazi site Daily Stormer.

“I woke up this morning in a bad mood and decided to kick them off the Internet,” he announced.

The company’s lawyers later explained that the move was meant as an “intellectual exercise” to start a conversation regarding censorship and free speech on the internet. However, this discussion went much further than Prince had planned.

For years, Cloudflare had a policy not to remove any accounts without a court order, so when this was exceeded, eyebrows were raised. In particular, copyright holders wondered why the company could terminate this account but not those of the most notorious pirate sites.

This is also why The Daily Stormer removal became an issue in the piracy liability case previously filed by adult entertainment publisher ALS Scan. After Cloudflare’s CEO was questioned on the matter, it could be raised before a jury during the trial as well.

Cloudflare didn’t fancy this prospect. In March, the company asked the court to preclude any evidence related to Daily Stormer or other hate groups from the upcoming trial, fearing that it would lead to “guilt by association.”

“The apparent reason that ALS seeks to offer is not for its probative value but rather for its distracting emotional impact,” Cloudflare argued.

“Given the strong feelings such evidence would almost certainly arouse among members of the jury, this evidence creates an unwarranted and impermissible risk of unfair prejudice to Cloudflare.”

However, California District Court Judge George Wu was not receptive to this argument. Following a hearing on the matter last week the Judge denied the motion, which means that ALS is allowed to use the Daily Stormer case at trial.

“[Cloudflare’s motion] to Exclude Evidence Relating to Provision or Termination of Services to Hate Groups is DENIED.”

Motion denied

In hindsight, Cloudflare’s decision to disconnect Daily Stormer the way it did might not have been the best option, but t’s too late now.

According to recent court filings AS and Cloudflare have tried to reach a settlement but thus far that hasn’t happened. This means that the case will move to the scheduled trial, unless both sides can make peace beforehand.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

MPAA-Seized Popcorn Time Domain Now Redirects to Pirate Site

2018-05-03 Ernesto

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-seized-popcorn-time-domain-now-redirects-to-pirate-site-180503/

Four years ago Popcorn Time took the Internet by storm.

The software amassed millions of users by offering BitTorrent-powered streaming in an easy-to-use Netflix-style interface.

While the original developers shut down their project after a few months, following pressure from Hollywood, others forked the application and took over.

PopcornTime.io swiftly became the main Popcorn Time fork. The spin-off soon had millions of users and updates were pushed out on a regular basis. At the end of 2015, however, this fork also disappeared from the web.

The MPAA took credit for the fall announcing that it had filed a lawsuit against several people in Canada. In response to these legal threats, several key developers backed out.

Soon after, the MPAA also assumed control of the main domain name, ensuring that it could not fall into the wrong hands.

This worked well, initially, but this week we noticed that PopcornTime.io is active again. The domain now links to the pirate streaming site Stream.cr, which welcomes its new visitors with a special message.

Redirection landing page

“Notice: If you’re looking for Popcorn Time(App) for it’s P2P torrent streaming, it’s over at popcorntime.sh. Otherwise, if you’re looking for streaming. Welcome to StreamCR!” a message on the site reads.

This is odd, considering that the PopcornTime.io domain name is still registered to the MPAA.

Popcorntime.io Whois

Adding to the intrigue is the fact that the PopcornTime.io domain registrar is listed as MarkMonitor, which is a well-known brand protection company, often used to prevent domain troubles.

“Protect your critical assets by partnering with a corporate-only domain registrar who has a strong security culture and is committed to providing the most secure and reliable solution in the industry,” MarkMonitor writes

However, since PopcornTime.io now links to a pirate site, something clearly went wrong.

It’s hard to say with certainty what happened. A likely option is that the domain’s nameservers, which point to DNS Made Easy, were not configured properly and that the people behind Stream.cr used that oversight to redirect the domain to their own site.

TorrentFreak spoke to a source unrelated to this case who says he was previously able to redirect traffic from a domain that was seized by the MPAA, simply by adding it to his own DNS Made Easy account. That worked, until the nameservers were updated to MarkMonitor’s DNS servers.

Whether the fault, in this instance, lies with the MPAA, MarkMonitor, or another party is hard to say without further details.

In any case, the MPAA is not going to be happy with the end result, and neither is MarkMonitor. The Stream.cr operators, meanwhile, are probably celebrating and they can enjoy the free traffic while it lasts.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

How to use Amazon AppStream 2.0 to reduce your bastion host attack surface

2018-05-02 Chaim Landau

Post Syndicated from Chaim Landau original https://aws.amazon.com/blogs/security/how-to-use-amazon-appstream-2-0-to-reduce-your-bastion-host-attack-surface/

To help protect their assets, many security-conscious enterprises require their system administrators to go through a “bastion” (or “jump”) host to gain administrative access to backend systems in protected or sensitive network segments.

A bastion host is a special-purpose instance that hosts a minimal number of administrative applications, such as RDP for Windows or Putty for Linux-based distributions. All other unnecessary services are removed. The host is typically placed in a segregated network (or “DMZ”), and is often protected with multi-factor authentication (MFA) and monitored with auditing tools. And most enterprises require that the access trail to the bastion host be auditable.

In this post, I demonstrate the use of Amazon AppStream 2.0 as a hardened and auto-scaled bastion host solution, and show how it could reduce the attack surface by stripping away the underlying OS and exposing only the necessary tools to system administrators that need access to protected network segments.

Prerequisites

A Virtual Private Cloud (VPC) with a dedicated subnet for AppStream 2.0.
An existing Active Directory (AD) domain. This may be on premises, on AWS EC2 for Windows, or AWS Directory Service for Microsoft Active Directory used as a user directory.
Active Directory Federation Services (ADFS).
A Linux or Windows instance for which AppStream 2.0 will be acting as a bastion host.

Solution overview

Amazon AppStream 2.0 is a fully managed application streaming service that provides users instant access to their desktop applications from anywhere by using an HTML5-compatible desktop browser. When a user requests access to an application, AppStream 2.0 uses a base image to deploy a streaming instance and destroys the instance after the user closes their session. This ensures the same consistent experience during each logon.

You can use AppStream 2.0 as a bastion solution to enable your system administrators to manage their environment without giving them a full bastion host. Because AppStream 2.0 freshly builds instances each time a user requests access, a compromised instance will only last for the duration of a user session. As soon as the user closes their session and the Disconnect Timeout period is reached, AppStream 2.0 terminates the instance and, with it, you’ve reduced your risks of compromised instances.

You will also potentially reduce your costs because AppStream 2.0 has built-in auto-scaling to increase and decrease capacity based on user demand. It allows you to take advantage of the pay-as-you-go model, where you only pay for what you use.

High-level AppStream 2.0 architecture

The diagram below depicts a high-level AppStream 2.0 architecture used as a bastion host for servers in another VPC.

There are three VPCs shown: AppStream 2.0 VPC, Bastion host VPC, and application VPC. The AppStream 2.0 VPC is an AWS-owned VPC where the AppStream 2.0 maintains its infrastructure. Customers are not responsible for this VPC and have no access to it. AppStream 2.0 builds each streaming instance with two Elastic Network Interfaces (ENI); one in the AppStream 2.0 VPC and one in the VPC where you choose to deploy your AppStream 2.0 instances. The third VPC is the application VPC where you would typically keep your backend servers.

The diagram also depicts the end-user process to access the AppStream 2.0 environment, which works as follow:

Using an HTML5 desktop browser the user logs on to a Single Sign-On URL. This authenticates the user against the corporate directory using SAML 2.0 federation and with optional MFA.
After successful authentication, the user will see a list of provisioned applications.
The user can launch applications, such as RDP and Putty, which are only visible within the browser and with its underlying OS hidden. The user is then able to connect to the backend systems over the ports that were opened through security groups. The user logs off and AppStream 2.0 destroys the instance used for the session.

Figure 1: Architecture diagram

Step-by-step instructions

This walk-through assumes you have created the following resources as prerequisites.

A single VPC with a /23 CIDR range and two private subnets in two AZs.

Note: “private” subnet refers to a subnet that has no internet gateway (IGW) attached.
- Bastion Subnets — used for the AppStream 2.0 instances that will be hosting the bastion applications.
- Apps Subnets — used for the servers for which the AppStream 2.0 instances will be acting as a bastion host.
  
  Figure 2: Screen shot of bastion and apps subnets
A peering connection to a VPC where the corporate Active Directory resides and with updated routing tables. This is only necessary if your AD resides in a different VPC.
Two EC2 instances with private IP addresses in the app subnet.

Phase 1: Create the DHCP Options Set

For the AppStream 2.0 instances to be able to join the corporate domain, they need to have their DNS entries point to the corporate domain controller(s). To accomplish this, you need to create a DHCP Options Set and assign it to the VPC:

Sign in to the AWS console, and then select VPC Dashboard > DHCP Option Sets > Create DHCP options set.
Give the DHCP Options Set a name, enter the domain name and DNS server(s) of your corporate domain controller(s), and then select Yes, Create.
Select your VPC Dashboard > your VPC > Actions > Edit DHCP Options Set.
Select the DHCP Options Set created in the previous step, and then select Save.

Figure 3: The “Edit DHCP Options Set” dialog

Phase 2: Create the AppStream 2.0 Stack

An AppStream 2.0 stack consists of a fleet, user access policies, and storage configuration. To create a stack, follow these steps:

Sign in to the AWS console and select AppStream 2.0 > Stack > Create Stack.
Give the stack a name, and then select Next.
Enable Home Folders, if you want persistent storage, and then select Review.

Figure 4: The “Enable Home Folders” dialog
Select Create.

Phase 3: Create the AppsStream 2.0 Directory Configuration

First create a directory configuration so you can join the AppStream 2.0 instances to an Organizational Unit (OU) in your corporate directory.

Note: AppStream 2.0 instances must be placed in an OU and can’t reside in the Computer Container.

To create a directory configuration, follow these steps:

Sign in to the AWS console and select AppStream 2.0 > Directory Configs > Create Directory Config.
Enter the following Directory Config information:
- Directory name: The FQDN of your corporate domain.
- Service Account Name: The account AppStream 2.0 uses to join the instances to the corporate domain. The required service account privileges are documented here.
- Organizational Unit (OU): The OUs where AppStream 2.0 will create your instances. You can add additional OUs by clicking the plus (+) sign.
Select Next, and then select Create.

Create Security Groups

Now, create AWS security groups for your AppStream 2.0 instances and backend servers.

BastionHostSecurityGroup

For your AppStream 2.0 instances, you must attach a “BastionHostSecurityGroup” in order to communicate to the backend servers. This security group is only used as a “source” by the security groups the backend servers are attached to and, therefore, they don’t require any inbound ports to be opened.

To create a security group, follow these steps:

Sign in to the AWS console and select VPC > Security Groups > Create Security Group.
Give your “BastionHostSecurityGroup” Security Group a name, select the VPC where you will place the AppStream 2.0 instances, and then select Yes, Create.

BastionHostAccessSecurityGroup

For your backend servers, you must attach a “BastionHostAccessSecurityGroup” that allows incoming traffic from the AppStream 2.0 instance. Unlike the “BastionHostSecurityGroup”, this one requires open inbound ports.

Sign in to the AWS console and select VPC > Security Groups > Create Security Group.
Give your “BastionHostAccessSecurityGroup” security group a name, select the correct VPC, and then select Yes, Create.
In the Security Group console, select the newly created security group, select the Inbound Rule tab, and then select Edit.
Add rules to open port 3389 and 22, use the previously-created security group as the source, and then select Save.

Figure 5: Opening ports 3389 and 22

Note: In addition to security groups, you can place Network ACLs (NACLs) around the subnet you use for AppStream 2.0 as an additional layer of security. The main differences between security groups and NACLs are that security groups are mandatory and you apply them to the instance level, while you apply NACLs to the subnet level and are optional. Another difference worth pointing out is that NACLs are “stateless” while security groups are “stateful.” This means that any port allowed inbound via NACLs will need a corresponding outbound rule. For more information on NACLs, refer to this documentation.

Phase 4: Build the AppStream 2.0 Image

An AppStream 2.0 image contains applications that you can stream to users. AppStream 2.0 uses the image to launch streaming instances that are part of an AppStream 2.0 fleet.

Once you have created the stack, create a custom image to make custom applications available to the users:

Sign in to the AWS console and select AppStream 2.0 > Images > Image Builder > Launch Image Builder.
Choose the image you want to use as a starting point, and then select Next. For this example, I chose a generic image from the General Purpose stock.

Figure 6: Choosing an image
Give your image a name, choose the instance family, and then select Next.
Choose the VPC and subnet you want to deploy the AppStream 2.0 instances in.
Select the security group you created for the AppStream 2.0 instances.
Select the directory configuration you created, the OU you want your AppStream 2.0 instances to reside in, and then select Review.
Select Launch.
Once the image is built and in a running state, select the image, and then select Connect. This will open a new browser tab where you’ll be able to connect to and manage the image.
Select Administrator and log in.

Figure 7: Log in as a local administrator
Once logged in as administrator, select the Image Assistant shortcut on the desktop.

Figure 8: The Image Assistant shortcut on the desktop
Add all the applications you want to make available to your users for streaming, and then select Next.

Note: If you need to upload installation or configuration files, you can use the My Files option in the Control menu. Any files uploaded through this method will show up under the X: drive on the Image Builder.

Figure 9: The “Control” menu
If you want to test the applications as a non-privileged user, follow the on-screen instructions to switch the user. Otherwise, select Next.

Figure 10: “Switch User” on-screen instructions
Select Launch to have the Image Assistant optimize the applications.
Give the image a name, and then select Next.
Select Disconnect and Create Image.
Go back to the AppStream 2.0 console and wait for the “snapshotting” to complete and for the image to be in an available state before continuing to the next step.

Phase 5: Create the AppStream 2.0 Fleet

Once you create your Stack and image, you need to create a Fleet and associate it with your Stack.

AppStream 2.0 fleets consist of streaming instances that run the image that you specify. The fleet type determines when your instances run and how you pay for them. You can specify a fleet type when you create a fleet, and you can’t change them once they’ve been created.

To create a fleet, follow these steps:

Sign in to the AWS console and select AppStream 2.0 > Fleets > Create Fleet.
Give your fleet a name, and then select Next.
Select the newly created image, and then select Next.
Choose your preferred settings, and then select Next.

Important: Pay special attention to the Fleet capacity value. Fleet capacity determines the number of running instances you have at any given time, and it affects your costs.

Figure 11: The “Fleet capacity” dialog
Select your VPC, subnet(s), security Group(s), Active Directory settings, and then select Next.
Review the information, and then select Create.

Figure 12: Review your settings

Associate the fleet with the stack

Follow these steps:

Sign in to the AWS console and select AppStream 2.0 > Stacks.
Select the stack, select Actions, and then select Associate Fleet.
Select the fleet, and then select Associate.

Phase 6: Configure ADFS for AppStream 2.0

To have users authenticate against the corporate directory prior to accessing AppStream 2.0, use a Single Sign-On solution. For this demo, I use ADFS. If you choose another solution, follow the instructions that come with the solution. For help with setting up ADFS with AppStream 2.0, review Enabling Identify Federation with ADSF and Amazon Appstream 2.0.

Note: If you use AWS Directory Service for Microsoft AD (AWS Managed Microsoft AD) as your user directory, you can use ADFS by following the ADFS set-up instructions in the blog on How to Enable Your Users to Access Office 365 with AWS Managed Microsoft AD Credentials.

End User Experience

This section shows you what the AppStream 2.0 end user experience is like when connecting to backend Windows and Linux instances.

Note: Make sure you have backend servers to connect to, as indicated in the prerequisites.

Process

Access the ADFS URL that you created as part of the ADFS setup.
Sign in using your corporate credentials.
Select Remote Desktop from the list of applications.
Enter your corporate credentials.
Enter the private IP address of the backend windows instance you want to remote in to.

Figure 13: Enter the private IP address

You’re now logged on to the backend Windows instance through AppStream 2.0.
To test in Linux, open putty. Select the Launch app icon in the Control menu, and then select putty.

Figure 14: The “Control” menu showing putty
Provide the private IP address of a backend Linux host you want to connect to, and then select Open.

Note: For putty to connect to a Linux instance on AWS, you will need to provide a KeyPair. For information on how to configure putty and KeyPairs, refer to this documentation.

You’re now logged on to a backend Linux host through AppStream 2.0.

Monitoring

You can monitor AppStream 2.0 use by default with the following AWS monitoring services.

Amazon CloudWatch is a monitoring service for AWS cloud resources. You can use CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources. For more information, refer to this documentation. Here’s a sample CloudWatch metric showing in-use capacity was 100% at 14:30, which indicates the Fleet capacity may need to be adjusted.

Figure 15: An example CloudWatch metric
AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. For more information, refer to this documentation. Here’s a sample CloudTrail event. For example, from this event you can see that user Bob logged on to AppStream 2.0 on March 4, 2018, and you can see his source IP.

Figure 16: An example CloudTrail event

Summary

Amazon AppStream 2.0 is a cost-effective way to provide administrators with a secure and auditable method to access their backend environments.

The AppStream 2.0 built-in auto-scaling feature offers a pay-as-you-go model, where the number of instances running is based on user demand. This allows you to keep costs down without compromising availability. Another cost-saving benefit of AppStream 2.0 is its underlying infrastructure being managed and maintained by AWS, so you can deploy AppStream 2.0 with minimal effort.

AppStream 2.0 helps with reducing the attack surface by hiding the shell of the streaming OS. This prevents administrators from interacting with executables that haven’t been made available to them through AppStream 2.0.

Another security benefit of AppStream 2.0 is that it destroys streaming instances after each use, reducing risks. This is a good mitigation strategy against compromised instances, as the lifespan of an instance is limited to the length of a user’s session.

AppStream 2.0 support for SAML provides yet another layer of security, allowing you to restrict access to SAML-federated URLs from corporate networks only, as well as the ability to enforce multi-factor authentication (MFA).

You can monitor the AppStream 2.0 environment through the use of AWS CloudTrail and Amazon CloudWatch, allowing you to monitor and trace the usage of AppStream 2.0.

For all of these reasons, AppStream 2.0 makes for a uniquely attractive bastion host solution.

For more information on the technologies mentioned in this blog, see the links below:

If you have comments about this post, submit them in the Comments section below. If you have questions about anything in this post, start a new thread on the Amazon AppStream 2.0 forum or contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Noise

Tag Archives: pc

HackSpace magazine 7: Internet of Everything

Internet of Things (IoT)

Awesome projects

Makerspaces

And there’s more!

Get your copy of HackSpace magazine

[$] What’s coming in OpenLDAP 2.5

Security updates for Monday

All Systems Go! 2018 CfP Open

EC2 Instance Update – C5 Instances with Local NVMe Storage (C5d)

timeShift(GrafanaBuzz, 1w) Issue 45

Extending Amazon Linux 2 with EPEL and Let’s Encrypt

Walkthrough

Prerequisites and costs

Step 1: Provision a VPC, Amazon Linux 2 instance, and LAMP stack

Step 2: Install and enable EPEL

Step 3: Install and configure Let’s Encrypt

Step: 4: Validate the installation

Step 5: Cleanup

Conclusion

Brutus 2: the gaming PC case of your dreams

der BRUTUS 2 by 3nb Gaming

What’s case modding?

The Brutus 2 case

The software

The crowdfunding campaign

The collective thoughts of the interwebz