Tag Archives: Manchester

Analyze Apache Parquet optimized data using Amazon Kinesis Data Firehose, Amazon Athena, and Amazon Redshift

Post Syndicated from Roy Hasson original https://aws.amazon.com/blogs/big-data/analyzing-apache-parquet-optimized-data-using-amazon-kinesis-data-firehose-amazon-athena-and-amazon-redshift/

Amazon Kinesis Data Firehose is the easiest way to capture and stream data into a data lake built on Amazon S3. This data can be anything—from AWS service logs like AWS CloudTrail log files, Amazon VPC Flow Logs, Application Load Balancer logs, and others. It can also be IoT events, game events, and much more. To efficiently query this data, a time-consuming ETL (extract, transform, and load) process is required to massage and convert the data to an optimal file format, which increases the time to insight. This situation is less than ideal, especially for real-time data that loses its value over time.

To solve this common challenge, Kinesis Data Firehose can now save data to Amazon S3 in Apache Parquet or Apache ORC format. These are optimized columnar formats that are highly recommended for best performance and cost-savings when querying data in S3. This feature directly benefits you if you use Amazon Athena, Amazon Redshift, AWS Glue, Amazon EMR, or any other big data tools that are available from the AWS Partner Network and through the open-source community.

Amazon Connect is a simple-to-use, cloud-based contact center service that makes it easy for any business to provide a great customer experience at a lower cost than common alternatives. Its open platform design enables easy integration with other systems. One of those systems is Amazon Kinesis—in particular, Kinesis Data Streams and Kinesis Data Firehose.

What’s really exciting is that you can now save events from Amazon Connect to S3 in Apache Parquet format. You can then perform analytics using Amazon Athena and Amazon Redshift Spectrum in real time, taking advantage of this key performance and cost optimization. Of course, Amazon Connect is only one example. This new capability opens the door for a great deal of opportunity, especially as organizations continue to build their data lakes.

Amazon Connect includes an array of analytics views in the Administrator dashboard. But you might want to run other types of analysis. In this post, I describe how to set up a data stream from Amazon Connect through Kinesis Data Streams and Kinesis Data Firehose and out to S3, and then perform analytics using Athena and Amazon Redshift Spectrum. I focus primarily on the Kinesis Data Firehose support for Parquet and its integration with the AWS Glue Data Catalog, Amazon Athena, and Amazon Redshift.

Solution overview

Here is how the solution is laid out:

 

 

The following sections walk you through each of these steps to set up the pipeline.

1. Define the schema

When Kinesis Data Firehose processes incoming events and converts the data to Parquet, it needs to know which schema to apply. The reason is that many times, incoming events contain all or some of the expected fields based on which values the producers are advertising. A typical process is to normalize the schema during a batch ETL job so that you end up with a consistent schema that can easily be understood and queried. Doing this introduces latency due to the nature of the batch process. To overcome this issue, Kinesis Data Firehose requires the schema to be defined in advance.

To see the available columns and structures, see Amazon Connect Agent Event Streams. For the purpose of simplicity, I opted to make all the columns of type String rather than create the nested structures. But you can definitely do that if you want.

The simplest way to define the schema is to create a table in the Amazon Athena console. Open the Athena console, and paste the following create table statement, substituting your own S3 bucket and prefix for where your event data will be stored. A Data Catalog database is a logical container that holds the different tables that you can create. The default database name shown here should already exist. If it doesn’t, you can create it or use another database that you’ve already created.

CREATE EXTERNAL TABLE default.kfhconnectblog (
  awsaccountid string,
  agentarn string,
  currentagentsnapshot string,
  eventid string,
  eventtimestamp string,
  eventtype string,
  instancearn string,
  previousagentsnapshot string,
  version string
)
STORED AS parquet
LOCATION 's3://your_bucket/kfhconnectblog/'
TBLPROPERTIES ("parquet.compression"="SNAPPY")

That’s all you have to do to prepare the schema for Kinesis Data Firehose.

2. Define the data streams

Next, you need to define the Kinesis data streams that will be used to stream the Amazon Connect events.  Open the Kinesis Data Streams console and create two streams.  You can configure them with only one shard each because you don’t have a lot of data right now.

3. Define the Kinesis Data Firehose delivery stream for Parquet

Let’s configure the Data Firehose delivery stream using the data stream as the source and Amazon S3 as the output. Start by opening the Kinesis Data Firehose console and creating a new data delivery stream. Give it a name, and associate it with the Kinesis data stream that you created in Step 2.

As shown in the following screenshot, enable Record format conversion (1) and choose Apache Parquet (2). As you can see, Apache ORC is also supported. Scroll down and provide the AWS Glue Data Catalog database name (3) and table names (4) that you created in Step 1. Choose Next.

To make things easier, the output S3 bucket and prefix fields are automatically populated using the values that you defined in the LOCATION parameter of the create table statement from Step 1. Pretty cool. Additionally, you have the option to save the raw events into another location as defined in the Source record S3 backup section. Don’t forget to add a trailing forward slash “ / “ so that Data Firehose creates the date partitions inside that prefix.

On the next page, in the S3 buffer conditions section, there is a note about configuring a large buffer size. The Parquet file format is highly efficient in how it stores and compresses data. Increasing the buffer size allows you to pack more rows into each output file, which is preferred and gives you the most benefit from Parquet.

Compression using Snappy is automatically enabled for both Parquet and ORC. You can modify the compression algorithm by using the Kinesis Data Firehose API and update the OutputFormatConfiguration.

Be sure to also enable Amazon CloudWatch Logs so that you can debug any issues that you might run into.

Lastly, finalize the creation of the Firehose delivery stream, and continue on to the next section.

4. Set up the Amazon Connect contact center

After setting up the Kinesis pipeline, you now need to set up a simple contact center in Amazon Connect. The Getting Started page provides clear instructions on how to set up your environment, acquire a phone number, and create an agent to accept calls.

After setting up the contact center, in the Amazon Connect console, choose your Instance Alias, and then choose Data Streaming. Under Agent Event, choose the Kinesis data stream that you created in Step 2, and then choose Save.

At this point, your pipeline is complete.  Agent events from Amazon Connect are generated as agents go about their day. Events are sent via Kinesis Data Streams to Kinesis Data Firehose, which converts the event data from JSON to Parquet and stores it in S3. Athena and Amazon Redshift Spectrum can simply query the data without any additional work.

So let’s generate some data. Go back into the Administrator console for your Amazon Connect contact center, and create an agent to handle incoming calls. In this example, I creatively named mine Agent One. After it is created, Agent One can get to work and log into their console and set their availability to Available so that they are ready to receive calls.

To make the data a bit more interesting, I also created a second agent, Agent Two. I then made some incoming and outgoing calls and caused some failures to occur, so I now have enough data available to analyze.

5. Analyze the data with Athena

Let’s open the Athena console and run some queries. One thing you’ll notice is that when we created the schema for the dataset, we defined some of the fields as Strings even though in the documentation they were complex structures.  The reason for doing that was simply to show some of the flexibility of Athena to be able to parse JSON data. However, you can define nested structures in your table schema so that Kinesis Data Firehose applies the appropriate schema to the Parquet file.

Let’s run the first query to see which agents have logged into the system.

The query might look complex, but it’s fairly straightforward:

WITH dataset AS (
  SELECT 
    from_iso8601_timestamp(eventtimestamp) AS event_ts,
    eventtype,
    -- CURRENT STATE
    json_extract_scalar(
      currentagentsnapshot,
      '$.agentstatus.name') AS current_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        currentagentsnapshot,
        '$.agentstatus.starttimestamp')) AS current_starttimestamp,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.firstname') AS current_firstname,
    json_extract_scalar(
      currentagentsnapshot,
      '$.configuration.lastname') AS current_lastname,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.username') AS current_username,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') AS               current_outboundqueue,
    json_extract_scalar(
      currentagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as current_inboundqueue,
    -- PREVIOUS STATE
    json_extract_scalar(
      previousagentsnapshot, 
      '$.agentstatus.name') as prev_status,
    from_iso8601_timestamp(
      json_extract_scalar(
        previousagentsnapshot, 
       '$.agentstatus.starttimestamp')) as prev_starttimestamp,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.firstname') as prev_firstname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.lastname') as prev_lastname,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.username') as prev_username,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.defaultoutboundqueue.name') as current_outboundqueue,
    json_extract_scalar(
      previousagentsnapshot, 
      '$.configuration.routingprofile.inboundqueues[0].name') as prev_inboundqueue
  from kfhconnectblog
  where eventtype <> 'HEART_BEAT'
)
SELECT
  current_status as status,
  current_username as username,
  event_ts
FROM dataset
WHERE eventtype = 'LOGIN' AND current_username <> ''
ORDER BY event_ts DESC

The query output looks something like this:

Here is another query that shows the sessions each of the agents engaged with. It tells us where they were incoming or outgoing, if they were completed, and where there were missed or failed calls.

WITH src AS (
  SELECT
     eventid,
     json_extract_scalar(currentagentsnapshot, '$.configuration.username') as username,
     cast(json_extract(currentagentsnapshot, '$.contacts') AS ARRAY(JSON)) as c,
     cast(json_extract(previousagentsnapshot, '$.contacts') AS ARRAY(JSON)) as p
  from kfhconnectblog
),
src2 AS (
  SELECT *
  FROM src CROSS JOIN UNNEST (c, p) AS contacts(c_item, p_item)
),
dataset AS (
SELECT 
  eventid,
  username,
  json_extract_scalar(c_item, '$.contactid') as c_contactid,
  json_extract_scalar(c_item, '$.channel') as c_channel,
  json_extract_scalar(c_item, '$.initiationmethod') as c_direction,
  json_extract_scalar(c_item, '$.queue.name') as c_queue,
  json_extract_scalar(c_item, '$.state') as c_state,
  from_iso8601_timestamp(json_extract_scalar(c_item, '$.statestarttimestamp')) as c_ts,
  
  json_extract_scalar(p_item, '$.contactid') as p_contactid,
  json_extract_scalar(p_item, '$.channel') as p_channel,
  json_extract_scalar(p_item, '$.initiationmethod') as p_direction,
  json_extract_scalar(p_item, '$.queue.name') as p_queue,
  json_extract_scalar(p_item, '$.state') as p_state,
  from_iso8601_timestamp(json_extract_scalar(p_item, '$.statestarttimestamp')) as p_ts
FROM src2
)
SELECT 
  username,
  c_channel as channel,
  c_direction as direction,
  p_state as prev_state,
  c_state as current_state,
  c_ts as current_ts,
  c_contactid as id
FROM dataset
WHERE c_contactid = p_contactid
ORDER BY id DESC, current_ts ASC

The query output looks similar to the following:

6. Analyze the data with Amazon Redshift Spectrum

With Amazon Redshift Spectrum, you can query data directly in S3 using your existing Amazon Redshift data warehouse cluster. Because the data is already in Parquet format, Redshift Spectrum gets the same great benefits that Athena does.

Here is a simple query to show querying the same data from Amazon Redshift. Note that to do this, you need to first create an external schema in Amazon Redshift that points to the AWS Glue Data Catalog.

SELECT 
  eventtype,
  json_extract_path_text(currentagentsnapshot,'agentstatus','name') AS current_status,
  json_extract_path_text(currentagentsnapshot, 'configuration','firstname') AS current_firstname,
  json_extract_path_text(currentagentsnapshot, 'configuration','lastname') AS current_lastname,
  json_extract_path_text(
    currentagentsnapshot,
    'configuration','routingprofile','defaultoutboundqueue','name') AS current_outboundqueue,
FROM default_schema.kfhconnectblog

The following shows the query output:

Summary

In this post, I showed you how to use Kinesis Data Firehose to ingest and convert data to columnar file format, enabling real-time analysis using Athena and Amazon Redshift. This great feature enables a level of optimization in both cost and performance that you need when storing and analyzing large amounts of data. This feature is equally important if you are investing in building data lakes on AWS.

 


Additional Reading

If you found this post useful, be sure to check out Analyzing VPC Flow Logs with Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight and Work with partitioned data in AWS Glue.


About the Author

Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan cheering his team on and hanging out with his family.

 

 

 

Raspberry Jam Big Birthday Weekend 2018 roundup

Post Syndicated from Ben Nuttall original https://www.raspberrypi.org/blog/big-birthday-weekend-2018-roundup/

A couple of weekends ago, we celebrated our sixth birthday by coordinating more than 100 simultaneous Raspberry Jam events around the world. The Big Birthday Weekend was a huge success: our fantastic community organised Jams in 40 countries, covering six continents!

We sent the Jams special birthday kits to help them celebrate in style, and a video message featuring a thank you from Philip and Eben:

Raspberry Jam Big Birthday Weekend 2018

To celebrate the Raspberry Pi’s sixth birthday, we coordinated Raspberry Jams all over the world to take place over the Raspberry Jam Big Birthday Weekend, 3-4 March 2018. A massive thank you to everyone who ran an event and attended.

The Raspberry Jam photo booth

I put together code for a Pi-powered photo booth which overlaid the Big Birthday Weekend logo onto photos and (optionally) tweeted them. We included an arcade button in the Jam kits so they could build one — and it seemed to be quite popular. Some Jams put great effort into housing their photo booth:



Here are some of my favourite photo booth tweets:

RGVSA on Twitter

PiParty photo booth @RGVSA & @ @Nerdvana_io #Rjam

Denis Stretton on Twitter

The @SouthendRPIJams #PiParty photo booth

rpijamtokyo on Twitter

PiParty photo booth

Preston Raspberry Jam on Twitter

Preston Raspberry Jam Photobooth #RJam #PiParty

If you want to try out the photo booth software yourself, find the code on GitHub.

The great Raspberry Jam bake-off

Traditionally, in the UK, people have a cake on their birthday. And we had a few! We saw (and tasted) a great selection of Pi-themed cakes and other baked goods throughout the weekend:






Raspberry Jams everywhere

We always say that every Jam is different, but there’s a common and recognisable theme amongst them. It was great to see so many different venues around the world filling up with like-minded Pi enthusiasts, Raspberry Jam–branded banners, and Raspberry Pi balloons!

Europe

Sergio Martinez on Twitter

Thank you so much to all the attendees of the Ikana Jam in Krakow past Saturday! We shared fun experiences, some of them… also painful 😉 A big thank you to @Raspberry_Pi for these global celebrations! And a big thank you to @hubraum for their hospitality! #PiParty #rjam

NI Raspberry Jam on Twitter

We also had a super successful set of wearables workshops using @adafruit Circuit Playground Express boards and conductive thread at today’s @Raspberry_Pi Jam! Very popular! #PiParty

Suzystar on Twitter

My SenseHAT workshop, going well! @SouthendRPiJams #PiParty

Worksop College Raspberry Jam on Twitter

Learning how to scare the zombies in case of an apocalypse- it worked on our young learners #PiParty @worksopcollege @Raspberry_Pi https://t.co/pntEm57TJl

Africa

Rita on Twitter

Being one of the two places in Kenya where the #PiParty took place, it was an amazing time spending the day with this team and getting to learn and have fun. @TaitaTavetaUni and @Raspberry_Pi thank you for your support. @TTUTechlady @mictecttu ch

GABRIEL ONIFADE on Twitter

@TheMagP1

GABRIEL ONIFADE on Twitter

@GABONIAVERACITY #PiParty Lagos Raspberry Jam 2018 Special International Celebration – 6th Raspberry-Pi Big Birthday! Lagos Nigeria @Raspberry_Pi @ben_nuttall #RJam #RaspberryJam #raspberrypi #physicalcomputing #robotics #edtech #coding #programming #edTechAfrica #veracityhouse https://t.co/V7yLxaYGNx

North America

Heidi Baynes on Twitter

The Riverside Raspberry Jam @Vocademy is underway! #piparty

Brad Derstine on Twitter

The Philly & Pi #PiParty event with @Bresslergroup and @TechGirlzorg was awesome! The Scratch and Pi workshop was amazing! It was overall a great day of fun and tech!!! Thank you everyone who came out!

Houston Raspi on Twitter

Thanks everyone who came out to the @Raspberry_Pi Big Birthday Jam! Special thanks to @PBFerrell @estefanniegg @pcsforme @pandafulmanda @colnels @bquentin3 couldn’t’ve put on this amazing community event without you guys!

Merge Robotics 2706 on Twitter

We are back at @SciTechMuseum for the second day of @OttawaPiJam! Our robot Mergius loves playing catch with the kids! #pijam #piparty #omgrobots

South America

Javier Garzón on Twitter

Así terminamos el #Raspberry Jam Big Birthday Weekend #Bogota 2018 #PiParty de #RaspberryJamBogota 2018 @Raspberry_Pi Nos vemos el 7 de marzo en #ArduinoDayBogota 2018 y #RaspberryJamBogota 2018

Asia

Fablab UP Cebu on Twitter

Happy 6th birthday, @Raspberry_Pi! Greetings all the way from CEBU,PH! #PiParty #IoTCebu Thanks @CebuXGeeks X Ramos for these awesome pics. #Fablab #UPCebu

福野泰介 on Twitter

ラズパイ、6才のお誕生日会スタート in Tokyo PCNブースで、いろいろ展示とhttps://t.co/L6E7KgyNHFとIchigoJamつないだ、こどもIoTハッカソンmini体験やってます at 東京蒲田駅近 https://t.co/yHEuqXHvqe #piparty #pipartytokyo #rjam #opendataday

Ren Camp on Twitter

Happy birthday @Raspberry_Pi! #piparty #iotcebu @coolnumber9 https://t.co/2ESVjfRJ2d

Oceania

Glenunga Raspberry Pi Club on Twitter

PiParty photo booth

Personally, I managed to get to three Jams over the weekend: two run by the same people who put on the first two Jams to ever take place, and also one brand-new one! The Preston Raspberry Jam team, who usually run their event on a Monday evening, wanted to do something extra special for the birthday, so they came up with the idea of putting on a Raspberry Jam Sandwich — on the Friday and Monday around the weekend! This meant I was able to visit them on Friday, then attend the Manchester Raspberry Jam on Saturday, and finally drop by the new Jam at Worksop College on my way home on Sunday.

Ben Nuttall on Twitter

I’m at my first Raspberry Jam #PiParty event of the big birthday weekend! @PrestonRJam has been running for nearly 6 years and is a great place to start the celebrations!

Ben Nuttall on Twitter

Back at @McrRaspJam at @DigInnMMU for #PiParty

Ben Nuttall on Twitter

Great to see mine & @Frans_facts Balloon Pi-Tay popper project in action at @worksopjam #rjam #PiParty https://t.co/GswFm0UuPg

Various members of the Foundation team attended Jams around the UK and US, and James from the Code Club International team visited AmsterJam.

hackerfemo on Twitter

Thanks to everyone who came to our Jam and everyone who helped out. @phoenixtogether thanks for amazing cake & hosting. Ademir you’re so cool. It was awesome to meet Craig Morley from @Raspberry_Pi too. #PiParty

Stuart Fox on Twitter

Great #PiParty today at the @cotswoldjam with bloody delicious cake and lots of raspberry goodness. Great to see @ClareSutcliffe @martinohanlon playing on my new pi powered arcade build:-)

Clare Sutcliffe on Twitter

Happy 6th Birthday @Raspberry_Pi from everyone at the #PiParty at #cotswoldjam in Cheltenham!

Code Club on Twitter

It’s @Raspberry_Pi 6th birthday and we’re celebrating by taking part in @amsterjam__! Happy Birthday Raspberry Pi, we’re so happy to be a part of the family! #PiParty

For more Jammy birthday goodness, check out the PiParty hashtag on Twitter!

The Jam makers!

A lot of preparation went into each Jam, and we really appreciate all the hard work the Jam makers put in to making these events happen, on the Big Birthday Weekend and all year round. Thanks also to all the teams that sent us a group photo:

Lots of the Jams that took place were brand-new events, so we hope to see them continue throughout 2018 and beyond, growing the Raspberry Pi community around the world and giving more people, particularly youths, the opportunity to learn digital making skills.

Philip Colligan on Twitter

So many wonderful people in the @Raspberry_Pi community. Thanks to everyone at #PottonPiAndPints for a great afternoon and for everything you do to help young people learn digital making. #PiParty

Special thanks to ModMyPi for shipping the special Raspberry Jam kits all over the world!

Don’t forget to check out our Jam page to find an event near you! This is also where you can find free resources to help you get a new Jam started, and download free starter projects made especially for Jam activities. These projects are available in English, Français, Français Canadien, Nederlands, Deutsch, Italiano, and 日本語. If you’d like to help us translate more content into these and other languages, please get in touch!

PS Some of the UK Jams were postponed due to heavy snowfall, so you may find there’s a belated sixth-birthday Jam coming up where you live!

S Organ on Twitter

@TheMagP1 Ours was rescheduled until later in the Spring due to the snow but here is Babbage enjoying the snow!

The post Raspberry Jam Big Birthday Weekend 2018 roundup appeared first on Raspberry Pi.

New – AWS Direct Connect Gateway – Inter-Region VPC Access

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-aws-direct-connect-gateway-inter-region-vpc-access/

As I was preparing to write this post, I took a nostalgic look at the blog post I wrote when we launched AWS Direct Connect back in 2012. We created Direct Connect after our enterprise customers asked us to allow them to establish dedicated connections to an AWS Region in pursuit of enhanced privacy, additional data transfer bandwidth, and more predictable data transfer performance. Starting from one AWS Region and a single colo, Direct Connect is now available in every public AWS Region and accessible from dozens of colos scattered across the world (over 60 locations at last count). Our customers have taken to Direct Connect wholeheartedly and we have added features such as Link Aggregation, Amazon EFS support, CloudWatch monitoring, and HIPAA eligibility. In the past five weeks alone we have added Direct Connect locations in Houston (Texas), Vancouver (Canada), Manchester (UK), Canberra (Australia), and Perth (Australia).

Today we are making Direct Connect simpler and more powerful with the addition of the Direct Connect Gateway. We are also giving Direct Connect customers in any Region the ability to create public virtual interfaces that receive our global IP routes and enable access to the public endpoints for our services and updating the Direct Connect pricing model.

Let’s take a look at each one!

New Direct Connect Gateway
You can use the new Direct Connect Gateway to establish connectivity that spans Virtual Private Clouds (VPCs) spread across multiple AWS Regions. You no longer need to establish multiple BGP sessions for each VPC; this reduces your administrative workload as well as the load on your network devices.

This feature also allows you to connect to any of the participating VPCs from any Direct Connect location, further reducing your costs for making using AWS services on a cross-region basis.

Here is a diagram that illustrates the simplification that you can achieve with a Direct Connect Gateway (each “lock” icon represents a Virtual Private Gateway). Start with this:

And end up like this:

The VPCs that reference a particular Direct Connect Gateway must have IP address ranges that do not overlap. Today, the VPCs must all be in the same AWS account; we plan to make this more flexible in the future.

Each Gateway is a global object that exists across all of the public AWS Regions. All communication between the Regions via the Gateways takes place across the AWS network backbone.

Creating a Direct Connect Gateway
You can create a Direct Connect Gateway from the Direct Connect Console or by calling the CreateDirectConnectGateway function from your code. I’ll use the Console!

I open the Direct Connect Console and click on Direct Connect Gateways to get started:

The list is empty since I don’t have any Gateways yet. Click on Create Direct Connect Gateway to change that:

I give my Gateway a name, enter a private ASN for my network, then click on Create. The ASN (Autonomous System Number) must be in one of the ranges defined as private in RFC 6996:

My new Gateway will appear in the other AWS Regions within a moment or two:

I have a Direct Connect Connection in Ohio that I will use to create my VIF:

Now I create a private VIF that references the Gateway and the Connection:

It is ready to use within seconds:

I already have a pair of VPCs with non-overlapping CIDRs, and a Virtual Private Gateway attached to each one. Here are the VPCs (since this is a demo I’ll show both in the same Region for convenience):

And the Virtual Private Gateways:

I return to the Direct Connect Console and navigate to the Direct Connect Gateways. I select my Gateway and choose Associate Virtual Private Gateway from the Actions menu:

Then I select both of my Virtual Private Gateways and click on Associate:

If, as would usually be the case, my VPCs are in distinct AWS Regions, the same procedure would apply. For this blog post it was easier to show you the operations once rather than twice.

The Virtual Gateway association is complete within a minute or so (the state starts out as associating):

When the state transitions to associated, traffic can flow between your on-premises network and your VPCs, over your AWS Direct Connect connection, regardless of the AWS Regions where your VPCs reside.

Public Virtual Interfaces for Service Endpoints
You can now create Public Virtual Interfaces that will allow you to access AWS public service endpoints for AWS services running in any AWS Region (except AWS China Region) over Direct Connect. These interfaces receive (via BGP) Amazon’s global IP routes. You can create these interfaces in the Direct Connect Console; start by selecting the Public option:

After you create it you will need to associate it with a VPC.

Updated Pricing Model
In light of the ever-expanding number of AWS Regions and AWS Direct Connect locations, data transfer pricing is now based on the location of the Direct Connect and the source AWS Region. The new pricing is simpler that the older model which was based on AWS Direct Connect locations.

Now Available
This new feature is available today and you can start to use it right now. You can create and use Direct Connect Gateways at no charge; you pay the usual Direct Connect prices for port hours and data transfer.

Jeff;

 

GNOME 3.26 released

Post Syndicated from ris original https://lwn.net/Articles/733587/rss

The GNOME Project has announced the release of GNOME 3.26 “Manchester”.
This release brings refinements to the system search, animations for
maximizing and unmaximizing windows and support for color Emoji.

Improvements to core GNOME applications include a redesigned Settings
application, a new display settings panel, Firefox sync in the Web browser,
and many more.” There are openSUSE nightly live images that include
GNOME 3.26.

GNOME turns 20

Post Syndicated from ris original https://lwn.net/Articles/731036/rss

The GNOME project was founded by Miguel de Icaza and Federico Mena Quintero
on August 15, 1997, so today the project celebrates
its 20th birthday. “There have been 33 stable releases since the initial release of GNOME 1.0 in 1999. The latest stable release, GNOME 3.24 “Portland,” was well-received. “Portland” included exciting new features like the GNOME Recipes application and Night Light, which helps users avoid eyestrain. The upcoming version of GNOME 3.26 “Manchester,” is scheduled for release in September of this year. With over 6,000 contributors, and 8 million lines of code, the GNOME Project continues to thrive in its twentieth year.

Raspberry Jam round-up: April

Post Syndicated from Ben Nuttall original https://www.raspberrypi.org/blog/raspberry-jam-round-up-april/

In case you missed it: in yesterday’s post, we released our Raspberry Jam Guidebook, a new Jam branding pack and some more resources to help people set up their own Raspberry Pi community events. Today I’m sharing some insights from Jams I’ve attended recently.

Raspberry Jam round-up April 2017

Preston Raspberry Jam

The Preston Jam is one of the most long-established Jams, and it recently ran its 58th event. It has achieved this by running like clockwork: on the first Monday evening of every month, without fail, the Jam takes place. A few months ago I decided to drop in to surprise the organiser, Alan O’Donohoe. The Jam is held at the Media Innovation Studio at the University of Central Lancashire. The format is quite informal, and it’s very welcoming to newcomers. The first half of the event allows people to mingle, and beginners can get support from more seasoned makers. I noticed a number of parents who’d brought their children along to find out more about the Pi and what can be done with it. It’s a great way to find out for real what people use their Pis for, and to get pointers on how to set up and where to start.

About half way through the evening, the organisers gather everyone round to watch a few short presentations. At the Jam I attended, most of these talks were from children, which was fantastic to see: Josh gave a demo in which he connected his Raspberry Pi to an Amazon Echo using the Alexa API, Cerys talked about her Jam in Staffordshire, and Elise told everyone about the workshops she ran at MozFest. All their talks were really well presented. The Preston Jam has done very well to keep going for so long and so consistently, and to provide such great opportunities and support for young people like Josh, Cerys and Elise to develop their digital making abilities (and presentation skills). Their next event is on Monday 1 May.



Manchester Raspberry Jam and CoderDojo

I set up the Manchester Jam back in 2012, around the same time that the Preston one started. Back then, you could only buy one Pi at a time, and only a handful of people in the area owned one. We ran a fairly small event at the local tech community space, MadLab, adopting the format of similar events I’d been to, which was very hands-on and project-based – people brought along their Pis and worked on their own builds. I ran the Jam for a year before moving to Cambridge to work for the Foundation, and I asked one of the regular attendees, Jack, if he’d run it in future. I hadn’t been back until last month, when Clare and I decided to visit.

The Jam is now held at The Shed, a digital innovation space at Manchester Metropolitan University, thanks to Darren Dancey, a computer science lecturer who claims he taught me everything I know (this claim is yet to be peer-reviewed). Jack, Darren, and Raspberry Pi Foundation co-founder and Trustee Pete Lomas put on an excellent event. They have a room for workshops, and a space for people to work on their own projects. It was wonderful to see some of the attendees from the early days still going along every month, as well as lots of new faces. Some of Darren’s students ran a Minecraft Pi workshop for beginners, and I ran one using traffic lights with GPIO Zero and guizero.



The next day, we went along to Manchester CoderDojo, a monthly event for young people learning to code and make things. The Dojo is held at The Sharp Project, and thanks to the broad range of skills of the volunteers, they provide a range of different activities: Raspberry Pi, Minecraft, LittleBits, Code Club Scratch projects, video editing, game making and lots more.

Raspberry Jam round-up April 2017

Manchester CoderDojo’s next event is on Sunday 14 May. Be sure to keep an eye on mcrraspjam.org.uk for the next Jam date!

CamJam and Pi Wars

The Cambridge Raspberry Jam is a big event that runs two or three times a year, with quite a different format to the smaller monthly Jams. They have a lecture theatre for talks, a space for workshops, lots of show-and-tell, and even a collection of retailers selling Pis and accessories. It’s a very social event, and always great fun to attend.

The organisers, Mike and Tim, who wrote the foreword for the Guidebook, also run Pi Wars: the annual Raspberry Pi robotics competition. Clare and I went along to this year’s event, where we got to see teams from all over the country (and even one from New Mexico, brought by one of our Certified Educators from Picademy USA, Kerry Bruce) take part in a whole host of robotic challenges. A few of the teams I spoke to have been working on their robots at their local Jams throughout the year. If you’re interested in taking part next year, you can get a team together now and start to make a plan for your 2018 robot! Keep an eye on camjam.me and piwars.org for announcements.

PiBorg on Twitter

Ely Cathedral has surprisingly good straight line speed for a cathedral. Great job Ely Makers! #PiWars

Raspberry Jam @ Pi Towers

As well as working on supporting other Jams, I’ve also been running my own for the last few months. Held at our own offices in Cambridge, Raspberry Jam @ Pi Towers is a monthly event for people of all ages. We run workshops, show-and-tell and other practical activities. If you’re in the area, our next event is on Saturday 13 May.

Ben Nuttall on Twitter

rjam @ Pi Towers

Raspberry Jamboree

In 2013 and 2014, Alan O’Donohoe organised the Raspberry Jamboree, which took place in Manchester to mark the first and second Raspberry Pi birthdays – and it’s coming back next month, this time organised by Claire Dodd Wicher and Les Pounder. It’s primarily an unconference, so the talks are given by the attendees and arranged on the day, which is a great way to allow anyone to participate. There will also be workshops and practical sessions, so don’t miss out! Unless, like me, you’re going to the new Norwich Jam instead…

Start a Jam near you

If there’s no Jam where you live, you can start your own! Download a copy of the brand new Raspberry Jam Guidebook for tips on how to get started. It’s not as hard as you’d think! And we’re on hand if you need any help.

Raspberry Jam round-up April 2017

Visiting Jams and hearing from Jam organisers are great ways for us to find out how we can best support our wonderful community. If you run a Jam and you’d like to tell us about what you do, or share your success stories, please don’t hesitate to get in touch. Email me at [email protected], and we’ll try to feature your stories on the blog in future.

The post Raspberry Jam round-up: April appeared first on Raspberry Pi.

Our full 2017 schedule of UK Picademy events

Post Syndicated from Dan Fisher original https://www.raspberrypi.org/blog/our-full-2017-schedule-of-uk-picademy-events/

Happy new year to everyone! We’re back with a new programme of Picademy events for 2017. All our UK events have been scheduled up to the end of the year, so you can look ahead and choose something at a location and date that is convenient.

An educator gets to grips with our Camera Module

For the uninitiated, Picademy is a free CPD programme that aims to give educators the skills and knowledge they need to get creative with computing, no matter what their level of experience. In fact, you don’t need any previous experience to apply, just an enthusiasm for teaching kids computing. Each course lasts for two full days and is a mixture of digital making workshops, project-based learning, and hacking. Delegates graduate as Raspberry Pi Certified Educators (RCEs).

Last year’s Picademy events yielded some wonderful moments. We trained over 540 educators in the UK and the US, so we had lots of highlights to choose from; I certainly witnessed many in person while delivering events in Glasgow. Two of my favourites included the educator who created music by coding DNA into Sonic Pi as note values (amazing!), and the project that used the Sense HAT to input notes to Sonic Pi and then convert them into coloured blocks in Minecraft for a digital disco.

It was so great to see the enthusiasm, the camaraderie, and the willingness of educators to be open to new experiences. You could see the cogs turning as they thought about how they could apply the new ideas to work in their own classrooms. It was also great to hear about things educators found less easy, and to answer questions about aspects of the computing curriculum. We find this feedback particularly useful as we are always looking for ways to improve our content and provide better support.

Below you’ll find details of the Picademy events we’re running across the UK in 2017:

CITYVENUEDATES
CardiffTramshed,
Clare Rd,
Cardiff,
CF11 6QP
21/22 February
ManchesterMadLab Manchester,
36-40 Edge St,
Manchester,
M4 1HN
14/15 March
02/03 October
BirminghamThe Learning Hub,
Birmingham Airport,
Birmingham,
B26 3QJ
10/11 April
04/05 December
CambridgeRaspberry Pi Foundation,
30 Station Road,
Cambridge,
CB1 2JH
15/16 May
LondonTBCLate May*
Late November*

* While London details are not fully confirmed, you can still apply for these events. We will email details to applicants later in 2017.

Who should apply?

We are looking for inspirational educators who are passionate about computing, enthusiastic about creating awesome learning experiences for their students, and proactive at sharing good practice.

While we’re primarily looking for primary, secondary, FE and HE teachers to apply, we’re also seeking other outstanding educators such as librarians, community educators, trainee teachers, and trainers of teachers.

We’re committed to running free high-quality training, and we invest substantial time (and money) in the educators that attend. Our hope is that our certified educators not only return home with a digital making mindset to inspire students and colleagues, but also have an impact on their wider education community through social media, meetups, or running their own training.

With this in mind, we should point out that Picademy events are often oversubscribed: for this reason, it’s really important that we get a sense of the person behind the application. We would therefore urge you to take your time when answering questions that ask you to reflect on your own experiences and reasons for applying.

A cohort of Picademy graduates in Manchester

How to apply

To apply for any of the events, fill in our Online Application Form. If you have any further questions, you can email [email protected] or post a message in the Picademy area on our forums.

The post Our full 2017 schedule of UK Picademy events appeared first on Raspberry Pi.

Presenting an Open Source Toolkit for Lightweight Multilingual Entity Linking

Post Syndicated from mikesefanov original https://yahooeng.tumblr.com/post/154168092396

yahooresearch:

By Aasish Pappu, Roi Blanco, and Amanda Stent

What’s the first thing you want to know about any kind of text document (like a Yahoo News or Yahoo Sports article)? What it’s about, of course! That means you want to know something about the people, organizations, and locations that are mentioned in the document. Systems that automatically surface this information are called named entity recognition and linking systems. These are one of the most useful components in text analytics as they are required for a wide variety of applications including search, recommender systems, question answering, and sentiment analysis.

Named entity recognition and linking systems use statistical models trained over large amounts of labeled text data. A major challenge is to be able to accurately detect entities, in new languages, at scale, with limited labeled data available, and while consuming a limited amount of resources (memory and processing power).

After researching and implementing solutions to enhance our own personalization technology, we are pleased to offer the open source community Fast Entity Linker, our unsupervised, accurate, and extensible multilingual named entity recognition and linking system, along with datapacks for English, Spanish, and Chinese.

For broad usability, our system links text entity mentions to Wikipedia. For example, in the sentence Yahoo is a company headquartered in Sunnyvale, CA with Marissa Mayer as CEO, our system would identify the following entities:

On the algorithmic side, we use entity embeddings, click-log data, and efficient clustering methods to achieve high precision. The system achieves a low memory footprint and fast execution times by using compressed data structures and aggressive hashing functions.

Entity embeddings are vector-based representations that capture how entities are referred to in context. We train entity embeddings using Wikipedia articles, and use hyperlinked terms in the articles to create canonical entities. The context of an entity and the context of a token are modeled using the neural network architecture in the figure below, where entity vectors are trained to predict not only their surrounding entities but also the global context of word sequences contained within them. In this way, one layer models entity context, and the other layer models token context. We connect these two layers using the same technique that (Quoc and Mikolov ‘14) used to train paragraph vectors.

image


Architecture for training word embeddings and entity embeddings simultaneously. Ent represents entities and W represents their context words.

Search click-log data gives very useful signals to disambiguate partial or ambiguous entity mentions. For example, if searchers for “Fox” tend to click on “Fox News” rather than “20th Century Fox,” we can use this data in order to identify “Fox” in a document. To disambiguate entity mentions and ensure a document has a consistent set of entities, our system supports three entity disambiguation algorithms:

*Currently, only the Forward Backward Algorithm is available in our open source release–the other two will be made available soon!

These algorithms are particularly helpful in accurately linking entities when a popular candidate is NOT the correct candidate for an entity mention. In the example below, these algorithms leverage the surrounding context to accurately link Manchester City, Swansea City, Liverpool, Chelsea, and Arsenal to their respective football clubs.



Ambiguous mentions that could refer to multiple entities are highlighted in red. For example, Chelsea could refer to Chelsea Football team or Chelsea neighborhood in New York or London. Unambiguous named entities are highlighted in green.


Examples of candidate retrieval process in Entity Linking for both ambiguous and unambiguous examples referred in the example above. The correct candidate is highlighted in green.


At this time, Fast Entity Linker is one of only three freely-available multilingual named entity recognition and linking systems (others are DBpedia Spotlight and Babelfy). In addition to a stand-alone entity linker, the software includes tools for creating and compressing word/entity embeddings and datapacks for different languages from Wikipedia data. As an example, the datapack containing information from all of English Wikipedia is only ~2GB.

The technical contributions of this system are described in two scientific papers:

There are numerous possible applications of the open-source toolkit. One of them is attributing sentiment to entities detected in the text, as opposed to the entire text itself. For example, consider the following actual review of the movie “Inferno” from a user on MetaCritic (revised for clarity): “While the great performance of Tom Hanks (wiki_Tom_Hanks) and company make for a mysterious and vivid movie, the plot is difficult to comprehend. Although the movie was a clever and fun ride, I expected more from Columbia (wiki_Columbia_Pictures).”  Though the review on balance is neutral, it conveys a positive sentiment about wiki_Tom_Hanks and a negative sentiment about wiki_Columbia_Pictures.

Many existing sentiment analysis tools collate the sentiment value associated with the text as a whole, which makes it difficult to track sentiment around any individual entity. With our toolkit, one could automatically extract “positive” and “negative” aspects within a given text, giving a clearer understanding of the sentiment surrounding its individual components.

Feel free to use the code, contribute to it, and come up with addtional applications; our system and models are available at https://github.com/yahoo/FEL.

Great work from our Yahoo Research team!

AWS Week in Review – August 29, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-august-29-2016/

This is the second community-driven edition of the AWS Week in Review. Special thanks are due to the 13 external contributors who helped to make this happen. If you would like to contribute, please take a look at the AWS Week in Review on GitHub. Adding relevant content is fast and easy and can be done from the comfort of your web browser! Just to be clear, it is perfectly fine for you to add content written by someone else. The goal is to catch it all, as they say!


Monday

August 29

Tuesday

August 30

Wednesday

August 31

Thursday

September 1

Friday

September 2

New & Notable Open Source

  • apilogs is a command-line utility to help aggregate, stream, and filter CloudWatch Log events produced by API Gateway and Lambda serverless APIs.
  • MoonMail is a fully Lambda / SES powered email marketing tool.

New SlideShare Presentations

New Customer Success Stories

  • Bustle uses AWS Lambda to process high volumes of data generated by the website in real-time, allowing the team to make faster, data-driven decisions. Bustle.com is a news, entertainment, lifestyle, and fashion website catering to women.
  • Graze continually improves its customers’ experience by staying agile—including in its infrastructure. The company sells healthy snacks through its website and via U.K. retailers. It runs all its infrastructure on AWS, including its customer-facing websites and all its internal systems from the factory floor to business intelligence.
  • Made.com migrated to AWS to support a record-breaking sales period with no downtime. The company provides a website that links home-furnishings designers directly to consumers. It now runs its e-commerce platform, website, and customer-facing applications on AWS, using services such as Amazon EC2, Amazon RDS, and Auto Scaling groups.
  • Sony DADC New Media Solutions (NMS) distributes hundreds of thousands of hours of video content monthly, spins up data analytics, renders solutions in days instead of months, and saves millions of dollars in hardware refresh costs by going all in on AWS. The organization distributes and delivers content to film studios, television broadcasters, and other providers across the globe. NMS runs its content distribution platform, broadcast playout services, and video archive on the AWS Cloud.
  • Upserve quickly develops and trains more than 100 learning models, streams restaurant sales and menu item data in real time, and gives restaurateurs the ability to predict their nightly business using Amazon Machine Learning. The company provides online payment and analytical software to thousands of restaurant owners throughout the U.S. Upserve uses Amazon Machine Learning to provide predictive analysis through its Shift Prep application.

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter, subscribe to the RSS feed, and contribute some content!

Jeff;

Human Sensor

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/human-sensor/

In collaboration with Professor Frank Kelly and the environmental scientists of King’s College London, artist Kasia Molga has created Human Sensor – a collection of hi-tech costumes that react to air pollution within the wearer’s environment.

Commissioned by Invisible Dust, an organisation supporting artists and scientists to create pieces that explore environmental and climate change, Molga took to the streets of Manchester with her army of Human Sensors to promote the invisible threat of air pollution in the industrial city.

Human Sensor

Angry little clouds of air pollution

Each suit is equipped with a small aerosol monitor that links to a Raspberry Pi and GPS watch. These components work together to collect pollution data from their location. Eventually, the suits will relay data back in real time to a publicly accessible website; for now, information is stored and submitted at a later date.

The Pi also works to control the LEDs within the suit, causing them to flash, pulse, and produce patterns and colours that morph in reaction to air conditions as they are read by the monitor.

Human Sensor

All of the lights…

The suit’s LED system responds to the presence of pollutant particles in the air, changing the colour of the white suit to reflect the positive or negative effect of the air around it. Walk past the grassy clearing of a local park, and the suit will turn green to match it. Stand behind the exhaust of a car, and you’ll find yourself pulsating red.

It’s unsurprising that the presence of the suits in Manchester was both well received and a shock to the system for the city’s residents. While articles are beginning to surface regarding the impact of air pollution on children’s mental health, and other aspects of the detrimental health effects of pollution have long been known, it’s a constant struggle for scientists to remind society of the importance of this invisible threat. By building a physical reminder, using the simple warning colour system of red and green, it’s hard not to take the threat seriously.

“The big challenge we have is that air pollution is mostly invisible. Art helps to makes it visible. We are trying to bring air pollution into the public realm. Scientific papers in journals work on one level, but this is a way to bring it into the street where the public are.” – Andrew Grieve, Senior Air Quality Analyst, King’s College

 

Human Sensor

23-29 July 2016 in Manchester Performers in hi tech illuminated costumes reveal changes in urban air pollution. Catch the extraordinary performances created by media artist Kasia Molga with Professor Frank Kelly from King’s College London. The hi-tech illuminated costumes reflect the air pollution you are breathing on your daily commute.

Human Sensor is supported by the Wellcome Trust’s Sustaining Excellence Award and by Arts Council England; Invisible Dust is working in partnership with Manchester, European City of Science.

The post Human Sensor appeared first on Raspberry Pi.