In the latest issue of Wireframe magazine, Mark Vanstone shows you how to turn a 3D shooter into a VR game for a variety of viewers, from Google Cardboard to gaming headsets.
Our shooter, now in VR format. Blast aliens from the sky as they fly over the mountains.
Browser development has really matured of late, with a number of exciting new features coming to the fore. Where WebGL was well supported, the new WebXR (previously WebVR) is now becoming standard in modern browsers. JavaScript ES5 has now been superseded by ES6, and with these new technologies, making browser-based apps – and specifically 3D games – is now a serious proposition. The other part of VR technology, the hardware, has also evolved – as mobile tech has become more powerful, so have the opportunities for cost-effective VR experiences.
With a mobile phone you can create a stereoscopic display for use with a simple Cardboard viewer.
If you have an old mobile phone, perhaps an Make your own virtual reality 3D Shooter B iPhone 6 or a Samsung S7, you can get a Google Cardboard headset for £6 and turn it into a rudimentary VR viewer. In this article, we’ll show you how to set up a 3D shooter to run not only on that hardware, but also on other viewers such as the Gear VR or Daydream, and even gaming headsets such as the Oculus, HTC, or Valve. All of these have web browsers built in which can support WebXR or a 3D display, and although there are some differences between how they work, the process of displaying a 3D scene is mostly the same on all of them, so we can use the Three.js JavaScript library.
There’s a wide range of VR viewers and headsets on the market, from Google Cardboard to pricey gaming headsets like the Oculus, HTC, HP, Microsoft, and Valve ranges.
To begin, we’ll start with the Three.js 3D shooter we made in Wireframe #32 – if you missed it, you can download a copy. We’ll use the same models and much of the same code. The first change, though, is to update the code to run as an ES6 module. The non-module version of Three.js is being phased out at the end of 2020, so it’s probably best to get with the times and use the new stuff. As with our earlier shooter, you’ll need to run this code from a secure web server, which, for mobile phones and gaming headsets, will mean uploading it to somewhere suitable, but if you want to see it running, you can play it at technovisual.co.uk/vr.
Basic VR viewers
Now we need to consider the hardware we’re going to use to run our game. Let’s start at our baseline, Google Cardboard, and work up from there. Available from many outlets online (including Google’s store), it’s a cut-out kit, which you fold up to create a viewer.
From a browser window inside a gaming headset environment, you can launch a Three.js VR experience
There are two lenses to look through, two magnets in a recess on the side, and velcro tabs to hold a mobile phone. The magnets on the side serve as a selection mechanism which we’ll explore later.
Next, we have Gear VR-style viewers. There are many different types, priced from around £12 to £40, and these are essentially a better-built plastic version of the Cardboard but with a button on top to act as a selector. Phones of varying sizes can be used, and as long as the device isn’t more than about four years old, it should be up-to-date enough to run the 3D software.
For example, the six-year-old Samsung S5 is capable of displaying VR, but it’s a bit too slow to make the experience pleasant, whereas a five-year-old iPhone 6 is quite capable of displaying simple VR scenes smoothly. (With iPhones, you may need to switch on Experimental Features in the Safari settings, however.)
Proper pro kit
Gaming headsets are a bit different, since they have a built-in screen in the headset, and – in the case of the Oculus Go and Quest – an Android computer in there as well. Tethered headsets use the power of a connected computer to generate the display, and all of them use a slightly different Three.js system from the cheaper viewers to generate the 3D display.
As time goes on, it’s likely that more mobile phones will be compatible with the VR software used by the untethered gaming headsets. Gaming headsets also have sensors that track your movement as well as the tilt of the headset, providing six degrees of freedom.
Get the rest of the tutorial in Wireframe #44
This is just a taste of the comprehensive guide included in the latest issue of Wireframe magazine. If you’re not a subscriber, you can download a PDF copy for free from the Wireframe magazine website. Start at page 50 and work your way through to create your own VR shooter game.
And if you want to take advantage of Wireframe magazine’s latest subscription deal, you can get it for just £10 at the official Raspberry Pi Press online store.
I’m happy to announce that Sumerian is now generally available. You can create realistic virtual environments and scenes without having to acquire or master specialized tools for 3D modeling, animation, lighting, audio editing, or programming. Once built, you can deploy your finished creation across multiple platforms without having to write custom code or deal with specialized deployment systems and processes.
Sumerian gives you a web-based editor that you can use to quickly and easily create realistic, professional-quality scenes. There’s a visual scripting tool that lets you build logic to control how objects and characters (Sumerian Hosts) respond to user actions. Sumerian also lets you create rich, natural interactions powered by AWS services such as Amazon Lex, Polly, AWS Lambda, AWS IoT, and Amazon DynamoDB.
Sumerian was designed to work on multiple platforms. The VR and AR apps that you create in Sumerian will run in browsers that supports WebGL or WebVR and on popular devices such as the Oculus Rift, HTC Vive, and those powered by iOS or Android.
During the preview period, we have been working with a broad spectrum of customers to put Sumerian to the test and to create proof of concept (PoC) projects designed to highlight an equally broad spectrum of use cases, including employee education, training simulations, field service productivity, virtual concierge, design and creative, and brand engagement. Fidelity Labs (the internal R&D unit of Fidelity Investments), was the first to use a Sumerian host to create an engaging VR experience. Cora (the host) lives within a virtual chart room. She can display stock quotes, pull up company charts, and answer questions about a company’s performance. This PoC uses Amazon Polly to implement text to speech and Amazon Lex for conversational chatbot functionality. Read their blog post and watch the video inside to see Cora in action:
Now that Sumerian is generally available, you have the power to create engaging AR, VR, and 3D experiences of your own. To learn more, visit the Amazon Sumerian home page and then spend some quality time with our extensive collection of Sumerian Tutorials.
If you’re not already familiar with building visualizations for quick access to business insights using Amazon QuickSight, consider this your introduction. In this post, we’ll walk through some common scenarios with sample datasets to provide an overview of how you can connect yuor data, perform advanced analysis and access the results from any web browser or mobile device.
The following visualizations are built from the public datasets available in the links below. Before we jump into that, let’s take a look at the supported data sources, file formats and a typical QuickSight workflow to build any visualization.
Which data sources does Amazon QuickSight support?
At the time of publication, you can use the following data methods:
Connect to AWS data sources, including:
Amazon RDS
Amazon Aurora
Amazon Redshift
Amazon Athena
Amazon S3
Upload Excel spreadsheets or flat files (CSV, TSV, CLF, and ELF)
Connect to on-premises databases like Teradata, SQL Server, MySQL, and PostgreSQL
Import data from SaaS applications like Salesforce and Snowflake
Use big data processing engines like Spark and Presto
SPICE is the Amazon QuickSight super-fast, parallel, in-memory calculation engine, designed specifically for ad hoc data visualization. SPICE stores your data in a system architected for high availability, where it is saved until you choose to delete it. Improve the performance of database datasets by importing the data into SPICE instead of using a direct database query. To calculate how much SPICE capacity your dataset needs, see Managing SPICE Capacity.
Typical Amazon QuickSight workflow
When you create an analysis, the typical workflow is as follows:
Connect to a data source, and then create a new dataset or choose an existing dataset.
(Optional) If you created a new dataset, prepare the data (for example, by changing field names or data types).
Create a new analysis.
Add a visual to the analysis by choosing the fields to visualize. Choose a specific visual type, or use AutoGraph and let Amazon QuickSight choose the most appropriate visual type, based on the number and data types of the fields that you select.
(Optional) Modify the visual to meet your requirements (for example, by adding a filter or changing the visual type).
(Optional) Add more visuals to the analysis.
(Optional) Add scenes to the default story to provide a narrative about some aspect of the analysis data.
(Optional) Publish the analysis as a dashboard to share insights with other users.
The following graphic illustrates a typical Amazon QuickSight workflow.
Visualizations created in Amazon QuickSight with sample datasets
Data catalog: The World Bank invests into multiple development projects at the national, regional, and global levels. It’s a great source of information for data analysts.
The following graph shows the percentage of the population that has access to electricity (rural and urban) during 2000 in Asia, Africa, the Middle East, and Latin America.
The following graph shows the share of healthcare costs that are paid out-of-pocket (private vs. public). Also, you can maneuver over the graph to get detailed statistics at a glance.
Data catalog: The DBG PDS project makes real-time data derived from Deutsche Börse’s trading market systems available to the public for free. This is the first time that such detailed financial market data has been shared freely and continually from the source provider.
The following graph shows the market trend of max trade volume for different EU banks. It builds on the data available on XETRA engines, which is made up of a variety of equities, funds, and derivative securities. This graph can be scrolled to visualize trade for a period of an hour or more.
The following graph shows the common stock beating the rest of the maximum trade volume over a period of time, grouped by security type.
Data catalog: Data derived from different sensor stations placed on the city bridges and surface streets are a core information source. The road weather information station has a temperature sensor that measures the temperature of the street surface. It also has a sensor that measures the ambient air temperature at the station each second.
The following graph shows the present max air temperature in Seattle from different RWI station sensors.
The following graph shows the minimum temperature of the road surface at different times, which helps predicts road conditions at a particular time of the year.
Data catalog: Kaggle has come up with a platform where people can donate open datasets. Data engineers and other community members can have open access to these datasets and can contribute to the open data movement. They have more than 350 datasets in total, with more than 200 as featured datasets. It has a few interesting datasets on the platform that are not present at other places, and it’s a platform to connect with other data enthusiasts.
The following graph shows the trending YouTube videos and presents the max likes for the top 20 channels. This is one of the most popular datasets for data engineers.
The following graph shows the YouTube daily statistics for the max views of video titles published during a specific time period.
Data catalog: NYC Open data hosts some very popular open data sets for all New Yorkers. This platform allows you to get involved in dive deep into the data set to pull some useful visualizations. 2016 Green taxi trip dataset includes trip records from all trips completed in green taxis in NYC in 2016. Records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.
The following graph presents maximum fare amount grouped by the passenger count during a period of time during a day. This can be further expanded to follow through different day of the month based on the business need.
The following graph shows the NewYork taxi data from January 2016, showing the dip in the number of taxis ridden on January 23, 2016 across all types of taxis.
A quick search for that date and location shows you the following news report:
Summary
Using Amazon QuickSight, you can see patterns across a time-series data by building visualizations, performing ad hoc analysis, and quickly generating insights. We hope you’ll give it a try today!
Karthik Odapally is a Sr. Solutions Architect in AWS. His passion is to build cost effective and highly scalable solutions on the cloud. In his spare time, he bakes cookies and cupcakes for family and friends here in the PNW. He loves vintage racing cars.
Pranabesh Mandal is a Solutions Architect in AWS. He has over a decade of IT experience. He is passionate about cloud technology and focuses on Analytics. In his spare time, he likes to hike and explore the beautiful nature and wild life of most divine national parks around the United States alongside his wife.
This post courtesy of Giedrius Praspaliauskas, AWS Solutions Architect
Even with best IVR systems, customers get frustrated. What if you knew that 10 callers in your Amazon Connect contact flow were likely to say “Agent!” in frustration in the next 30 seconds? Would you like to get to them before that happens? What if your bot was smart enough to admit, “I’m sorry this isn’t helping. Let me find someone for you.”?
Setting up a Lambda function for sentiment analysis
There are multiple natural language and text processing frameworks or services available to use with Lambda, including but not limited to Amazon Comprehend, TextBlob, Pattern, and NLTK. Pick one based on the nature of your system: the type of interaction, languages supported, and so on. For this post, I picked Amazon Comprehend, which uses natural language processing (NLP) to extract insights and relationships in text.
The walkthrough in this post is just an example. In a full-scale implementation, you would likely implement a more nuanced approach. For example, you could keep the overall sentiment score through the conversation and act only when it reaches a certain threshold. It is worth noting that this Lambda function is not called for missed utterances, so there may be a gap between what is being analyzed and what was actually said.
The Lambda function is straightforward. It analyses the input transcript field of the Amazon Lex event. Based on the overall sentiment value, it generates a response message with next step instructions. When the sentiment is neutral, positive, or mixed, the response leaves it to Amazon Lex to decide what the next steps should be. It adds to the response overall sentiment value as an additional session attribute, along with slots’ values received as an input.
When the overall sentiment is negative, the function returns the dialog action, pointing to an escalation intent (specified in the environment variable ESCALATION_INTENT_NAME) or returns the fulfillment closure action with a failure state when the intent is not specified. In addition to actions or intents, the function returns a message, or prompt, to be provided to the customer before taking the next step. Based on the returned action, Amazon Connect can select the appropriate next step in a contact flow.
For this walkthrough, you create a Lambda function using the AWS Management Console:
Open the Lambda console.
Choose Create Function.
Choose Author from scratch (no blueprint).
For Runtime, choose Python 3.6.
For Role, choose Create a custom role. The custom execution role allows the function to detect sentiments, create a log group, stream log events, and store the log events.
Enter the following values:
For Role Description, enter Lambda execution role permissions.
For IAM Role, choose Create an IAM role.
For Role Name, enter LexSentimentAnalysisLambdaRole.
Copy/paste the following code to the editor window
import os, boto3
ESCALATION_INTENT_MESSAGE="Seems that you are having troubles with our service. Would you like to be transferred to the associate?"
FULFILMENT_CLOSURE_MESSAGE="Seems that you are having troubles with our service. Let me transfer you to the associate."
escalation_intent_name = os.getenv('ESACALATION_INTENT_NAME', None)
client = boto3.client('comprehend')
def lambda_handler(event, context):
sentiment=client.detect_sentiment(Text=event['inputTranscript'],LanguageCode='en')['Sentiment']
if sentiment=='NEGATIVE':
if escalation_intent_name:
result = {
"sessionAttributes": {
"sentiment": sentiment
},
"dialogAction": {
"type": "ConfirmIntent",
"message": {
"contentType": "PlainText",
"content": ESCALATION_INTENT_MESSAGE
},
"intentName": escalation_intent_name
}
}
else:
result = {
"sessionAttributes": {
"sentiment": sentiment
},
"dialogAction": {
"type": "Close",
"fulfillmentState": "Failed",
"message": {
"contentType": "PlainText",
"content": FULFILMENT_CLOSURE_MESSAGE
}
}
}
else:
result ={
"sessionAttributes": {
"sentiment": sentiment
},
"dialogAction": {
"type": "Delegate",
"slots" : event["currentIntent"]["slots"]
}
}
return result
Below the code editor specify the environment variable ESCALATION_INTENT_NAME with a value of Escalate.
Click on Save in the top right of the console.
Now you can test your function.
Click Test at the top of the console.
Configure a new test event using the following test event JSON:
This message should return a response from Lambda with a sentiment session attribute of NEUTRAL.
However, if you change the input to “This is garbage!”, Lambda changes the dialog action to the escalation intent specified in the environment variable ESCALATION_INTENT_NAME.
Setting up Amazon Lex
Now that you have your Lambda function running, it is time to create the Amazon Lex bot. Use the BookTrip sample bot and call it BookSomething. The IAM role is automatically created on your behalf. Indicate that this bot is not subject to the COPPA, and choose Create. A few minutes later, the bot is ready.
Make the following changes to the default configuration of the bot:
Add an intent with no associated slots. Name it Escalate.
Specify the Lambda function for initialization and validation in the existing two intents (“BookCar” and “BookHotel”), at the same time giving Amazon Lex permission to invoke it.
Leave the other configuration settings as they are and save the intents.
You are ready to build and publish this bot. Set a new alias, BookSomethingWithSentimentAnalysis. When the build finishes, test it.
After the instance is created, you need to integrate the Amazon Lex bot created in the previous step. For more information, see the Amazon Lex section in the Configuring Your Amazon Connect Instance topic. You may also want to look at the excellent post by Randall Hunt, New – Amazon Connect and Amazon Lex Integration.
Create a new contact flow, “Sentiment analysis walkthrough”:
Log in into the Amazon Connect instance.
Choose Create contact flow, Create transfer to agent flow.
Add a Get customer input block, open the icon in the top left corner, and specify your Amazon Lex bot and its intents.
Select the Text to speech audio prompt type and enter text for Amazon Connect to play at the beginning of the dialog.
Choose Amazon Lex, enter your Amazon Lex bot name and the alias.
Specify the intents to be used as dialog branches that a customer can choose: BookHotel, BookTrip, or Escalate.
Add two Play prompt blocks and connect them to the customer input block.
If booking hotel or car intent is returned from the bot flow, play the corresponding prompt (“OK, will book it for you”) and initiate booking (in this walkthrough, just hang up after the prompt).
However, if escalation intent is returned (caused by the sentiment analysis results in the bot), play the prompt (“OK, transferring to an agent”) and initiate the transfer.
Save and publish the contact flow.
As a result, you have a contact flow with a single customer input step and a text-to-speech prompt that uses the Amazon Lex bot. You expect one of the three intents returned:
Edit the phone number to associate the contact flow that you just created. It is now ready for testing. Call the phone number and check how your contact flow works.
Cleanup
Don’t forget to delete all the resources created during this walkthrough to avoid incurring any more costs:
Amazon Connect instance
Amazon Lex bot
Lambda function
IAM role LexSentimentAnalysisLambdaRole
Summary
In this walkthrough, you implemented sentiment analysis with a Lambda function. The function can be integrated into Amazon Lex and, as a result, into Amazon Connect. This approach gives you the flexibility to analyze user input and then act. You may find the following potential use cases of this approach to be of interest:
Extend the Lambda function to identify “hot” topics in the user input even if the sentiment is not negative and take action proactively. For example, switch to an escalation intent if a user mentioned “where is my order,” which may signal potential frustration.
Use Amazon Connect Streams to provide agent sentiment analysis results along with call transfer. Enable service tailored towards particular customer needs and sentiments.
Route calls to agents based on both skill set and sentiment.
Prioritize calls based on sentiment using multiple Amazon Connect queues instead of transferring directly to an agent.
Monitor quality and flag for review contact flows that result in high overall negative sentiment.
Implement sentiment and AI/ML based call analysis, such as a real-time recommendation engine. For more details, see Machine Learning on AWS.
If you have questions or suggestions, please comment below.
The Rust team has announced the release of Rust 1.25.0. “The last few releases have been relatively minor, but Rust 1.25 contains a bunch of stuff! The first one is straightforward: we’ve upgraded to LLVM 6 from LLVM 4. This has a number of effects, a major one being a step closer to AVR support.” See the release notes for details.
With the explosion in virtual reality (VR) technologies over the past few years, we’ve had an increasing number of customers ask us for advice and best practices around deploying their VR-based products and service offerings on the AWS Cloud. It soon became apparent that while the VR ecosystem is large in both scope and depth of types of workloads (gaming, e-medicine, security analytics, live streaming events, etc.), many of the workloads followed repeatable patterns, with storage and delivery of live and on-demand immersive video at the top of the list.
Looking at consumer trends, the desire for live and on-demand immersive video is fairly self-explanatory. VR has ushered in convenient and low-cost access for consumers and businesses to a wide variety of options for consuming content, ranging from browser playback of live and on-demand 360º video, all the way up to positional tracking systems with a high degree of immersion. All of these scenarios contain one lowest common denominator: video.
Which brings us to the topic of this post. We set out to build a solution that could support both live and on-demand events, bring with it a high degree of scalability, be flexible enough to support transformation of video if required, run at a low cost, and use open-source software to every extent possible.
In this post, we describe the reference architecture we created to solve this challenge, using Amazon EC2 Spot Instances, Amazon S3, Elastic Load Balancing, Amazon CloudFront, AWS CloudFormation, and Amazon CloudWatch, with open-source software such as NGINX, FFMPEG, and JavaScript-based client-side playback technologies. We step you through deployment of the solution and how the components work, as well as the capture, processing, and playback of the underlying live and on-demand immersive media streams.
This GitHub repository includes the source code necessary to follow along. We’ve also provided a self-paced workshop, from AWS re:Invent 2017 that breaks down this architecture even further. If you experience any issues or would like to suggest an enhancement, please use the GitHub issue tracker.
Prerequisites
As a side note, you’ll also need a few additional components to take best advantage of the infrastructure:
A camera/capture device capable of encoding and streaming RTMP video
A browser to consume the content.
You’re going to generate HTML5-compatible video (Apple HLS to be exact), but there are many other native iOS and Android options for consuming the media that you create. It’s also worth noting that your playback device should support projection of your input stream. We’ll talk more about that in the next section.
How does immersive media work?
At its core, any flavor of media, be that audio or video, can be viewed with some level of immersion. The ability to interact passively or actively with the content brings with it a further level of immersion. When you look at VR devices with rotational and positional tracking, you naturally need more than an ability to interact with a flat plane of video. The challenge for any creative thus becomes a tradeoff between immersion features (degrees of freedom, monoscopic 2D or stereoscopic 3D, resolution, framerate) and overall complexity.
Where can you start from a simple and effective point of view, that enables you to build out a fairly modular solution and test it? There are a few areas we chose to be prescriptive with our solution.
Source capture from the Ricoh Theta S
First, monoscopic 360-degree video is currently one of the most commonly consumed formats on consumer devices. We explicitly chose to focus on this format, although the infrastructure is not limited to it. More on this later.
Second, if you look at most consumer-level cameras that provide live streaming ability, and even many professional rigs, there are at least two lenses or cameras at a minimum. The figure above illustrates a single capture from a Ricoh Theta S in monoscopic 2D. The left image captures 180 degrees of the field of view, and the right image captures the other 180 degrees.
For this post, we chose a typical midlevel camera (the Ricoh Theta S), and used a laptop with open-source software (Open Broadcaster Software) to encode and stream the content. Again, the solution infrastructure is not limited to this particular brand of camera. Any camera or encoder that outputs 360º video and encodes to H264+AAC with an RTMP transport will work.
Third, capturing and streaming multiple camera feeds brings additional requirements around stream synchronization and cost of infrastructure. There is also a requirement to stitch media in real time, which can be CPU and GPU-intensive. Many devices and platforms do this either on the device, or via outboard processing that sits close to the camera location. If you stitch and deliver a single stream, you can save the costs of infrastructure and bitrate/connectivity requirements. We chose to keep these aspects on the encoder side to save on cost and reduce infrastructure complexity.
Last, the most common delivery format that requires little to no processing on the infrastructure side is equirectangular projection, as per the above figure. By stitching and unwrapping the spherical coordinates into a flat plane, you can easily deliver the video exactly as you would with any other live or on-demand stream. The only caveat is that resolution and bit rate are of utmost importance. The higher you can push these (high bit rate @ 4K resolution), the more immersive the experience is for viewers. This is due to the increase in sharpness and reduction of compression artifacts.
Knowing that we would be transcoding potentially at 4K on the source camera, but in a format that could be transmuxed without an encoding penalty on the origin servers, we implemented a pass-through for the highest bit rate, and elected to only transcode lower bitrates. This requires some level of configuration on the source encoder, but saves on cost and infrastructure. Because you can conform the source stream, you may as well take advantage of that!
For this post, we chose not to focus on ways to optimize projection. However, the reference architecture does support this with additional open source components compiled into the FFMPEG toolchain. A number of options are available to this end, such as open source equirectangular to cubic transformation filters. There is a tradeoff, however, in that reprojection implies that all streams must be transcoded.
Processing and origination stack
To get started, we’ve provided a CloudFormation template that you can launch directly into your own AWS account. We quickly review how it works, the solution’s components, key features, processing steps, and examine the main configuration files. Following this, you launch the stack, and then proceed with camera and encoder setup.
Immersive streaming reference architecture
The event encoder publishes the RTMP source to multiple origin elastic IP addresses for packaging into the HLS adaptive bitrate.
The client requests the live stream through the CloudFront CDN.
The origin responds with the appropriate HLS stream.
The edge fleet caches media requests from clients and elastically scales across both Availability Zones to meet peak demand.
CloudFront caches media at local edge PoPs to improve performance for users and reduce the origin load.
When the live event is finished, the VOD asset is published to S3. An S3 event is then published to SQS.
The encoding fleet processes the read messages from the SQS queue, processes the VOD clips, and stores them in the S3 bucket.
How it works
A camera captures content, and with the help of a contribution encoder, publishes a live stream in equirectangular format. The stream is encoded at a high bit rate (at least 2.5 Mbps, but typically 16+ Mbps for 4K) using H264 video and AAC audio compression codecs, and delivered to a primary origin via the RTMP protocol. Streams may transit over the internet or dedicated links to the origins. Typically, for live events in the field, internet or bonded cellular are the most widely used.
The encoder is typically configured to push the live stream to a primary URI, with the ability (depending on the source encoding software/hardware) to roll over to a backup publishing point origin if the primary fails. Because you run across multiple Availability Zones, this architecture could handle an entire zone outage with minor disruption to live events. The primary and backup origins handle the ingestion of the live stream as well as transcoding to H264+AAC-based adaptive bit rate sets. After transcode, they package the streams into HLS for delivery and create a master-level manifest that references all adaptive bit rates.
The edge cache fleet pulls segments and manifests from the active origin on demand, and supports failover from primary to backup if the primary origin fails. By adding this caching tier, you effectively separate the encoding backend tier from the cache tier that responds to client or CDN requests. In addition to origin protection, this separation allows you to independently monitor, configure, and scale these components.
Viewers can use the sample HTML5 player (or compatible desktop, iOS or Android application) to view the streams. Navigation in the 360-degree view is handled either natively via device-based gyroscope, positionally via more advanced devices such as a head mount display, or via mouse drag on the desktop. Adaptive bit rate is key here, as this allows you to target multiple device types, giving the player on each device the option of selecting an optimum stream based on network conditions or device profile.
Solution components
When you deploy the CloudFormation template, all the architecture services referenced above are created and launched. This includes:
The compute tier running on Spot Instances for the corresponding components:
the primary and backup ingest origins
the edge cache fleet
the transcoding fleet
the test source
The CloudFront distribution
S3 buckets for storage of on-demand VOD assets
An Application Load Balancer for load balancing the service
An Amazon ECS cluster and container for the test source
The template also provisions the underlying dependencies:
A VPC
Security groups
IAM policies and roles
Elastic network interfaces
Elastic IP addresses
The edge cache fleet instances need some way to discover the primary and backup origin locations. You use elastic network interfaces and elastic IP addresses for this purpose.
As each component of the infrastructure is provisioned, software required to transcode and process the streams across the Spot Instances is automatically deployed. This includes NGiNX-RTMP for ingest of live streams, FFMPEG for transcoding, NGINX for serving, and helper scripts to handle various tasks (potential Spot Instance interruptions, queueing, moving content to S3). Metrics and logs are available through CloudWatch and you can manage the deployment using the CloudFormation console or AWS CLI.
Key features include:
Live and video-on-demand recording
You’re supporting both live and on-demand. On-demand content is created automatically when the encoder stops publishing to the origin.
Cost-optimization and operating at scale using Spot Instances
Spot Instances are used exclusively for infrastructure to optimize cost and scale throughput.
Midtier caching
To protect the origin servers, the midtier cache fleet pulls, caches, and delivers to downstream CDNs.
Distribution via CloudFront or multi-CDN
The Application Load Balancer endpoint allows CloudFront or any third-party CDN to source content from the edge fleet and, indirectly, the origin.
FFMPEG + NGINX + NGiNX-RTMP
These three components form the core of the stream ingest, transcode, packaging, and delivery infrastructure, as well as the VOD-processing component for creating transcoded VOD content on-demand.
Simple deployment using a CloudFormation template
All infrastructure can be easily created and modified using CloudFormation.
Prototype player page
To provide an end-to-end experience right away, we’ve included a test player page hosted as a static site on S3. This page uses A-Frame, a cross-platform, open-source framework for building VR experiences in the browser. Though A-Frame provides many features, it’s used here to render a sphere that acts as a 3D canvas for your live stream.
Spot Instance considerations
At this stage, and before we discuss processing, it is important to understand how the architecture operates with Spot Instances.
Spot Instances are spare compute capacity in the AWS Cloud available to you at steep discounts compared to On-Demand prices. Spot Instances enables you to optimize your costs on the AWS Cloud and scale your application’s throughput up to 10X for the same budget. By selecting Spot Instances, you can save up-to 90% on On-Demand prices. This allows you to greatly reduce the cost of running the solution because, outside of S3 for storage and CloudFront for delivery, this solution is almost entirely dependent on Spot Instances for infrastructure requirements.
We also know that customers running events look to deploy streaming infrastructure at the lowest price point, so it makes sense to take advantage of it wherever possible. A potential challenge when using Spot Instances for live streaming and on-demand processing is that you need to proactively deal with potential Spot Instance interruptions. How can you best deal with this?
First, the origin is deployed in a primary/backup deployment. If a Spot Instance interruption happens on the primary origin, you can fail over to the backup with a brief interruption. Should a potential interruption not be acceptable, then either Reserved Instances or On-Demand options (or a combination) can be used at this tier.
Second, the edge cache fleet runs a job (started automatically at system boot) that periodically queries the local instance metadata to detect if an interruption is scheduled to occur. Spot Instance Interruption Notices provide a two-minute warning of a pending interruption. If you poll every 5 seconds, you have almost 2 full minutes to detach from the Load Balancer and drain or stop any traffic directed to your instance.
Lastly, use an SQS queue when transcoding. If a transcode for a Spot Instance is interrupted, the stale item falls back into the SQS queue and is eventually re-surfaced into the processing pipeline. Only remove items from the queue after the transcoded files have been successfully moved to the destination S3 bucket.
Processing
As discussed in the previous sections, you pass through the video for the highest bit rate to save on having to increase the instance size to transcode the 4K or similar high bit rate or resolution content.
We’ve selected a handful of bitrates for the adaptive bit rate stack. You can customize any of these to suit the requirements for your event. The default ABR stack includes:
2160p (4K)
1080p
540p
480p
These can be modified by editing the /etc/nginx/rtmp.d/rtmp.conf NGINX configuration file on the origin or the CloudFormation template.
It’s important to understand where and how streams are transcoded. When the source high bit rate stream enters the primary or backup origin at the /live RTMP application entry point, it is recorded on stop and start of publishing. On completion, it is moved to S3 by a cleanup script, and a message is placed in your SQS queue for workers to use. These workers transcode the media and push it to a playout location bucket.
This solution uses Spot Fleet with automatic scaling to drive the fleet size. You can customize it based on CloudWatch metrics, such as simple utilization metrics to drive the size of the fleet. Why use Spot Instances for the transcode option instead of Amazon Elastic Transcoder? This allows you to implement reprojection of the input stream via FFMPEG filters in the future.
The origins handle all the heavy live streaming work. Edges only store and forward the segments and manifests, and provide scaling plus reduction of burden on the origin. This lets you customize the origin to the right compute capacity without having to rely on a ‘high watermark’ for compute sizing, thus saving additional costs.
Loopback is an important concept for the live origins. The incoming stream entering /live is transcoded by FFMPEG to multiple bit rates, which are streamed back to the same host via RTMP, on a secondary publishing point /show. The secondary publishing point is transparent to the user and encoder, but handles HLS segment generation and cleanup, and keeps a sliding window of live segments and constantly updating manifests.
Configuration
Our solution provides two key points of configuration that can be used to customize the solution to accommodate ingest, recording, transcoding, and delivery, all controlled via origin and edge configuration files, which are described later. In addition, a number of job scripts run on the instances to provide hooks into Spot Instance interruption events and the VOD SQS-based processing queue.
Origin instances
The rtmp.conf excerpt below also shows additional parameters that can be customized, such as maximum recording file size in Kbytes, HLS Fragment length, and Playlist sizes. We’ve created these in accordance with general industry best practices to ensure the reliable streaming and delivery of your content.
rtmp {
server {
listen 1935;
chunk_size 4000;
application live {
live on;
record all;
record_path /var/lib/nginx/rec;
record_max_size 128000K;
exec_record_done /usr/local/bin/record-postprocess.sh $path $basename;
exec /usr/local/bin/ffmpeg <…parameters…>;
}
application show {
live on;
hls on;
...
hls_type live;
hls_fragment 10s;
hls_playlist_length 60s;
...
}
}
}
This exposes a few URL endpoints for debugging and general status. In production, you would most likely turn these off:
/stat provides a statistics endpoint accessible via any standard web browser.
/control enables control of RTMP streams and publishing points.
You also control the TTLs, as previously discussed. It’s important to note here that you are setting TTLs explicitly at the origin, instead of in CloudFront’s distribution configuration. While both are valid, this approach allows you to reconfigure and restart the service on the fly without having to push changes through CloudFront. This is useful for debugging any caching or playback issues.
record-postprocess.sh – Ensures that recorded files on the origin are well-formed, and transfers them to S3 for processing.
ffmpeg.sh – Transcodes content on the encoding fleet, pulling source media from your S3 ingress bucket, based on SQS queue entries, and pushing transcoded adaptive bit rate segments and manifests to your VOD playout egress bucket.
For more details, see the Delivery and Playback section later in this post.
Camera source
With the processing and origination infrastructure running, you need to configure your camera and encoder.
As discussed, we chose to use a Ricoh Theta S camera and Open Broadcaster Software (OBS) to stitch and deliver a stream into the infrastructure. Ricoh provides a free ‘blender’ driver, which allows you to transform, stitch, encode, and deliver both transformed equirectangular (used for this post) video as well as spherical (two camera) video. The Theta provides an easy way to get capturing for under $300, and OBS is a free and open-source software application for capturing and live streaming on a budget. It is quick, cheap, and enjoys wide use by the gaming community. OBS lowers the barrier to getting started with immersive streaming.
While the resolution and bit rate of the Theta may not be 4K, it still provides us with a way to test the functionality of the entire pipeline end to end, without having to invest in a more expensive camera rig. One could also use this type of model to target smaller events, which may involve mobile devices with smaller display profiles, such as phones and potentially smaller sized tablets.
Looking for a more professional solution? Nokia, GoPro, Samsung, and many others have options ranging from $500 to $50,000. This solution is based around the Theta S capabilities, but we’d encourage you to extend it to meet your specific needs.
If your device can support equirectangular RTMP, then it can deliver media through the reference architecture (dependent on instance sizing for higher bit rate sources, of course). If additional features are required such as camera stitching, mixing, or device bonding, we’d recommend exploring a commercial solution such as Teradek Sphere.
Teradek Rig (Teradek)
Ricoh Theta (CNET)
All cameras have varied PC connectivity support. We chose the Ricoh Theta S due to the real-time video connectivity that it provides through software drivers on macOS and PC. If you plan to purchase a camera to use with a PC, confirm that it supports real-time capabilities as a peripheral device.
Encoding and publishing
Now that you have a camera, encoder, and AWS stack running, you can finally publish a live stream.
To start streaming with OBS, configure the source camera and set a publishing point. Use the RTMP application name /live on port 1935 to ingest into the primary origin’s Elastic IP address provided as the CloudFormation output: primaryOriginElasticIp.
You also need to choose a stream name or stream key in OBS. You can use any stream name, but keep the naming short and lowercase, and use only alphanumeric characters. This avoids any parsing issues on client-side player frameworks. There’s no publish point protection in your deployment, so any stream key works with the default NGiNX-RTMP configuration. For more information about stream keys, publishing point security, and extending the NGiNX-RTMP module, see the NGiNX-RTMP Wiki.
You should end up with a configuration similar to the following:
OBS Stream Settings
The Output settings dialog allows us to rescale the Video canvas and encode it for delivery to our AWS infrastructure. In the dialog below, we’ve set the Theta to encode at 5 Mbps in CBR mode using a preset optimized for low CPU utilization. We chose these settings in accordance with best practices for the stream pass-through at the origin for the initial incoming bit rate. You may notice that they largely match the FFMPEG encoding settings we use on the origin – namely constant bit rate, a single audio track, and x264 encoding with the ‘veryfast’ encoding profile.
OBS Output Settings
Live to On-Demand
As you may have noticed, an on-demand component is included in the solution architecture. When talking to customers, one frequent request that we see is that they would like to record the incoming stream with as little effort as possible.
NGINX-RTMP’s recording directives provide an easy way to accomplish this. We record any newly published stream on stream start at the primary or backup origins, using the incoming source stream, which also happens to be the highest bit rate. When the encoder stops broadcasting, NGINX-RTMP executes an exec_record_done script – record-postprocess.sh (described in the Configuration section earlier), which ensures that the content is well-formed, and then moves it to an S3 ingest bucket for processing.
Transcoding of content to make it ready for VOD as adaptive bit rate is a multi-step pipeline. First, Spot Instances in the transcoding cluster periodically poll the SQS queue for new jobs. Items on the queue are pulled off on demand by processing instances, and transcoded via FFMPEG into adaptive bit rate HLS. This allows you to also extend FFMPEG using filters for cubic and other bitrate-optimizing 360-specific transforms. Finally, transcoded content is moved from the ingest bucket to an egress bucket, making them ready for playback via your CloudFront distribution.
Separate ingest and egress by bucket to provide hard security boundaries between source recordings (which are highest quality and unencrypted), and destination derivatives (which may be lower quality and potentially require encryption). Bucket separation also allows you to order and archive input and output content using different taxonomies, which is common when moving content from an asset management and archival pipeline (the ingest bucket) to a consumer-facing playback pipeline (the egress bucket, and any other attached infrastructure or services, such as CMS, Mobile applications, and so forth).
Because streams are pushed over the internet, there is always the chance that an interruption could occur in the network path, or even at the origin side of the equation (primary to backup roll-over). Both of these scenarios could result in malformed or partial recordings being created. For the best level of reliability, encoding should always be recorded locally on-site as a precaution to deal with potential stream interruptions.
Delivery and playback
With the camera turned on and OBS streaming to AWS, the final step is to play the live stream. We’ve primarily tested the prototype player on the latest Chrome and Firefox browsers on macOS, so your mileage may vary on different browsers or operating systems. For those looking to try the livestream on Google Cardboard, or similar headsets, native apps for iOS (VRPlayer) and Android exist that can play back HLS streams.
The prototype player is hosted in an S3 bucket and can be found from the CloudFormation output clientWebsiteUrl. It requires a stream URL provided as a query parameter ?url=<stream_url> to begin playback. This stream URL is determined by the RTMP stream configuration in OBS. For example, if OBS is publishing to rtmp://x.x.x.x:1935/live/foo, the resulting playback URL would be:
https://<cloudFrontDistribution>/hls/foo.m3u8
The combined player URL and playback URL results in a path like this one:
To assist in setup/debugging, we’ve provided a test source as part of the CloudFormation template. A color bar pattern with timecode and audio is being generated by FFmpeg running as an ECS task. Much like OBS, FFmpeg is streaming the test pattern to the primary origin over the RTMP protocol. The prototype player and test HLS stream can be accessed by opening the clientTestPatternUrl CloudFormation output link.
Test Stream Playback
What’s next?
In this post, we walked you through the design and implementation of a full end-to-end immersive streaming solution architecture. As you may have noticed, there are a number of areas this could expand into, and we intend to do this in follow-up posts around the topic of virtual reality media workloads in the cloud. We’ve identified a number of topics such as load testing, content protection, client-side metrics and analytics, and CI/CD infrastructure for 24/7 live streams. If you have any requests, please drop us a line.
We would like to extend extra-special thanks to Scott Malkie and Chad Neal for their help and contributions to this post and reference architecture.
We have been busy adding new features and capabilities to Amazon Redshift, and we wanted to give you a glimpse of what we’ve been doing over the past year. In this article, we recap a few of our enhancements and provide a set of resources that you can use to learn more and get the most out of your Amazon Redshift implementation.
In 2017, we made more than 30 announcements about Amazon Redshift. We listened to you, our customers, and delivered Redshift Spectrum, a feature of Amazon Redshift, that gives you the ability to extend analytics to your data lake—without moving data. We launched new DC2 nodes, doubling performance at the same price. We also announced many new features that provide greater scalability, better performance, more automation, and easier ways to manage your analytics workloads.
To see a full list of our launches, visit our what’s new page—and be sure to subscribe to our RSS feed.
Major launches in 2017
Amazon Redshift Spectrum—extend analytics to your data lake, without moving data
We launched Amazon Redshift Spectrum to give you the freedom to store data in Amazon S3, in open file formats, and have it available for analytics without the need to load it into your Amazon Redshift cluster. It enables you to easily join datasets across Redshift clusters and S3 to provide unique insights that you would not be able to obtain by querying independent data silos.
With Redshift Spectrum, you can run SQL queries against data in an Amazon S3 data lake as easily as you analyze data stored in Amazon Redshift. And you can do it without loading data or resizing the Amazon Redshift cluster based on growing data volumes. Redshift Spectrum separates compute and storage to meet workload demands for data size, concurrency, and performance. Redshift Spectrum scales processing across thousands of nodes, so results are fast, even with massive datasets and complex queries. You can query open file formats that you already use—such as Apache Avro, CSV, Grok, ORC, Apache Parquet, RCFile, RegexSerDe, SequenceFile, TextFile, and TSV—directly in Amazon S3, without any data movement.
“For complex queries, Redshift Spectrum provided a 67 percent performance gain,” said Rafi Ton, CEO, NUVIAD. “Using the Parquet data format, Redshift Spectrum delivered an 80 percent performance improvement. For us, this was substantial.”
DC2 nodes—twice the performance of DC1 at the same price
We launched second-generation Dense Compute (DC2) nodes to provide low latency and high throughput for demanding data warehousing workloads. DC2 nodes feature powerful Intel E5-2686 v4 (Broadwell) CPUs, fast DDR4 memory, and NVMe-based solid state disks (SSDs). We’ve tuned Amazon Redshift to take advantage of the better CPU, network, and disk on DC2 nodes, providing up to twice the performance of DC1 at the same price. Our DC2.8xlarge instances now provide twice the memory per slice of data and an optimized storage layout with 30 percent better storage utilization.
“Redshift allows us to quickly spin up clusters and provide our data scientists with a fast and easy method to access data and generate insights,” said Bradley Todd, technology architect at Liberty Mutual. “We saw a 9x reduction in month-end reporting time with Redshift DC2 nodes as compared to DC1.”
On average, our customers are seeing 3x to 5x performance gains for most of their critical workloads.
We introduced short query acceleration to speed up execution of queries such as reports, dashboards, and interactive analysis. Short query acceleration uses machine learning to predict the execution time of a query, and to move short running queries to an express short query queue for faster processing.
We launched results caching to deliver sub-second response times for queries that are repeated, such as dashboards, visualizations, and those from BI tools. Results caching has an added benefit of freeing up resources to improve the performance of all other queries.
We also introduced late materialization to reduce the amount of data scanned for queries with predicate filters by batching and factoring in the filtering of predicates before fetching data blocks in the next column. For example, if only 10 percent of the table rows satisfy the predicate filters, Amazon Redshift can potentially save 90 percent of the I/O for the remaining columns to improve query performance.
We launched query monitoring rules and pre-defined rule templates. These features make it easier for you to set metrics-based performance boundaries for workload management (WLM) queries, and specify what action to take when a query goes beyond those boundaries. For example, for a queue that’s dedicated to short-running queries, you might create a rule that aborts queries that run for more than 60 seconds. To track poorly designed queries, you might have another rule that logs queries that contain nested loops.
Customer insights
Amazon Redshift and Redshift Spectrum serve customers across a variety of industries and sizes, from startups to large enterprises. Visit our customer page to see the success that customers are having with our recent enhancements. Learn how companies like Liberty Mutual Insurance saw a 9x reduction in month-end reporting time using DC2 nodes. On this page, you can find case studies, videos, and other content that show how our customers are using Amazon Redshift to drive innovation and business results.
In addition, check out these resources to learn about the success our customers are having building out a data warehouse and data lake integration solution with Amazon Redshift:
You can enhance your Amazon Redshift data warehouse by working with industry-leading experts. Our AWS Partner Network (APN) Partners have certified their solutions to work with Amazon Redshift. They offer software, tools, integration, and consulting services to help you at every step. Visit our Amazon Redshift Partner page and choose an APN Partner. Or, use AWS Marketplace to find and immediately start using third-party software.
To see what our Partners are saying about Amazon Redshift Spectrum and our DC2 nodes mentioned earlier, read these blog posts:
If you are evaluating or considering a proof of concept with Amazon Redshift, or you need assistance migrating your on-premises or other cloud-based data warehouse to Amazon Redshift, our team of product experts and solutions architects can help you with architecting, sizing, and optimizing your data warehouse. Contact us using this support request form, and let us know how we can assist you.
If you are an Amazon Redshift customer, we offer a no-cost health check program. Our team of database engineers and solutions architects give you recommendations for optimizing Amazon Redshift and Amazon Redshift Spectrum for your specific workloads. To learn more, email us at [email protected].
Larry Heathcote is a Principle Product Marketing Manager at Amazon Web Services for data warehousing and analytics. Larry is passionate about seeing the results of data-driven insights on business outcomes. He enjoys family time, home projects, grilling out and the taste of classic barbeque.
If you’re an educator from the UK, chances are you’ve heard of Bett. For everyone else: Bett stands for British Education Technology Tradeshow. It’s the El Dorado of edtech, where every street is adorned with interactive whiteboards, VR headsets, and new technologies for the classroom. Every year since 2014, the Raspberry Pi Foundation has been going to the event hosted in the ExCeL London to chat to thousands of lovely educators about our free programmes and resources.
On a mission
Our setup this year consisted of four pods (imagine tables on steroids) in the STEAM village, and the mission of our highly trained team of education agents was to establish a new world record for Highest number of teachers talked to in a four-day period. I’m only half-joking.
Educators with a mission
Meeting educators
The best thing about being at Bett is meeting the educators who use our free content and training materials. It’s easy to get wrapped up in the everyday tasks of the office without stopping to ask: “Hey, have we asked our users what they want recently?” Events like Bett help us to connect with our audience, creating some lovely moments for both sides. We had plenty of Hello World authors visit us, including Gary Stager, co-author of Invent to Learn, a must-read for any computing educator. More than 700 people signed up for a digital subscription, we had numerous lovely conversations about our content and about ideas for new articles, and we met many new authors expressing an interest in writing for us in the future.
We also talked to lots of Raspberry Pi Certified Educators who we’d trained in our free Picademy programme — new dates in Belfast and Dublin now! — and who are now doing exciting and innovative things in their local areas. For example, Chris Snowden came to tell us about the great digital making outreach work he has been doing with the Eureka! museum in Yorkshire.
Raspberry Pi Certified Educator Chris Snowden
Digital making for kids
The other best thing about being at Bett is running workshops for young learners and seeing the delight on their faces when they accomplish something they believed to be impossible only five minutes ago. On the Saturday, we ran a massive Raspberry Jam/Code Club where over 250 children, parents, and curious onlookers got stuck into some of our computing activities. We were super happy to find out that we’d won the Bett Kids’ Choice Award for Best Hands-on Experience — a fantastic end to a busy four days. With Bett over for another year, our tired and happy ‘rebel alliance’ from across the Foundation still had the energy to pose for a group photo.
On January 3, the world learned about a series of major security vulnerabilities in modern microprocessors. Called Spectre and Meltdown, these vulnerabilities were discovered by several different researchers last summer, disclosed to the microprocessors’ manufacturers, and patched — at least to the extent possible.
This news isn’t really any different from the usual endless stream of security vulnerabilities and patches, but it’s also a harbinger of the sorts of security problems we’re going to be seeing in the coming years. These are vulnerabilities in computer hardware, not software. They affect virtually all high-end microprocessors produced in the last 20 years. Patching them requires large-scale coordination across the industry, and in some cases drastically affects the performance of the computers. And sometimes patching isn’t possible; the vulnerability will remain until the computer is discarded.
Spectre and Meltdown aren’t anomalies. They represent a new area to look for vulnerabilities and a new avenue of attack. They’re the future of security — and it doesn’t look good for the defenders.
Modern computers do lots of things at the same time. Your computer and your phone simultaneously run several applications — or apps. Your browser has several windows open. A cloud computer runs applications for many different computers. All of those applications need to be isolated from each other. For security, one application isn’t supposed to be able to peek at what another one is doing, except in very controlled circumstances. Otherwise, a malicious advertisement on a website you’re visiting could eavesdrop on your banking details, or the cloud service purchased by some foreign intelligence organization could eavesdrop on every other cloud customer, and so on. The companies that write browsers, operating systems, and cloud infrastructure spend a lot of time making sure this isolation works.
Both Spectre and Meltdown break that isolation, deep down at the microprocessor level, by exploiting performance optimizations that have been implemented for the past decade or so. Basically, microprocessors have become so fast that they spend a lot of time waiting for data to move in and out of memory. To increase performance, these processors guess what data they’re going to receive and execute instructions based on that. If the guess turns out to be correct, it’s a performance win. If it’s wrong, the microprocessors throw away what they’ve done without losing any time. This feature is called speculative execution.
Spectre and Meltdown attack speculative execution in different ways. Meltdown is more of a conventional vulnerability; the designers of the speculative-execution process made a mistake, so they just needed to fix it. Spectre is worse; it’s a flaw in the very concept of speculative execution. There’s no way to patch that vulnerability; the chips need to be redesigned in such a way as to eliminate it.
Since the announcement, manufacturers have been rolling out patches to these vulnerabilities to the extent possible. Operating systems have been patched so that attackers can’t make use of the vulnerabilities. Web browsers have been patched. Chips have been patched. From the user’s perspective, these are routine fixes. But several aspects of these vulnerabilities illustrate the sorts of security problems we’re only going to be seeing more of.
First, attacks against hardware, as opposed to software, will become more common. Last fall, vulnerabilities were discovered in Intel’s Management Engine, a remote-administration feature on its microprocessors. Like Spectre and Meltdown, they affected how the chips operate. Looking for vulnerabilities on computer chips is new. Now that researchers know this is a fruitful area to explore, security researchers, foreign intelligence agencies, and criminals will be on the hunt.
Second, because microprocessors are fundamental parts of computers, patching requires coordination between many companies. Even when manufacturers like Intel and AMD can write a patch for a vulnerability, computer makers and application vendors still have to customize and push the patch out to the users. This makes it much harder to keep vulnerabilities secret while patches are being written. Spectre and Meltdown were announced prematurely because details were leaking and rumors were swirling. Situations like this give malicious actors more opportunity to attack systems before they’re guarded.
Third, these vulnerabilities will affect computers’ functionality. In some cases, the patches for Spectre and Meltdown result in significant reductions in speed. The press initially reported 30%, but that only seems true for certain servers running in the cloud. For your personal computer or phone, the performance hit from the patch is minimal. But as more vulnerabilities are discovered in hardware, patches will affect performance in noticeable ways.
And then there are the unpatchable vulnerabilities. For decades, the computer industry has kept things secure by finding vulnerabilities in fielded products and quickly patching them. Now there are cases where that doesn’t work. Sometimes it’s because computers are in cheap products that don’t have a patch mechanism, like many of the DVRs and webcams that are vulnerable to the Mirai (and other) botnets — groups of Internet-connected devices sabotaged for coordinated digital attacks. Sometimes it’s because a computer chip’s functionality is so core to a computer’s design that patching it effectively means turning the computer off. This, too, is becoming more common.
Increasingly, everything is a computer: not just your laptop and phone, but your car, your appliances, your medical devices, and global infrastructure. These computers are and always will be vulnerable, but Spectre and Meltdown represent a new class of vulnerability. Unpatchable vulnerabilities in the deepest recesses of the world’s computer hardware is the new normal. It’s going to leave us all much more vulnerable in the future.
Last week I attended a talk given by Bryan Mistele, president of Seattle-based INRIX. Bryan’s talk provided a glimpse into the future of transportation, centering around four principle attributes, often abbreviated as ACES:
Autonomous – Cars and trucks are gaining the ability to scan and to make sense of their environments and to navigate without human input.
Connected – Vehicles of all types have the ability to take advantage of bidirectional connections (either full-time or intermittent) to other cars and to cloud-based resources. They can upload road and performance data, communicate with each other to run in packs, and take advantage of traffic and weather data.
Electric – Continued development of battery and motor technology, will make electrics vehicles more convenient, cost-effective, and environmentally friendly.
Shared – Ride-sharing services will change usage from an ownership model to an as-a-service model (sound familiar?).
Individually and in combination, these emerging attributes mean that the cars and trucks we will see and use in the decade to come will be markedly different than those of the past.
On the Road with AWS AWS customers are already using our AWS IoT, edge computing, Amazon Machine Learning, and Alexa products to bring this future to life – vehicle manufacturers, their tier 1 suppliers, and AutoTech startups all use AWS for their ACES initiatives. AWS Greengrass is playing an important role here, attracting design wins and helping our customers to add processing power and machine learning inferencing at the edge.
AWS customer Aptiv (formerly Delphi) talked about their Automated Mobility on Demand (AMoD) smart vehicle architecture in a AWS re:Invent session. Aptiv’s AMoD platform will use Greengrass and microservices to drive the onboard user experience, along with edge processing, monitoring, and control. Here’s an overview:
Another customer, Denso of Japan (one of the world’s largest suppliers of auto components and software) is using Greengrass and AWS IoT to support their vision of Mobility as a Service (MaaS). Here’s a video:
AWS at CES The AWS team will be out in force at CES in Las Vegas and would love to talk to you. They’ll be running demos that show how AWS can help to bring innovation and personalization to connected and autonomous vehicles.
Personalized In-Vehicle Experience – This demo shows how AWS AI and Machine Learning can be used to create a highly personalized and branded in-vehicle experience. It makes use of Amazon Lex, Polly, and Amazon Rekognition, but the design is flexible and can be used with other services as well. The demo encompasses driver registration, login and startup (including facial recognition), voice assistance for contextual guidance, personalized e-commerce, and vehicle control. Here’s the architecture for the voice assistance:
Connected Vehicle Solution – This demo shows how a connected vehicle can combine local and cloud intelligence, using edge computing and machine learning at the edge. It handles intermittent connections and uses AWS DeepLens to train a model that responds to distracted drivers. Here’s the overall architecture, as described in our Connected Vehicle Solution:
Digital Content Delivery – This demo will show how a customer uses a web-based 3D configurator to build and personalize their vehicle. It will also show high resolution (4K) 3D image and an optional immersive AR/VR experience, both designed for use within a dealership.
Autonomous Driving – This demo will showcase the AWS services that can be used to build autonomous vehicles. There’s a 1/16th scale model vehicle powered and driven by Greengrass and an overview of a new AWS Autonomous Toolkit. As part of the demo, attendees drive the car, training a model via Amazon SageMaker for subsequent on-board inferencing, powered by Greengrass ML Inferencing.
To speak to one of my colleagues or to set up a time to see the demos, check out the Visit AWS at CES 2018 page.
Some Resources If you are interested in this topic and want to learn more, the AWS for Automotive page is a great starting point, with discussions on connected vehicles & mobility, autonomous vehicle development, and digital customer engagement.
When you are ready to start building a connected vehicle, the AWS Connected Vehicle Solution contains a reference architecture that combines local computing, sophisticated event rules, and cloud-based data processing and storage. You can use this solution to accelerate your own connected vehicle projects.
Тази статия бе написана за Дневник и публикувана там на 8 ноември 2017 под заглавие „Желанието за независимост на Каталуня не идва от учебниците“. Не съм търсил и получавал хонорар за този текст – написах го по свое желание и настоях да бъде публикуван заради предходен материал, към който имах сериозни критики и за който става дума в текста. Препубликувам го и тук, в блога си, за да остане за личния ми архив.
Не съм изоставил темата Каталуния – макар да съзнавам напълно, че това се превръща в моя лична и самотна битка. Напоследък не смогвам с всичко, което е важно за мен, а и ситуацията в собствената ни превзета държава не е по-различна от театралната демокрация в кралство Испания. Тъжно е, че Каталуния извади на яве симптом на много тежка болест, свързана с егоизма на европейците и липсата на обикновена човешка солидарност в Европа. Симптом, който не предвещава нищо добро за бъдещето ни. Но ще го осъзнаем късно.
Важността да се направи езика един и същ винаги е била признавана за нещо велико и е признак на доминация или превъзходство на принцове или нации… но тъй като всяка нация чувства собствения си език като дар от Природата, това прави трудно завладяването им и ще ви е нужно известно време да го направите, особено когато хората, както в случая – каталунците, са упорити, арогантни и влюбени в своята страна, затова е добре внимателно да се помисли и да се дават скрити инструкции и съвети, така че целта да се постигне незабелязано…
Хосе Родриго Вилялпандо, старши служител в Съвета на Кастилия, 1716 г., извадка от “Тайни инструкции” до магистратите в Каталуния
В наш интерес е да “испанизираме” каталунските деца.
Хосе Игнасио Верт, испански министър на образованието, 2012 г.
Дори по математика се учи за Испания
Каталуния е една от автономните области на ИспанияДон КихотСервантес и Дон Кихот
“Нивото на университетското образование е високо, студентите имат много богат избор от дисциплини и относително малък брой „задължителни“ лекции. В университета всички предмети се преподават най-малко на два езика (каталунски и испански). Много често има възможност и за обучение на английски.” Това ми казва мой приятел, който от три години живее в едно от по-малките градчета на север от Барселона и завършва докторантура.
Поводът да говорим по темата е статия, публикувана в Дневник и озаглавена Испания отсъства, монархията е зло, Европа се проваля – какво пише в каталунските учебници, която преразказва прочита на испанската медия ABC на едно изследване. Самото изследване е валидно и съдържа анализ на учебен материал за V и VI клас, заедно със забележки и препоръки, но преразказът на ABC е далеч от обективен и преднамерено внушава напълно недостоверни неща: като например, че в каталунските училища всичко се учи на каталунски, учебният материал е манипулиран и манипулативен, испаноговорящите деца са в неравностойно положение, учениците дори не знаели кой е Дон Кихот и биват облъчвани с антиевропейски послания.
За жалост, това е част от активна кампания, обслужваща пряко тезата на испанското централно правителство на Народната партия, че каталунското движение за независимост е подхранвано от манипулирани исторически факти и систематично промиване на съзнания. В конфликта между Барселона и Мадрид медиите играят активна, но невинаги достатъчно чистоплътна роля.
Затова помолих няколко събеседници, които имат преки наблюдения върху образователната система в Каталуния, да споделят своята гледна точка.
Валентина е българка, която живее от 17 години в каталунската област Тарагона. Майка е на две деца (на 9 и 11 години), нейни са снимките от учебниците за IV, V и VI клас от обикновено държавно училище, които илюстрират тази статия.
Румен също е българин и е баща на две деца. Живее от две години и половина с цялото си семейство много близо до Барселона, а по-големият му син учи в частно международно училище.
Роса е каталунка, живее вече 48 години в град от област Барселона. Има две деца (на 8 и на 11 години) и е преподавател в системата на държавното образование на Каталуния. Комуникацията ни с Роса е на испански.
Споделям им, че в България се прокрадва тезата, че каталунският стремеж за независимост е подклаждан от манипулации в образователната система и моля за техните гледни точки.
“Мисля, че статията е едностранчива, защото учениците имат по два комплекта учебници (испански и каталунски), като първите се спускат от Мадрид. Освен това има сериозен контрол от централното правителство.”, споделя Румен.
“Тази информация е невярна и не отговаря на действителността.”, още по-категорична е Роса. “Учебниците не индоктринират. Те обясняват съвременната история с исторически факти, съвсем коректно и без да вменяват какви намерения е имала една или друга страна. Учителите също не индоктринират, най-малкото нямат никакво време за това. Всяко ново правителство променя образователния закон и учебните програми дори без съобразяване с децата със специални нужди, например. Има паралелки с повече от 30 деца (в тази на дъщеря ми са 31), но ако дете зададе въпрос на испански, ще му бъде отговорено на испански, дори и в час по каталунски.”
А Валентина допълва, че проблемът е всъщност обратен и “борбата винаги е била и е на основата да се запази езикът, културата и традициите. „Залитането“ на Каталуния към независимост няма нищо общо с това, което се изучава в училищата, по-скоро с политиката на испанските власти, която подтиска всякаква възможност за развитие и подобряване условията на живот в Каталуния.”
Питам Румен как се обучават децата в частно международно училище и на колко езика се преподава и общува. “Там основно ги обучават на английски език, но все пак имат минимум 30% от учебния материал на испански и каталунски, предимно по география, история и социални науки. Предметите се преподават на три основни езика и един чужд, приблизително 60% английски, 20% испански, 10% каталунски и 10% френски (чуждия език). Единственият сериозен недостатък, който съм забелязал в образователната им система, е много ниското ниво на владеене (когато въобще го има) на английски сред местното население и най-вече младежите под 25 г. Живял съм в три европейски държави освен в България и мога да кажа с чисто сърце, че Испания е най-назад във владеенето на чужди езици.”
В едно свое есе Пере Маянс Балселс, който е професор по каталунски език и литература, разказва как още през 1979 г. Каталунският автономен статут (регионалния основен закон) установява каталунския език като официален (поделяйки, разбира се, този статус с испанския), а Законът за езиково нормиране в Каталуния от 1983 г, има за цел да изравни положението на каталунския език, който дотогава на всички нива е бил поставен неоспоримо по-ниско от испанския. С този закон каталунският език става главният език, използван в основното и средното образование. “Той също признаваше правото на децата да получават начално образование на своя първи език, дали каталунски или испански, и изискваше двата езика да бъдат преподавани на всички нива на неуниверситетското образование, държеше заедно учениците с различен езиков произход и осигуряваше на всички деца в Каталуния на училищна възраст, независимо какъв език са ползвали при започване на училище, в края на своето основно обучение да могат да си служат и с каталунския, и с испанския език правилно и свободно.”, пише проф. Балселс. “Общо диагностично оценяване от 2010 г., публикувано от испанското министерство на образованието, показва, че езиковата компетенция на каталунските ученици по испански език е еднаква със средната за испанската държава, и в действителност е по-голяма от тази в някои едноезични общности като Канарските острови, Екстремадура или Андалусия, или тази в двуезични общности, където испанският има много по-силно присъствие като език на преподаване (като в Балеарските острови, Галисия и област Валенсия).”
Между другото, каталунският езиков модел е признат за добра практика в областта на преподаването в многоезични общности и отличен от групата на високо равнище по въпросите на многоезичието, създадена от Европейската комисия през 2005 г.
Реалната борба е да се опази каталунския език, който е застрашен. “Реалният проблем тук в Каталуния не е дали се изучава испански език и дали децата могат да общуват на него свободно.”, уточнява Валентина. “Реалният проблем е, че децата се връщат вкъщи с речник, пълен с испанизирани думи (espanyolismes) по вина на самите учители, които нямат добър каталунски, и, разбира се, на съученици, с които не могат да говорят на каталунски, защото те не искат да го говорят. Свидетели сме на това всеки ден от много време насам. Реалната борба е да се опази каталунския език, който е застрашен. Истината е, че малко хора говорят правилен каталунски, използват се много испански фрази и се забравят традиционните каталунски фрази и изрази. Нужно е учителите да имат по високо ниво за да се даде по-добра основа за бъдещото образование. Определено е много странно, когато получиш съобщение от преподавател в бележника на детето ти с правописни грешки, видни дори за мен, която съм българка с ниво С по каталунски.”
Съгласно данни на официалната анкета за употреба на езиците EULP (Enquesta d’usos lingüístics de la població), проведена през 2013 г., 94,3% от жителите над 14-годишна възраст заявяват, че разбират каталунски, 80,4% го говорят, 82,4% могат да четат и 60,4% могат да пишат на него. Но трябва да се има предвид, че Каталуния е общност на имигранти и едва малко повече от половината от анкетираните са родени в Каталуния, а около четвърт от тях на друго място в Испания – главно в региони, в които се говори кастилски, 18% са от чужбина.
Ф. Чавие Вила (доцент в Барселонския университет по каталунска филология и доктор по лингвистика от Vrije Universiteit в Брюксел) пише: “През 2010 г. Конституционният съд постанови, че кастилският език ще е задължителен в Каталуния, но не и самият каталунски. И на базата на това решение, две години по-късно испанският министър на образованието подпали Каталуния, когато се похвали, че неговата цел е да се “испанизират” (españolizar) каталунските ученици. Той представи законопроект, който не само посегна на каталунската юрисдикция, но позволи каталунските деца да бъдат обучавани едноезично на кастилски – позиция, която беше възприета като колониална и напълно неприемлива от повечето каталунци.”
Жузеп Мария Ганиет, компютърен инженер от Барселонския автономен университет, специализирал Изкуствен интелект, твърди, че “испанската държава, съзнавайки силното чувство за идентичност, което езикът поражда у хората, винаги е гледала на некастилските езици и култури като на историческа аномалия, вместо да ги приеме като общо културно наследство, което следва да бъде запазено и защитено.”
В заключение питам събеседниците си, дали считат, че учебниците провокират анти-европейски нагласи сред учениците или внушават, че Европа е провален проект?
Валентина споделя: “Определено не вярвам в това, тук хората са с точно противното мнение. Макар че след последните събития и позицията си относно 1 октомври, смятам, че ЕС загуби много почитатели, както и част от желанието за членство на Република Каталуния в ЕС.”
Румен допълва: “Според мен каталунците са абсолютно проевропейски настроени и това е една от причините сепаратизмът да не е толкова разпространен, защото ЕС е по-важен за каталунците от независимостта им. Твърдя това въз основа на многобройни разговори и с про-, и с антинастроените към отцепването. Историята на Испания, както и на България, има много гледни точки, така че дали е преподавана манипулативно е страшно трудно да се установи обективно, вероятно има такива елементи, но не вярвам спорните моменти да са много.”
Роса обобщава: “Убедена съм, че образованието в Каталуния защитава демокрацията, свободата и мира, и разказва за Европа на многообразието и плурализма и в никакъв случай за Европа като за провален проект.” Питам я накрая и за гледната ѝ точка за политическата криза между централното и автономното правителство и тя споделя: “Каталунското правителство действа съгласно демократичния мандат, който получи след изборите през септември 2015 г. и действията им бяха отговорни. Има неща, които бих искала да бяха направили по-добре, но и Мадрид не остана по-назад, особено провокирайки ситуации, които можеха да бъдат избегнати. Аз твърдо вярвам, че независимостта ще се случи. Ще отнеме повече или по-малко усилия, но ще се случи. Когато едно гражданско общество с постоянство и мир направи крачка към по-добро общество за себе си и децата си… няма връщане назад. Достатъчно е да си спомним, че всичките многолюдни демонстрации бяха като празненства, по лицата на хората грееха усмивки, завладяни от мечтата за по-добър проект. Това не може да бъде спряно. Гневът и конфронтацията, или отрицанието в името на самото отрицание, не могат да спрат това.”
В този материал са използвани извадки от есета, включени в книгата “Какво става с Каталуния?”, преведена и издадена на български език от Сдружение “Диалози” като свободна за разпространение електронна книга. Снимките са от учебници за IV, V и VI клас и опровергават твърденията, че всички учебни материали са само на каталунски език, избягват споменаването на Испания и Европейския съюз, а децата не учат кой е Сервантес и Дон Кихот. Направени са от Валентина Григорова. Благодаря за съдействието на Юлияна Кръстева при подготвянето на този материал.
Тази статия бе написана за Дневник и публикувана там на 8 ноември 2017 под заглавие „Желанието за независимост на Каталуня не идва от учебниците“. Не съм търсил и получавал хонорар за този текст – написах го по свое желание и настоях да бъде публикуван заради предходен материал, към който имах сериозни критики и за който става дума в текста. Препубликувам го и тук, в блога си, за да остане за личния ми архив.
Не съм изоставил темата Каталуния – макар да съзнавам напълно, че това се превръща в моя лична и самотна битка. Напоследък не смогвам с всичко, което е важно за мен, а и ситуацията в собствената ни превзета държава не е по-различна от театралната демокрация в кралство Испания. Тъжно е, че Каталуния извади на яве симптом на много тежка болест, свързана с егоизма на европейците и липсата на обикновена човешка солидарност в Европа. Симптом, който не предвещава нищо добро за бъдещето ни. Но ще го осъзнаем късно.
Важността да се направи езика един и същ винаги е била признавана за нещо велико и е признак на доминация или превъзходство на принцове или нации… но тъй като всяка нация чувства собствения си език като дар от Природата, това прави трудно завладяването им и ще ви е нужно известно време да го направите, особено когато хората, както в случая – каталунците, са упорити, арогантни и влюбени в своята страна, затова е добре внимателно да се помисли и да се дават скрити инструкции и съвети, така че целта да се постигне незабелязано…
Хосе Родриго Вилялпандо, старши служител в Съвета на Кастилия, 1716 г., извадка от “Тайни инструкции” до магистратите в Каталуния
В наш интерес е да “испанизираме” каталунските деца.
Хосе Игнасио Верт, испански министър на образованието, 2012 г.
Дори по математика се учи за Испания
Каталуния е една от автономните области на ИспанияДон КихотСервантес и Дон Кихот
“Нивото на университетското образование е високо, студентите имат много богат избор от дисциплини и относително малък брой „задължителни“ лекции. В университета всички предмети се преподават най-малко на два езика (каталунски и испански). Много често има възможност и за обучение на английски.” Това ми казва мой приятел, който от три години живее в едно от по-малките градчета на север от Барселона и завършва докторантура.
Поводът да говорим по темата е статия, публикувана в Дневник и озаглавена Испания отсъства, монархията е зло, Европа се проваля – какво пише в каталунските учебници, която преразказва прочита на испанската медия ABC на едно изследване. Самото изследване е валидно и съдържа анализ на учебен материал за V и VI клас, заедно със забележки и препоръки, но преразказът на ABC е далеч от обективен и преднамерено внушава напълно недостоверни неща: като например, че в каталунските училища всичко се учи на каталунски, учебният материал е манипулиран и манипулативен, испаноговорящите деца са в неравностойно положение, учениците дори не знаели кой е Дон Кихот и биват облъчвани с антиевропейски послания.
За жалост, това е част от активна кампания, обслужваща пряко тезата на испанското централно правителство на Народната партия, че каталунското движение за независимост е подхранвано от манипулирани исторически факти и систематично промиване на съзнания. В конфликта между Барселона и Мадрид медиите играят активна, но невинаги достатъчно чистоплътна роля.
Затова помолих няколко събеседници, които имат преки наблюдения върху образователната система в Каталуния, да споделят своята гледна точка.
Валентина е българка, която живее от 17 години в каталунската област Тарагона. Майка е на две деца (на 9 и 11 години), нейни са снимките от учебниците за IV, V и VI клас от обикновено държавно училище, които илюстрират тази статия.
Румен също е българин и е баща на две деца. Живее от две години и половина с цялото си семейство много близо до Барселона, а по-големият му син учи в частно международно училище.
Роса е каталунка, живее вече 48 години в град от област Барселона. Има две деца (на 8 и на 11 години) и е преподавател в системата на държавното образование на Каталуния. Комуникацията ни с Роса е на испански.
Споделям им, че в България се прокрадва тезата, че каталунският стремеж за независимост е подклаждан от манипулации в образователната система и моля за техните гледни точки.
“Мисля, че статията е едностранчива, защото учениците имат по два комплекта учебници (испански и каталунски), като първите се спускат от Мадрид. Освен това има сериозен контрол от централното правителство.”, споделя Румен.
“Тази информация е невярна и не отговаря на действителността.”, още по-категорична е Роса. “Учебниците не индоктринират. Те обясняват съвременната история с исторически факти, съвсем коректно и без да вменяват какви намерения е имала една или друга страна. Учителите също не индоктринират, най-малкото нямат никакво време за това. Всяко ново правителство променя образователния закон и учебните програми дори без съобразяване с децата със специални нужди, например. Има паралелки с повече от 30 деца (в тази на дъщеря ми са 31), но ако дете зададе въпрос на испански, ще му бъде отговорено на испански, дори и в час по каталунски.”
А Валентина допълва, че проблемът е всъщност обратен и “борбата винаги е била и е на основата да се запази езикът, културата и традициите. „Залитането“ на Каталуния към независимост няма нищо общо с това, което се изучава в училищата, по-скоро с политиката на испанските власти, която подтиска всякаква възможност за развитие и подобряване условията на живот в Каталуния.”
Питам Румен как се обучават децата в частно международно училище и на колко езика се преподава и общува. “Там основно ги обучават на английски език, но все пак имат минимум 30% от учебния материал на испански и каталунски, предимно по география, история и социални науки. Предметите се преподават на три основни езика и един чужд, приблизително 60% английски, 20% испански, 10% каталунски и 10% френски (чуждия език). Единственият сериозен недостатък, който съм забелязал в образователната им система, е много ниското ниво на владеене (когато въобще го има) на английски сред местното население и най-вече младежите под 25 г. Живял съм в три европейски държави освен в България и мога да кажа с чисто сърце, че Испания е най-назад във владеенето на чужди езици.”
В едно свое есе Пере Маянс Балселс, който е професор по каталунски език и литература, разказва как още през 1979 г. Каталунският автономен статут (регионалния основен закон) установява каталунския език като официален (поделяйки, разбира се, този статус с испанския), а Законът за езиково нормиране в Каталуния от 1983 г, има за цел да изравни положението на каталунския език, който дотогава на всички нива е бил поставен неоспоримо по-ниско от испанския. С този закон каталунският език става главният език, използван в основното и средното образование. “Той също признаваше правото на децата да получават начално образование на своя първи език, дали каталунски или испански, и изискваше двата езика да бъдат преподавани на всички нива на неуниверситетското образование, държеше заедно учениците с различен езиков произход и осигуряваше на всички деца в Каталуния на училищна възраст, независимо какъв език са ползвали при започване на училище, в края на своето основно обучение да могат да си служат и с каталунския, и с испанския език правилно и свободно.”, пише проф. Балселс. “Общо диагностично оценяване от 2010 г., публикувано от испанското министерство на образованието, показва, че езиковата компетенция на каталунските ученици по испански език е еднаква със средната за испанската държава, и в действителност е по-голяма от тази в някои едноезични общности като Канарските острови, Екстремадура или Андалусия, или тази в двуезични общности, където испанският има много по-силно присъствие като език на преподаване (като в Балеарските острови, Галисия и област Валенсия).”
Между другото, каталунският езиков модел е признат за добра практика в областта на преподаването в многоезични общности и отличен от групата на високо равнище по въпросите на многоезичието, създадена от Европейската комисия през 2005 г.
Реалната борба е да се опази каталунския език, който е застрашен. “Реалният проблем тук в Каталуния не е дали се изучава испански език и дали децата могат да общуват на него свободно.”, уточнява Валентина. “Реалният проблем е, че децата се връщат вкъщи с речник, пълен с испанизирани думи (espanyolismes) по вина на самите учители, които нямат добър каталунски, и, разбира се, на съученици, с които не могат да говорят на каталунски, защото те не искат да го говорят. Свидетели сме на това всеки ден от много време насам. Реалната борба е да се опази каталунския език, който е застрашен. Истината е, че малко хора говорят правилен каталунски, използват се много испански фрази и се забравят традиционните каталунски фрази и изрази. Нужно е учителите да имат по високо ниво за да се даде по-добра основа за бъдещото образование. Определено е много странно, когато получиш съобщение от преподавател в бележника на детето ти с правописни грешки, видни дори за мен, която съм българка с ниво С по каталунски.”
Съгласно данни на официалната анкета за употреба на езиците EULP (Enquesta d’usos lingüístics de la població), проведена през 2013 г., 94,3% от жителите над 14-годишна възраст заявяват, че разбират каталунски, 80,4% го говорят, 82,4% могат да четат и 60,4% могат да пишат на него. Но трябва да се има предвид, че Каталуния е общност на имигранти и едва малко повече от половината от анкетираните са родени в Каталуния, а около четвърт от тях на друго място в Испания – главно в региони, в които се говори кастилски, 18% са от чужбина.
Ф. Чавие Вила (доцент в Барселонския университет по каталунска филология и доктор по лингвистика от Vrije Universiteit в Брюксел) пише: “През 2010 г. Конституционният съд постанови, че кастилският език ще е задължителен в Каталуния, но не и самият каталунски. И на базата на това решение, две години по-късно испанският министър на образованието подпали Каталуния, когато се похвали, че неговата цел е да се “испанизират” (españolizar) каталунските ученици. Той представи законопроект, който не само посегна на каталунската юрисдикция, но позволи каталунските деца да бъдат обучавани едноезично на кастилски – позиция, която беше възприета като колониална и напълно неприемлива от повечето каталунци.”
Жузеп Мария Ганиет, компютърен инженер от Барселонския автономен университет, специализирал Изкуствен интелект, твърди, че “испанската държава, съзнавайки силното чувство за идентичност, което езикът поражда у хората, винаги е гледала на некастилските езици и култури като на историческа аномалия, вместо да ги приеме като общо културно наследство, което следва да бъде запазено и защитено.”
В заключение питам събеседниците си, дали считат, че учебниците провокират анти-европейски нагласи сред учениците или внушават, че Европа е провален проект?
Валентина споделя: “Определено не вярвам в това, тук хората са с точно противното мнение. Макар че след последните събития и позицията си относно 1 октомври, смятам, че ЕС загуби много почитатели, както и част от желанието за членство на Република Каталуния в ЕС.”
Румен допълва: “Според мен каталунците са абсолютно проевропейски настроени и това е една от причините сепаратизмът да не е толкова разпространен, защото ЕС е по-важен за каталунците от независимостта им. Твърдя това въз основа на многобройни разговори и с про-, и с антинастроените към отцепването. Историята на Испания, както и на България, има много гледни точки, така че дали е преподавана манипулативно е страшно трудно да се установи обективно, вероятно има такива елементи, но не вярвам спорните моменти да са много.”
Роса обобщава: “Убедена съм, че образованието в Каталуния защитава демокрацията, свободата и мира, и разказва за Европа на многообразието и плурализма и в никакъв случай за Европа като за провален проект.” Питам я накрая и за гледната ѝ точка за политическата криза между централното и автономното правителство и тя споделя: “Каталунското правителство действа съгласно демократичния мандат, който получи след изборите през септември 2015 г. и действията им бяха отговорни. Има неща, които бих искала да бяха направили по-добре, но и Мадрид не остана по-назад, особено провокирайки ситуации, които можеха да бъдат избегнати. Аз твърдо вярвам, че независимостта ще се случи. Ще отнеме повече или по-малко усилия, но ще се случи. Когато едно гражданско общество с постоянство и мир направи крачка към по-добро общество за себе си и децата си… няма връщане назад. Достатъчно е да си спомним, че всичките многолюдни демонстрации бяха като празненства, по лицата на хората грееха усмивки, завладяни от мечтата за по-добър проект. Това не може да бъде спряно. Гневът и конфронтацията, или отрицанието в името на самото отрицание, не могат да спрат това.”
В този материал са използвани извадки от есета, включени в книгата “Какво става с Каталуния?”, преведена и издадена на български език от Сдружение “Диалози” като свободна за разпространение електронна книга. Снимките са от учебници за IV, V и VI клас и опровергават твърденията, че всички учебни материали са само на каталунски език, избягват споменаването на Испания и Европейския съюз, а децата не учат кой е Сервантес и Дон Кихот. Направени са от Валентина Григорова. Благодаря за съдействието на Юлияна Кръстева при подготвянето на този материал.
Тази статия бе написана за Дневник и публикувана там на 8 ноември 2017 под заглавие „Желанието за независимост на Каталуня не идва от учебниците“. Не съм търсил и получавал хонорар за този текст – написах го по свое желание и настоях да бъде публикуван заради предходен материал, към който имах сериозни критики и за който става дума в текста. Препубликувам го и тук, в блога си, за да остане за личния ми архив.
Не съм изоставил темата Каталуния – макар да съзнавам напълно, че това се превръща в моя лична и самотна битка. Напоследък не смогвам с всичко, което е важно за мен, а и ситуацията в собствената ни превзета държава не е по-различна от театралната демокрация в кралство Испания. Тъжно е, че Каталуния извади на яве симптом на много тежка болест, свързана с егоизма на европейците и липсата на обикновена човешка солидарност в Европа. Симптом, който не предвещава нищо добро за бъдещето ни. Но ще го осъзнаем късно.
Важността да се направи езика един и същ винаги е била признавана за нещо велико и е признак на доминация или превъзходство на принцове или нации… но тъй като всяка нация чувства собствения си език като дар от Природата, това прави трудно завладяването им и ще ви е нужно известно време да го направите, особено когато хората, както в случая – каталунците, са упорити, арогантни и влюбени в своята страна, затова е добре внимателно да се помисли и да се дават скрити инструкции и съвети, така че целта да се постигне незабелязано…
Хосе Родриго Вилялпандо, старши служител в Съвета на Кастилия, 1716 г., извадка от “Тайни инструкции” до магистратите в Каталуния
В наш интерес е да “испанизираме” каталунските деца.
Хосе Игнасио Верт, испански министър на образованието, 2012 г.
“Нивото на университетското образование е високо, студентите имат много богат избор от дисциплини и относително малък брой „задължителни“ лекции. В университета всички предмети се преподават най-малко на два езика (каталунски и испански). Много често има възможност и за обучение на английски.” Това ми казва мой приятел, който от три години живее в едно от по-малките градчета на север от Барселона и завършва докторантура.
Поводът да говорим по темата е статия, публикувана в Дневник и озаглавена Испания отсъства, монархията е зло, Европа се проваля – какво пише в каталунските учебници, която преразказва прочита на испанската медия ABC на едно изследване. Самото изследване е валидно и съдържа анализ на учебен материал за V и VI клас, заедно със забележки и препоръки, но преразказът на ABC е далеч от обективен и преднамерено внушава напълно недостоверни неща: като например, че в каталунските училища всичко се учи на каталунски, учебният материал е манипулиран и манипулативен, испаноговорящите деца са в неравностойно положение, учениците дори не знаели кой е Дон Кихот и биват облъчвани с антиевропейски послания.
За жалост, това е част от активна кампания, обслужваща пряко тезата на испанското централно правителство на Народната партия, че каталунското движение за независимост е подхранвано от манипулирани исторически факти и систематично промиване на съзнания. В конфликта между Барселона и Мадрид медиите играят активна, но невинаги достатъчно чистоплътна роля.
Затова помолих няколко събеседници, които имат преки наблюдения върху образователната система в Каталуния, да споделят своята гледна точка.
Валентина е българка, която живее от 17 години в каталунската област Тарагона. Майка е на две деца (на 9 и 11 години), нейни са снимките от учебниците за IV, V и VI клас от обикновено държавно училище, които илюстрират тази статия.
Румен също е българин и е баща на две деца. Живее от две години и половина с цялото си семейство много близо до Барселона, а по-големият му син учи в частно международно училище.
Роса е каталунка, живее вече 48 години в град от област Барселона. Има две деца (на 8 и на 11 години) и е преподавател в системата на държавното образование на Каталуния. Комуникацията ни с Роса е на испански.
Споделям им, че в България се прокрадва тезата, че каталунският стремеж за независимост е подклаждан от манипулации в образователната система и моля за техните гледни точки.
“Мисля, че статията е едностранчива, защото учениците имат по два комплекта учебници (испански и каталунски), като първите се спускат от Мадрид. Освен това има сериозен контрол от централното правителство.”, споделя Румен.
“Тази информация е невярна и не отговаря на действителността.”, още по-категорична е Роса. “Учебниците не индоктринират. Те обясняват съвременната история с исторически факти, съвсем коректно и без да вменяват какви намерения е имала една или друга страна. Учителите също не индоктринират, най-малкото нямат никакво време за това. Всяко ново правителство променя образователния закон и учебните програми дори без съобразяване с децата със специални нужди, например. Има паралелки с повече от 30 деца (в тази на дъщеря ми са 31), но ако дете зададе въпрос на испански, ще му бъде отговорено на испански, дори и в час по каталунски.”
А Валентина допълва, че проблемът е всъщност обратен и “борбата винаги е била и е на основата да се запази езикът, културата и традициите. „Залитането“ на Каталуния към независимост няма нищо общо с това, което се изучава в училищата, по-скоро с политиката на испанските власти, която подтиска всякаква възможност за развитие и подобряване условията на живот в Каталуния.”
Питам Румен как се обучават децата в частно международно училище и на колко езика се преподава и общува. “Там основно ги обучават на английски език, но все пак имат минимум 30% от учебния материал на испански и каталунски, предимно по география, история и социални науки. Предметите се преподават на три основни езика и един чужд, приблизително 60% английски, 20% испански, 10% каталунски и 10% френски (чуждия език). Единственият сериозен недостатък, който съм забелязал в образователната им система, е много ниското ниво на владеене (когато въобще го има) на английски сред местното население и най-вече младежите под 25 г. Живял съм в три европейски държави освен в България и мога да кажа с чисто сърце, че Испания е най-назад във владеенето на чужди езици.”
В едно свое есе Пере Маянс Балселс, който е професор по каталунски език и литература, разказва как още през 1979 г. Каталунският автономен статут (регионалния основен закон) установява каталунския език като официален (поделяйки, разбира се, този статус с испанския), а Законът за езиково нормиране в Каталуния от 1983 г, има за цел да изравни положението на каталунския език, който дотогава на всички нива е бил поставен неоспоримо по-ниско от испанския. С този закон каталунският език става главният език, използван в основното и средното образование. “Той също признаваше правото на децата да получават начално образование на своя първи език, дали каталунски или испански, и изискваше двата езика да бъдат преподавани на всички нива на неуниверситетското образование, държеше заедно учениците с различен езиков произход и осигуряваше на всички деца в Каталуния на училищна възраст, независимо какъв език са ползвали при започване на училище, в края на своето основно обучение да могат да си служат и с каталунския, и с испанския език правилно и свободно.”, пише проф. Балселс. “Общо диагностично оценяване от 2010 г., публикувано от испанското министерство на образованието, показва, че езиковата компетенция на каталунските ученици по испански език е еднаква със средната за испанската държава, и в действителност е по-голяма от тази в някои едноезични общности като Канарските острови, Екстремадура или Андалусия, или тази в двуезични общности, където испанският има много по-силно присъствие като език на преподаване (като в Балеарските острови, Галисия и област Валенсия).”
Между другото, каталунският езиков модел е признат за добра практика в областта на преподаването в многоезични общности и отличен от групата на високо равнище по въпросите на многоезичието, създадена от Европейската комисия през 2005 г.
Реалната борба е да се опази каталунския език, който е застрашен. “Реалният проблем тук в Каталуния не е дали се изучава испански език и дали децата могат да общуват на него свободно.”, уточнява Валентина. “Реалният проблем е, че децата се връщат вкъщи с речник, пълен с испанизирани думи (espanyolismes) по вина на самите учители, които нямат добър каталунски, и, разбира се, на съученици, с които не могат да говорят на каталунски, защото те не искат да го говорят. Свидетели сме на това всеки ден от много време насам. Реалната борба е да се опази каталунския език, който е застрашен. Истината е, че малко хора говорят правилен каталунски, използват се много испански фрази и се забравят традиционните каталунски фрази и изрази. Нужно е учителите да имат по високо ниво за да се даде по-добра основа за бъдещото образование. Определено е много странно, когато получиш съобщение от преподавател в бележника на детето ти с правописни грешки, видни дори за мен, която съм българка с ниво С по каталунски.”
Съгласно данни на официалната анкета за употреба на езиците EULP (Enquesta d’usos lingüístics de la població), проведена през 2013 г., 94,3% от жителите над 14-годишна възраст заявяват, че разбират каталунски, 80,4% го говорят, 82,4% могат да четат и 60,4% могат да пишат на него. Но трябва да се има предвид, че Каталуния е общност на имигранти и едва малко повече от половината от анкетираните са родени в Каталуния, а около четвърт от тях на друго място в Испания – главно в региони, в които се говори кастилски, 18% са от чужбина.
Ф. Чавие Вила (доцент в Барселонския университет по каталунска филология и доктор по лингвистика от Vrije Universiteit в Брюксел) пише: “През 2010 г. Конституционният съд постанови, че кастилският език ще е задължителен в Каталуния, но не и самият каталунски. И на базата на това решение, две години по-късно испанският министър на образованието подпали Каталуния, когато се похвали, че неговата цел е да се “испанизират” (españolizar) каталунските ученици. Той представи законопроект, който не само посегна на каталунската юрисдикция, но позволи каталунските деца да бъдат обучавани едноезично на кастилски – позиция, която беше възприета като колониална и напълно неприемлива от повечето каталунци.”
Жузеп Мария Ганиет, компютърен инженер от Барселонския автономен университет, специализирал Изкуствен интелект, твърди, че “испанската държава, съзнавайки силното чувство за идентичност, което езикът поражда у хората, винаги е гледала на некастилските езици и култури като на историческа аномалия, вместо да ги приеме като общо културно наследство, което следва да бъде запазено и защитено.”
В заключение питам събеседниците си, дали считат, че учебниците провокират анти-европейски нагласи сред учениците или внушават, че Европа е провален проект?
Валентина споделя: “Определено не вярвам в това, тук хората са с точно противното мнение. Макар че след последните събития и позицията си относно 1 октомври, смятам, че ЕС загуби много почитатели, както и част от желанието за членство на Република Каталуния в ЕС.”
Румен допълва: “Според мен каталунците са абсолютно проевропейски настроени и това е една от причините сепаратизмът да не е толкова разпространен, защото ЕС е по-важен за каталунците от независимостта им. Твърдя това въз основа на многобройни разговори и с про-, и с антинастроените към отцепването. Историята на Испания, както и на България, има много гледни точки, така че дали е преподавана манипулативно е страшно трудно да се установи обективно, вероятно има такива елементи, но не вярвам спорните моменти да са много.”
Роса обобщава: “Убедена съм, че образованието в Каталуния защитава демокрацията, свободата и мира, и разказва за Европа на многообразието и плурализма и в никакъв случай за Европа като за провален проект.” Питам я накрая и за гледната ѝ точка за политическата криза между централното и автономното правителство и тя споделя: “Каталунското правителство действа съгласно демократичния мандат, който получи след изборите през септември 2015 г. и действията им бяха отговорни. Има неща, които бих искала да бяха направили по-добре, но и Мадрид не остана по-назад, особено провокирайки ситуации, които можеха да бъдат избегнати. Аз твърдо вярвам, че независимостта ще се случи. Ще отнеме повече или по-малко усилия, но ще се случи. Когато едно гражданско общество с постоянство и мир направи крачка към по-добро общество за себе си и децата си… няма връщане назад. Достатъчно е да си спомним, че всичките многолюдни демонстрации бяха като празненства, по лицата на хората грееха усмивки, завладяни от мечтата за по-добър проект. Това не може да бъде спряно. Гневът и конфронтацията, или отрицанието в името на самото отрицание, не могат да спрат това.”
В този материал са използвани извадки от есета, включени в книгата “Какво става с Каталуния?”, преведена и издадена на български език от Сдружение “Диалози” като свободна за разпространение електронна книга. Снимките са от учебници за IV, V и VI клас и опровергават твърденията, че всички учебни материали са само на каталунски език, избягват споменаването на Испания и Европейския съюз, а децата не учат кой е Сервантес и Дон Кихот. Направени са от Валентина Григорова. Благодаря за съдействието на Юлияна Кръстева при подготвянето на този материал.
Glenn Gore here, Chief Architect for AWS. I’m in Las Vegas this week — with 43K others — for re:Invent 2017. We have a lot of exciting announcements this week. I’m going to post to the AWS Architecture blog each day with my take on what’s interesting about some of the announcements from a cloud architectural perspective.
Why not start at the beginning? At the Midnight Madness launch on Sunday night, we announced Amazon Sumerian, our platform for VR, AR, and mixed reality. The hype around VR/AR has existed for many years, though for me, it is a perfect example of how a working end-to-end solution often requires innovation from multiple sources. For AR/VR to be successful, we need many components to come together in a coherent manner to provide a great experience.
First, we need lightweight, high-definition goggles with motion tracking that are comfortable to wear. Second, we need to track movement of our body and hands in a 3-D space so that we can interact with virtual objects in the virtual world. Third, we need to build the virtual world itself and populate it with assets and define how the interactions will work and connect with various other systems.
There has been rapid development of the physical devices for AR/VR, ranging from iOS devices to Oculus Rift and HTC Vive, which provide excellent capabilities for the first and second components defined above. With the launch of Amazon Sumerian we are solving for the third area, which will help developers easily build their own virtual worlds and start experimenting and innovating with how to apply AR/VR in new ways.
Already, within 48 hours of Amazon Sumerian being announced, I have had multiple discussions with customers and partners around some cool use cases where VR can help in training simulations, remote-operator controls, or with new ideas around interacting with complex visual data sets, which starts bringing concepts straight out of sci-fi movies into the real (virtual) world. I am really excited to see how Sumerian will unlock the creative potential of developers and where this will lead.
Amazon MQ I am a huge fan of distributed architectures where asynchronous messaging is the backbone of connecting the discrete components together. Amazon Simple Queue Service (Amazon SQS) is one of my favorite services due to its simplicity, scalability, performance, and the incredible flexibility of how you can use Amazon SQS in so many different ways to solve complex queuing scenarios.
While Amazon SQS is easy to use when building cloud-native applications on AWS, many of our customers running existing applications on-premises required support for different messaging protocols such as: Java Message Service (JMS), .Net Messaging Service (NMS), Advanced Message Queuing Protocol (AMQP), MQ Telemetry Transport(MQTT), Simple (or Streaming) Text Orientated Messaging Protocol (STOMP), and WebSockets. One of the most popular applications for on-premise message brokers is Apache ActiveMQ. With the release of Amazon MQ, you can now run Apache ActiveMQ on AWS as a managed service similar to what we did with Amazon ElastiCache back in 2012. For me, there are two compelling, major benefits that Amazon MQ provides:
Integrate existing applications with cloud-native applications without having to change a line of application code if using one of the supported messaging protocols. This removes one of the biggest blockers for integration between the old and the new.
Remove the complexity of configuring Multi-AZ resilient message broker services as Amazon MQ provides out-of-the-box redundancy by always storing messages redundantly across Availability Zones. Protection is provided against failure of a broker through to complete failure of an Availability Zone.
I believe that Amazon MQ is a major component in the tools required to help you migrate your existing applications to AWS. Having set up cross-data center Apache ActiveMQ clusters in the past myself and then testing to ensure they work as expected during critical failure scenarios, technical staff working on migrations to AWS benefit from the ease of deploying a fully redundant, managed Apache ActiveMQ cluster within minutes.
Who would have thought I would have been so excited to revisit Apache ActiveMQ in 2017 after using SQS for many, many years? Choice is a wonderful thing.
Amazon GuardDuty Maintaining application and information security in the modern world is increasingly complex and is constantly evolving and changing as new threats emerge. This is due to the scale, variety, and distribution of services required in a competitive online world.
At Amazon, security is our number one priority. Thus, we are always looking at how we can increase security detection and protection while simplifying the implementation of advanced security practices for our customers. As a result, we released Amazon GuardDuty, which provides intelligent threat detection by using a combination of multiple information sources, transactional telemetry, and the application of machine learning models developed by AWS. One of the biggest benefits of Amazon GuardDuty that I appreciate is that enabling this service requires zero software, agents, sensors, or network choke points. which can all impact performance or reliability of the service you are trying to protect. Amazon GuardDuty works by monitoring your VPC flow logs, AWS CloudTrail events, DNS logs, as well as combing other sources of security threats that AWS is aggregating from our own internal and external sources.
The use of machine learning in Amazon GuardDuty allows it to identify changes in behavior, which could be suspicious and require additional investigation. Amazon GuardDuty works across all of your AWS accounts allowing for an aggregated analysis and ensuring centralized management of detected threats across accounts. This is important for our larger customers who can be running many hundreds of AWS accounts across their organization, as providing a single common threat detection of their organizational use of AWS is critical to ensuring they are protecting themselves.
Detection, though, is only the beginning of what Amazon GuardDuty enables. When a threat is identified in Amazon GuardDuty, you can configure remediation scripts or trigger Lambda functions where you have custom responses that enable you to start building automated responses to a variety of different common threats. Speed of response is required when a security incident may be taking place. For example, Amazon GuardDuty detects that an Amazon Elastic Compute Cloud (Amazon EC2) instance might be compromised due to traffic from a known set of malicious IP addresses. Upon detection of a compromised EC2 instance, we could apply an access control entry restricting outbound traffic for that instance, which stops loss of data until a security engineer can assess what has occurred.
Whether you are a customer running a single service in a single account, or a global customer with hundreds of accounts with thousands of applications, or a startup with hundreds of micro-services with hourly release cycle in a devops world, I recommend enabling Amazon GuardDuty. We have a 30-day free trial available for all new customers of this service. As it is a monitor of events, there is no change required to your architecture within AWS.
Stay tuned for tomorrow’s post on AWS Media Services and Amazon Neptune.
If you have had an opportunity to read any of my blog posts or attended any session I’ve conducted at various conferences, you are probably aware that I am definitively a geek girl. I am absolutely enamored with all of the latest advancements that have been made in technology areas like cloud, artificial intelligence, internet of things and the maker space, as well as, with virtual reality and augmented reality. In my opinion, it is a wonderful time to be a geek. All the things that we dreamed about building while we sweated through our algorithms and discrete mathematics classes or the technology we marveled at when watching Star Wars and Star Trek are now coming to fruition. So hopefully this means it will only be a matter of time before I can hyperdrive to other galaxies in space, but until then I can at least build the 3D virtual reality and augmented reality characters and images like those featured in some of my favorite shows.
Amazon Sumerian provides tools and resources that allows anyone to create and run augmented reality (AR), virtual reality (VR), and 3D applications with ease. With Sumerian, you can build multi-platform experiences that run on hardware like the Oculus, HTC Vive, and iOS devices using WebVR compatible browsers and with support for ARCore on Android devices coming soon.
This exciting new service, currently in preview, delivers features to allow you to design highly immersive and interactive 3D experiences from your browser. Some of these features are:
Editor: A web-based editor for constructing 3D scenes, importing assets, scripting interactions and special effects, with cross-platform publishing.
Object Library: a library of pre-built objects and templates.
Asset Import: Upload 3D assets to use in your scene. Sumerian supports importing FBX, OBJ, and coming soon Unity projects.
Scripting Library: provides a JavaScript scripting library via its 3D engine for advanced scripting capabilities.
Hosts: animated, lifelike 3D characters that can be customized for gender, voice, and language.
AWS Services Integration: baked in integration with Amazon Polly and Amazon Lex to add speech and natural language to into Sumerian hosts. Additionally, the scripting library can be used with AWS Lambda allowing use of the full range of AWS services.
Since Amazon Sumerian doesn’t require you to have 3D graphics or programming experience to build rich, interactive VR and AR scenes, let’s take a quick run to the Sumerian Dashboard and check it out.
From the Sumerian Dashboard, I can easily create a new scene with a push of a button.
A default view of the new scene opens and is displayed in the Sumerian Editor. With the Tara Blog Scene opened in the editor, I can easily import assets into my scene.
I’ll click the Import Asset button and pick an asset, View Room, to import into the scene. With the desired asset selected, I’ll click the Add button to import it.
Excellent, my asset was successfully imported into the Sumerian Editor and is shown in the Asset panel. Now, I have the option to add the View Room object into my scene by selecting it in the Asset panel and then dragging it onto the editor’s canvas.
I’ll repeat the import asset process and this time I will add the Mannequin asset to the scene.
Additionally, with Sumerian, I can add scripting to Entity assets to make my scene even more exciting by adding a ScriptComponent to an entity and creating a script. I can use the provided built-in scripts or create my own custom scripts. If I create a new custom script, I will get a blank script with some base JavaScript code that looks similar to the code below.
'use strict';
/* global sumerian */
//This is Me – trying out the custom scripts - Tara
var setup = function (args, ctx) {
// Called when play mode starts.
};
var fixedUpdate = function (args, ctx) {
// Called on every physics update, after setup().
};
var update = function (args, ctx) {
// Called on every render frame, after setup().
};
var lateUpdate = function (args, ctx) {
// Called after all script "update" methods in the scene has been called.
};
var cleanup = function (args, ctx) {
// Called when play mode stops.
};
var parameters = [];
Very cool, I just created a 3D scene using Amazon Sumerian in a matter of minutes and I have only scratched the surface.
Summary
The Amazon Sumerian service enables you to create, build, and run virtual reality (VR), augmented reality (AR), and 3D applications with ease. You don’t need any 3D graphics or specialized programming knowledge to get started building scenes and immersive experiences. You can import FBX, OBJ, and Unity projects in Sumerian, as well as upload your own 3D assets for use in your scene. In addition, you can create digital characters to narrate your scene and with these digital assets, you have choices for the character’s appearance, speech and behavior.
You can learn more about Amazon Sumerian and sign up for the preview to get started with the new service on the product page. I can’t wait to see what rich experiences you all will build.
This post courtesy of Aaron Friedman, Healthcare and Life Sciences Partner Solutions Architect, AWS and Angel Pizarro, Genomics and Life Sciences Senior Solutions Architect, AWS
Precision medicine is tailored to individuals based on quantitative signatures, including genomics, lifestyle, and environment. It is often considered to be the driving force behind the next wave of human health. Through new initiatives and technologies such as population-scale genomics sequencing and IoT-backed wearables, researchers and clinicians in both commercial and public sectors are gaining new, previously inaccessible insights.
Many of these precision medicine initiatives are already happening on AWS. A few of these include:
PrecisionFDA – This initiative is led by the US Food and Drug Administration. The goal is to define the next-generation standard of care for genomics in precision medicine.
Deloitte ConvergeHEALTH – Gives healthcare and life sciences organizations the ability to analyze their disparate datasets on a singular real world evidence platform.
Central to many of these initiatives is genomics, which gives healthcare organizations the ability to establish a baseline for longitudinal studies. Due to its wide applicability in precision medicine initiatives—from rare disease diagnosis to improving outcomes of clinical trials—genomics data is growing at a larger rate than Moore’s law across the globe. Many expect these datasets to grow to be in the range of tens of exabytes by 2025.
Genomics data is also regularly re-analyzed by the community as researchers develop new computational methods or compare older data with newer genome references. These trends are driving innovations in data analysis methods and algorithms to address the massive increase of computational requirements.
Edico Genome, an AWS Partner Network (APN) Partner, has developed a novel solution that accelerates genomics analysis using field-programmable gate arrays, or FPGAs. Historically, Edico Genome deployed their FPGA appliances on-premises. When AWS announced the Amazon EC2 F1 PGA-based instance family in December 2016, Edico Genome adopted a cloud-first strategy, became a F1 launch partner, and was one of the first partners to deploy FPGA-enabled applications on AWS.
On October 19, 2017, Edico Genome partnered with the Children’s Hospital of Philadelphia (CHOP) to demonstrate their FPGA-accelerated genomic pipeline software, called DRAGEN. It can significantly reduce time-to-insight for patient genomes, and analyzed 1,000 genomes from the Center for Applied Genomics Biobank in the shortest time possible. This set a Guinness World Record for the fastest analysis of 1000 whole human genomes, and they did this using 1000 EC2 f1.2xlarge instances in a single AWS region. Not only were they able to analyze genomes at high throughput, they did so averaging approximately $3 per whole human genome of AWS compute for the analysis.
The version of DRAGEN that Edico Genome used for this analysis was also the same one used in the precisionFDA Hidden Treasures – Warm Up challenge, where they were one of the top performers in every assessment.
In the remainder of this post, we walk through the architecture used by Edico Genome, combining EC2 F1 instances and AWS Batch to achieve this milestone.
EC2 F1 instances and Edico’s DRAGEN
EC2 F1 instances provide access to programmable hardware-acceleration using FPGAs at a cloud scale. AWS customers use F1 instances for a wide variety of applications, including big data, financial analytics and risk analysis, image and video processing, engineering simulations, AR/VR, and accelerated genomics. Edico Genome’s FPGA-backed DRAGEN Bio-IT Platform is now integrated with EC2 F1 instances. You can access the accuracy, speed, flexibility, and low compute cost of DRAGEN through a number of third-party platforms, AWS Marketplace, and Edico Genome’s own platform. The DRAGEN platform offers a scalable, accelerated, and cost-efficient secondary analysis solution for a wide variety of genomics applications. Edico Genome also provides a highly optimized mechanism for the efficient storage of genomic data.
Scaling DRAGEN on AWS
Edico Genome used 1,000 EC2 F1 instances to help their customer, the Children’s Hospital of Philadelphia (CHOP), to process and analyze all 1,000 whole human genomes in parallel. They used AWS Batch to provision compute resources and orchestrate DRAGEN compute jobs across the 1,000 EC2 F1 instances. This solution successfully addressed the challenge of creating a scalable genomic processing pipeline that can easily scale to thousands of engines running in parallel.
Architecture
A simplified view of the architecture used for the analysis is shown in the following diagram:
DRAGEN’s portal uses Elastic Load Balancing and Auto Scaling groups to scale out EC2 instances that submitted jobs to AWS Batch.
Job metadata is stored in their Workflow Management (WFM) database, built on top of Amazon Aurora.
The DRAGEN Workflow Manager API submits jobs to AWS Batch.
These jobs are executed on the AWS Batch managed compute environment that was responsible for launching the EC2 F1 instances.
These jobs run as Docker containers that have the requisite DRAGEN binaries for whole genome analysis.
As each job runs, it retrieves and stores genomics data that is staged in Amazon S3.
The steps listed previously can also be bucketed into the following higher-level layers:
Workflow: Edico Genome used their Workflow Management API to orchestrate the submission of AWS Batch jobs. Metadata for the jobs (such as the S3 locations of the genomes, etc.) resides in the Workflow Management Database backed by Amazon Aurora.
Batch execution: AWS Batch launches EC2 F1 instances and coordinates the execution of DRAGEN jobs on these compute resources. AWS Batch enabled Edico to quickly and easily scale up to the full number of instances they needed as jobs were submitted. They also scaled back down as each job was completed, to optimize for both cost and performance.
Compute/job: Edico Genome stored their binaries in a Docker container that AWS Batch deployed onto each of the F1 instances, giving each instance the ability to run DRAGEN without the need to pre-install the core executables. The AWS based DRAGEN solution streams all genomics data from S3 for local computation and then writes the results to a destination bucket. They used an AWS Batch job role that specified the IAM permissions. The role ensured that DRAGEN only had access to the buckets or S3 key space it needed for the analysis. Jobs didn’t need to embed AWS credentials.
In the following sections, we dive deeper into several tasks that enabled Edico Genome’s scalable FPGA genome analysis on AWS:
Prepare your Amazon FPGA Image for AWS Batch
Create a Dockerfile and build your Docker image
Set up your AWS Batch FPGA compute environment
Prerequisites
In brief, you need a modern Linux distribution (3.10+), Amazon ECS Container Agent, awslogs driver, and Docker configured on your image. There are additional recommendations in the Compute Resource AMI specification.
Preparing your Amazon FPGA Image for AWS Batch
You can use any Amazon Machine Image (AMI) or Amazon FPGA Image (AFI) with AWS Batch, provided that it meets the Compute Resource AMI specification. This gives you the ability to customize any workload by increasing the size of root or data volumes, adding instance stores, and connecting with the FPGA (F) and GPU (G and P) instance families.
Next, install the AWS CLI:
pip install awscli
Add any additional software required to interact with the FPGAs on the F1 instances.
As a starting point, AWS publishes an FPGA Developer AMI in the AWS Marketplace. It is based on a CentOS Linux image and includes pre-integrated FPGA development tools. It also includes the runtime tools required to develop and use custom FPGAs for hardware acceleration applications.
For more information about how to set up custom AMIs for your AWS Batch managed compute environments, see Creating a Compute Resource AMI.
Building your Dockerfile
There are two common methods for connecting to AWS Batch to run FPGA-enabled algorithms. The first method, which is the route Edico Genome took, involves storing your binaries in the Docker container itself and running that on top of an F1 instance with Docker installed. The following code example is what a Dockerfile to build your container might look like for this scenario.
# DRAGEN_EXEC Docker image generator –
# Run this Dockerfile from a local directory that contains the latest release of
# - Dragen RPM and Linux DMA Driver available from Edico
# - Edico's Dragen WFMS Wrapper files
FROM centos:centos7
RUN rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# Install Basic packages needed for Dragen
RUN yum -y install \
perl \
sos \
coreutils \
gdb \
time \
systemd-libs \
bzip2-libs \
R \
ca-certificates \
ipmitool \
smartmontools \
rsync
# Install the Dragen RPM
RUN mkdir -m777 -p /var/log/dragen /var/run/dragen
ADD . /root
RUN rpm -Uvh /root/edico_driver*.rpm || true
RUN rpm -Uvh /root/dragen-aws*.rpm || true
# Auto generate the Dragen license
RUN /opt/edico/bin/dragen_lic -i auto
#########################################################
# Now install the Edico WFMS "Wrapper" functions
# Add development tools needed for some util
RUN yum groupinstall -y "Development Tools"
# Install necessary standard packages
RUN yum -y install \
dstat \
git \
python-devel \
python-pip \
time \
tree && \
pip install – upgrade pip && \
easy_install requests && \
pip install psutil && \
pip install python-dateutil && \
pip install constants && \
easy_install boto3
# Setup Python path used by the wrapper
RUN mkdir -p /opt/workflow/python/bin
RUN ln -s /usr/bin/python /opt/workflow/python/bin/python2.7
RUN ln -s /usr/bin/python /opt/workflow/python/bin/python
# Install d_haul and dragen_job_execute wrapper functions and associated packages
RUN mkdir -p /root/wfms/trunk/scheduler/scheduler
COPY scheduler/d_haul /root/wfms/trunk/scheduler/
COPY scheduler/dragen_job_execute /root/wfms/trunk/scheduler/
COPY scheduler/scheduler/aws_utils.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/constants.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/job_utils.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/logger.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/scheduler_utils.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/webapi.py /root/wfms/trunk/scheduler/scheduler/
COPY scheduler/scheduler/wfms_exception.py /root/wfms/trunk/scheduler/scheduler/
RUN touch /root/wfms/trunk/scheduler/scheduler/__init__.py
# Landing directory should be where DJX is located
WORKDIR "/root/wfms/trunk/scheduler/"
# Debug print of container's directories
RUN tree /root/wfms/trunk/scheduler
# Default behaviour. Over-ride with – entrypoint on docker run cmd line
ENTRYPOINT ["/root/wfms/trunk/scheduler/dragen_job_execute"]
CMD []
Note: Edico Genome’s custom Python wrapper functions for its Workflow Management System (WFMS) in the latter part of this Dockerfile should be replaced with functions that are specific to your workflow.
The second method is to install binaries and then use Docker as a lightweight connector between AWS Batch and the AFI. For example, this might be a route you would choose to use if you were provisioning DRAGEN from the AWS Marketplace.
In this case, the Dockerfile would not contain the installation of the binaries to run DRAGEN, but would contain any other packages necessary for job completion. When you run your Docker container, you enable Docker to access the underlying file system.
Connecting to AWS Batch
AWS Batch provisions compute resources and runs your jobs, choosing the right instance types based on your job requirements and scaling down resources as work is completed. AWS Batch users submit a job, based on a template or “job definition” to an AWS Batch job queue.
Job queues are mapped to one or more compute environments that describe the quantity and types of resources that AWS Batch can provision. In this case, Edico created a managed compute environment that was able to launch 1,000 EC2 F1 instances across multiple Availability Zones in us-east-1. As jobs are submitted to a job queue, the service launches the required quantity and types of instances that are needed. As instances become available, AWS Batch then runs each job within appropriately sized Docker containers.
The Edico Genome workflow manager API submits jobs to an AWS Batch job queue. This job queue maps to an AWS Batch managed compute environment containing On-Demand F1 instances. In this section, you can set this up yourself.
To create the compute environment that DRAGEN can use:
An f1.2xlarge EC2 instance contains one FPGA, eight vCPUs, and 122-GiB RAM. As DRAGEN requires an entire FPGA to run, Edico Genome needed to ensure that only one analysis per time executed on an instance. By using the f1.2xlarge vCPUs and memory as a proxy in their AWS Batch job definition, Edico Genome could ensure that only one job runs on an instance at a time. Here’s what that looks like in the AWS CLI:
You can query the status of your DRAGEN job with the following command:
aws batch describe-jobs – jobs <the job ID from the above command>
The logs for your job are written to the /aws/batch/job CloudWatch log group.
Conclusion
In this post, we demonstrated how to set up an environment with AWS Batch that can run DRAGEN on EC2 F1 instances at scale. If you followed the walkthrough, you’ve replicated much of the architecture Edico Genome used to set the Guinness World Record.
There are several ways in which you can harness the computational power of DRAGEN to analyze genomes at scale. First, DRAGEN is available through several different genomics platforms, such as the DNAnexus Platform. DRAGEN is also available on the AWS Marketplace. You can apply the architecture presented in this post to build a scalable solution that is both performant and cost-optimized.
For more information about how AWS Batch can facilitate genomics processing at scale, be sure to check out our aws-batch-genomics GitHub repo on high-throughput genomics on AWS.
A data lake is an increasingly popular way to store and analyze data that addresses the challenges of dealing with massive volumes of heterogeneous data. A data lake allows organizations to store all their data—structured and unstructured—in one centralized repository. Because data can be stored as-is, there is no need to convert it to a predefined schema.
Many organizations understand the benefits of using AWS as their data lake. For example, Amazon S3 is a highly durable, cost-effective object start that supports Open Data Formats while decoupling storage from compute, and it works with all the AWS analytic services. Although Amazon S3 provides the foundation of a data lake, you can add other services to tailor the data lake to your business needs. For more information about building data lakes on AWS, see What is a Data Lake?
Because one of the main challenges of using a data lake is finding the data and understanding the schema and data format, Amazon recently introduced AWS Glue. AWS Glue significantly reduces the time and effort that it takes to derive business insights quickly from an Amazon S3 data lake by discovering the structure and form of your data. AWS Glue automatically crawls your Amazon S3 data, identifies data formats, and then suggests schemas for use with other AWS analytic services.
This post walks you through the process of using AWS Glue to crawl your data on Amazon S3 and build a metadata store that can be used with other AWS offerings.
AWS Glue features
AWS Glue is a fully managed data catalog and ETL (extract, transform, and load) service that simplifies and automates the difficult and time-consuming tasks of data discovery, conversion, and job scheduling. AWS Glue crawls your data sources and constructs a data catalog using pre-built classifiers for popular data formats and data types, including CSV, Apache Parquet, JSON, and more.
The AWS Glue Data Catalog is compatible with Apache Hive Metastore and supports popular tools such as Hive, Presto, Apache Spark, and Apache Pig. It also integrates directly with Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.
In addition, the AWS Glue Data Catalog features the following extensions for ease-of-use and data-management functionality:
AWS Glue is an essential component of an Amazon S3 data lake, providing the data catalog and transformation services for modern data analytics.
In the preceding figure, data is staged for different analytic use cases. Initially, the data is ingested in its raw format, which is the immutable copy of the data. The data is then transformed and enriched to make it more valuable for each use case. In this example, the raw CSV files are transformed into Apache Parquet for use by Amazon Athena to improve performance and reduce cost.
The data can also be enriched by blending it with other datasets to provide additional insights. An AWS Glue crawler creates a table for each stage of the data based on a job trigger or a predefined schedule. In this example, an AWS Lambda function is used to trigger the ETL process every time a new file is added to the Raw Data S3 bucket. The tables can be used by Amazon Athena, Amazon Redshift Spectrum, and Amazon EMR to query the data at any stage using standard SQL or Apache Hive. This configuration is a popular design pattern that delivers Agile Business Intelligence to derive business value from a variety of data quickly and easily.
Walkthrough
In this walkthrough, you define a database, configure a crawler to explore data in an Amazon S3 bucket, create a table, transform the CSV file into Parquet, create a table for the Parquet data, and query the data with Amazon Athena.
Discover the data
Sign in to the AWS Management Console and open the AWS Glue console. You can find AWS Glue in the Analytics section. AWS Glue is currently available in US East (N. Virginia), US East (Ohio), and US West (Oregon). Additional AWS Regions are added frequently.
The first step to discovering the data is to add a database. A database is a collection of tables.
In the console, choose Add database. In Database name, type nycitytaxi, and choose Create.
Choose Tables in the navigation pane. A table consists of the names of columns, data type definitions, and other metadata about a dataset.
Add a table to the database nycitytaxi.You can add a table manually or by using a crawler. A crawler is a program that connects to a data store and progresses through a prioritized list of classifiers to determine the schema for your data. AWS Glue provides classifiers for common file types like CSV, JSON, Avro, and others. You can also write your own classifier using a grok pattern.
To add a crawler, enter the data source: an Amazon S3 bucket named s3://aws-bigdata-blog/artifacts/glue-data-lake/data/. This S3 bucket contains the data file consisting of all the rides for the green taxis for the month of January 2017.
Choose Next.
For IAM role, choose the default role AWSGlueServiceRoleDefault in the drop-down list.
For Frequency, choose Run on demand. The crawler can be run on demand or set to run on a schedule.
For Database, choose nycitytaxi.It is important to understand how AWS Glue deals with schema changes so that you can select the appropriate method. In this example, the table is updated with any change. For more information about schema changes, see Cataloging Tables with a Crawler in the AWS Glue Developer Guide.
Review the steps, and choose Finish. The crawler is ready to run. Choose Run it now. When the crawler has finished, one table has been added.
Choose Tables in the left navigation pane, and then choose data. This screen describes the table, including schema, properties, and other valuable information.
Transform the data from CSV to Parquet format
Now you can configure and run a job to transform the data from CSV to Parquet. Parquet is a columnar format that is well suited for AWS analytics services like Amazon Athena and Amazon Redshift Spectrum.
Under ETL in the left navigation pane, choose Jobs, and then choose Add job.
For the Name, type nytaxi-csv-parquet.
For the IAM role, choose AWSGlueServiceRoleDefault.
For This job runs, choose A proposed script generated by AWS Glue.
Provide a unique Amazon S3 path to store the scripts.
Provide a unique Amazon S3 directory for a temporary directory.
Choose Next.
Choose data as the data source.
Choose Create tables in your data target.
Choose Parquet as the format.
Choose a new location (a new prefix location without any existing objects) to store the results.
Verify the schema mapping, and choose Finish.
View the job.This screen provides a complete view of the job and allows you to edit, save, and run the job.AWS Glue created this script. However, if required, you can create your own.
Choose Save, and then choose Run job.
Add the Parquet table and crawler
When the job has finished, add a new table for the Parquet data using a crawler.
For Crawler name, type nytaxiparquet.
Choose S3 as the Data store.
Include the Amazon S3 path chosen in the ETL
For the IAM role, choose AWSGlueServiceRoleDefault.
For Database, choose nycitytaxi.
For Frequency, choose Run on demand.
After the crawler has finished, there are two tables in the nycitytaxi database: a table for the raw CSV data and a table for the transformed Parquet data.
Analyze the data with Amazon Athena
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is capable of querying CSV data. However, the Parquet file format significantly reduces the time and cost of querying the data. For more information, see the blog post Analyzing Data in Amazon S3 using Amazon Athena.
To use AWS Glue with Amazon Athena, you must upgrade your Athena data catalog to the AWS Glue Data Catalog. For more information about upgrading your Athena data catalog, see this step-by-step guide.
Open the AWS Management Console for Athena. The Query Editor displays both tables in the nycitytaxi
You can query the data using standard SQL.
Choose the nytaxigreenparquet
Type Select * From "nycitytaxi"."data" limit 10;
Choose Run Query.
Conclusion
This post demonstrates how easy it is to build the foundation of a data lake using AWS Glue and Amazon S3. By using AWS Glue to crawl your data on Amazon S3 and build an Apache Hive-compatible metadata store, you can use the metadata across the AWS analytic services and popular Hadoop ecosystem tools. This combination of AWS services is powerful and easy to use, allowing you to get to business insights faster.
If you have questions or suggestions, please comment below.
Additional reading
See the following blog posts for more information:
Gordon Heinrich is a Solutions Architect working with global systems integrators. He works with our partners and customers to provide them architectural guidance for building data lakes and using AWS analytic services. In his spare time, he enjoys spending time with his family, skiing, hiking, and mountain biking in Colorado.
Halloween is almost upon us! In honour of one of the maker community’s favourite howlidays, here are some posts from enthusiastic makers on Twitter to get you inspired and prepared for the big event.
Using a @Raspberry_Pi with @pimoroni tilt hat to make a cool puppet for #Halloween https://t.co/pOeTFZ0r29
Made with a Pimoroni Pan-Tilt HAT, a Raspberry Pi, and some VR software on her phone, Lorraine Underwood‘s puppet is going to be a rather fitting doorman to interact with this year’s trick-or-treaters. Follow her project’s progress as she posts it on her blog.
Making my house super spooky for Halloween! https://t.co/w553l40BT0
Harnessing the one song guaranteed to earworm its way into my mind this October, Firr has upgraded his house to sing for all those daring enough to approach it this coming All Hallows’ Eve.
Firr used resources from Adafruit, along with three projectors, two Raspberry Pis, and some speakers, to create this semi-interactive display.
While the eyes can move on their own, a joystick can be added for direct control. Firr created a switch that goes between autonomous animation and direct control.
First #pumpkin of the season for Friday the 13th! @PaintYourDragon’s snake eyes bonnet for the #RaspberryPi to handle the eye animation. https://t.co/TSlUUxYP5Q
The Animated Snake Eyes Bonnet is definitely one of the freakiest products to come from the Adafruit lab, and it’s the perfect upgrade for any carved pumpkin this Halloween. Attach the bonnet to a Raspberry Pi 3, or the smaller Zero or Zero W, and thus add animated eyes to your scary orange masterpiece, as Justin Smith demonstrates in his video. The effect will terrify even the bravest of trick-or-treaters! Just make sure you don’t light a candle in there too…we’re not sure how fire-proof the tech is.
It is a truth universally acknowledged that a single man in possession of the zombie virus must be in want of braaaaaaains.
No matter whether you share your Halloween builds on Twitter, Facebook, G+, Instagram, or YouTube, we want to see them — make sure to tag us in your posts. We also have a comment section below this post, so go ahead and fill it with your ideas, links to completed projects, and general chat about the world of RasBOOrry Pi!
…sorry, that’s a hideous play on words. I apologise.
Robot-builder extraordinaire Clément Didier is ushering in the era of our cybernetic overlords. Future generations will remember him as the creator of robots constructed from cardboard and conductive paint which are so easy to replicate that a robot could do it. Welcome to the singularity.
This cool robot was made with the #PiCap, conductive paint and @Raspberry_Pi by @clementdidier. Full tutorial: https://t.co/AcQVTS4vr2 https://t.co/D04U5UGR0P
Simple interface
To assemble the robot, Clément made use of a Pi Cap board, a motor driver, and most importantly, a tube of Bare Conductive Electric Paint. He painted the control interface onto the cardboard surface of the robot, allowing a human, replicant, or superior robot to direct its movements simply by touching the paint.
Clever design
The Raspberry Pi 3, the motor control board, and the painted input buttons interface via the GPIO breakout pins on the Pi Cap. Crocodile clips connect the Pi Cap to the cardboard-and-paint control surface, while jumper wires connect it to the motor control board.
Sing with me: ‘The Raspberry Pi’s connected to the Pi Cap, and the Pi Cap’s connected to the inputs, and…’
Two battery packs provide power to the Raspberry Pi, and to the four independently driven motors. Software, written in Python, allows the robot to respond to inputs from the conductive paint. The motors drive wheels attached to a plastic chassis, moving and turning the robot at the touch of a square of black paint.
Artistic circuit
Clément used masking tape and a paintbrush to create the control buttons. For a human, this is obviously a fiddly process which relies on the blocking properties of the masking tape and a steady hand. For a robot, however, the process would be a simple, freehand one, resulting in neatly painted circuits on every single one of countless robotic minions. Cybernetic domination is at (metallic) hand.
One fiddly job for a human, one easy task for robotkind
The instructions and code for Clément’s build can be found here.
Low-tech solutions
Here at Pi Towers, we love seeing the high-tech Raspberry Pi integrated so successfully with low-tech components. In addition to conductive paint, we’ve seen cardboard laptops, toilet roll robots, fruit drum kits, chocolate box robots, and hamster-wheel-triggered cameras. Have you integrated low-tech elements into your projects (and potentially accelerated the robot apocalypse in the process)? Tell us about it in the comments!
Many customers use Amazon Kinesis to ingest, analyze, and persist their streaming data. One of the easiest ways to gain real-time insights into your streaming data is to use Kinesis Analytics. It enables you to query the data in your stream or build entire streaming applications using SQL. Customers use Kinesis Analytics for things like filtering, aggregation, and anomaly detection.
Kinesis Analytics now gives you the option to preprocess your data with AWS Lambda. This gives you a great deal of flexibility in defining what data gets analyzed by your Kinesis Analytics application. You can also define how that data is structured before it is queried by your SQL.
In this post, I discuss some common use cases for preprocessing, and walk you through an example to help highlight its applicability.
Common use cases
There are many reasons why you might choose to preprocess data before starting your analysis. Because you build your preprocessing logic with Lambda, your preprocessor can do anything supported by Lambda. However, there are some specific use cases that lend themselves well to preprocessing, such as data enrichment and data transformation.
Enrichment
In some scenarios, you may need to enhance your streaming data with additional information, before you perform your SQL analysis. Kinesis Analytics gives you the ability to use data from Amazon S3 in your Kinesis Analytics application, using the Reference Data feature. However, you cannot use other data sources from within your SQL query.
To add dynamic data to your streaming data, you can preprocess with a Lambda function, and retrieve the data from the data store of your choosing. For example, consider a scenario where you’re streaming some data about users of your application, including their IP address. You want to do some real-time analysis on the geographic locations of your customers. In this example, your preprocessing Lambda function uses your data source for geolocation information to retrieve the user’s city, state or province, and country, based on the IP address that was included in the streaming record. You then enrich the record with that information and now your SQL query can use those attributes in its aggregation.
Transformation
Because Kinesis Analytics uses SQL to analyze your data, the structure of your streaming records must be mapped to a schema. If your records are JSON or CSV, Kinesis Analytics automatically creates a schema. However, if your JSON records contain complex nested arrays, you may need to customize how the record structure is mapped to a flattened schema. Further, Kinesis Analytics is unable to automatically parse formats such as GZIP, protobuf, or Avro.
If your input records are unstructured text, Kinesis Analytics creates a schema, but it consists of a single column representing your entire record. To remedy these complexities, use Lambda to transform and convert your streaming data so that it more easily maps to a schema that can be queried by the SQL in your Kinesis Analytics application.
Assume that you’re streaming raw Apache access log data from a web fleet to a Kinesis stream, and you want to use Kinesis Analytics to detect anomalies in your HTTP response codes. In this example, you want to detect when your stream contains an unusually large number of 500 response codes. This may indicate that something has gone wrong somewhere in your application, and as a result, Apache is returning 500 responses to clients. This is typically not a good customer experience.
An example Apache access log record looks like this:
231.55.150.184 - - [28/Sep/2017:11:18:59 -0400] "PUT /explore HTTP/1.1" 200 2742 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.3) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/20.0.872.0 Safari/538.0.1"
Although its structure is well-defined, it is not JSON or CSV, so it doesn’t map to a defined schema in Kinesis Analytics. To use Kinesis Analytics with raw Apache log records, you can transform them to JSON or CSV with a preprocessing Lambda function. For example, you can convert it to a simple JSON documents that easily maps to a schema:
{
"client_ip": "231.55.150.184",
"request_time": "28/Sep/2017:11:18:59 -0400",
"method": "PUT",
"resource": "/explore",
"protocol": "HTTP/1.1",
"response": 200,
"response_size": 2742,
"user-agent": "Mozilla/5.0 (Windows; U; Windows NT 6.3) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/20.0.872.0 Safari/538.0.1"
}
Architecture
To illustrate where the preprocessing step takes place within Kinesis Analytics, take a high-level look at the architecture.
Kinesis Analytics continuously reads data from your Kinesis stream or Kinesis Firehose delivery stream. For each batch of records that it retrieves, the Lambda processor subsystem manages how each batch gets passed to your Lambda function. Your function receives a list of records as input. Within your function, you iterate through the list and apply your business logic to accomplish your preprocessing requirements (such as data transformation).
The input model to your preprocessing function varies slightly, depending on whether the data was received from a stream or delivery stream.
If the source is Kinesis Firehose, the input model is:
{
"invocationId" : Lambda invocation ID (random GUID)
"applicationArn" : Kinesis Analytics application ARN,
"streamArn" : Source delivery stream ARN of the records,
"records": [
{
"recordId" : random GUID,
"kinesisFirehoseRecordMetadata" : {
"approximateArrivalTimestamp" : Approximate time that the delivery stream received the record,
},
"data" : base64 encoded user payload
}
]
}
If the source is Kinesis Streams, the input data contains a few extra attributes specific to streams:
{
"invocationId" : Lambda invocation ID (random GUID)
"applicationArn" : Kinesis Analytics application ARN,
"streamArn" : Source stream ARN of the records,
"records": [
{
"recordId" : random GUID,
"kinesisStreamRecordMetadata" : {
"sequenceNumber" : from the Kinesis Record,
"partitionKey" : from the Kinesis Record,
"shardId" : from the Kinesis Record
"approximateArrivalTimestamp" : from the Kinesis Record
},
"data" : base64 encoded user payload
}
]
}
As you can see, the records attribute is an array of record objects. In your Lambda function, you iterate through each element of the array. Each element’s data attribute contains the base64-encoded record that was retrieved from your input stream or delivery stream. Your function is required to return the modified batch of records using the following model:
{
"records": [
{
"recordId" : record ID that was passed,
"result" : string with value - Ok, Dropped, ProcessingFailed
"data" : processed base64-encoded user payload
}
]
}
For each input record, the output must contain a corresponding record with the same recordId value. The result attribute for each record varies, depending on the logic that you applied in your function. The acceptable values for result are Ok, Dropped, and ProcessingFailed. If the result is Ok, then the value returned in the data attribute continues in the pipeline to the SQL processor. By setting the value of result to Dropped, you can filter out unwanted records.
If your preprocessing function determines that it cannot process a particular record, it can set result to ProcessingFailed, and Kinesis Analytics writes the original record to its error stream. You can access the contents of the error stream to review failed records and take appropriate action.
The Lambda processor in Kinesis Analytics also manages failures and retries of your Lambda function. In cases where your Lambda function returns a result of ProcessingFailed, Kinesis Analytics writes the original input records to its error stream. Therefore, it’s important that you configure a destination for the error stream in your Kinesis Analytics application. By configuring a destination, such as a delivery stream with an S3 destination, all failed records get delivered to a bucket in S3 for later debugging and analysis.
Example Lambda function
Now that you understand some common preprocessing scenarios, and you know where preprocessing fits in the sequence of your Kinesis Analytics application, put it all together into an example Lambda function.
Assume that a data producer is compressing JSON records before sending them to a Kinesis stream or a Kinesis Firehose delivery stream. You want to use Kinesis Analytics to analyze these compressed records. Before you can use SQL to perform the analysis, you must first decompress each input record so that it’s represented as decompressed JSON. This enables it to map to the schema you’ve created in the Kinesis Analytics application. The function below, written in Node.js, receives a list of records, decompresses each, and returns the complete list of decompressed records:
'use strict';
console.log('Loading function');
const zlib = require('zlib');
exports.handler = (event, context, callback) => {
let success = 0; // Number of valid entries found
let failure = 0; // Number of invalid entries found
/* Process the list of records */
const output = event.records.map((record) => {
/* Data is base64-encoded, so decode here */
const compressedData = Buffer.from(record.data, 'base64');
try {
const decompressedData = zlib.unzipSync(compressedData);
/* Encode decompressed JSON or CSV */
const result = (Buffer.from(decompressedData, 'utf8')).toString('base64');
success++;
return {
recordId: record.recordId,
result: 'Ok',
data: result,
};
} catch (err) {
failure++;
return {
recordId: record.recordId,
result: 'ProcessingFailed',
data: record.data,
};
}
});
console.log('Processing completed. Successful records ${success}, Failed records ${failure}.');
callback(null, {
records: output,
});
};
Notice how the function sets the result to ProcessingFailed for any record that fails to decompress. In this scenario, Kinesis Analytics writes that failed record to its error stream.
Conclusion
The ability to preprocess your streaming data with Lambda enables you to use Kinesis Analytics to analyze a limitless variety of data. To help you get started, we’ve created several Lambda blueprints in both Python and Node.js. The decompression example used earlier has also been included as a blueprint. The Lambda blueprints can be found in the AWS Lambda Management Console.
Allan MacInnis is a Solutions Architect at Amazon Web Services. He works with our customers to help them build streaming data solutions using Amazon Kinesis. In his spare time, he enjoys mountain biking and spending time with his family.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
‼️ https://t.co/qLfpLLiXqZ
