Tag Archives: MXNet

Introducing Gluon: a new library for machine learning from AWS and Microsoft

Post Syndicated from Ana Visneski original https://aws.amazon.com/blogs/aws/introducing-gluon-a-new-library-for-machine-learning-from-aws-and-microsoft/

Post by Dr. Matt Wood

Today, AWS and Microsoft announced Gluon, a new open source deep learning interface which allows developers to more easily and quickly build machine learning models, without compromising performance.

Gluon Logo

Gluon provides a clear, concise API for defining machine learning models using a collection of pre-built, optimized neural network components. Developers who are new to machine learning will find this interface more familiar to traditional code, since machine learning models can be defined and manipulated just like any other data structure. More seasoned data scientists and researchers will value the ability to build prototypes quickly and utilize dynamic neural network graphs for entirely new model architectures, all without sacrificing training speed.

Gluon is available in Apache MXNet today, a forthcoming Microsoft Cognitive Toolkit release, and in more frameworks over time.

Neural Networks vs Developers
Machine learning with neural networks (including ‘deep learning’) has three main components: data for training; a neural network model, and an algorithm which trains the neural network. You can think of the neural network in a similar way to a directed graph; it has a series of inputs (which represent the data), which connect to a series of outputs (the prediction), through a series of connected layers and weights. During training, the algorithm adjusts the weights in the network based on the error in the network output. This is the process by which the network learns; it is a memory and compute intensive process which can take days.

Deep learning frameworks such as Caffe2, Cognitive Toolkit, TensorFlow, and Apache MXNet are, in part, an answer to the question ‘how can we speed this process up? Just like query optimizers in databases, the more a training engine knows about the network and the algorithm, the more optimizations it can make to the training process (for example, it can infer what needs to be re-computed on the graph based on what else has changed, and skip the unaffected weights to speed things up). These frameworks also provide parallelization to distribute the computation process, and reduce the overall training time.

However, in order to achieve these optimizations, most frameworks require the developer to do some extra work: specifically, by providing a formal definition of the network graph, up-front, and then ‘freezing’ the graph, and just adjusting the weights.

The network definition, which can be large and complex with millions of connections, usually has to be constructed by hand. Not only are deep learning networks unwieldy, but they can be difficult to debug and it’s hard to re-use the code between projects.

The result of this complexity can be difficult for beginners and is a time-consuming task for more experienced researchers. At AWS, we’ve been experimenting with some ideas in MXNet around new, flexible, more approachable ways to define and train neural networks. Microsoft is also a contributor to the open source MXNet project, and were interested in some of these same ideas. Based on this, we got talking, and found we had a similar vision: to use these techniques to reduce the complexity of machine learning, making it accessible to more developers.

Enter Gluon: dynamic graphs, rapid iteration, scalable training
Gluon introduces four key innovations.

  1. Friendly API: Gluon networks can be defined using a simple, clear, concise code – this is easier for developers to learn, and much easier to understand than some of the more arcane and formal ways of defining networks and their associated weighted scoring functions.
  2. Dynamic networks: the network definition in Gluon is dynamic: it can bend and flex just like any other data structure. This is in contrast to the more common, formal, symbolic definition of a network which the deep learning framework has to effectively carve into stone in order to be able to effectively optimizing computation during training. Dynamic networks are easier to manage, and with Gluon, developers can easily ‘hybridize’ between these fast symbolic representations and the more friendly, dynamic ‘imperative’ definitions of the network and algorithms.
  3. The algorithm can define the network: the model and the training algorithm are brought much closer together. Instead of separate definitions, the algorithm can adjust the network dynamically during definition and training. Not only does this mean that developers can use standard programming loops, and conditionals to create these networks, but researchers can now define even more sophisticated algorithms and models which were not possible before. They are all easier to create, change, and debug.
  4. High performance operators for training: which makes it possible to have a friendly, concise API and dynamic graphs, without sacrificing training speed. This is a huge step forward in machine learning. Some frameworks bring a friendly API or dynamic graphs to deep learning, but these previous methods all incur a cost in terms of training speed. As with other areas of software, abstraction can slow down computation since it needs to be negotiated and interpreted at run time. Gluon can efficiently blend together a concise API with the formal definition under the hood, without the developer having to know about the specific details or to accommodate the compiler optimizations manually.

The team here at AWS, and our collaborators at Microsoft, couldn’t be more excited to bring these improvements to developers through Gluon. We’re already seeing quite a bit of excitement from developers and researchers alike.

Getting started with Gluon
Gluon is available today in Apache MXNet, with support coming for the Microsoft Cognitive Toolkit in a future release. We’re also publishing the front-end interface and the low-level API specifications so it can be included in other frameworks in the fullness of time.

You can get started with Gluon today. Fire up the AWS Deep Learning AMI with a single click and jump into one of 50 fully worked, notebook examples. If you’re a contributor to a machine learning framework, check out the interface specs on GitHub.

-Dr. Matt Wood

Journey into Deep Learning with AWS

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/journey-into-deep-learning-with-aws/

If you are anything like me, Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning are completely fascinating and exciting topics. As AI, ML, and Deep Learning become more widely used, for me it means that the science fiction written by Dr. Issac Asimov, the robotics and medical advancements in Star Wars, and the technologies that enabled Captain Kirk and his Star Trek crew “to boldly go where no man has gone before” can become achievable realities.

 

Most people interested in the aforementioned topics are familiar with the AI and ML solutions enabled by Deep Learning, such as Convolutional Neural Networks for Image and Video Classification, Speech Recognition, Natural Language interfaces, and Recommendation Engines. However, it is not always an easy task setting up the infrastructure, environment, and tools to enable data scientists, machine learning practitioners, research scientists, and deep learning hobbyists/advocates to dive into these technologies. Most developers desire to go quickly from getting started with deep learning to training models and developing solutions using deep learning technologies.

For these reasons, I would like to share some resources that will help to quickly build deep learning solutions whether you are an experienced data scientist or a curious developer wanting to get started.

Deep Learning Resources

The Apache MXNet is Amazon’s deep learning framework of choice. With the power of Apache MXNet framework and NVIDIA GPU computing, you can launch your scalable deep learning projects and solutions easily on the AWS Cloud. As you get started on your MxNet deep learning quest, there are a variety of self-service tutorials and datasets available to you:

  • Launch an AWS Deep Learning AMI: This guide walks you through the steps to launch the AWS Deep Learning AMI with Ubuntu
  • MXNet – Create a computer vision application: This hands-on tutorial uses a pre-built notebook to walk you through using neural networks to build a computer vision application to identify handwritten digits
  • AWS Machine Learning Datasets: AWS hosts datasets for Machine Learning on the AWS Marketplace that you can access for free. These large datasets are available for anyone to analyze the data without requiring the data to be downloaded or stored.
  • Predict and Extract – Learn to use pre-trained models for predictions: This hands-on tutorial will walk you through how to use pre-trained model for predicting and feature extraction using the full Imagenet dataset.

 

AWS Deep Learning AMIs

AWS offers Amazon Machine Images (AMIs) for use on Amazon EC2 for quick deployment of an infrastructure needed to start your deep learning journey. The AWS Deep Learning AMIs are pre-configured with popular deep learning frameworks built using Amazon EC2 instances on Amazon Linux, and Ubuntu that can be launched for AI targeted solutions and models. The deep learning frameworks supported and pre-configured on the deep learning AMI are:

  • Apache MXNet
  • TensorFlow
  • Microsoft Cognitive Toolkit (CNTK)
  • Caffe
  • Caffe2
  • Theano
  • Torch
  • Keras

Additionally, the AWS Deep Learning AMIs install preconfigured libraries for Jupyter notebooks with Python 2.7/3.4, AWS SDK for Python, and other data science related python packages and dependencies. The AMIs also come with NVIDIA CUDA and NVIDIA CUDA Deep Neural Network (cuDNN) libraries preinstalled with all the supported deep learning frameworks and the Intel Math Kernel Library is installed for Apache MXNet framework. You can launch any of the Deep Learning AMIs by visiting the AWS Marketplace using the Try the Deep Learning AMIs link.

Summary

It is a great time to dive into Deep Learning. You can accelerate your work in deep learning by using the AWS Deep Learning AMIs running on the AWS cloud to get your deep learning environment running quickly or get started learning more about Deep Learning on AWS with MXNet using the AWS self-service resources.  Of course, you can learn even more information about Deep Learning, Machine Learning, and Artificial Intelligence on AWS by reviewing the AWS Deep Learning page, the Amazon AI product page, and the AWS AI Blog.

May the Deep Learning Force be with you all.

Tara

AWS Online Tech Talks – June 2017

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/aws-online-tech-talks-june-2017/

As the sixth month of the year, June is significant in that it is not only my birth month (very special), but it contains the summer solstice in the Northern Hemisphere, the day with the most daylight hours, and the winter solstice in the Southern Hemisphere, the day with the fewest daylight hours. In the United States, June is also the month in which we celebrate our dads with Father’s Day and have month-long celebrations of music, heritage, and the great outdoors.

Therefore, the month of June can be filled with lots of excitement. So why not add even more delight to the month, by enhancing your cloud computing skills. This month’s AWS Online Tech Talks features sessions on Artificial Intelligence (AI), Storage, Big Data, and Compute among other great topics.

June 2017 – Schedule

Noted below are the upcoming scheduled live, online technical sessions being held during the month of June. Make sure to register ahead of time so you won’t miss out on these free talks conducted by AWS subject matter experts. All schedule times for the online tech talks are shown in the Pacific Time (PDT) time zone.

Webinars featured this month are:

Thursday, June 1

Storage

9:00 AM – 10:00 AM: Deep Dive on Amazon Elastic File System

Big Data

10:30 AM – 11:30 AM: Migrating Big Data Workloads to Amazon EMR

Serverless

12:00 Noon – 1:00 PM: Building AWS Lambda Applications with the AWS Serverless Application Model (AWS SAM)

 

Monday, June 5

Artificial Intelligence

9:00 AM – 9:40 AM: Exploring the Business Use Cases for Amazon Lex

 

Tuesday, June 6

Management Tools

9:00 AM – 9:40 AM: Automated Compliance and Governance with AWS Config and AWS CloudTrail

 

Wednesday, June 7

Storage

9:00 AM – 9:40 AM: Backing up Amazon EC2 with Amazon EBS Snapshots

Big Data

10:30 AM – 11:10 AM: Intro to Amazon Redshift Spectrum: Quickly Query Exabytes of Data in S3

DevOps

12:00 Noon – 12:40 PM: Introduction to AWS CodeStar: Quickly Develop, Build, and Deploy Applications on AWS

 

Thursday, June 8

Artificial Intelligence

9:00 AM – 9:40 AM: Exploring the Business Use Cases for Amazon Polly

10:30 AM – 11:10 AM: Exploring the Business Use Cases for Amazon Rekognition

 

Monday, June 12

Artificial Intelligence

9:00 AM – 9:40 AM: Exploring the Business Use Cases for Amazon Machine Learning

 

Tuesday, June 13

Compute

9:00 AM – 9:40 AM: DevOps with Visual Studio, .NET and AWS

IoT

10:30 AM – 11:10 AM: Create, with Intel, an IoT Gateway and Establish a Data Pipeline to AWS IoT

Big Data

12:00 Noon – 12:40 PM: Real-Time Log Analytics using Amazon Kinesis and Amazon Elasticsearch Service

 

Wednesday, June 14

Containers

9:00 AM – 9:40 AM: Batch Processing with Containers on AWS

Security & Identity

12:00 Noon – 12:40 PM: Using Microsoft Active Directory across On-premises and Cloud Workloads

 

Thursday, June 15

Big Data

12:00 Noon – 1:00 PM: Building Big Data Applications with Serverless Architectures

 

Monday, June 19

Artificial Intelligence

9:00 AM – 9:40 AM: Deep Learning for Data Scientists: Using Apache MxNet and R on AWS

 

Tuesday, June 20

Storage

9:00 AM – 9:40 AM: Cloud Backup & Recovery Options with AWS Partner Solutions

Artificial Intelligence

10:30 AM – 11:10 AM: An Overview of AI on the AWS Platform

 

The AWS Online Tech Talks series covers a broad range of topics at varying technical levels. These sessions feature live demonstrations & customer examples led by AWS engineers and Solution Architects. Check out the AWS YouTube channel for more on-demand webinars on AWS technologies.

Tara

Deep Learning on AWS Batch

Post Syndicated from Chris Barclay original https://aws.amazon.com/blogs/compute/deep-learning-on-aws-batch/

Thanks to my colleague Kiuk Chung for this great post on Deep Learning using AWS Batch.

—-

GPU instances naturally pair with deep learning as neural network algorithms can take advantage of their massive parallel processing power. AWS provides GPU instance families, such as g2 and p2, which allow customers to run scalable GPU workloads. You can leverage such scalability efficiently with AWS Batch.

AWS Batch manages the underlying compute resources on-your behalf, allowing you to focus on modeling tasks without the overhead of resource management. Compute environments (that is, clusters) in AWS Batch are pools of instances in your account, which AWS Batch dynamically scales up and down, provisioning and terminating instances with respect to the numbers of jobs. This minimizes idle instances, which in turn optimizes cost.

Moreover, AWS Batch ensures that submitted jobs are scheduled and placed onto the appropriate instance, hence managing the lifecycle of the jobs. With the addition of customer-provided AMIs, AWS Batch users can now take advantage of this elasticity and convenience for jobs that require GPU.

This post illustrates how you can run GPU-based deep learning workloads on AWS Batch. I walk you through an example of training a convolutional neural network (the LeNet architecture), using Apache MXNet to recognize handwritten digits using the MNIST dataset.

Running an MXNet job in AWS Batch

Apache MXNet is a full-featured, flexibly programmable, and highly scalable deep learning framework that supports state-of-the-art deep models, including convolutional neural networks (CNNs) and long short-term memory networks (LSTMs).

There are three steps to running an AWS Batch job:

  • Create a custom AMI
  • Create AWS Batch entities
  • Submit a training job

Create a custom AMI

Start by creating an AMI that includes the NVIDIA driver and the Amazon ECS agent. In AWS Batch, instances can be launched with the specific AMI of your choice by specifying imageId when you create your compute environment. Because you are running a job that requires GPU, you need an AMI that has the NVIDIA driver installed.

Choose Launch Stack to launch the CloudFormation template in us-east-1 in your account:

As shown below, take note of the AMI value in the Outputs tab of the CloudFormation stack. You use this as the imageId value when creating the compute environment in the next section.

Alternatively, you may follow the AWS Batch documentation to create a GPU-enabled AMI.

Create AWS Batch resources

After you have built the AMI, create the following resources:

A compute environment, is a collection of instances (compute resources) of the same or different instance types. In this case, you create a managed compute environment in which the instances are of type p2.xlarge. For imageId, specify the AMI you built in the previous section.

Then, create a job queue. In AWS Batch, jobs are submitted to a job queue that are associated to an ordered list of compute environments. After a lower order compute environment is filled, jobs spill over to the next compute environment. For this example, you associate a single compute environment to the job queue.

Finally, create a job definition, which is a template for a job specification. For those familiar with Amazon ECS, this is analogous to task definitions. You mount the directory containing the NVIDIA driver on the host to /usr/local/nvidia on the container. You also need to set the privileged flag on the container properties.

The following code creates the aforementioned resources in AWS Batch. For more information, see the AWS Batch User Guide.

git clone https://github.com/awslabs/aws-batch-helpers
cd aws-batch-helpers/gpu-example

python create-batch-entities.py\
 --subnets <subnet1,subnet2,…>\
 --security-groups <sg1,sg2,…>\
 --key-pair \
 --instance-role \
 --image-id \
 --service-role 

Submit a training job

Now you submit a job that trains a convolutional neural network model for handwritten digit recognition. Much like Amazon ECS tasks, jobs in AWS Batch are run as commands in a Docker container. To use MXNet as your deep learning library, you need a Docker image containing MXNet. For this example, use mxnet/python:gpu.

The submit-job.py script submits the job, and tails the output from CloudWatch Logs.

# cd aws-batch-helpers/gpu-example
python submit-job.py --wait

You should see an output that looks like the following:

Submitted job [train_imagenet - e1bccebc-76d9-4cd1-885b-667ef93eb1f5] to the job queue [gpu_queue]
Job [train_imagenet - e1bccebc-76d9-4cd1-885b-667ef93eb1f5] is RUNNING.
Output [train_imagenet/e1bccebc-76d9-4cd1-885b-667ef93eb1f5/12030dd3-0734-42bf-a3d1-d99118b401eb]:
 ================================================================================

[2017-04-25T19:02:57.076Z] INFO:root:Epoch[0] Batch [100]	Speed: 15554.63 samples/sec Train-accuracy=0.861077
[2017-04-25T19:02:57.428Z] INFO:root:Epoch[0] Batch [200]	Speed: 18224.89 samples/sec Train-accuracy=0.954688
[2017-04-25T19:02:57.755Z] INFO:root:Epoch[0] Batch [300]	Speed: 19551.42 samples/sec Train-accuracy=0.965313
[2017-04-25T19:02:58.080Z] INFO:root:Epoch[0] Batch [400]	Speed: 19697.65 samples/sec Train-accuracy=0.969531
[2017-04-25T19:02:58.405Z] INFO:root:Epoch[0] Batch [500]	Speed: 19705.82 samples/sec Train-accuracy=0.968281
[2017-04-25T19:02:58.734Z] INFO:root:Epoch[0] Batch [600]	Speed: 19486.54 samples/sec Train-accuracy=0.971719
[2017-04-25T19:02:59.058Z] INFO:root:Epoch[0] Batch [700]	Speed: 19735.59 samples/sec Train-accuracy=0.973281
[2017-04-25T19:02:59.384Z] INFO:root:Epoch[0] Batch [800]	Speed: 19631.17 samples/sec Train-accuracy=0.976562
[2017-04-25T19:02:59.713Z] INFO:root:Epoch[0] Batch [900]	Speed: 19490.74 samples/sec Train-accuracy=0.979062
[2017-04-25T19:02:59.834Z] INFO:root:Epoch[0] Train-accuracy=0.976774
[2017-04-25T19:02:59.834Z] INFO:root:Epoch[0] Time cost=3.190
[2017-04-25T19:02:59.850Z] INFO:root:Saved checkpoint to "/mnt/model/mnist-0001.params"
[2017-04-25T19:03:00.079Z] INFO:root:Epoch[0] Validation-accuracy=0.969148

================================================================================
Job [train_imagenet - e1bccebc-76d9-4cd1-885b-667ef93eb1f5] SUCCEEDED

In reality, you may want to modify the job command to save the trained model artifact to Amazon S3 so that subsequent prediction jobs can generate predictions against the model. For information about how to reference objects in Amazon S3 in your jobs, see the Creating a Simple “Fetch & Run” AWS Batch Job post.

Conclusion

In this post, I walked you through an example of running a GPU-enabled job in AWS Batch, using MXNet as the deep learning library. AWS Batch exposes primitives to allow you to focus on implementing the most efficient algorithm for your workload. It enables you to manage the lifecycle of submitted jobs and dynamically adapt the infrastructure requirements of your jobs within the specified bounds. It’s easy to take advantage of the horizontal scalability of compute instances provided by AWS in a cost-efficient manner.

MXNet, on the other hand, provides a rich set of highly optimized and scalable building blocks to start implementing your own deep learning algorithms. Together, you can not only solve problems requiring large neural network models, but also cut down on iteration time by harnessing the seemingly unlimited compute resources in Amazon EC2.

With AWS Batch managing the resources on your behalf, you can easily implement workloads such as hyper-parameter optimization to fan out tens or even hundreds of searches in parallel to find the best set of model parameters for your problem space. Moreover, because your jobs are run inside Docker containers, you may choose the tools and libraries that best fit your needs, build a Docker image, and submit your jobs using the image of your choice.

We encourage you to try it yourself and let us know what you think!

AWS Enables Consortium Science to Accelerate Discovery

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-enables-consortium-science-to-accelerate-discovery/

My colleague Mia Champion is a scientist (check out her publications), an AWS Certified Solutions Architect, and an AWS Certified Developer. The time that she spent doing research on large-data datasets gave her an appreciation for the value of cloud computing in the bioinformatics space, which she summarizes and explains in the guest post below!

Jeff;


Technological advances in scientific research continue to enable the collection of exponentially growing datasets that are also increasing in the complexity of their content. The global pace of innovation is now also fueled by the recent cloud-computing revolution, which provides researchers with a seemingly boundless scalable and agile infrastructure. Now, researchers can remove the hindrances of having to own and maintain their own sequencers, microscopes, compute clusters, and more. Using the cloud, scientists can easily store, manage, process and share datasets for millions of patient samples with gigabytes and more of data for each individual. As American physicist, John Bardeen once said: “Science is a collaborative effort. The combined results of several people working together is much more effective than could be that of an individual scientist working alone”.

Prioritizing Reproducible Innovation, Democratization, and Data Protection
Today, we have many individual researchers and organizations leveraging secure cloud enabled data sharing on an unprecedented scale and producing innovative, customized analytical solutions using the AWS cloud.  But, can secure data sharing and analytics be done on such a collaborative scale as to revolutionize the way science is done across a domain of interest or even across discipline/s of science? Can building a cloud-enabled consortium of resources remove the analytical variability that leads to diminished reproducibility, which has long plagued the interpretability and impact of research discoveries? The answers to these questions are ‘yes’ and initiatives such as the Neuro Cloud Consortium, The Global Alliance for Genomics and Health (GA4GH), and The Sage Bionetworks Synapse platform, which powers many research consortiums including the DREAM challenges, are starting to put into practice model cloud-initiatives that will not only provide impactful discoveries in the areas of neuroscience, infectious disease, and cancer, but are also revolutionizing the way in which scientific research is done.

Bringing Crowd Developed Models, Algorithms, and Functions to the Data
Collaborative projects have traditionally allowed investigators to download datasets such as those used for comparative sequence analysis or for training a deep learning algorithm on medical imaging data. Investigators were then able to develop and execute their analysis using institutional clusters, local workstations, or even laptops:

This method of collaboration is problematic for many reasons. The first concern is data security, since dataset download essentially permits “chain-data-sharing” with any number of recipients. Second, analytics done using compute environments that are not templated at some level introduces the risk of variable analytics that itself is not reproducible by a different investigator, or even the same investigator using a different compute environment. Third, the required data dump, processing, and then re-upload or distribution to the collaborative group is highly inefficient and dependent upon each individual’s networking and compute capabilities. Overall, traditional methods of scientific collaboration have introduced methods in which security is compromised and time to discovery is hampered.

Using the AWS cloud, collaborative researchers can share datasets easily and securely by taking advantage of Identity and Access Management (IAM) policy restrictions for user bucket access as well as S3 bucket policies or Access Control Lists (ACLs). To streamline analysis and ensure data security, many researchers are eliminating the necessity to download datasets entirely by leveraging resources that facilitate moving the analytics to the data source and/or taking advantage of remote API requests to access a shared database or data lake. One way our customers are accomplishing this is to leverage container based Docker technology to provide collaborators with a way to submit algorithms or models for execution on the system hosting the shared datasets:

Docker container images have all of the application’s dependencies bundled together, and therefore provide a high degree of versatility and portability, which is a significant advantage over using other executable-based approaches. In the case of collaborative machine learning projects, each docker container will contain applications, language runtime, packages and libraries, as well as any of the more popular deep learning frameworks commonly used by researchers including: MXNet, Caffe, TensorFlow, and Theano.

A common feature in these frameworks is the ability to leverage a host machine’s Graphical Processing Units (GPUs) for significant acceleration of the matrix and vector operations involved in the machine learning computations. As such, researchers with these objectives can leverage EC2’s new P2 instance types in order to power execution of submitted machine learning models. In addition, GPUs can be mounted directly to containers using the NVIDIA Docker tool and appear at the system level as additional devices. By leveraging Amazon EC2 Container Service and the EC2 Container Registry, collaborators are able to execute analytical solutions submitted to the project repository by their colleagues in a reproducible fashion as well as continue to build on their existing environment.  Researchers can also architect a continuous deployment pipeline to run their docker-enabled workflows.

In conclusion, emerging cloud-enabled consortium initiatives serve as models for the broader research community for how cloud-enabled community science can expedite discoveries in Precision Medicine while also providing a platform where data security and discovery reproducibility is inherent to the project execution.

Mia D. Champion, Ph.D.

 

AWS Monthly Online Tech Talks – March, 2017

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/aws-monthly-online-tech-talks-march-2017/

Unbelievably it is March already, as you enter into the madness of March don’t forget to take some time and learning more about the latest service innovations from AWS. Each month, we have a series of webinars targeting best practices and new service features in the AWS Cloud.

I have shared below the schedule for the live, online technical sessions scheduled for the month of March. Remember these talks are free, but they fill up quickly so register ahead of time. The online tech talks scheduled times are shown in Pacific Time (PT) time zone.

Webinars featured this month are as follows:

Tuesday, March 21

Big Data

9:00 AM – 10:00 AM: Deploying a Data Lake in AWS

Databases

10:30 AM – 11:30 AM: Optimizing the Data Tier for Serverless Web Applications

IoT

12:00 Noon – 1:00 PM: One Click Enterprise IoT Services

 

Wednesday, March 22

Databases

10:30 – 11:30 AM: ElastiCache Deep Dive: Best Practices and Usage Patterns

Mobile

12:00 Noon – 1:00 PM: A Deeper Dive into Apache MXNet on AWS

 

Thursday, March 23

IoT

9:00 – 10:00 AM: Developing Applications with the IoT Button

Compute

10:30 – 11:30 AM: Automating Management of Amazon EC2 Instances with Auto Scaling

 

Friday, March 24

Compute

10:30 – 11:30 AM: An Overview of Designing Microservices Based Applications on AWS

 

Monday, March 27

AI

9:00 – 10:00 AM: How to get the most out of Amazon Polly, a text-to-speech service

 

Tuesday, March 28

Compute

10:30 AM – 11:30 AM: Getting the Most Out of the New Amazon EC2 Reserved Instances Enhancements

Getting Started

12:00 Noon – 1:30 PM: Getting Started with AWS

 

Wednesday, March 29

Security

9:00 – 10:00 AM: Best Practices for Managing Security Operations in AWS

Storage

10:30 – 11:30 AM: Deep Dive on Amazon S3

Big Data

12:00 Noon – 1:00 PM: Log Analytics with Amazon Elasticsearch Service and Amazon Kinesis

 

Thursday, March 30

Storage

9:00 – 10:00 AM: Active Archiving with Amazon S3 and Tiering to Amazon Glacier

Mobile

10:30 AM – 11:30 AM: Deep Dive on Amazon Cognito

Compute

12:00 Noon – 1:00 PM: Building a Development Workflow for Serverless Applications

 

The AWS Online Tech Talks series covers a broad range of topics at varying technical levels. These technical sessions are led by AWS solutions architects and engineers and feature live demonstrations & customer examples. You can also check out the AWS on-demand webinar series on the AWS YouTube channel.

Tara

 

Join us next week at Strata + Hadoop World in San Jose, CA

Post Syndicated from Jorge A. Lopez original https://aws.amazon.com/blogs/big-data/join-us-next-week-at-strata-hadoop-world-in-san-jose-ca/

We’re back in San Jose for the Strata conference, March 13-16, 2017, to talk all things big data at AWS and show you some of our latest innovations. Come meet the AWS Big Data team at booth #928, where big data experts will be happy to answer your questions, hear about your requirements, and help you with your big data initiatives.

New this year, we’re hosting a hands-on tutorial on Tuesday, where big data solutions architects will guide you through creating a sample big data application using services such as Amazon Kinesis, Amazon Athena, and AWS Lambda.

You can catch the tutorial on Tuesday March 14, 9am – 12:30pm. Please note that you may need to register in advance.

As previous years, Strata will host a number of technical sessions where you can get best practices and learn more about big data on AWS. Here are some recommendations to get you started.

Amazon Kinesis Data Streaming Services
Roger Barga, AWS
11:50am Wednesday, March 15, 2017. Room LL20 C

Serverless Big Data Architectures: Design Patterns and Best Practices
Siva Raghupathy & Ben Snively, AWS
5:10pm Wednesday, March 15, 2017. Room 210 B/F

Distributed deep learning on AWS using MXNet
Anima Anandkumar, AWS
11:00 AM Thursday, March 16, 2017. Room 230 C

Feature Engineering for Diverse Data Types
Alice Zheng, Amazon
5:10pm–5:50pm Wednesday, March 15, 2017. Room 230 C

The Netflix data platform: Now and in the future
Kurt Brown, Netflix
11:50am–12:30pm Thursday, March 16, 2017. Room LL20 A

Going real time: Creating online datasets for personalization.
Christopher Colburn & Monal Daxini, Netflix
11:50am–12:30pm Wednesday, March 15, 2017. Room LL20 A

Zillow: Transforming real estate through big data and data science
Jasjeet Thind, Zillow
11:50am–12:30pm Wednesday, March 15, 2017. Room 230 A

Recommending 1+ billion items to 100+ million users in real time: Harnessing the structure of the user-to-object graph to extract ranking signals at scale
Jure Leskovec, Pinterest
11:50am–12:30pm Wednesday, March 15, 2017. Room 230 C

Shifting left for continuous quality in an Agile data world
Avinash Padmanabhan, Intuit
1:50pm–2:30pm Wednesday, March 15, 2017. Room LL20 A

Looking forward to seeing you at Strata + Hadoop World!

Excited about MXNet joining Apache!

Post Syndicated from Ana Visneski original https://aws.amazon.com/blogs/aws/excited-about-mxnet-joining-apache/

Post by Dr. Matt Wood

From Alexa to Amazon Go, we use deep learning extensively across all areas of Amazon, and we’ve tried a lot of deep learning engines along the way. One has emerged as the most scalable, efficient way to perform deep learning, and for these reasons, we have selected MXNet as our engine of choice at Amazon.

MXNet is an open source, state of the art deep learning engine, which allows developers to build sophisticated, custom artificial intelligence systems. Training these systems is significantly faster in MXNet, due to its scale and performance. For example, for the popular image recognition network, Resnet, MXNet has 2X the throughput compared to other engines, letting you train equivalent models in half the time. MXNet also shows close to linear scaling across hundreds of GPUs, while the performance of other engines show diminishing returns at scale.

We have a significant team at Amazon working with the MXNet community to continue to evolve it. The team proposed MXNet joining the Apache Incubator to take advantage of the Apache Software Foundation’s process, stewardship, outreach, and community events. We’re excited to announce that it has been accepted.

We’re at the start what we’ll be investing in Apache MXNet, and look forward to partnering with the community to keep extending its already significant utility.

If you’d like to get started with MXNet, take a look at the keynote presentation I gave at AWS Re:Invent, and fire up an instance (or an entire cluster), of the AWS Deep Learning AMI which includes MXNet with example code, pre-compiled and ready to rock. You should also watch Leo’s presentation and tutorial on recommendation modeling.

You should follow @apachemxnet on Twitter, or check out the new Apache MXNet page for updates from the open source project.

 

AWS Reinvent 2016: Embiggen your business with Amazon Web Services

Post Syndicated from Bart Thomas original https://www.anchor.com.au/blog/2016/12/aws-reinvent-2016/

Three weeks ago, Amazon Web Services ran their annual love-fest in Las Vegas and it was quite a remarkable week. On arrival, attendees (all 32,000+ of them) were given a shiny new Alexa Echo Dot, Amazon’s latest entrant into the growing market for voice controlled, AI-based smart assistants, a segment that includes Apple’s Siri, Google Assistant and Microsoft’s Cortana.

Amazon have now made it clear that they’re taking Artificial Intelligence (AI) and Machine Learning very seriously, with four brand new, developer-focussed AI related services (Polly, Rekognition, Lex and MXNet) announced during the week. The free Alexa Echo Dots yet another incentive for developers to start building apps that make use of (and ultimately contribute to) Amazon’s efforts in this space.

The week was brought to a close with a spectacular party, headlined by Martin Garrix, named the world’s top DJ in 2016 by djmag.com. Goes to show that some of the worlds biggest geeks and code cutters are also capable of cutting some serious rug:

While the potential of artificial intelligence and machine learning technologies are both exciting and somewhat scary, there was plenty more to consider over the course of the week with a bevy of announcements such as new server instance types, enhanced support and orchestration for containers (Blox), low cost, simple to launch virtual servers from $5 per month (Lightsail), free of charge DDoS protection (AWS Shield), application performance monitoring and debugging (X-Ray), a new ”Internet of Things” (IoT) play to help developers build and manage smart, connected devices (Greengrass) and a fully managed continuous integration (CI) service (CodeBuild) that neatly rounds off Amazon’s DevOps-friendly suite of CI/CD services — and that’s just scratching the surface.

There’s a summary of the announcements here: https://aws.amazon.com/new/reinvent/

With videos of the sessions here: https://www.youtube.com/user/AmazonWebServices

For me, the main takeaway was that the pace of technology-enabled change is continuing to accelerate and Amazon Web Services is very likely to be at the heart of it.

AWS pace of innovation 2016

AWS pace of innovation 2016

AWS is a sales and innovation machine, continuing to put distance between themselves and their competitors — their sheer pace of innovation would appear almost impossible to compete with. The public clouds of Microsoft, IBM and Google would need years to catch up and that’s assuming AWS were sporting enough to stand still for that long.

In 2016, AWS announced around 1000 new services and updates – simply incredible if you’re company whose product and development teams are making use of the platform, and quite simply terrifying if you’re just about anyone else. As AWS continue their march up the value chain, those in the business of infrastructure services, monitoring, BI, data analytics, CI/CD developer tools, network security and even artificial intelligence (AI) all have very good reason to be concerned.

Interestingly, AWS reported an annual revenue run rate of nearly $13 billion with an incredible growth rate of 55% this past year, while the traditional big IT vendors – VMware, HP, Oracle, Cisco, Dell, EMC and IBM have gone backwards — dropping from a collective $221 billion revenue in 2012, to $206 billion in 2016.

Momentum for the public cloud keeps growing, and it’s easy to see why.

AWS is without doubt the leader in the field, and according to Andy Jassy (AWS CEO and pleasingly the very same guy who first presented Jeff Bezos with the AWS business plan) they are the fastest growing, US$1 billion-plus technology company ever, with Gartner estimating in 2015 that AWS is more than ten times the size of the next 14 competitors in the public cloud space combined – Microsoft, Google and IBM included.

Just look at these revenue and YOY growth numbers:
aws-revenueaws-yoy-growth
Source – https://www.statista.com/statistics/250520/forecast-of-amazon-web-services-revenue/

If you’re an application developer looking to win in your market, you would be remiss not to give careful consideration to building your application on top of AWS. Legacy IT infrastructure still has its place, but if your business is looking to the future then the cloud with all its automation and as-a-service goodness is where it’s at.

AWS’ API-driven infrastructure services enable you to take your development processes and application smarts to the next level. Adopting continuous delivery allows your product and development teams to move many orders of magnitude faster than they do today, reducing outages, improving software quality and security. And once your applications are infrastructure aware (aka “cloud native”), they’ll auto-scale seamlessly with the peaks and troughs of customer demand, self-heal when things go wrong and deliver a a great experience to your customers – no matter where they are in the world.

If you’re serious about embiggening your business, you need to embiggen your product and software development capabilities, and you need to do it quickly. Wondering where you’ll get the biggest bang for your buck? Where you’ll find the most efficiency gains? AWS looks like a pretty safe bet to me.

The post AWS Reinvent 2016: Embiggen your business with Amazon Web Services appeared first on AWS Managed Services by Anchor.