Tag Archives: startup

How SikSin improved customer engagement with AWS Data Lab and Amazon Personalize

Post Syndicated from Byungjun Choi original https://aws.amazon.com/blogs/big-data/how-siksin-improved-customer-engagement-with-aws-data-lab-and-amazon-personalize/

This post is co-written with Byungjun Choi and Sangha Yang from SikSin.

SikSin is a technology platform connecting customers with restaurant partners serving their multiple needs. Customers use the SikSin platform to search and discover restaurants, read and write reviews, and view photos. From the restaurateurs’ perspective, SikSin enables restaurant partners to engage and acquire customers in order to grow their business. SikSin has a partnership with 850 corporate companies and more than 50,000 restaurants. They issue restaurant e-vouchers to more than 220,000 members, including individuals as well as corporate members. The SikSin platform receives more than 3 million users in a month. SikSin was listed in the top 100 of the Financial Times’s Asia-Pacific region’s high-growth companies in 2022.

SikSin was looking to deliver improved customer experiences and increase customer engagement. SikSin confronted two business challenges:

  • Customer engagement – SikSin maintains data on more than 750,000 restaurants and has more than 4,000 restaurant articles (and growing). SikSin was looking for a personalized and customized approach to provide restaurant recommendations for their customers and get them engaged with the content, thereby providing a personalized customer experience.
  • Data analysis activities – The SikSin Food Service team experienced difficulties in regards to report generation due to scattered data across multiple systems. The team previously had to submit a request to the IT team and then wait for answers that might be outdated. For the IT team, they needed to manually pull data out of files, databases, and applications, and then combine them upon every request, which is a time-consuming activity. The SikSin Food Service team wanted to view web analytics log data by multiple dimensions, such as customer profiles and places. Examples include page view, conversion rate, and channels.

To overcome these two challenges, SikSin participated in the AWS Data Lab program to assist them in building a prototype solution. The AWS Data Lab offers accelerated, joint-engineering engagements between customers and AWS technical resources to create tangible deliverables that accelerate data and analytics modernization initiatives. The Build Lab is a 2–5-day intensive build with a technical customer team.

In this post, we share how SikSin built the basis for accelerating their data project with the help of the Data Lab and Amazon Personalize.

Use cases

The Data Lab team and SikSin team had three consecutive meetings to discuss business and technical requirements, and decided to work on two uses cases to resolve their two business challenges:

  • Build personalized recommendations – SikSin wanted to deploy a machine learning (ML) model to produce personalized content on the landing page of the platform, particularly restaurants and restaurant articles. The success criteria was to increase the number of page views per session and membership subscription, reduce their bounce rate, and ultimately engage more visitors and members in SikSin’s contents.
  • Establish self-service analytics – SikSin’s business users wanted to reduce time to insight by making data more accessible while removing the reliance on the IT team by giving business users the ability to query data. The key was to consolidate web logs from BigQuery and operational business data from Amazon Relational Data Service (Amazon RDS) into a single place and analyze data whenever they need.

Solution overview

The following architecture depicts what the SikSin team built in the 4-day Build Lab. There are two parts in the solution to address SikSin’s business and technical requirements. The first part (1–8) is for building personalized recommendations, and the second part (A–D) is for establishing self-service analytics.

SikSin Solution Architecture

SikSin deployed an ML model to produce personalized content recommendations by using the following AWS services:

  1. AWS Database Migration Service (AWS DMS) helps migrate databases to AWS quickly and securely with minimal downtime. The SikSin team used AWS DMS to perform full load to bring data from the database tables into Amazon Simple Storage Service (Amazon S3) as a target. Amazon S3 is an object storage service offering industry-leading scalability, data availability, security, and performance. An AWS Glue crawler populates the AWS Glue Data Catalog with the data schema definitions (in a landing folder).
  2. An AWS Lambda function checks if any previous files still exist in the landing folder and archives the files into a backup folder, if any.
  3. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, ML, and application development. The SikSin team created AWS Glue Spark extract, transform, and load (ETL) jobs to prepare input datasets for ML models. These datasets are used to train ML models in bulk mode. There are a total of five datasets for training and two datasets for batch inference jobs.
  4. Amazon Personalize allows developers to quickly build and deploy curated recommendations and intelligent user segmentation at scale using ML. Because Amazon Personalize can be tailored to your individual needs, you can deliver the right customer experience at the right time and in the right place. Also, users will select existing ML models (also known as recipes), train models, and run batch inference to make recommendations.
  5. An Amazon Personalize job predicts for each line of input data (restaurants and restaurant articles) and produces ML-generated recommendations in the designated S3 output folder. The recommendation records are surfaced using interaction data, product data, and predictive models. An AWS Glue crawler populates the AWS Glue Data Catalog with the data schema definitions (in an output folder).
  6. The SikSin team applied business logics and filters in an AWS Glue job to prepare the final datasets for recommendations.
  7. AWS Step Functions enables you to build scalable, distributed applications using state machines. The SikSin team used AWS Step Functions Workflow Studio to visually create, run, and debug workflow runs. This workflow is triggered based on a schedule. The process includes data ingestion, cleansing, processing, and all steps defined in Amazon Personalize. This also involves managing run dependencies, scheduling, error-catching, and concurrency in accordance with the logical flow of the pipeline.
  8. Amazon Simple Notification Service (Amazon SNS) sends notifications. The SikSin team used Amazon SNS to send a notification via email and Google Hangouts with a Lambda function as a target.

To establish a self-service analytics environment to enable business users to perform data analysis, SikSin used the following services:

  1. The Google BigQuery Connector for AWS Glue simplifies the process of connecting AWS Glue jobs to extract data from BigQuery. The SikSin team used the connector to extract web analytics logs from BigQuery and load them to an S3 bucket.
  2. AWS Glue DataBrew is a visual data preparation tool that makes it easy for data analysts and data scientists to clean and normalize data to prepare it for analytics and ML. You can choose from over 250 pre-built transformations to automate data preparation tasks, all without the need to write any code. The SikSin Food Service team used it to visually inspect large datasets and shape the data for their data analysis activities. An S3 bucket (in the intermediate folder) contains business operational data such as customers, places, articles, and products, and reference data loaded from AWS DMS and web analytics logs and data by AWS Glue jobs.
  3. An AWS Glue Python shell runs a job to cleanse and join data, and apply business rules to prepare the data for queries. The SikSin team used AWS SDK Pandas, an AWS Professional Service open-source Python initiative, which extends the power of the Pandas library to AWS, connecting DataFrames and AWS data related services. The output files are stored in an Apache Parquet format in a single folder. An AWS Glue crawler populates the data schema definitions (in an output folder) into the AWS Glue Data Catalog.
  4. The SikSin Food Service team used Amazon Athena and Amazon Quicksight to query and visualize the data analysis. Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. QuickSight is an ML-powered business intelligence service built for the cloud.

Business outcomes

The SikSin Food Service team is now able to access the available data for performing data analysis and manipulation operations efficiently, as well as for getting insights on their own. This immediately allows the team as well as other lines of business to understand how customers are interacting with SikSin’s contents and services on the platform and make decisions sooner. For example, with the data output, the Food Service team was able to provide insights and data points for their external stakeholder and customer to initiate a new business idea. Moreover, the team shared, “We anticipate the recommendations and personalized content will increase conversion rates and customer engagement.”

The AWS Data Lab enabled SikSin to review and assess thoroughly what data is actually usable and available. With SikSin’s objective to successfully build a data pipeline for data analytics purposes, the SikSin team came to realize the importance of data cleansing, categorization, and standardization. “Only fruitful analysis and recommendation are possible when data is intact and properly cleansed,” said Byungjun Choi (the Head of SikSin’s Food Service Team). After completing the Data Lab, SikSin completed and set up an internal process that can streamline the data cleansing pipeline.

SikSin was stuck in the research phase of looking for a solution to solve their personalization challenges. The AWS Data Lab enabled the SikSin IT Team to get hands-on with the technology and build a minimum viable product (MVP) to explore how Amazon Personalize would work in their environment with their data. They achieved this via the Data Lab by adopting AWS DMS, AWS Glue, Amazon Personalize, and Step Functions. “Though it is still the early stage of building a prototype, I am very confident with the right enablement provided from AWS that an effective recommendation system can be adopted on production level very soon,” commented Sangha Yang (the Head of SikSin IT Team).

Conclusion

As a result of the 4-day Build Lab, the SikSin team left with a working prototype that is custom fit to their needs, gaining a clear path forward for enabling end-users to gain valuable insights into its data. The Data Lab allowed the SikSin team to accelerate the architectural design and prototype build of this solution by months. Based on the lessons and learnings obtained from Data Lab, SikSin is planning to launch a Global News Content Platform equipped with a recommendation feature in FY23.

As demonstrated by SikSin’s achievements, Amazon Personalize allows developers to quickly build and deploy curated recommendations and intelligent user segmentation at scale using ML. Because Amazon Personalize can be tailored to your individual needs, you can deliver the right customer experience at the right time and in the right place. Whether you want to optimize recommendations, target customers more accurately, maximize your data’s value, or promote items using business rules.

To accelerate your digital transformation with ML, the Data Lab program is available to support you by providing prescriptive architectural guidance on a particular use case, sharing best practices, and removing technical roadblocks. You’ll leave the engagement with an architecture or working prototype that is custom fit to your needs, a path to production, and deeper knowledge of AWS services.

Please contact your AWS Account Manager or Solutions Architect to get started. If you don’t have an AWS Account Manager, please contact Sales.


About the Authors

bdb-2857-BJByungjun Choi is the Head of SikSin Food Service at SikSin.

bdb-2857-SHSangha Yang is the Head of IT team at SinSin.

bdb-2857-youngguYounggu Yun is a Senior Data Lab Architect at AWS. He works with customers around the APAC region to help them achieve business goals and solve technical problems by providing prescriptive architectural guidance, sharing best practices, and building innovative solutions together.

Junwoo Lee is an Account Manager at AWS. He provides technical and business support to help customer resolve their problems and enrich customer journey by introducing local and global programs for his customers.

bdb-2857-jinwooJinwoo Park is a Senior Solutions Architect at AWS. He provides technical support for AWS customers to succeed with their cloud journey. He helps customers build more secure, efficient, and cost-optimized architectures and solutions, and delivers best practices and workshops.

AWS Week in Review – October 17, 2022

Post Syndicated from Steve Roberts original https://aws.amazon.com/blogs/aws/aws-week-in-review-october-17-2020/

This post is part of our Week in Review series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Monday means it’s time for another Week in Review post, so, without further ado, let’s dive right in!

Last Week’s Launches
Here’s some launch announcements from last week you may have missed.

AWS Directory Service for Microsoft Active Directory is now available on Windows Server 2019, and all new directories will run on this server platform. Those of you with existing directories can choose to update with either a few clicks on the AWS Managed Microsoft AD console, or you can update programmatically using an API. With either approach, you can update at a time convenient to you and your organization between now and March 2023. After March 2023, directories will be updated automatically.

Users of SAP Solution Manager can now use automated deployments to provision it, in accordance with AWS and SAP best practices, to both single-node and distributed architectures using AWS Launch Wizard.

AWS Activate is a program that offers free tools, resources, and the opportunity to apply for credits to smaller early stage businesses and also more advanced digital businesses, helping them get started quickly on AWS. The program is now open to any self-identified startup.

Amazon QuickSight users who employ row-level security (RLS) to control access to restricted datasets will be interested in a new feature that enables you to ask questions against topics in these datasets. User-based rules control the answers received to questions and any auto-complete suggestions provided when the questions are being framed. This ensures that users only ever receive answer data that they are granted permission to access.

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Other AWS News
This interesting blog post focus on the startup Pieces Technologies, who are putting predictive artificial intelligence (AI) and machine learning (ML) tools to work on AWS to predict and offer clinical insights on patient outcomes such as such as projected discharge dates, anticipated clinical and non-clinical barriers to discharge, and risk of readmission. To help healthcare teams work more efficiently, the insights are provided in natural language and seek to optimize overall clarity of a patient’s clinical issues.

As usual, there’s another AWS open-source and updates newsletter. The newsletter is published weekly to bring you up to date on the latest news on open-source projects, posts, and events.

Upcoming Events
Speaking of upcoming events, the following are some you may be interested in joining, especially if you work with .NET:

Looking to modernize .NET workloads using Windows containers on AWS? There’s a free webinar, with follow-along lab, in just a couple of days on October 20. You can find more details and register here.

My .NET colleagues are also hosting another webinar on November 2 related to building modern .NET applications on AWS. If you’re curious about the hosting and development capabilities of AWS for .NET applications, this is a webinar you should definitely check out. You’ll find further information and registration here.

And finally, a reminder that reserved seating for sessions at AWS re:Invent 2022 is now open. We’re now just 6 weeks away from the event! There are lots of great sessions for your attention, but those of particular interest to me are the ones related to .NET, and at this year’s event we have seven breakouts, three chalk talks, and a workshop for you. You can find all the details using the .NET filter in the session catalog (the sessions all start with the prefix XNT, by the way).

That’s all for this week. Check back next Monday for another AWS Week in Review!

— Steve

How Epos Now modernized their data platform by building an end-to-end data lake with the AWS Data Lab

Post Syndicated from Debadatta Mohapatra original https://aws.amazon.com/blogs/big-data/how-epos-now-modernized-their-data-platform-by-building-an-end-to-end-data-lake-with-the-aws-data-lab/

Epos Now provides point of sale and payment solutions to over 40,000 hospitality and retailers across 71 countries. Their mission is to help businesses of all sizes reach their full potential through the power of cloud technology, with solutions that are affordable, efficient, and accessible. Their solutions allow businesses to leverage actionable insights, manage their business from anywhere, and reach customers both in-store and online.

Epos Now currently provides real-time and near-real-time reports and dashboards to their merchants on top of their operational database (Microsoft SQL Server). With a growing customer base and new data needs, the team started to see some issues in the current platform.

First, they observed performance degradation for serving the reporting requirements from the same OLTP database with the current data model. A few metrics that needed to be delivered in real time (seconds after a transaction was complete) and a few metrics that needed to be reflected in the dashboard in near-real-time (minutes) took several attempts to load in the dashboard.

This started to cause operational issues for their merchants. The end consumers of reports couldn’t access the dashboard in a timely manner.

Cost and scalability also became a major problem because one single database instance was trying to serve many different use cases.

Epos Now needed a strategic solution to address these issues. Additionally, they didn’t have a dedicated data platform for doing machine learning and advanced analytics use cases, so they decided on two parallel strategies to resolve their data problems and better serve merchants:

  • The first was to rearchitect the near-real-time reporting feature by moving it to a dedicated Amazon Aurora PostgreSQL-Compatible Edition database, with a specific reporting data model to serve to end consumers. This will improve performance, uptime, and cost.
  • The second was to build out a new data platform for reporting, dashboards, and advanced analytics. This will enable use cases for internal data analysts and data scientists to experiment and create multiple data products, ultimately exposing these insights to end customers.

In this post, we discuss how Epos Now designed the overall solution with support from the AWS Data Lab. Having developed a strong strategic relationship with AWS over the last 3 years, Epos Now opted to take advantage of the AWS Data lab program to speed up the process of building a reliable, performant, and cost-effective data platform. The AWS Data Lab program offers accelerated, joint-engineering engagements between customers and AWS technical resources to create tangible deliverables that accelerate data and analytics modernization initiatives.

Working with an AWS Data Lab Architect, Epos Now commenced weekly cadence calls to come up with a high-level architecture. After the objective, success criteria, and stretch goals were clearly defined, the final step was to draft a detailed task list for the upcoming 3-day build phase.

Overview of solution

As part of the 3-day build exercise, Epos Now built the following solution with the ongoing support of their AWS Data Lab Architect.

Epos Now Arch Image

The platform consists of an end-to-end data pipeline with three main components:

  • Data lake – As a central source of truth
  • Data warehouse – For analytics and reporting needs
  • Fast access layer – To serve near-real-time reports to merchants

We chose three different storage solutions:

  • Amazon Simple Storage Service (Amazon S3) for raw data landing and a curated data layer to build the foundation of the data lake
  • Amazon Redshift to create a federated data warehouse with conformed dimensions and star schemas for consumption by Microsoft Power BI, running on AWS
  • Aurora PostgreSQL to store all the data for near-real-time reporting as a fast access layer

In the following sections, we go into each component and supporting services in more detail.

Data lake

The first component of the data pipeline involved ingesting the data from an Amazon Managed Streaming for Apache Kafka (Amazon MSK) topic using Amazon MSK Connect to land the data into an S3 bucket (landing zone). The Epos Now team used the Confluent Amazon S3 sink connector to sink the data to Amazon S3. To make the sink process more resilient, Epos Now added the required configuration for dead-letter queues to redirect the bad messages to another topic. The following code is a sample configuration for a dead-letter queue in Amazon MSK Connect:

Because Epos Now was ingesting from multiple data sources, they used Airbyte to transfer the data to a landing zone in batches. A subsequent AWS Glue job reads the data from the landing bucket , performs data transformation, and moves the data to a curated zone of Amazon S3 in optimal format and layout. This curated layer then became the source of truth for all other use cases. Then Epos Now used an AWS Glue crawler to update the AWS Glue Data Catalog. This was augmented by the use of Amazon Athena for doing data analysis. To optimize for cost, Epos Now defined an optimal data retention policy on different layers of the data lake to save money as well as keep the dataset relevant.

Data warehouse

After the data lake foundation was established, Epos Now used a subsequent AWS Glue job to load the data from the S3 curated layer to Amazon Redshift. We used Amazon Redshift to make the data queryable in both Amazon Redshift (internal tables) and Amazon Redshift Spectrum. The team then used dbt as an extract, load, and transform (ELT) engine to create the target data model and store it in target tables and views for internal business intelligence reporting. The Epos Now team wanted to use their SQL knowledge to do all ELT operations in Amazon Redshift, so they chose dbt to perform all the joins, aggregations, and other transformations after the data was loaded into the staging tables in Amazon Redshift. Epos Now is currently using Power BI for reporting, which was migrated to the AWS Cloud and connected to Amazon Redshift clusters running inside Epos Now’s VPC.

Fast access layer

To build the fast access layer to deliver the metrics to Epos Now’s retail and hospitality merchants in near-real time, we decided to create a separate pipeline. This required developing a microservice running a Kafka consumer job to subscribe to the same Kafka topic in an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. The microservice received the messages, conducted the transformations, and wrote the data to a target data model hosted on Aurora PostgreSQL. This data was delivered to the UI layer through an API also hosted on Amazon EKS, exposed through Amazon API Gateway.

Outcome

The Epos Now team is currently building both the fast access layer and a centralized lakehouse architecture-based data platform on Amazon S3 and Amazon Redshift for advanced analytics use cases. The new data platform is best positioned to address scalability issues and support new use cases. The Epos Now team has also started offloading some of the real-time reporting requirements to the new target data model hosted in Aurora. The team has a clear strategy around the choice of different storage solutions for the right access patterns: Amazon S3 stores all the raw data, and Aurora hosts all the metrics to serve real-time and near-real-time reporting requirements. The Epos Now team will also enhance the overall solution by applying data retention policies in different layers of the data platform. This will address the platform cost without losing any historical datasets. The data model and structure (data partitioning, columnar file format) we designed greatly improved query performance and overall platform stability.

Conclusion

Epos Now revolutionized their data analytics capabilities, taking advantage of the breadth and depth of the AWS Cloud. They’re now able to serve insights to internal business users, and scale their data platform in a reliable, performant, and cost-effective manner.

The AWS Data Lab engagement enabled Epos Now to move from idea to proof of concept in 3 days using several previously unfamiliar AWS analytics services, including AWS Glue, Amazon MSK, Amazon Redshift, and Amazon API Gateway.

Epos Now is currently in the process of implementing the full data lake architecture, with a rollout to customers planned for late 2022. Once live, they will deliver on their strategic goal to provide real-time transactional data and put insights directly in the hands of their merchants.


About the Authors

Jason Downing is VP of Data and Insights at Epos Now. He is responsible for the Epos Now data platform and product direction. He specializes in product management across a range of industries, including POS systems, mobile money, payments, and eWallets.

Debadatta Mohapatra is an AWS Data Lab Architect. He has extensive experience across big data, data science, and IoT, across consulting and industrials. He is an advocate of cloud-native data platforms and the value they can drive for customers across industries.

Essential security for everyone: Building a secure AWS foundation

Post Syndicated from Byron Pogson original https://aws.amazon.com/blogs/security/essential-security-for-everyone-building-a-secure-aws-foundation/

In this post, I will show you how teams of all sizes can gain access to world-class security in the cloud without a dedicated security person in your organization. I look at how small teams can build securely on Amazon Web Services (AWS) in a way that’s cost effective and time efficient. I show you the key elements to create a foundation with good security controls, and how you can then use that foundation as a base to build a secure workload upon. In this post, I will also share a lab guide to get you started today. It may look like a lot of work but I ran this as a day-long workshop across Australia in 2019 reaching many start-ups and small businesses. The majority of them implemented the guide by mid-afternoon.

Many large organizations run their regulated workloads on AWS and customers of all sizes have the same security controls available to them. These large organizations have gone through a rigorous process to ensure that the right security controls are available to them. If you go to the AWS Startups Blog, you can read the story of two Australian customers and their journeys to set up a secure foundation on AWS: Tic:Toc, an Australian scaleup in the financial services industry and FYI, a start-up with their document and process management system for accounting practices.

The Well-Architected Framework has been developed to help cloud architects build secure, high performance, resilient, and efficient infrastructure for their applications. Based on five pillars—operational excellence, security, reliability, performance efficiency, and cost optimization—the Framework provides a consistent approach for customers and partners to evaluate architectures and implement designs that will scale over time. In this post, I will discuss the key areas from the security pillar to help you build a secure foundation. These areas are:

  • Security foundations. You can use an AWS account as a coarse boundary for isolating resources and use cross-account roles to share common infrastructure. Protect your AWS accounts and use tools like AWS Control Tower to help you get started quickly.
  • Identity and access management. Be deliberate about who has access to what.
  • Detection. Start with the implementation of baseline logging and monitoring. Do this in a way that’s implemented automatically so it is scalable. When incidents occur, this will help to ensure that basic log data is in place to aid your investigations. Configure alerts for key events and define your response plan so you are prepared to take action.
  • Infrastructure and data protection. Apply defense in depth, starting with the features that AWS provides you, to help build a secure application.
  • Incidence response. Ensure your team is prepared to respond to incidents by educating your team, creating a response plan, simulating scenarios so your team knows what to do before it happens and iterating to improve your plan.

Small teams want to move fast and deliver value. To support that, you want to build a secure foundation. This post focuses on the key initial steps to help you achieve that. To help guide you through the content in this post and implement your foundation faster, we have a Quick Steps to Security Success quest in our Well-Architected Labs.

Security Foundations

With a strong foundation in place to support your workload, you can look at how to build securely on top of it. Security is part of every feature, not a separate feature to be implemented later. Teams need to be comfortable with the idea that a feature isn’t complete just when it’s tested and in production. Adjust your culture to think of complete as meaning tested and secure in production.

An AWS account is a boundary within which resources are deployed. You can open multiple AWS accounts for different purposes. For example, to separate different applications you operate by splitting different workloads across multiple environments in different accounts, to provide developer sandbox accounts or to isolated resources such as a security account. A workload is a collection of systems and applications to meet a specific business objective and could be a useful guide for determining what needs to be deployed into separate accounts. From a security point of view, being able to use an AWS account as a boundary helps isolate different parts of your workloads. The account boundary acts as a coarse isolation boundary and you have to be deliberate about how you allow access to resources in it. For human access, this can form a basis for providing least privilege access – an IAM best practice for ensuring that users only have permissions required to fulfil their tasks.

A best practice is to keep users away from data – least privilege could start with not providing access to the production environment. One way to achieve this is to create a separate account for your production workload and ensure that all regular operations are performed at a distance through tools such as pipelines or ticketing systems. Where human access is essential, only grant temporary human access for a fixed period of time. In addition to limited IAM policies, you can give people access only to AWS accounts containing the workload they need access to. For machine-to-machine access you can apply the same concepts and use cross-account access.

At a minimum, it’s a best practice to have a separate organizational management account that’s only used to establish controls across your set of accounts and for configuring identity and access management within your organization. The same identity configuration is then used across accounts. Also, set up a dedicated account for logging to more securely store data such as audit logs. To increase security, create an audit account that has read-only access to the logs and other accounts used by your security team. Then create different accounts for different environments and workloads.

The easiest way to get started creating and organizing accounts is to use AWS Control Tower, which will set up a separated logging and audit account, an AWS Single Sign-On (AWS SSO) directory—which supports identity federation with SAML 2.0—as well as a few basic guardrails. AWS SSO can also give users a single view of all the accounts and roles within those accounts that they have access to. AWS Control Tower also includes a basic account-creation tool—the Account Factory—that you can use to create additional accounts within your AWS account structure.

Guardrails are an important mechanism that customers can implement to help maintain security in the cloud. AWS Control Tower provides two types of guardrails: preventive and detective.

Preventive guardrails are designed to prevent users from performing certain actions; for example, preventing a user from disabling security logging. You can implement preventative guardrails through AWS Control Tower, which provides a feature of AWS Organizations called Service Control Policies (SCP) that you can use to set the maximum boundary for what is allowed in an account. These guardrails are either enforced or disabled.

Detective guardrails look at the state of resources in an account using AWS Config rules and indicate if resources are compliant to those rules or not. For example, looking for Amazon Simple Storage Service (Amazon S3) buckets that are publicly accessible. If you need to have data publicly available, be deliberate about how you do it.

AWS Control Tower has a number of mandatory guardrails that are necessary for the operation of AWS Control Tower as well as a number of strongly recommended and elective guardrails. The strongly recommended and elective guardrails help to ensure that you’re building a strong security posture as soon as you enable them.

There is no additional charge to use AWS Control Tower. However, when you set up AWS Control Tower, you will begin to incur costs for AWS services configured to set up your landing zone and mandatory guardrails. For further details see the AWS Control Tower pricing.

Identity and access management

Identity forms the basis of validating that users are who they say they are and how you give them permission to operate in your environment.

When you sign up for an AWS account, the first login you receive is the root user credentials. The root user credentials are very powerful and allows full access to all resources in the account. It’s critical that you protect your root account from unauthorized access, starting with multi-factor authentication. Multi-factor authentication uses a password (something you know) plus something you have (such as a one-time key or a hardware token) to create a more secure login. After you set up multi-factor authentication, both factors are required to access the root account. After that, use the root account only in emergencies, not in day-to-day operations. Moving from using the root account to using centralized identities allows you to manage your identities centrally and tie every action taken in your environment back an individual. The most effective way to enable connecting all actions to individual users is through federation.

Federation lets you reuse your existing identities, such as those you have in your organization’s identity directory. When a user joins your organization, the first thing you’re likely to do is to give them an identity (so they can do things like access your email systems) and when they leave, you would remove that identity and therefore the access. By federating your AWS accounts with your existing identity directory, you can use the same mechanisms that are tied to your business processes to provide AWS access. Using tools like AWS Single Sign-On (AWS SSO) enables you to quickly federate access for your users and maintain a mapping of the AWS IAM roles (an identity with specific permissions that can be assigned to or assumed by other identities) they have access to across accounts in your organization. If you don’t have an existing identity store you can still achieve a central identity store by using the built-in provider in AWS SSO. When you are assigning permissions, be deliberate with what access you give different users. Ensure that you’re creating and assigning roles based on least privilege—giving only as much access as users need to perform their tasks.

IAM is a feature of your AWS account provided at no additional charge and AWS SSO is offered at no extra charge. Implementing SSO is a low effort way to build a strong identity foundation. If you’ve been operating for a while on AWS, you should perform an audit of your existing AWS Identity and Access Management (IAM) users with a goal to move to a centralized model. An audit of your IAM resources (and centralized identities) will help you to understand who has access to your AWS environment, clear unused credentials, and check that users are assigned permissions that are relevant for their role. IAM access advisor will allow you to see when services were last accessed. Tools such as the IAM Access Analyzer will help you identify the resources in your organization and accounts, such as Amazon S3 buckets or IAM roles, that are shared with an external entity. At the same time, make sure that your account contacts are up to date so that you don’t miss any important information from AWS. You can update these details under AWS billing and management in the console.

Detection

After some baseline controls are in place, you need to add controls to ensure that you are aware of what is happening in the environment and that actions are logged. To help you with governance, compliance and auditing your AWS environment you can configure AWS CloudTrail. A CloudTrail log shows you who attempted to take what actions against resources in your AWS account and if the action was allowed or denied. Having a secure store of these logs provides you with an audit history of who did what in your environment. AWS Control Tower configures a secure log store for you in the logging account.

Amazon GuardDuty is a security service that uses intelligent threat detection to alert you to unusual activity in your environment. GuardDuty uses CloudTrail logs to alert you to malicious activity and unauthorized behavior in addition to DNS logs and VPC Flow Logs—which are similar to network flow logs—to analyze the behavior of your workload. GuardDuty builds a baseline over time of activity in your account and alerts you when behavior that strays from the baseline is detected. For example, GuardDuty sends an alert when a user tries to escalate their privilege. These events can be configured in Amazon CloudWatch Events for alerting and triggering automatic actions—for example by triggering an AWS Lambda function to disable the user trying to escalate their privilege until you can contact them.

Implementing manual dashboards though Amazon CloudWatch or those provided with detective tools such as Amazon GuardDuty can give you a clear idea of what’s happening in your environment, but you should also configure alerting for key events. An initial, temporary way of achieving this could be by creating an Amazon CloudWatch Rule with an Amazon SNS topic as the destination and have your team subscribe their email to the SNS topic. As part of setting up alerts, ensure that there’s a remediation process defined for each alert that includes what action to take when an alert is triggered. In the longer term, as your cloud skills mature, you can evolve this to filter out alerts appropriately and iterate your response and remediation processes.

Having a single view of what’s happening in your infrastructure across all accounts and relevant regions gives you a clear picture of the overall state of your environment. Consider using AWS Security Hub to bring together alerts from GuardDuty, other AWS services such as AWS Inspector (for network availability and common vulnerabilities and exposures analysis), and partner products. Security Hub lets you consolidate findings from multiple sources and normalize them so they are comparable. This allows you to have a single view of where you need to take action and what high priority actions are required. Security Hub also allows you to enable compliance checks on your AWS infrastructure to help you adhere to best practices. A great starting point is AWS Foundational Security Best Practices standard.

Both GuardDuty and Security Hub include a free trial period and scale with usage after you turn them on. You can use the trial period to estimate what they will cost to use in all your AWS accounts.

Infrastructure and data protection

Build protection in layers and be aware of what features are available in the services that you’re using. Many AWS services include a specific section on security as part of their developer documentation. Before you add an AWS service, read the security section of the documentation and understand what options are available to you. Make sure that you are understand the cloud-native AWS security services that integrate with the services you use. AWS Key Management Service integrates with many AWS services to enable encryption at rest. For example, you can enable default encryption for all EBS volumes in a region. AWS Certificate Manager provides public certificates which integrate with Elastic Load Balancing and Amazon CloudFront to encrypt data in transit. Public SSL/TLS certificates provisioned through AWS Certificate Manager are free. You pay only for the AWS resources you create to run your application. You can implement AWS WAF (web application firewall) and AWS Shield to protect your HTTPS endpoints. Where implement services that manage resources, such as Amazon RDS, AWS Lambda, and Amazon ECS, to reduce your security maintenance tasks as part of the shared responsibility model.

Incident response

Once you have your baseline security controls in place your team needs to be prepared to respond effectively during an incident. This includes designing your incident response goals, educating your team and preparing to respond. Simulating events helps the team to learn your processes and tools. Always iterate to improve the process for the future. As a start, consider using the GuardDuty finding types as the basis for what you should be able to respond to. Have a look through the finding types, identify which findings are most applicable and write a runbook outlining the steps on how you would respond. For each finding type, test your response process. In doing so your team will ensure they have the right tools available, the right emergency access, and know who they need to escalate to and collaborate with. By simulating your response process, your team becomes practiced in how to respond and will reduce the time to recovery if an incident should occur.

When comfortable with the process, automate it. For example, create a Lambda function to perform remediation without you having to wake up in the middle of the night to take action. This can be built up over time as you build a baseline of events. Spending some time thinking through priority events for your environment will help you develop a playbook to respond to them. When you’re comfortable with what incident responses you need, you can automate those responses so remediation is triggered when an event occurs, though you may also want a human to verify before triggering a potentially impactful response.

For example, one of the GuardDuty finding types identifies when an EC2 instance is querying an IP address that is associated with cryptocurrency-related activity. The suggested remediation is to investigate the instance, create a snapshot, consider stopping and starting a new instance and raise a support case. Your runbook could outline how to do each of those steps or you could use CloudWatch to trigger a Lambda function which will place the instance in an isolated security group with no internet access for investigation later. Further examples of automation can be found in the Getting Hands on with Amazon GuardDuty labs.

Conclusion

In this post, I’ve shown you some of the techniques and services that can be used to build a secure foundation. Build a strong security foundation and have a multi-account strategy that allows you to isolate different workloads within your organization. A strong identity foundation ensures that you know who is doing what in your environment. Logging and monitoring ensures that you are ready to take action. Building security in layers and using the service features available as you build ensures that you’re using the security controls available to all customers on the platform. Be prepared to respond to incidents and regularly practice your response process so your team is ready if an incident should occur.

A secure foundation is just the start. Remember that security is not a separate feature and new features are not complete until they’re tested and securely in production. Build a security culture of continuous improvement, and take action to ensure that you remain secure as you build out your workloads. Iterate to continue to reduce risk. Use the AWS Well-Architected Tool which allows you and your team to review your workload against best practices which can be paired with the Well-Architected labs for hands on learning. As mentioned above, a lab to help you implement the content in this blog post yourself can be found in the Quick Steps to Security Success lab. Don’t forget that you can also read stories from two AWS customers—Tic:Toc and FYI—on the AWS Startups Blog.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on one of the AWS Security, Identity, and Compliance forums or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Byron Pogson

Byron helps organizations use technology to solve their biggest problems. With Amazon Web Services he works with teams across West and South Australia to help refine their technology strategies in this fast-changing environment. You can often find him talking about security best practices to help organizations of all sizes to move fast and stay secure.