Tag Archives: auditing

AWS achieves its second ISMAP authorization in Japan

2022-09-21 Hidetoshi Takeuchi

Post Syndicated from Hidetoshi Takeuchi original https://aws.amazon.com/blogs/security/aws-achieves-its-second-ismap-authorization-in-japan/

Earning and maintaining customer trust is an ongoing commitment at Amazon Web Services (AWS). Our customers’ security requirements drive the scope and portfolio of the compliance reports, attestations, and certifications we pursue. We’re excited to announce that AWS has achieved authorization under the Information System Security Management and Assessment Program (ISMAP) program, effective from April 1, 2022 to March 31, 2023. The authorization scope covers a total of 145 AWS services (an increase of 22 services over the previous authorization) across 22 AWS Regions, including the Asia Pacific (Tokyo) Region and the Asia Pacific (Osaka) Region. This is the second time AWS has undergone an assessment since ISMAP was first published by the ISMAP steering committee in March 2020.

ISMAP is a Japanese government program for assessing the security of public cloud services. The purpose of ISMAP is to provide a common set of security standards for cloud service providers (CSPs) to comply with as a baseline requirement for government procurement. ISMAP introduces security requirements for cloud domains, practices, and procedures that CSPs must implement. CSPs must engage with an ISMAP-approved third-party assessor to assess compliance with the ISMAP security requirements in order to apply as an ISMAP-registered CSP. The ISMAP program will evaluate the security of each CSP and register those that satisfy the Japanese government’s security requirements. Upon successful ISMAP registration of CSPs, government procurement departments and agencies can accelerate their engagement with the registered CSPs and contribute to the smooth introduction of cloud services in government information systems.

The achievement of this authorization demonstrates the proactive approach AWS has taken to help customers meet compliance requirements set by the Japanese government and to deliver secure AWS services to our customers. Service providers and customers of AWS can use the ISMAP authorization of AWS services to support their own ISMAP authorization programs. The full list of 145 ISMAP-authorized AWS services is available on the AWS Services in Scope by Compliance Program webpage, and you can also use the ISMAP Customer Package on AWS Artifact. You can confirm the AWS ISMAP authorization status and find detailed scope information on the ISMAP Portal.

As always, we are committed to bringing new services and Regions into the scope of our ISMAP program, based on your business needs. If you have any questions, don’t hesitate to contact your AWS Account Manager.

If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS achieves TISAX certification (Information with Very High Protection Needs (AL3)

2022-07-15 Janice Leung

Post Syndicated from Janice Leung original https://aws.amazon.com/blogs/security/aws-achieves-tisax-certification-information-with-very-high-protection-needs-al3/

We’re excited to announce the completion of the Trusted Information Security Assessment Exchange (TISAX) certification on June 30, 2022 for 19 AWS Regions. These Regions achieved the Information with Very High Protection Needs (AL3) label for the control domains Information Handling and Data Protection. This alignment with TISAX requirements demonstrates our continued commitment to adhere to the heightened expectations for cloud service providers. AWS automotive customers can run their applications in the AWS Cloud certified Regions in confidence.

The following 19 Regions are currently TISAX certified:

US East (Ohio)
US East (Northern Virginia)
US West (Oregon)
Africa (Cape Town)
Asia Pacific (Hong Kong)
Asia Pacific (Mumbai)
Asia Pacific (Osaka)
Asia Pacific (Korea)
Asia Pacific (Singapore)
Asia Pacific (Sydney)
Asia Pacific (Tokyo)
Canada (Central)
Europe (Frankfurt)
Europe (Ireland)
Europe (London)
Europe (Milan)
Europe (Paris)
Europe (Stockholm)
South America (Sao Paulo)

TISAX is a European automotive industry-standard information security assessment (ISA) catalog based on key aspects of information security, such as data protection and connection to third parties.

AWS was evaluated and certified by independent third-party auditors on June 30, 2022. The Certificate of Compliance demonstrating the AWS compliance status is available on the European Network Exchange (ENX) Portal (the scope ID and assessment ID are SM22TH and AYA2D4-1, respectively) and through AWS Artifact. AWS Artifact is a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact.

For up-to-date information, including when additional Regions are added, see the AWS Compliance Program, and choose TISAX.

AWS strives to continuously bring services into scope of its compliance programs to help you meet your architectural and regulatory needs. Please reach out to your AWS account team if you have questions or feedback about TISAX compliance.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS achieves HDS certification to three additional Regions

2022-07-15 Janice Leung

Post Syndicated from Janice Leung original https://aws.amazon.com/blogs/security/aws-achieves-hds-certification-to-three-additional-regions/

We’re excited to announce that three additional AWS Regions—Asia Pacific (Korea), Europe (London), and Europe (Stockholm)—have been granted the Health Data Hosting (Hébergeur de Données de Santé, HDS) certification. This alignment with the HDS requirements demonstrates our continued commitment to adhere to the heightened expectations for cloud service providers. AWS customers who handle personal health data can be hosted in the AWS Cloud certified Regions with confidence.

The following 16 Regions are now in scope of this certification:

US East (Ohio)
US East (Northern Virginia)
US West (Northern California)
US West (Oregon)
Asia Pacific (Mumbai)
Asia Pacific (Korea)
Asia Pacific (Singapore)
Asia Pacific (Sydney)
Asia Pacific (Tokyo)
Canada (Central)
Europe (Frankfurt)
Europe (Ireland)
Europe (London)
Europe (Paris)
Europe (Stockholm)
South America (Sao Paulo)

Introduced by the French governmental agency for health, Agence Française de la Santé Numérique (ASIP Santé), HDS certification aims to strengthen the security and protection of personal health data. Achieving this certification demonstrates that AWS provides a framework for technical and governance measures to secure and protect personal health data, governed by French law.

AWS was evaluated and certified by independent third-party auditors on June 30, 2022. The Certificate of Compliance demonstrating the AWS compliance status is available on the Agence du Numérique en Santé (ANS) website and through AWS Artifact. AWS Artifact is a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact.

For up-to-date information, including when additional Regions are added, see the AWS Compliance Program, and choose HDS.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

C5 Type 2 attestation report now available with 141 services in scope

2022-02-11 Mercy Kanengoni

Post Syndicated from Mercy Kanengoni original https://aws.amazon.com/blogs/security/c5-type-2-attestation-report-now-available-with-141-services-in-scope/

Amazon Web Services (AWS) is pleased to announce the issuance of the new Cloud Computing Compliance Controls Catalogue (C5) Type 2 attestation report. We added 18 additional services and service features to the scope of the 2021 report.

Germany’s national cybersecurity authority, Bundesamt für Sicherheit in der Informationstechnik (BSI), established C5 to define a reference standard for German cloud security requirements. The C5 Type 2 report covers the time period from October 1, 2020, through September 30, 2021. It was issued by an independent third-party attestation organization, and assesses the design and the operational effectiveness of AWS’s controls against the new version C5:2020’s basic and additional criteria.

Customers in Germany and other European countries can use AWS’s attestation report to confirm that AWS meets the security requirements of the C5:2020 framework, and to review the details of the tested controls. This attestation demonstrates our commitment to meet and exceed the security expectations for cloud service providers set by the BSI.

AWS has added the following 18 services and service features to the new C5 scope:

AWS App Mesh
AWS Audit Manager
Amazon Augmented AI [Excludes Public Workforce and Vendor Workforce for all features]
AWS Chatbot
AWS Cloud9
AWS Cloud Map
Amazon CloudWatch SDK Metrics for Enterprise Support
Amazon Detective
Amazon EC2 Image Builder
Amazon FinSpace
Amazon Fraud Detector
AWS Glue DataBrew
Amazon Kendra
Amazon KeySpaces (for Apache Cassandra)
Amazon Macie Classic
AWS Network Firewall
AWS Single Sign-On
Amazon Timestream

You can see a current list of the services in scope for C5 on the AWS Services in Scope by Compliance Program page.

The C5 report and Continuing Operations Letter is available to AWS customers through AWS Artifact. For more information, see Cloud Computing Compliance Controls Catalogue (C5).

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Security Hub forum. To start your 30-day free trial of Security Hub, visit AWS Security Hub.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS achieves GSMA Security Certification for Europe (Paris) Region

2021-11-10 Janice Leung

Post Syndicated from Janice Leung original https://aws.amazon.com/blogs/security/aws-achieves-gsma-security-certification-for-europe-paris-region/

We continue to expand the scope of our assurance programs at Amazon Web Services (AWS) and are pleased to announce that our Europe (Paris) Region is now certified by the GSM Association (GSMA) under its Security Accreditation Scheme Subscription Management (SAS-SM) with scope Data Center Operations and Management (DCOM). This is an addition to our US East (Ohio) Region, which received certification in September 2021. This alignment with GSMA requirements demonstrates our continuous commitment to adhere to the heightened expectations for cloud service providers. AWS customers who provide embedded Universal Integrated Circuit Card (eUICC) for mobile devices can run their remote provisioning applications with confidence in the AWS Cloud in the GSMA-certified Regions.

As of this writing, 72 services offered in the Europe (Paris) Region and 128 services offered in the US East (Ohio) Region are in scope of this certification. For up-to-date information, including when additional services are added, see the AWS Services in Scope by Compliance Program and choose GSMA.

AWS was evaluated by independent third-party auditors chosen by GSMA. The Certificate of Compliance that shows that AWS achieved GSMA compliance status is available on the GSMA Website and through AWS Artifact. AWS Artifact is a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact.

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page. Or if you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS achieves GSMA security certification for US East (Ohio) Region

2021-09-24 Janice Leung

Post Syndicated from Janice Leung original https://aws.amazon.com/blogs/security/aws-achieves-gsma-security-certification-for-us-east-ohio-region/

We continue to expand the scope of our assurance programs at Amazon Web Services (AWS) and are pleased to announce that our US East (Ohio) Region (us-east-2) is now certified by the GSM Association (GSMA) under its Security Accreditation Scheme Subscription Management (SAS-SM) with scope Data Center Operations and Management (DCOM). This alignment with GSMA requirements demonstrates our continuous commitment to adhere to the heightened expectations for cloud service providers. AWS customers who provide embedded Universal Integrated Circuit Card (eUICC) for mobile devices can run their remote provisioning applications with confidence in the AWS Cloud in the GSMA-certified US East (Ohio) Region.

As of this writing, 128 services offered in the US East (Ohio) Region are in scope of this certification. For up-to-date information, including when additional services are added, see the AWS Services in Scope by Compliance Program and choose GSMA.

AWS was evaluated by independent third-party auditors chosen by GSMA. The Certificate of Compliance illustrating the AWS GSMA compliance status is available on the GSMA website and through AWS Artifact. AWS Artifact is a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

C5 Type 2 attestation report now available with one new Region and 123 services in scope

2021-03-03 Mercy Kanengoni

Post Syndicated from Mercy Kanengoni original https://aws.amazon.com/blogs/security/c5-type-2-attestation-report-available-one-new-region-123-services-in-scope/

Amazon Web Services (AWS) is pleased to announce the issuance of the 2020 Cloud Computing Compliance Controls Catalogue (C5) Type 2 attestation report. We added one new AWS Region (Europe-Milan) and 21 additional services and service features to the scope of the 2020 report.

Germany’s national cybersecurity authority, Bundesamt für Sicherheit in der Informationstechnik (BSI), established C5 to define a reference standard for German cloud security requirements. Customers in Germany and other European countries can use AWS’s attestation report to help them meet local security requirements of the C5 framework.

The C5 Type 2 report covers the time period October 1, 2019, through September 30, 2020. It was issued by an independent third-party attestation organization and assesses the design and the operational effectiveness of AWS’s controls against C5’s basic and additional criteria. This attestation demonstrates our commitment to meet the security expectations for cloud service providers set by the BSI in Germany.

We continue to add new Regions and services to the C5 compliance scope so that you have more services to choose from that meet regulatory and compliance requirements. AWS has added the Europe (Milan) Region and the following 21 services and service features to this year’s C5 scope:

You can see a current list of the services in scope for C5 on the AWS Services in Scope by Compliance Program page. The C5 report and Continuing Operations Letter is available to AWS customers through AWS Artifact. For more information, see Cloud Computing Compliance Controls Catalogue (C5).

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS is the first global cloud service provider to comply with the new K-ISMS-P standard

2021-01-26 Seulun Sung

Post Syndicated from Seulun Sung original https://aws.amazon.com/blogs/security/aws-is-the-first-global-cloud-service-provider-to-comply-with-the-new-k-isms-p-standard/

We’re excited to announce that Amazon Web Services (AWS) has achieved certification under the Korea-Personal Information & Information Security Management System (K-ISMS-P) standard (effective from December 16, 2020 to December 15, 2023). The assessment by the Korea Internet & Security Agency (KISA) covered the operation of infrastructure (including compute, storage, networking, databases, and security) in the AWS Asia Pacific (Seoul) Region. AWS was the first global cloud service provider (CSP) to obtain K-ISMS certification (the previous version of K-ISMS-P) back in 2017. Now AWS is the first global CSP to achieve compliance with the K-ISMS portion of the new K-ISMS-P standard.

Sponsored by KISA and affiliated with the Korean Ministry of Science and ICT (MSIT), K-ISMS-P serves as a standard for evaluating whether enterprises and organizations operate and manage their information security management systems consistently and securely, such that they thoroughly protect their information assets. The new K-ISMS-P standard combined the K-ISMS and K-PIMS (Personal Information Management System) standards with updated control items. Accordingly, the new K-ISMS certification and K-ISMS-P certification (personal information–focused) are introduced under the updated standard.

In this year’s audit, 110 services running in the Asia Pacific (Seoul) Region are included. The newly launched Availability Zone in 2020 is also added to the certification scope.

This certification helps enterprises and organizations across South Korea, regardless of industry, meet KISA compliance requirements more efficiently. Achieving this certification demonstrates the proactive approach AWS has taken to meet compliance set by the South Korean government and to deliver secure AWS services to customers. In addition, we’ve launched Quick Start and Operational Best Practices (conformance pack) pages to provide customers with a compliance framework that they can utilize for their K-ISMS-P compliance needs. Enterprises and organizations can use these toolkits and AWS certification to reduce the effort and cost of getting their own K-ISMS-P certification. You can download the AWS K-ISMS certification under the K-ISMS-P standard from AWS Artifact. To learn more about the AWS K-ISMS certification, see the AWS K-ISMS page. If you have any questions, don’t hesitate to contact your AWS Account Manager.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Announcing Cloud Audit Academy AWS-specific for audit and compliance teams

2020-12-01 Chad Woolf

Post Syndicated from Chad Woolf original https://aws.amazon.com/blogs/security/announcing-cloud-audit-academy-aws-specific-for-audit-and-compliance-teams/

Today, I’m pleased to announce the launch of Cloud Audit Academy AWS-specific (CAA AWS-specific). This is a new, accelerated training program for auditing AWS Cloud implementations, and is designed for auditors, regulators, or anyone working within a control framework.

Over the past few years, auditing security in the cloud has become one of the fastest growing questions among Amazon Web Services (AWS) customers, across multiple industries and all around the world. Here are the two pain points that I hear about most often:

Engineering teams want to move regulatory frameworks compliant workloads to AWS to take advantage of its innovation capabilities, but security and risk teams are uncertain how AWS can help them meet their compliance requirements through audits.
Compliance teams want to effectively audit the cloud environments and take advantage of the available security control options that are built into the cloud, but the legacy audit processes and control frameworks are built for an on-premises environment. The differences require some reconciliation and improvement work to be done on compliance programs, audit processes, and auditor training.

To help address these issues for not only AWS customers but for any auditor or compliance team facing cloud migration, we announced Cloud Audit Academy Cloud Agnostic (CAA Cloud Agnostic) at re:Inforce 2019. This foundational, first-of-its-kind, course provides baseline knowledge on auditing in the cloud and in understanding the differences in control operation, design, and auditing. It is cloud agnostic and can benefit security and compliance professionals in any industry—including independent third-party auditors. Since its launch in June 2019, 1,400 students have followed this cloud audit learning path, with 91 percent of participants saying that they would recommend the workshop to others.

So today we’re releasing the next phase of that education program, Cloud Audit Academy AWS-specific. Offered virtually or in-person, CAA AWS-specific is an instructor-led workshop on addressing risks and auditing security in the AWS Cloud, with a focus on the security and audit tools provided by AWS. All instructors have professional audit industry experience, current audit credentials, and maintain AWS Solutions Architect credentials.

Here are four things to know about CAA AWS-specific and what it has to offer audit and compliance teams:

Content was created with PricewaterhouseCoopers (PwC)
PricewaterhouseCoopers worked with us to develop the curriculum content, bringing their expertise in independent risk and control auditing.

“With so many of our customers already in the cloud—or ready to be—we’ve seen a huge increase in the need to meet regulatory and compliance requirements. We’re excited to have combined our risk and controls experience with the power of AWS to create a curriculum in which customers can not only [leverage AWS to help them] meet their compliance needs, but unlock the total value of their cloud investment.” – Paige Hayes, Global Account Leader at PwC
Attendees earn continuing professional education credits
Based on feedback from CAA Cloud Agnostic, we now offer continuing professional education (CPE) credits to attendees. Completion of CAA AWS-specific will allow attendees to earn 28 CPE credits towards any of the International Information System Security Certification Consortium, or (ISC)², certifications, and 18 CPE credits towards any Global Information Assurance Certification (GIAC).
Training helps boost confidence when auditing the AWS cloud
Our customers have proven repeatedly that running sensitive workloads in AWS can be more secure than in on-premises environments. However, a lack of knowledge and updated processes for implementing, monitoring, and proving compliance in the cloud has caused some difficulty. Through CAA AWS-specific, you will get critical training to become more comfortable and confident knowing how to audit the AWS environment with precision.

“Our FSI customer conversations are often focused on security and compliance controls. Leveraging the Cloud Audit Academy enables our team to educate the internal and external auditors of our customers. CAA provides them the necessary tools and knowledge to evaluate and gain comfort with their AWS control environment firsthand. The varying depth and levels focus on everything from basic cloud auditing to diving deeper into the domains which align with our governance and control domains. We reference key AWS services that customers can utilize to create an effective control environment that [helps to meet their] regulatory and audit expectations.” – Jeff (Axe) Axelrad, Compliance Manager, AWS Financial Services
Training enables the governance, risk, and compliance professional
In four days of CAA AWS-specific, you’ll become more comfortable with topics like control domains, network management, vulnerability management, logging and monitoring, incident response, and general knowledge about compliance controls in the cloud.

“In addition to [using AWS to help support and maintain their compliance], our customers need to be able to clearly communicate with their external auditors and regulators HOW compliance is achieved. CAA doesn’t teach auditors how to audit, but rather accelerates the learning necessary to understand specifically how the control landscape changes.” – Jesse Skibbe, Sr. Practice Manager, AWS Professional Services

CAA Cloud Agnostic provides some foundational concepts and is a prerequisite to CAA AWS-specific. It is available for free online at our AWS Training and Certification learning library, or you can contact your account manager to have a one-day instructor-led training session in person.

If it sounds like Cloud Audit Academy training would benefit you and your team, contact our AWS Security Assurance Services team or contact your AWS account manager. For more information, check out the newly updated Security Audit Learning Path.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

How to use the AWS Security Hub PCI DSS v3.2.1 standard

2020-02-17 Rima Tanash

Post Syndicated from Rima Tanash original https://aws.amazon.com/blogs/security/how-to-use-the-aws-security-hub-pci-dss-v3-2-1-standard/

On February 13, 2020, AWS added partial support for the Payment Card Industry Data Security Standard (PCI DSS) version 3.2.1 requirements to AWS Security Hub.

This update enables you to validate a subset of PCI DSS’s requirements and helps with ongoing PCI DSS security activities by conducting continuous and automated checks. The new Security Hub standard also makes it easier to proactively monitor AWS resources, which is critical for any company involved with the storage, processing, or transmission of cardholder data. There’s also a Security score feature for the Security Hub standard, which can help support preparations for PCI DSS assessment.

Use this post to learn how to:

Enable the AWS Security Hub PCI DSS v3.2.1 standard and navigating results
Interpret your security score
Remediate failed security checks
Understand requirements related to findings

Enable Security Hub’s PCI DSS v3.2.1 standard and navigate results

Note: This section assumes that you have Security Hub enabled in one or more accounts. To learn how to enable Security Hub, follow these instructions. If you don’t have Security Hub enabled, the first time you enable Security Hub you will be given the option to enable PCI DSS v3.2.1.

To enable the PCI DSS v3.2.1 security standard in Security Hub:

Open Security Hub and enable PCI DSS v3.2.1 Security standards.
(Once enabled, Security Hub will begin evaluating related resources in the current AWS account and region against the AWS controls within the standard. The scope of the assessment is the current AWS account).
When the evaluation completes, select View results.
Now you are on the PCI DSS v3.2.1 page (Figure 1). You can see all 32 currently-implemented security controls in this standard, their severities, and their status for this account and region. Use search and filters to narrow down the controls by status, severity, title, or related requirement.

Figure 1: PCI DSS v3.2.1 standard results page
Select the name of the control to review detailed information about it. This action will take you to the control’s detail page (Figure 2), which gives you related findings.

Figure 2: Detailed control information
If a specific control is not relevant for you, you can disable the control by selecting Disable and providing a Reason for disabling. (See Disabling Individual Compliance Controls for instructions).

How to interpret and improve your “Security score”

After enabling the PCI DSS v3.2.1 standard in Security Hub, you will notice a Security score appear for the standard itself, and for your account overall. These scores range between 0% and 100%.

Figure 3: Security score for PCI DSS standard (left) and overall (right)

The PCI DSS standard’s Security score represents the proportion of passed PCI DSS controls over enabled PCI DSS controls. The score is displayed as a percentage. Similarly, the overall Security score represents the proportion of passed controls over enabled controls, including controls from every enabled Security Hub standard, displayed as a percentage.

Your aim should be to pass all enabled security checks to reach a score of 100%. Reaching a 100% security score for the AWS Security Hub PCI DSS standard will help you prepare for a PCI DSS assessment. The PCI DSS Compliance Standard in Security Hub is designed to help you with your ongoing PCI DSS security activities.

An important note, the controls cannot verify whether your systems are compliant with the PCI DSS standard. They can neither replace internal efforts nor guarantee that you will pass a PCI DSS assessment.

Remediating failed security checks

To remediate a failed control, you need to remediate every failed finding for that control.

To prioritize remediation, we recommend filtering by Failed controls and then remediating issues starting with critical– and ending with low severity controls.
Identify a control you want to remediate and visit the control detail page.
Follow the Remediation instructions link, and then follow the step-by-step remediation instructions, applying them for every failed finding.

Figure 4: The control detail page, with a link to the remediation instructions

How to interpret “Related requirements”

Every control displays Related requirements in the control card and in the control’s detail page. For PCI DSS, the Related requirements show which PCI DSS requirements are related to the Security Hub PCI DSS control. A single AWS control might relate to multiple PCI DSS requirements.

Figure 5: Related requirements in the control detail page

The user guide lists the related PCI DSS requirements and explains how the specific Security Hub PCI DSS control is related to the requirement.

For example, the AWS Config rule cmk-backing-key-rotation-enabled checks that key rotation is enabled for each customer master key (CMK), but it doesn’t check for CMKs that are using key material imported with the AWS Key Management Service (AWS KMS) BYOK mechanism. The related PCI DSS requirement that is mapped to this rule is PCI DSS 3.6.4 – “Cryptographic keys should be changed once they have reached the end of their cryptoperiod.” Although PCI DSS doesn’t specify the time frame for cryptoperiods, this rule is mapped because, if key rotation is enabled, rotation occurs annually by default with a customer-managed CMK.

Conclusion

The new AWS Security Hub PCI DSS v3.2.1 standard is fundamental for any company involved with storing, processing, or transmitting cardholder data. In this post, you learned how to enable the standard to begin proactively monitoring your AWS resources against the Security Hub PCI DSS controls. You also learned how to navigate the PCI DSS results within Security Hub. By frequently reviewing failed security checks, prioritizing their remediation, and aiming to achieve a 100% security score for PCI DSS within Security Hub, you’ll be better prepared for a PCI DSS assessment.

Smartphone Election in Washington State

2020-01-27 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2020/01/smartphone_elec.html

This year:

King County voters will be able to use their name and birthdate to log in to a Web portal through the Internet browser on their phones, says Bryan Finney, the CEO of Democracy Live, the Seattle-based voting company providing the technology.

Once voters have completed their ballots, they must verify their submissions and then submit a signature on the touch screen of their device.

Finney says election officials in Washington are adept at signature verification because the state votes entirely by mail. That will be the way people are caught if they log in to the system under false pretenses and try to vote as someone else.

The King County elections office plans to print out the ballots submitted electronically by voters whose signatures match and count the papers alongside the votes submitted through traditional routes.

While advocates say this creates an auditable paper trail, many security experts say that because the ballots cross the Internet before they are printed, any subsequent audits on them would be moot. If a cyberattack occurred, an audit could essentially require double-checking ballots that may already have been altered, says Buell.

Of course it’s not an auditable paper trail. There’s a reason why security experts use the phrase “voter-verifiable paper ballots.” A centralized printout of a received Internet message is not voter verifiable.

Another news article.

Securing Tiffany’s Move

2020-01-16 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2020/01/securing_tiffan.html

Story of how Tiffany & Company moved all of its inventory from one store to another. Short summary: careful auditing and a lot of police.

On Financial Fraud

2018-07-25 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/07/on_financial_fr.html

There are some good lessons in this article on financial fraud:

That’s how we got it so wrong. We were looking for incidental breaches of technical regulations, not systematic crime. And the thing is, that’s normal. The nature of fraud is that it works outside your field of vision, subverting the normal checks and balances so that the world changes while the picture stays the same. People in financial markets have been missing the wood for the trees for as long as there have been markets.

[..]

Trust — particularly between complete strangers, with no interactions beside relatively anonymous market transactions — is the basis of the modern industrial economy. And the story of the development of the modern economy is in large part the story of the invention and improvement of technologies and institutions for managing that trust.

And as industrial society develops, it becomes easier to be a victim. In The Wealth of Nations, Adam Smith described how prosperity derived from the division of labour — the 18 distinct operations that went into the manufacture of a pin, for example. While this was going on, the modern world also saw a growing division of trust. The more a society benefits from the division of labour in checking up on things, the further you can go into a con game before you realise that you’re in one.

[…]

Libor teaches us a valuable lesson about commercial fraud — that unlike other crimes, it has a problem of denial as well as one of detection. There are very few other criminal acts where the victim not only consents to the criminal act, but voluntarily transfers the money or valuable goods to the criminal. And the hierarchies, status distinctions and networks that make up a modern economy also create powerful psychological barriers against seeing fraud when it is happening. White-collar crime is partly defined by the kind of person who commits it: a person of high status in the community, the kind of person who is always given the benefit of the doubt.

[…]

Fraudsters don’t play on moral weaknesses, greed or fear; they play on weaknesses in the system of checks and balances — the audit processes that are meant to supplement an overall environment of trust. One point that comes up again and again when looking at famous and large-scale frauds is that, in many cases, everything could have been brought to a halt at a very early stage if anyone had taken care to confirm all the facts. But nobody does confirm all the facts. There are just too bloody many of them. Even after the financial rubble has settled and the arrests been made, this is a huge problem.

Store, Protect, Optimize Your Healthcare Data with AWS

2018-04-26 Stephen Jepsen

Post Syndicated from Stephen Jepsen original https://aws.amazon.com/blogs/architecture/store-protect-optimize-your-healthcare-data-with-aws/

This blog post was co-authored by Ujjwal Ratan, a senior AI/ML solutions architect on the global life sciences team.

Healthcare data is generated at an ever-increasing rate and is predicted to reach 35 zettabytes by 2020. Being able to cost-effectively and securely manage this data whether for patient care, research or legal reasons is increasingly important for healthcare providers.

Healthcare providers must have the ability to ingest, store and protect large volumes of data including clinical, genomic, device, financial, supply chain, and claims. AWS is well-suited to this data deluge with a wide variety of ingestion, storage and security services (e.g. AWS Direct Connect, Amazon Kinesis Streams, Amazon S3, Amazon Macie) for customers to handle their healthcare data. In a recent Healthcare IT News article, healthcare thought-leader, John Halamka, noted, “I predict that five years from now none of us will have datacenters. We’re going to go out to the cloud to find EHRs, clinical decision support, analytics.”

I realize simply storing this data is challenging enough. Magnifying the problem is the fact that healthcare data is increasingly attractive to cyber attackers, making security a top priority. According to Mariya Yao in her Forbes column, it is estimated that individual medical records can be worth hundreds or even thousands of dollars on the black market.

In this first of a 2-part post, I will address the value that AWS can bring to customers for ingesting, storing and protecting provider’s healthcare data. I will describe key components of any cloud-based healthcare workload and the services AWS provides to meet these requirements. In part 2 of this post we will dive deep into the AWS services used for advanced analytics, artificial intelligence and machine learning.

The data tsunami is upon us

So where is this data coming from? In addition to the ubiquitous electronic health record (EHR), the sources of this data include:

genomic sequencers
devices such as MRIs, x-rays and ultrasounds
sensors and wearables for patients
medical equipment telemetry
mobile applications

Additional sources of data come from non-clinical, operational systems such as:

human resources
finance
supply chain
claims and billing

Data from these sources can be structured (e.g., claims data) as well as unstructured (e.g., clinician notes). Some data comes across in streams such as that taken from patient monitors, while some comes in batch form. Still other data comes in near-real time such as HL7 messages. All of this data has retention policies dictating how long it must be stored. Much of this data is stored in perpetuity as many systems in use today have no purge mechanism. AWS has services to manage all these data types as well as their retention, security and access policies.

Imaging is a significant contributor to this data tsunami. Increasing demand for early-stage diagnoses along with aging populations drive increasing demand for images from CT, PET, MRI, ultrasound, digital pathology, X-ray and fluoroscopy. For example, a thin-slice CT image can be hundreds of megabytes. Increasing demand and strict retention policies make storage costly.

Due to the plummeting cost of gene sequencing, molecular diagnostics (including liquid biopsy) is a large contributor to this data deluge. Many predict that as the value of molecular testing becomes more identifiable, the reimbursement models will change and it will increasingly become the standard of care. According to the Washington Post article “Sequencing the Genome Creates so Much Data We Don’t Know What to do with It,”

“Some researchers predict that up to one billion people will have their genome sequenced by 2025 generating up to 40 exabytes of data per year.”

Although genomics is primarily used for oncology diagnostics today, it’s also used for other purposes, pharmacogenomics — used to understand how an individual will metabolize a medication.

Reference Architecture

It is increasingly challenging for the typical hospital, clinic or physician practice to securely store, process and manage this data without cloud adoption.

Amazon has a variety of ingestion techniques depending on the nature of the data including size, frequency and structure. AWS Snowball and AWS Snowmachine are appropriate for extremely-large, secure data transfers whether one time or episodic. AWS Glue is a fully-managed ETL service for securely moving data from on-premise to AWS and Amazon Kinesis can be used for ingesting streaming data.

Amazon S3, Amazon S3 IA, and Amazon Glacier are economical, data-storage services with a pay-as-you-go pricing model that expand (or shrink) with the customer’s requirements.

The above architecture has four distinct components – ingestion, storage, security, and analytics. In this post I will dive deeper into the first three components, namely ingestion, storage and security. In part 2, I will look at how to use AWS’ analytics services to draw value on, and optimize, your healthcare data.

Ingestion

A typical provider data center will consist of many systems with varied datasets. AWS provides multiple tools and services to effectively and securely connect to these data sources and ingest data in various formats. The customers can choose from a range of services and use them in accordance with the use case.

For use cases involving one-time (or periodic), very large data migrations into AWS, customers can take advantage of AWS Snowball devices. These devices come in two sizes, 50 TB and 80 TB and can be combined together to create a petabyte scale data transfer solution.

The devices are easy to connect and load and they are shipped to AWS avoiding the network bottlenecks associated with such large-scale data migrations. The devices are extremely secure supporting 256-bit encryption and come in a tamper-resistant enclosure. AWS Snowball imports data in Amazon S3 which can then interface with other AWS compute services to process that data in a scalable manner.

For use cases involving a need to store a portion of datasets on premises for active use and offload the rest on AWS, the Amazon storage gateway service can be used. The service allows you to seamlessly integrate on premises applications via standard storage protocols like iSCSI or NFS mounted on a gateway appliance. It supports a file interface, a volume interface and a tape interface which can be utilized for a range of use cases like disaster recovery, backup and archiving, cloud bursting, storage tiering and migration.

The AWS Storage Gateway appliance can use the AWS Direct Connect service to establish a dedicated network connection from the on premises data center to AWS.

Specific Industry Use Cases

By using the AWS proposed reference architecture for disaster recovery, healthcare providers can ensure their data assets are securely stored on the cloud and are easily accessible in the event of a disaster. The “AWS Disaster Recovery” whitepaper includes details on options available to customers based on their desired recovery time objective (RTO) and recovery point objective (RPO).

AWS is an ideal destination for offloading large volumes of less-frequently-accessed data. These datasets are rarely used in active compute operations but are exceedingly important to retain for reasons like compliance. By storing these datasets on AWS, customers can take advantage of the highly-durable platform to securely store their data and also retrieve them easily when they need to. For more details on how AWS enables customers to run back and archival use cases on AWS, please refer to the following set of whitepapers.

A healthcare provider may have a variety of databases spread throughout the hospital system supporting critical applications such as EHR, PACS, finance and many more. These datasets often need to be aggregated to derive information and calculate metrics to optimize business processes. AWS Glue is a fully-managed Extract, Transform and Load (ETL) service that can read data from a JDBC-enabled, on-premise database and transfer the datasets into AWS services like Amazon S3, Amazon Redshift and Amazon RDS. This allows customers to create transformation workflows that integrate smaller datasets from multiple sources and aggregates them on AWS.

Healthcare providers deal with a variety of streaming datasets which often have to be analyzed in near real time. These datasets come from a variety of sources such as sensors, messaging buses and social media, and often do not adhere to an industry standard. The Amazon Kinesis suite of services, that includes Amazon Kinesis Streams, Amazon Kinesis Firehose, and Amazon Kinesis Analytics, are the ideal set of services to accomplish the task of deriving value from streaming data.

Example: Using AWS Glue to de-identify and ingest healthcare data into S3
Let’s consider a scenario in which a provider maintains patient records in a database they want to ingest into S3. The provider also wants to de-identify the data by stripping personally- identifiable attributes and store the non-identifiable information in an S3 bucket. This bucket is different from the one that contains identifiable information. Doing this allows the healthcare provider to separate sensitive information with more restrictions set up via S3 bucket policies.

To ingest records into S3, we create a Glue job that reads from the source database using a Glue connection. The connection is also used by a Glue crawler to populate the Glue data catalog with the schema of the source database. We will use the Glue development endpoint and a zeppelin notebook server on EC2 to develop and execute the job.

Step 1: Import the necessary libraries and also set a glue context which is a wrapper on the spark context:

Step 2: Create a dataframe from the source data. I call the dataframe “readmissionsdata”. Here is what the schema would look like:

Step 3: Now select the columns that contains indentifiable information and store it in a new dataframe. Call the new dataframe “phi”.

Step 4: Non-PHI columns are stored in a separate dataframe. Call this dataframe “nonphi”.

Step 5: Write the two dataframes into two separate S3 buckets

Once successfully executed, the PHI and non-PHI attributes are stored in two separate files in two separate buckets that can be individually maintained.

Storage

In 2016, 327 healthcare providers reported a protected health information (PHI) breach, affecting 16.4m patient records[1]. There have been 342 data breaches reported in 2017 — involving 3.2 million patient records.[2]

To date, AWS has released 51 HIPAA-eligible services to help customers address security challenges and is in the process of making many more services HIPAA-eligible. These HIPAA-eligible services (along with all other AWS services) help customers build solutions that comply with HIPAA security and auditing requirements. A catalogue of HIPAA-enabled services can be found at AWS HIPAA-eligible services. It is important to note that AWS manages physical and logical access controls for the AWS boundary. However, the overall security of your workloads is a shared responsibility, where you are responsible for controlling user access to content on your AWS accounts.

AWS storage services allow you to store data efficiently while maintaining high durability and scalability. By using Amazon S3 as the central storage layer, you can take advantage of the Amazon S3 storage management features to get operational metrics on your data sets and transition them between various storage classes to save costs. By tagging objects on Amazon S3, you can build a governance layer on Amazon S3 to grant role based access to objects using Amazon IAM and Amazon S3 bucket policies.

To learn more about the Amazon S3 storage management features, see the following link.

Security

In the example above, we are storing the PHI information in a bucket named “phi.” Now, we want to protect this information to make sure its encrypted, does not have unauthorized access, and all access requests to the data are logged.

Encryption: S3 provides settings to enable default encryption on a bucket. This ensures any object in the bucket is encrypted by default.

Logging: S3 provides object level logging that can be used to capture all API calls to the object. The API calls are logged in cloudtrail for easy access and consolidation. Moreover, it also supports events to proactively alert customers of read and write operations.

Access control: Customers can use S3 bucket policies and IAM policies to restrict access to the phi bucket. It can also put a restriction to enforce multi-factor authentication on the bucket. For example, the following policy enforces multi-factor authentication on the phi bucket:

"Version": "2012-10-17",
    "Id": "123",
    "Statement": [
      {
        "Sid": "",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::phi/*",
        "Condition": { "Null": { "aws:MultiFactorAuthAge": true } }
      },
      {
        "Sid": "",
        "Effect": "Allow",
        "Principal": "*",
        "Action": ["s3:GetObject"],
        "Resource": "arn:aws:s3:::phi/*"
      }
    ]
 }

Conclusion

In Part 1 of this blog, we detailed the ingestion, storage, security and management of healthcare data on AWS. Stay tuned for part two where we are going to dive deep into optimizing the data for analytics and machine learning.

Please visit Healthcare Providers and Insurers in the Cloud for additional information.

[1] “Largest Healthcare Data Breaches of 2016.” HIPAA Journal, 29 Aug. 2017
[2] “Largest Healthcare Data Breaches of 2017.” HIPAA Journal, 8 Mar. 2018

About the Author

Stephen Jepsen is a Global HCLS Practice Manager in AWS Professional Services.

Securing Elections

2018-04-20 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/04/securing_electi_1.html

Elections serve two purposes. The first, and obvious, purpose is to accurately choose the winner. But the second is equally important: to convince the loser. To the extent that an election system is not transparently and auditably accurate, it fails in that second purpose. Our election systems are failing, and we need to fix them.

Today, we conduct our elections on computers. Our registration lists are in computer databases. We vote on computerized voting machines. And our tabulation and reporting is done on computers. We do this for a lot of good reasons, but a side effect is that elections now have all the insecurities inherent in computers. The only way to reliably protect elections from both malice and accident is to use something that is not hackable or unreliable at scale; the best way to do that is to back up as much of the system as possible with paper.

Recently, there have been two graphic demonstrations of how bad our computerized voting system is. In 2007, the states of California and Ohio conducted audits of their electronic voting machines. Expert review teams found exploitable vulnerabilities in almost every component they examined. The researchers were able to undetectably alter vote tallies, erase audit logs, and load malware on to the systems. Some of their attacks could be implemented by a single individual with no greater access than a normal poll worker; others could be done remotely.

Last year, the Defcon hackers’ conference sponsored a Voting Village. Organizers collected 25 pieces of voting equipment, including voting machines and electronic poll books. By the end of the weekend, conference attendees had found ways to compromise every piece of test equipment: to load malicious software, compromise vote tallies and audit logs, or cause equipment to fail.

It’s important to understand that these were not well-funded nation-state attackers. These were not even academics who had been studying the problem for weeks. These were bored hackers, with no experience with voting machines, playing around between parties one weekend.

It shouldn’t be any surprise that voting equipment, including voting machines, voter registration databases, and vote tabulation systems, are that hackable. They’re computers — often ancient computers running operating systems no longer supported by the manufacturers — and they don’t have any magical security technology that the rest of the industry isn’t privy to. If anything, they’re less secure than the computers we generally use, because their manufacturers hide any flaws behind the proprietary nature of their equipment.

We’re not just worried about altering the vote. Sometimes causing widespread failures, or even just sowing mistrust in the system, is enough. And an election whose results are not trusted or believed is a failed election.

Voting systems have another requirement that makes security even harder to achieve: the requirement for a secret ballot. Because we have to securely separate the election-roll system that determines who can vote from the system that collects and tabulates the votes, we can’t use the security systems available to banking and other high-value applications.

We can securely bank online, but can’t securely vote online. If we could do away with anonymity — if everyone could check that their vote was counted correctly — then it would be easy to secure the vote. But that would lead to other problems. Before the US had the secret ballot, voter coercion and vote-buying were widespread.

We can’t, so we need to accept that our voting systems are insecure. We need an election system that is resilient to the threats. And for many parts of the system, that means paper.

Let’s start with the voter rolls. We know they’ve already been targeted. In 2016, someone changed the party affiliation of hundreds of voters before the Republican primary. That’s just one possibility. A well-executed attack that deletes, for example, one in five voters at random — or changes their addresses — would cause chaos on election day.

Yes, we need to shore up the security of these systems. We need better computer, network, and database security for the various state voter organizations. We also need to better secure the voter registration websites, with better design and better internet security. We need better security for the companies that build and sell all this equipment.

Multiple, unchangeable backups are essential. A record of every addition, deletion, and change needs to be stored on a separate system, on write-only media like a DVD. Copies of that DVD, or — even better — a paper printout of the voter rolls, should be available at every polling place on election day. We need to be ready for anything.

Next, the voting machines themselves. Security researchers agree that the gold standard is a voter-verified paper ballot. The easiest (and cheapest) way to achieve this is through optical-scan voting. Voters mark paper ballots by hand; they are fed into a machine and counted automatically. That paper ballot is saved, and serves as a final true record in a recount in case of problems. Touch-screen machines that print a paper ballot to drop in a ballot box can also work for voters with disabilities, as long as the ballot can be easily read and verified by the voter.

Finally, the tabulation and reporting systems. Here again we need more security in the process, but we must always use those paper ballots as checks on the computers. A manual, post-election, risk-limiting audit varies the number of ballots examined according to the margin of victory. Conducting this audit after every election, before the results are certified, gives us confidence that the election outcome is correct, even if the voting machines and tabulation computers have been tampered with. Additionally, we need better coordination and communications when incidents occur.

It’s vital to agree on these procedures and policies before an election. Before the fact, when anyone can win and no one knows whose votes might be changed, it’s easy to agree on strong security. But after the vote, someone is the presumptive winner — and then everything changes. Half of the country wants the result to stand, and half wants it reversed. At that point, it’s too late to agree on anything.

The politicians running in the election shouldn’t have to argue their challenges in court. Getting elections right is in the interest of all citizens. Many countries have independent election commissions that are charged with conducting elections and ensuring their security. We don’t do that in the US.

Instead, we have representatives from each of our two parties in the room, keeping an eye on each other. That provided acceptable security against 20th-century threats, but is totally inadequate to secure our elections in the 21st century. And the belief that the diversity of voting systems in the US provides a measure of security is a dangerous myth, because few districts can be decisive and there are so few voting-machine vendors.

We can do better. In 2017, the Department of Homeland Security declared elections to be critical infrastructure, allowing the department to focus on securing them. On 23 March, Congress allocated $380m to states to upgrade election security.

These are good starts, but don’t go nearly far enough. The constitution delegates elections to the states but allows Congress to “make or alter such Regulations”. In 1845, Congress set a nationwide election day. Today, we need it to set uniform and strict election standards.

This essay originally appeared in the Guardian.

Easily manage table metadata for Presto running on Amazon EMR using the AWS Glue Data Catalog

2018-03-10 Radhika Ravirala

Post Syndicated from Radhika Ravirala original https://aws.amazon.com/blogs/big-data/easily-manage-table-metadata-for-presto-running-on-amazon-emr-using-the-aws-glue-data-catalog/

Amazon EMR empowers many customers to build big data processing applications quickly and cost-effectively, using popular distributed frameworks such as Apache Spark, Apache HBase, Presto, and Apache Flink. For organizations that are crafting their analytical applications on Amazon EMR, there is a growing need to keep their data assets organized in an automated fashion. Because datasets tend to grow exponentially, using cataloging tools is essential to automating data discovery and organizing data assets.

AWS Glue Data Catalog provides this essential capability, allowing you to automatically discover and catalog metadata about your data stores in a central repository. Since Amazon EMR 5.8.0, customers have been using the AWS Glue Data Catalog as a metadata store for Apache Hive and Spark SQL applications that are running on Amazon EMR. Starting with Amazon EMR 5.10.0, you can catalog datasets using AWS Glue and run queries using Presto on Amazon EMR from the Hue (Hadoop User Experience) and Apache Zeppelin UIs.

You might wonder what scenarios warrant using Presto running on Amazon EMR and when to choose Amazon Athena (which uses Presto as the query engine under the hood). It is important to note that both are excellent tools for querying massive amounts of data and addressing different needs and use cases.

Amazon Athena provides the easiest way to run interactive queries for data in Amazon S3 without needing to set up or manage any servers. Presto running on Amazon EMR gives you much more flexibility in how you configure and run your queries, providing the ability to federate to other data sources if needed. For example, you might have a use case that requires LDAP authentication for clients such as the Presto CLI or JDBC/ODBC drivers. Or you might have a workflow where you need to join data between different systems like MySQL/Amazon Redshift/Apache Cassandra and Hive. In these examples, Presto running on Amazon EMR is the right tool to use because it can be configured to enable LDAP authentication in addition to the desired database connectors at cluster launch.

Now, let’s look at how metadata management for Presto works with AWS Glue.

Using an AWS Glue crawler to discover datasets

The AWS Glue Data Catalog is a reference to the location, schema, and runtime metrics of your datasets. To create this reference metadata, AWS Glue needs to crawl your datasets. In this exercise, we use an AWS Glue crawler to populate tables in the Data Catalog for the NYC taxi rides dataset.

The following are the steps for adding a crawler:

Sign in to the AWS Management Console, and open the AWS Glue console. In the navigation pane, choose Crawlers. Then choose Add crawler.
On the Add a data store page, specify the location of the NYC taxi rides dataset.

In the next step, choose an existing IAM role if one is available, or create a new role. Then choose Next.

On the scheduling page, for Frequency, choose Run on demand.
On the Configure the crawler’s output page, choose Add database. Specify blog-db as the database name. (You can specify a name of your choice, but be sure to choose the correct database name when running queries.)
Follow the remaining steps using the default values to create a crawler.

When the crawler displays the Ready state, navigate to the Databases (Choose blog-db from the list of databases, or search for it by specifying it as a filter, as shown in the following screenshot.) Then choose Tables. You should see the three tables created by the crawler, as follows.

(Optional) The discovered data is classified as CSV files. You can optionally convert this data into Parquet format for better response times on your queries.

Launching an Amazon EMR cluster

With the dataset discovered and organized, we can now walk through different options for launching Presto on an Amazon EMR cluster to use the AWS Glue Data Catalog.

Option 1: From the Amazon EMR console

On the Amazon EMR console, choose Create cluster.
In Create Cluster – Quick Options, choose EMR release 10.0 or greater.
Choose Presto as an application.
Under AWS Glue Data Catalog settings, select Use for Presto table metadata.

Option 2: From the AWS CLI

Create a classification configuration as shown following, and save it as a JSON file (presto-emr-config.json).

 [
  {
   "Classification": "presto-connector-hive",
   "Properties": {
     "hive.metastore.glue.datacatalog.enabled": "true"
    }
  }
]

Create the cluster using the AWS CLI as follows:

aws emr create-cluster --name "<your-cluster-name>" --configurations file:///<local-folder>/presto-emr-config.json --release-label emr-5.10.0 --use-default-roles --ec2-attributes KeyName=<your-key-name> --applications Name=Hadoop Name=Spark Name=Hive Name=PRESTO Name=HUE --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=3,InstanceType=m3.xlarge

Running queries with Presto on Amazon EMR

After you’ve set up the Amazon EMR cluster with Presto, the AWS Glue Data Catalog is available through a default “hive” catalog. To change between the Hive and Glue metastores, you have to manually update hive.properties and restart the Presto server. Connect to the master node on your EMR cluster using SSH, and run the Presto CLI to start running queries interactively.

$ presto-cli --catalog hive

Begin with a simple query to sample a few rows:

presto> SELECT * FROM “blog-db”.taxi limit 10;

The query shows a few sample rows as follows:

Query the average fare for trips at each hour of the day and for each day of the month on the Parquet version of the taxi dataset.

presto> SELECT EXTRACT (HOUR FROM pickup_datetime) AS hour, avg(fare_amount) AS average_fare FROM “blog-db”.taxi_parquet GROUP BY 1 ORDER BY 1;

The following image shows the results:

More interestingly, you can compute the number of trips that gave tips in the 10 percent, 15 percent, or higher percentage range:

presto> -- Tip Percent Category
SELECT TipPrctCtgry
, COUNT (DISTINCT TripID) TripCt
FROM
(SELECT TripID
, (CASE 
WHEN fare_prct < 0.7 THEN 'FL70'
WHEN fare_prct < 0.8 THEN 'FL80'
WHEN fare_prct < 0.9 THEN 'FL90'
ELSE 'FL100'
END) FarePrctCtgry
, (CASE 
WHEN tip_prct < 0.1 THEN 'TL10'
WHEN tip_prct < 0.15 THEN 'TL15'
WHEN tip_prct < 0.2 THEN 'TL20'
ELSE 'TG20'
END) TipPrctCtgry

FROM
(SELECT TripID
, (fare_amount / total_amount) as fare_prct
, (extra / total_amount) as extra_prct
, (mta_tax / total_amount) as tip_prct
, (tolls_amount / total_amount) as mta_taxprct
, (tip_amount / total_amount) as tolls_prct
, (improvement_surcharge / total_amount) as imprv_suchrgprct
, total_amount
FROM 
(SELECT *
 , (cast(pickup_longitude AS VARCHAR(100)) || '_' || cast(pickup_latitude AS VARCHAR(100))) as TripID
 from "blog-db”.taxi_parquet 
 WHERE total_amount > 0
 ) as t
) as t
) ct
GROUP BY TipPrctCtgry;

The results are as follows:

While the preceding query is running, navigate to the web interface for Presto on Amazon EMR at <http://master-public-dns-name:8889/. Here you can look into the query metrics, such as active worker nodes, number of rows read per second, reserved memory, and parallelism.

Running queries in the Presto Editor on Hue

If you installed Hue with your Amazon EMR launch, you can also run queries on Hue’s Presto Editor. On the Amazon EMR Cluster console, choose Enable Web Connection, and follow the instructions to access the web interfaces for Hue and Zeppelin.

After the web connection is enabled, choose the Hue link to open the web interface. At the login screen, if you are the administrator logging in for the first time, type a user name and password to create your Hue superuser account. Then choose Create account. Otherwise, type your user name and password and choose Create account, or type the credentials provided by your administrator.

Choose the Presto Editor from the menu. You can run Presto queries against your tables in the AWS Glue Data Catalog.

Conclusion

Having a shared data catalog for applications on Amazon EMR alleviates a myriad of data-related challenges that organizations face today—including discovery, governance, auditability, and collaboration. In this post, we explored how the AWS Glue Data Catalog addresses discoverability and manageability for table metadata for Presto on Amazon EMR. Go ahead, give this a try, and share your experience with us!

Additional Reading

If you found this post useful, be sure to check out Custom Log Presto Query Events on Amazon EMR for Auditing and Performance Insights and Build a Multi-Tenant Amazon EMR Cluster with Kerberos, Microsoft Active Directory Integration and EMRFS Authorization.

About the Author

Radhika Ravirala is a Solutions Architect at Amazon Web Services where she helps customers craft distributed big data applications on the AWS platform. Prior to her cloud journey, she worked as a software engineer and designer for technology companies in Silicon Valley. She holds a M.S in computer science from San Jose State University.

Amazon Relational Database Service – Looking Back at 2017

2018-02-13 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-relational-database-service-looking-back-at-2017/

The Amazon RDS team launched nearly 80 features in 2017. Some of them were covered in this blog, others on the AWS Database Blog, and the rest in What’s New or Forum posts. To wrap up my week, I thought it would be worthwhile to give you an organized recap. So here we go!

Certification & Security

January – FedRAMP Certification for Amazon RDS for MySQL, Oracle, and PostgreSQL.
February – Forced SSL Support for SQL Server.
April – Access Management for MySQL and Aurora Using IAM.
June – TDE Encrypted Cross-Region Snapshots for RDS SQL Server and Oracle.
June – RDS Encryption at Rest for Additional T2 Instance Types.
September – Migration of Encrypted RDS MySQL to Aurora.
September – HIPAA Eligibility for RDS for SQL Server and RDS for MariaDB.

Features

January – Advanced Auditing for Amazon Aurora.
January – RDS MySQL to Aurora Replication in Same Region.
January – Cross-Region Read Replicas of Encrypted Database Instances.
January – Outbound Oracle Network Access Using Custom DNS Servers.
March – Cross-Region Snapshot Copy, Cross-Region Replication, Cross-Account Snapshot Sharing for Aurora.
April – Fast, Online DDL Processing for Aurora.
May – Linux Huge Page Support for Amazon RDS for PostgreSQL.
June – Amazon Aurora Select Into Amazon S3.
June – Stopping and Starting of RDS Database Instances.
June – Amazon Aurora Copy on Write Database Cloning.
August – Amazon Aurora Fast Database Cloning.
August – 16 TB Maximum Database Storage Size for SQL Server.
September – Open Preview of Performance Insights for Aurora PostgreSQL.
September – Amazon Aurora Audit Log Upload to CloudWatch Logs.
October – Size Flexibility for Amazon RDS Reserved Instances.
October – General Availability of Amazon Aurora with PostgreSQL Compatibility.
October – Amazon Aurora for MySQL Asynchronous Key Prefetch.
October – Amazon RDS For Oracle Support for Oracle SQLT Diagnostics Tool.
November – Restore MySQL Backup to Amazon RDS MySQL Database.
November – Auto Scaling for Aurora Replicas.
November – Scaling of Storage Size for Amazon RDS for SQL Server.
November – Up to 16 TB of Storage & Faster Scaling for Amazon RDS for MySQL, MariaDB, Oracle, and PostgreSQL.
November – Amazon RDS for SQL Server Storage Reconfiguration on Snapshot Restore.
November – Amazon RDS for PostgreSQL Support for the pgrouting and postrgresql-hll Extensions, HyperLogLog Data, and decoder_raw Module.
December – Batched Scans and Hashed Joins for Aurora with MySQL Compatibility.

Engine Versions & Features

January – MariaDB 10.0.28.
April – Open Preview of Aurora for PostgreSQL.
May – Amazon RDS Support for Versions 9.6.2, 9.5.6, 9.4.11 and 9.3.16 of PostgreSQL.
May – Oracle Patch Set Upgrades (PSU) for April 2017.
June – Amazon RDS Support for Versions 9.6.2, 9.5.6, 9.4.11, and 9.3.16 in the AWS GovCloud (US) Region.
June – SQL Server 2016 Service Pack 1 (SP1).
July – Amazon RDS for Oracle Supports HugePages.
July – Amazon RDS for SQL Server Enterprise Edition with License Included in More Regions.
July – Oracle SE2 License Included Available in AWS GovCloud (US).
July – MariaDB 10.1.23 and 10.0.31.
July – PostgreSQL 9.6.3, 9.5.7, 9.4.12, and 9.3.17.
September – PostgreSQL 9.6.3, 9.5.7, 9.4.12, and 9.3.17 in AWS GovCloud (US).
September – Amazon RDS for Oracle Support for Oracle APEX.
September – Amazon RDS for Oracle Support for Oracle Multimedia, Spatial, and Locator Options.
September – Amazon RDS for Oracle Support for Customer-Initiated Snapshot Upgrades for Deprecated Database Versions.
November – Amazon RDS for Oracle Support for Oracle Enterprise Manager Cloud Control 13c1 and 13c2.
November – PostgreSQL 9.6.5, 9.5.9, 9.4.14, and 9.3.19 in Commercial and AWS GovCloud (US).
November – Oracle Patch Set Upgrades (PSU) for July 2017.
November – SQL Server 2016 SP1 CU5 and SQL Server 2017.
November – MariaDB 10.1.26 and 10.0.32.
December – Oracle Patch Set Upgrades (PSU) for October 2017.

Regional Support

March – Aurora in the US West (N. California) Region.
June – Aurora in the EU (Frankfurt) Region.
July – Windows Authentication for RDS for SQL Server in More Regions.
August – Multi-AZ for RDS SQL Server in the EU (Frankfurt) Region.
November – Amazon Aurora for PostgreSQL in the Canada (Central), EU (Frankfurt), Asia Pacific (Sydney), and Asia Pacific (Mumbai) Regions.
December – Amazon Aurora for PostgreSQL in the EU (Paris) Region.

Instance Support

March – Aurora on T2.Small Instances.
June – RDS on M4 Instances in the São Paulo Region.
October – Amazon RDS for Oracle on New R4, T2, and M4 Instance Types.
October – Amazon Aurora on R4 Instances.
November – Amazon RDS for MySQL and MariaDB on R4, T2, and M4 Instance Types.
November – Amazon RDS for SQL Server on R4 and M4 Instance Types.
December – Amazon RDS for PostgreSQL on R4, T2, and M4 Instance Types.

Price Reductions

July – Price Reduction for R3 Instances running SQL Server Enterprise License Included.

And That’s a Wrap
I’m pretty sure that’s everything. As you can see, 2017 was quite the year! I can’t wait to see what the team delivers in 2018.

— Jeff;

Announcing our new beta for the AWS Certified Security – Specialty exam

2018-01-11 Janna Pellegrino

Post Syndicated from Janna Pellegrino original https://aws.amazon.com/blogs/architecture/announcing-our-new-beta-for-the-aws-certified-security-specialty-exam/

Take the AWS Certified Security – Specialty beta exam for the chance to be among the first to hold this new AWS Certification. This beta exam allows experienced cloud security professionals to demonstrate and validate their expertise. Register today – this beta exam will only be available from January 15 to March 2!

About the exam

This beta exam validates that the successful candidate can effectively demonstrate knowledge of how to secure the AWS platform. The exam covers incident response, logging and monitoring, infrastructure security, identity and access management, and data protection.

The exam validates:

Familiarity with regional- and country-specific security and compliance regulations and meta issues that these regulations embody.
An understanding of specialized data classifications and AWS data protection mechanisms.
An understanding of data encryption methods and AWS mechanisms to implement them.
An understanding of secure Internet protocols and AWS mechanisms to implement them.
A working knowledge of AWS security services and features of services to provide a secure production environment.
Competency gained from two or more years of production deployment experience using AWS security services and features.
Ability to make tradeoff decisions with regard to cost, security, and deployment complexity given a set of application requirements.
An understanding of security operations and risk.

Learn more and register >>

Who is eligible

The beta is open to anyone who currently holds an Associate or Cloud Practitioner certification. We recommend candidates have five years of IT security experience designing and implementing security solutions, and at least two years of hands-on experience securing AWS workloads.

How to prepare

We have training and other resources to help you prepare for the beta exam:

AWS Security Fundamentals Digital| 3 Hours
This course introduces you to fundamental cloud computing and AWS security concepts, including AWS access control and management, governance, logging, and encryption methods. It also covers security-related compliance protocols and risk management strategies, as well as procedures related to auditing your AWS security infrastructure.

Security Operations on AWS Classroom | 3 Days
This course demonstrates how to efficiently use AWS security services to stay secure and compliant in the AWS Cloud. The course focuses on the AWS-recommended security best practices that you can implement to enhance the security of your data and systems in the cloud. The course highlights the security features of AWS key services including compute, storage, networking, and database services.

Online resources for Cloud Security and Compliance

Review documentation, whitepapers, and articles & tutorials related to cloud security and compliance.

Learn more and register >>

Please contact us if you have questions about exam registration.

Good luck!

Validate Your IT Security Expertise with the New AWS Certified Security – Specialty Beta Exam

2018-01-11 Sara Snedeker

Post Syndicated from Sara Snedeker original https://aws.amazon.com/blogs/security/validate-your-it-security-expertise-with-the-new-aws-certified-security-specialty-beta-exam/

AWS Training and Certification image

If you are an experienced cloud security professional, you can demonstrate and validate your expertise with the new AWS Certified Security – Specialty beta exam. This exam allows you to demonstrate your knowledge of incident response, logging and monitoring, infrastructure security, identity and access management, and data protection. Register today – this beta exam will be available only from January 15 to March 2, 2018.

By taking this exam, you can validate your:

Familiarity with region-specific and country-specific security and compliance regulations and meta issues that these regulations include.
Understanding of data encryption methods and secure internet protocols, and the AWS mechanisms to implement them.
Working knowledge of AWS security services to provide a secure production environment.
Ability to make trade-off decisions with regard to cost, security, and deployment complexity when given a set of application requirements.

See the full list of security knowledge you can validate by taking this beta exam.

Who is eligible?

The beta exam is open to anyone who currently holds an AWS Associate or Cloud Practitioner certification. We recommend candidates have five years of IT security experience designing and implementing security solutions, and at least two years of hands-on experience securing AWS workloads.

How to prepare

You can take the following courses and use AWS cloud security resources and compliance resources to prepare for this exam.

AWS Security Fundamentals (digital, 3 hours)
This digital course introduces you to fundamental cloud computing and AWS security concepts, including AWS access control and management, governance, logging, and encryption methods. It also covers security-related compliance protocols and risk management strategies, as well as procedures related to auditing your AWS security infrastructure.

Security Operations on AWS (classroom, 3 days)
This instructor-led course demonstrates how to efficiently use AWS security services to help stay secure and compliant in the AWS Cloud. The course focuses on the AWS-recommended security best practices that you can implement to enhance the security of your AWS resources. The course highlights the security features of AWS compute, storage, networking, and database services.

If you have questions about this new beta exam, contact us.

Good luck with the exam!

– Sara

Use Kerberos Authentication to Integrate Amazon EMR with Microsoft Active Directory

2018-01-10 Bruno Faria

Post Syndicated from Bruno Faria original https://aws.amazon.com/blogs/big-data/use-kerberos-authentication-to-integrate-amazon-emr-with-microsoft-active-directory/

Many enterprises use Microsoft Active Directory to manage users, groups, and computers in a network. And a question is asked frequently: How can Active Directory users access big data workloads running on Amazon EMR with the same single sign-on (SSO) experience they have when accessing resources in the Active Directory network?

This post walks you through the process of using AWS CloudFormation to set up a cross-realm trust and extend authentication from an Active Directory network into an Amazon EMR cluster with Kerberos enabled. By establishing a cross-realm trust, Active Directory users can use their Active Directory credentials to access an Amazon EMR cluster and run jobs as themselves.

Walkthrough overview

In this example, you build a solution that allows Active Directory users to seamlessly access Amazon EMR clusters and run big data jobs. Here’s what you need before setting up this solution:

An AWS account
An Amazon EC2 key pair
A possible limit increase for your account (Note: Usually a limit increase will not be necessary. See the AWS Service Limits documentation if you encounter a limit error while building the solution.)

To make it easier for you to get started, I created AWS CloudFormation templates that automatically configure and deploy the solution for you. The following steps and resources are involved in setting up the solution:

Create and configure an Amazon Virtual Private Cloud (Amazon VPC).
Launch an Amazon EC2 Windows instance (Active Directory domain controller).
Create an Amazon EMR security configuration for Kerberos and cross-realm trust.
Launch an Amazon EMR cluster with Kerberos enabled and a cross-realm trust configuration.

You can use the AWS CloudFormation templates to complete each step individually, or you can deploy the entire solution through a single step.

To skip the basics and deploy the entire solution through the single-step AWS CloudFormation template, go to the Single-step solution deployment
To set up each component individually, go to the Deploying each component individually

Note: If you want to manually create and configure the components for this solution without using AWS CloudFormation, refer to the Amazon EMR cross-realm documentation.
IMPORTANT: The AWS CloudFormation templates used in this post are designed to work only in the us-east-1 (N. Virginia) Region. They are not intended for production use without modification.

Single-step solution deployment

If you don’t want to set up each component individually, you can use the single-step AWS CloudFormation template. The single-step template is a master template that uses nested stacks (additional templates) to launch and configure all the resources for the solution in one go.

To deploy the single-step template into your account, choose Launch Stack:

This takes you to the Create stack wizard in the AWS CloudFormation console. The template is launched in the US East (N. Virginia) Region by default. Do not change to a different Region because the template is designed to work only in us-east-1 (N. Virginia).

On the Select Template page, keep the default URL for the AWS CloudFormation template, and then choose Next.

On the Specify Details page, review the parameters for the template. Provide values for the parameters that require input (for more information, see the parameters table that follows).

The following parameters are available in this template.

Parameter	Default	Description
Domain Controller name	DC1	NetBIOS (hostname) name of the Active Directory server. This name can be up to 15 characters long.
Active Directory domain	example.com	Fully qualified domain name (FQDN) of the forest root domain (for example, `example.com`).
Domain NetBIOS name	EXAMPLE	NetBIOS name of the domain for users of earlier versions of Windows. This name can be up to 15 characters long.
Domain admin user	CrossRealmAdmin	User name for the account that is added as domain administrator. This account is separate from the default administrator account.
Domain admin password	Requires input	Password for the domain admin user. Must be at least eight characters including letters, numbers, and symbols.
Key pair name	Requires input	Name of an existing key pair, which enables you to connect securely to your instance after it launches.
Instance type	m4.xlarge	Instance type for the domain controller and the Amazon EMR cluster.
Allowed IP address	10.0.0.0/16	The client IP address can that can reach your cluster. Specify an IP address range in CIDR notation (for example, 203.0.113.5/32). By default, only the VPC CIDR (10.0.0.0/16) can reach the cluster. Be sure to add your client IP range so that you can connect to the cluster using SSH.
EMR Kerberos realm	EC2.INTERNAL	Cluster’s Kerberos realm name. By default, the realm name is derived from the cluster’s VPC domain name in uppercase letters (for example, `EC2.INTERNAL` is the default VPC domain name in the us-east-1 Region).
Trusted AD domain	EXAMPLE.COM	The Active Directory (AD) domain that you want to trust. This is the same as the “Active Directory domain.” However, it must use all uppercase letters (for example, `EXAMPLE.COM`).
Cross-realm trust password	Requires input	Password that you want to use for your cross-realm trust.
Instance count	2	The number of instances (core nodes) for the cluster.
EMR applications	Hadoop, Spark, Ganglia, Hive	Comma separated list of applications to install on the cluster.

After you specify the template details, choose Next. On the Options page, choose Next again. On the Review page, select the I acknowledge that AWS CloudFormation might create IAM resources with custom names check box, and then choose Create.

It takes approximately 45 minutes for the deployment to complete. When the stack launch is complete, it will return outputs with information about the resources that were created. Note the outputs and skip to the Managing and testing the solution section. You can view the stack outputs on the AWS Management Console or by using the following AWS CLI command:

$ aws cloudformation describe-stacks --stack-name <Stack_Name> --region us-east-1 --query Stacks[0].Outputs

Deploying each component individually

This section describes how to use AWS CloudFormation templates to perform each step separately in the solution.

Create and configure an Amazon VPC

In order for you to establish a cross-realm trust between an Amazon EMR Kerberos realm and an Active Directory domain, your Amazon VPC must meet the following requirements:

The subnet used for the Amazon EMR cluster must have a CIDR block of fewer than nine digits (for example, 10.0.1.0/24).
Both DNS resolution and DNS hostnames must be enabled (set to “yes”).
The Active Directory domain controller must be the DNS server for instances in the Amazon VPC (this is configured in the next step).

To use the AWS CloudFormation template to create and configure an Amazon VPC with the prerequisites listed previously, choose Launch Stack:

Note: If you want to create the VPC manually (without using AWS CloudFormation), see Set Up the VPC and Subnet in the Amazon EMR documentation.

Launching this stack creates the following AWS resources:

Amazon VPC with CIDR block 10.0.0.0/16 (Name: CrossRealmVPC)
Internet Gateway (Name: CrossRealmGateway)
Public subnet with CIDR block 10.0.1.0/24 (Name: CrossRealmSubnet)
Security group allowing inbound access from the VPC’s subnets (Name tag: CrossRealmSecurityGroup)

When the stack launch is complete, it should return outputs similar to the following.

Key	Value example	Description
SubnetID	subnet-xxxxxxxx	The subnet for the Active Directory domain controller and the EMR cluster.
SecurityGroup	sg-xxxxxxxx	The security group for the Active Directory domain controller.
VPCID	vpc-xxxxxxxx	The Active Directory domain controller and EMR cluster will be launched on this VPC.

Note the outputs because they are used in the next step. You can view the stack outputs on the AWS Management Console or by using the following AWS CLI command:

$ aws cloudformation describe-stacks --stack-name <Stack_Name> --region us-east-1 --query Stacks[0].Outputs

Launch and configure an Active Directory domain controller

In this step, you use an AWS CloudFormation template to automatically launch and configure a new Active Directory domain controller and cross-realm trust.

Note: There are various ways to install and configure an Active Directory domain controller. For details on manually launching and installing a domain controller without AWS CloudFormation, see Step 2: Launch and Install the AD Domain Controller in the Amazon EMR documentation.

In addition to launching and configuring an Active Directory domain controller and cross-realm trust, this AWS CloudFormation template also sets the domain controller as the DNS server (name server) for your Amazon VPC. In other words, the template creates a new DHCP option-set for the VPC where it’s being deployed to, and it sets the private IP of the domain controller as the name server for that new DHCP option set.

IMPORTANT: You should not use this template on a production VPC with existing resources like Amazon EC2 instances. When you launch this stack, make sure that you use the new environment and resources (Amazon VPC, subnet, and security group) that were created in the Create and configure an Amazon VPC step.

To launch this stack, choose Launch Stack:

The following table contains information about the parameters available in this template. Review the parameters for the template and provide values for those that require input.

Parameter	Default	Description
VPC ID	Requires input	Launch the domain controller on this VPC (for example, use the VPC created in the Create and configure an Amazon VPC step).
Subnet ID	Requires input	Subnet used for the domain controller (for example, use the subnet created in the Create and configure an Amazon VPC step).
Security group ID	Requires input	Security group (SG) for the domain controller (for example, use the SG created in the Create and configure an Amazon VPC step).
Domain Controller name	DC1	NetBIOS name of the Active Directory server (up to 15 characters).
Active Directory domain	example.com	Fully qualified domain name (FQDN) of the forest root domain (for example, `example.com`).
Domain NetBIOS name	EXAMPLE	NetBIOS name of the domain for users of earlier versions of Windows. This name can be up to 15 characters long.
Domain admin user	CrossRealmAdmin	User name for the account that is added as domain administrator. This account is separate from the default administrator account.
Domain admin password	Requires input	Password for the domain admin user. Must be at least eight characters including letters, numbers, and symbols.
Key pair name	Requires input	Name of an existing EC2 key pair to enable access to the domain controller instance.
Instance type	m4.xlarge	Instance type for the domain controller.
EMR Kerberos realm	EC2.INTERNAL	Cluster’s Kerberos realm name. By default, the realm name is derived from the cluster’s VPC domain name in uppercase letters (for example, `EC2.INTERNAL` is the default VPC domain name in the us-east-1 Region).
Cross-realm trust password	Requires input	Password that you want to use for your cross-realm trust.

It takes 25–30 minutes for this stack to be created. When it’s complete, note the stack’s outputs, and then move to the next step: Launch an EMR cluster with Kerberos enabled.

Create a security configuration and launch an Amazon EMR cluster with Kerberos enabled

To launch a kerberized Amazon EMR cluster, you first need to create a security configuration containing the cross-realm trust configuration. You then specify cluster-specific Kerberos attributes when launching the cluster.

In this step, you use AWS CloudFormation to launch and configure a kerberized Amazon EMR cluster with a cross-realm trust. If you want to manually launch and configure a cluster with Kerberos enabled, see Step 6: Launch a Kerberized EMR Cluster in the Amazon EMR documentation.

Note: At the time of this writing, AWS CloudFormation does not yet support launching Amazon EMR clusters with Kerberos authentication enabled. To overcome this limitation, I created a template that uses an AWS Lambda-backed custom resource to launch and configure the Amazon EMR cluster with Kerberos enabled. If you use this template, there’s nothing else that you need to do. Just keep in mind that the template creates and invokes an AWS Lambda function (custom resource) to launch the cluster.

To create a cross-realm trust security configuration and launch a kerberized Amazon EMR cluster using AWS CloudFormation, choose Launch Stack:

The following table lists and describes the template parameters for deploying a kerberized Amazon EMR cluster and configuring a cross-realm trust.

Parameter	Default	Description
Active Directory domain	example.com	The Active Directory domain that you want to establish the cross-realm trust with.
Domain admin user (joiner user)	CrossRealmAdmin	The user name of an Active Directory domain user with privileges to join domains/computers to the Active Directory domain (joiner user).
Domain admin password	Requires input	Password of the joiner user.
Cross-realm trust password	Requires input	Password of your cross-realm trust.
EC2 key pair name	Requires input	Name of an existing key pair, which enables you to connect securely to your cluster after it launches.
Subnet ID	Requires input	Subnet that you want to use for your Amazon EMR cluster (for example, choose the subnet created in the Create and configure an Amazon VPC step).
Security group ID	Requires input	Security group that you want to use for your Amazon EMR cluster (for example, choose the security group created in the Create and configure an Amazon VPC step).
Instance type	m4.xlarge	The instance type that you want to use for the cluster nodes.
Instance count	2	The number of instances (core nodes) for the cluster.
Allowed IP address	10.0.0.0/16	The client IP address can that can reach your cluster. Specify an IP address range in CIDR notation (for example, 203.0.113.5/32). By default, only the VPC CIDR (10.0.0.0/16) can reach the cluster. Be sure to add your client IP range so that you can connect to the cluster using SSH.
EMR applications	Hadoop, Spark, Ganglia, Hive	Comma separated list of the applications that you want installed on the cluster.
EMR Kerberos realm	EC2.INTERNAL	Cluster’s Kerberos realm name. By default, the realm name is derived from the cluster’s VPC domain name in uppercase letters (for example, `EC2.INTERNAL` is the default VPC domain name in the us-east-1 Region).
Trusted AD domain	EXAMPLE.COM	The Active Directory domain that you want to trust. This name is the same as the “AD domain name.” However, it must use all uppercase letters (for example, `EXAMPLE.COM`).

It takes 10–15 minutes for this stack to be created. When it’s complete, note the stack’s outputs, and then move to the next section: Managing and testing the solution.

Managing and testing the solution

Now that you’ve configured and built the solution, it’s time to test it by connecting to a cluster using Active Directory credentials.

SSH to a cluster using Active Directory credentials (single sign-on)

After you launch a kerberized Amazon EMR cluster, if you used the AWS CloudFormation templates and added your client IP range to the Allowed IP address parameter, you should be able to connect to the cluster using an SSH client and your Active Directory user credentials. If you have trouble connecting to the cluster using SSH, check the cluster’s security group to make sure that it allows inbound SSH connection (TCP port 22) from your client’s IP address (source).

The following steps assume that you’re using a client such as OpenSSH. If you’re using a different SSH application (for example, PuTTY), consult the application-specific documentation.

Note: Because the cluster was launched with a cross-realm trust configuration, you don’t need to use a private key (.pem file) when you connect to it as a domain user using SSH.

To connect to your Amazon EMR cluster as an Active Directory user using SSH, run the following command. Replace ad_user with the domain admin user that you created while setting up the domain controller and replace master_node_URL with the cluster’s URL (see the stack’s outputs to find this information):

$ ssh -l <ad_user> <master_node_URL>

If your SSH client is configured to use a key as the preferred authentication method, the login might fail. If that’s the case, you can add the following options to your SSH command to force the SSH connection to use password authentication:

$ ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no -l <ad_user> <master_node_URL>

After a domain user connects to the cluster using SSH, if this is the first that the user is connecting, a local home directory is created for that user. In addition to creating a local home directory, if you used the create-hdfs-home-ba.sh bootstrap action when launching the cluster (done by default if you used the AWS CloudFormation template to launch a kerberized cluster), an HDFS user home directory is also automatically created.

Note: If you manually launched the cluster and did not use the create-hdfs-home-ba.sh bootstrap action, then you’ll need to manually create HDFS user home directories for your users.

When you connect to the cluster using SSH for the first time (as a domain user), you should see the following messages if the HDFS home directory for your domain user was successfully created:

Running jobs on a kerberized Amazon EMR cluster

To run a job on a kerberized cluster, the user submitting the job must first be authenticated. If you followed the previous section to connect to your cluster as an Active Directory user using SSH, the user should be authenticated automatically.

If running the klist command returns a “No credentials cache found” message, it means that the user is not authenticated (the user doesn’t have a Kerberos ticket). You can re-authenticate a user at any time by running the following command (be sure to use all uppercase letters for the Active Directory domain):

$ kinit <username>@<AD_DOMAIN>

When the user is authenticated, they can submit jobs just like they would on a non-kerberized cluster.

Auditing jobs

Another advantage that Kerberos can provide is that you can easily tell which user ran a particular job. For example, connect (using SSH) to a kerberized cluster with an Active Directory user, and submit the SparkPi sample application:

$ spark-example SparkPi

After running the SparkPi application, go to the Amazon EMR console and choose your cluster. Then choose the Application history tab. There you can see information about the application, including the user that submitted the job:

Common issues

Although it would be hard to cover every possible Kerberos issue, this section covers some of the more common issues that might occur and ways to fix them.

Issue 1: You can successfully connect and get authenticated on a cluster. However, whenever you try running job, it fails with an error similar to the following:

org.apache.hadoop.security.AccessControlException: Permission denied

Solution: Make sure that an HDFS home directory for the user was created and that it has the right permissions.

Issue 2: You can successfully connect to the cluster, but you can’t run any Hadoop or HDFS commands.

Solution: Use the klist command to confirm whether the user is authenticated and has a valid Kerberos ticket. Use the kinit command to re-authenticate a user.

Issue 3: You can’t connect (using SSH) to the cluster using Active Directory user credentials, but you can manually authenticate the user with kinit.

Solution: Make sure that the Active Directory domain controller is the DNS server (name server) for the cluster nodes.

Cleaning up

After completing and testing this solution, remember to clean up the resources. If you used the AWS CloudFormation templates to create the resources, then use the AWS CloudFormation console or AWS CLI/SDK to delete the stacks. Deleting a stack also deletes the resources created by that stack.

If one of your stacks does not delete, make sure that there are no dependencies on the resources created by that stack. For example, if you deployed an Amazon VPC using AWS CloudFormation and then deployed a domain controller into that VPC using a different AWS CloudFormation stack, you must first delete the domain controller stack before the VPC stack can be deleted.

Summary

The ability to authenticate users and services with Kerberos not only allows you to secure your big data applications, but it also enables you to easily integrate Amazon EMR clusters with an Active Directory environment. This post showed how you can use Kerberos on Amazon EMR to create a single sign-on solution where Active Directory domain users can seamlessly access Amazon EMR clusters and run big data applications. We also showed how you can use AWS CloudFormation to automate the deployment of this solution.

Noise

Tag Archives: auditing

AWS achieves its second ISMAP authorization in Japan

AWS achieves TISAX certification (Information with Very High Protection Needs (AL3)

AWS achieves HDS certification to three additional Regions

C5 Type 2 attestation report now available with 141 services in scope

AWS achieves GSMA Security Certification for Europe (Paris) Region

AWS achieves GSMA security certification for US East (Ohio) Region

C5 Type 2 attestation report now available with one new Region and 123 services in scope

AWS is the first global cloud service provider to comply with the new K-ISMS-P standard

How to use the AWS Security Hub PCI DSS v3.2.1 standard

Enable Security Hub’s PCI DSS v3.2.1 standard and navigate results

How to interpret and improve your “Security score”

Remediating failed security checks

How to interpret “Related requirements”

Conclusion

Further reading

Smartphone Election in Washington State

Securing Tiffany’s Move

On Financial Fraud

Securing Elections

The collective thoughts of the interwebz