Using AWS Shield Advanced protection groups to improve DDoS detection and mitigation

Post Syndicated from Joe Viggiano original https://aws.amazon.com/blogs/security/using-aws-shield-advanced-protection-groups-to-improve-ddos-detection-and-mitigation/

Amazon Web Services (AWS) customers can use AWS Shield Advanced to detect and mitigate distributed denial of service (DDoS) attacks that target their applications running on Amazon Elastic Compute Cloud (Amazon EC2), Elastic Local Balancing (ELB), Amazon CloudFront, AWS Global Accelerator, and Amazon Route 53. By using protection groups for Shield Advanced, you can logically group your collections of Shield Advanced protected resources. In this blog post, you will learn how you can use protection groups to customize the scope of DDoS detection for application layer events, and accelerate mitigation for infrastructure layer events.

What is a protection group?

A protection group is a resource that you create by grouping your Shield Advanced protected resources, so that the service considers them to be a single protected entity. A protection group can contain many different resources that compose your application, and the resources may be part of multiple protection groups spanning different AWS Regions within an AWS account. Common patterns that you might use when designing protection groups include aligning resources to applications, application teams, or environments (such as production and staging), and by product tiers (such as free or paid). For more information about setting up protection groups, see Managing AWS Shield Advanced protection groups.

Why should you consider using a protection group?

The benefits of protection groups differ for infrastructure layer (layer 3 and layer 4) events and application layer (layer 7) events. For layer 3 and layer 4 events, protection groups can reduce the time it takes for Shield Advanced to begin mitigations. For layer 7 events, protection groups add an additional reporting mechanism. There is no change in the mechanism that Shield Advanced uses internally for detection of an event, and you do not lose the functionality of individual resource-level detections. You receive both group-level and individual resource-level Amazon CloudWatch metrics to consume for operational use. Let’s look at the benefits for each layer in more detail.

Layers 3 and 4: Accelerate time to mitigate for DDoS events

For infrastructure layer (layer 3 and layer 4) events, Shield Advanced monitors the traffic volume to your protected resource. An abnormal traffic deviation signals the possibility of a DDoS attack, and Shield Advanced then puts mitigations in place. By default, Shield Advanced observes the elevation of traffic to a resource over multiple consecutive time intervals to establish confidence that a layer 3/layer 4 event is under way. In the absence of a protection group, Shield Advanced follows the default behavior of waiting to establish confidence before it puts mitigation in place for each resource. However, if the resources are part of a protection group, and if the service detects that one resource in a group is targeted, Shield Advanced uses that confidence for other resources in the group. This can accelerate the process of putting mitigations in place for those resources.

Consider a case where you have an application deployed in different AWS Regions, and each stack is fronted with a Network Load Balancer (NLB). When you enable Shield Advanced on the Elastic IP addresses associated with the NLB in each Region, you can optionally add those Elastic IP addresses to a protection group. If an actor targets one of the NLBs in the protection group and a DDoS attack is detected, Shield Advanced will lower the threshold for implementing mitigations on the other NLBs associated with the protection group. If the scope of the attack shifts to target the other NLBs, Shield Advanced can potentially mitigate the attack faster than if the NLB was not in the protection group.

Note: This benefit applies only to Elastic IP addresses and Global Accelerator resource types.

Layer 7: Reduce false positives and improve accuracy of detection for DDoS events

Shield Advanced detects application layer (layer 7) events when you associate a web access control list (web ACL) in AWS WAF with it. Shield Advanced consumes request data for the associated web ACL, analyzes it, and builds a traffic baseline for your application. The service then uses this baseline to detect anomalies in traffic patterns that might indicate a DDoS attack.

When you group resources in a protection group, Shield Advanced aggregates the data from individual resources and creates the baseline for the whole group. It then uses this aggregated baseline to detect layer 7 events for the group resource. It also continues to monitor and report for the resources individually, regardless of whether they are part of protection groups or not.

Shield Advanced provides three types of aggregation to choose from (sum, mean, and max) to aggregate the volume data of individual resources to use as a baseline for the whole group. We’ll look at the three types of aggregation, with a use case for each, in the next section.

Note: Traffic aggregation is applicable only for layer 7 detection.

Case 1: Blue/green deployments

Blue/green is a popular deployment strategy that increases application availability and reduces deployment risk when rolling out changes. The blue environment runs the current application version, and the green environment runs the new application version. When testing is complete, live application traffic is directed to the green environment, and the blue environment is dismantled.

During blue/green deployments, the traffic to your green resources can go from zero load to full load in a short period of time. Shield Advanced layer 7 detection uses traffic baselining for individual resources, so newly created resources like an Application Load Balancer (ALB) that are part of a blue/green operation would have no baseline, and the rapid increase in traffic could cause Shield Advanced to declare a DDoS event. In this scenario, the DDoS event could be a false positive.

Figure 1: A blue/green deployment with ALBs in a protection group. Shield is using the sum of total traffic to the group to baseline layer 7 traffic for the group as a single unit

Figure 1: A blue/green deployment with ALBs in a protection group. Shield is using the sum of total traffic to the group to baseline layer 7 traffic for the group as a single unit

In the example architecture shown in Figure 1, we have configured Shield to include all resources of type ALB in a single protection group with aggregation type sum. Shield Advanced will use the sum of traffic to all resources in the protection group as an additional baseline. We have only one ALB (called blue) to begin with. When you add the green ALB as part of your deployment, you can optionally add it to the protection group. As traffic shifts from blue to green, the total traffic to the protection group remains the same even though the volume of traffic changes for the individual resources that make up the group. After the blue ALB is deleted, the Shield Advanced baseline for that ALB is deleted with it. At this point, the green ALB hasn’t existed for sufficient time to have its own accurate baseline, but the protection group baseline persists. You could still receive a DDoSDetected CloudWatch metric with a value of 1 for individual resources, but with a protection group you have the flexibility to set one or more alarms based on the group-level DDoSDetected metric. Depending on your application’s use case, this can reduce non-actionable event notifications.

Note: You might already have alarms set for individual resources, because the onboarding wizard in Shield Advanced provides you an option to create alarms when you add protection to a resource. So, you should review the alarms you already have configured before you create a protection group. Simply adding a resource to a protection group will not reduce false positives.

Case 2: Resources that have traffic patterns similar to each other

Client applications might interact with multiple services as part of a single transaction or workflow. These services can be behind their own dedicated ALBs or CloudFront distributions and can have traffic patterns similar to each other. In the example architecture shown in Figure 2, we have two services that are always called to satisfy a user request. Consider a case where you add a new service to the mix. Before protection groups existed, setting up such a new protected resource, such as ALB or CloudFront, required Shield Advanced to build a brand-new baseline. You had to wait for a certain minimum period before Shield Advanced could start monitoring the resource, and the service would need to monitor traffic for a few days in order to be accurate.

Figure 2: Deploying a new service and including it in a protection group with an existing baseline. Shield is using the mean aggregation type to baseline traffic for the group.

Figure 2: Deploying a new service and including it in a protection group with an existing baseline. Shield is using the mean aggregation type to baseline traffic for the group.

For improved accuracy of detection of level 7 events, you can cause Shield Advanced to inherit the baseline of existing services that are part of the same transaction or workflow. To do so, you can put your new resource in a protection group along with an existing service or services, and set the aggregation type to mean. Shield Advanced will take some time to build up an accurate baseline for the new service. However, the protection group has an established baseline, so the new service won’t be susceptible to decreased accuracy of detection for that period of time. Note that this setting will not stop Shield Advanced from sending notifications for the new service individually; however, you might prefer to take corrective action based on the detection for the group instead.

Case 3: Resources that share traffic in a non-uniform way

Consider the case of a CloudFront distribution with an ALB as origin. If the content is cached in CloudFront edge locations, the traffic reaching the application will be lower than that received by the edge locations. Similarly, if there are multiple origins of a CloudFront distribution, the traffic volumes of individual origins will not reflect the aggregate traffic for the application. Scenarios like invalidation of cache or an origin failover can result in increased traffic at one of the ALB origins. This could cause Shield Advanced to send “1” as the value for the DDoSDetected CloudWatch metric for that ALB. However, you might not want to initiate an alarm or take corrective action in this case.

Figure 3: CloudFront and ALBs in a protection group with aggregation type max. Shield is using CloudFront’s baseline for the group

Figure 3: CloudFront and ALBs in a protection group with aggregation type max. Shield is using CloudFront’s baseline for the group

You can combine the CloudFront distribution and origin (or origins) in a protection group with the aggregation type set to max. Shield Advanced will consider the CloudFront distribution’s traffic volume as the baseline for the protection group as a whole. In the example architecture in Figure 3, a CloudFront distribution fronts two ALBs and balances the load between the two. We have bundled all three resources (CloudFront and two ALBs) into a protection group. In case one ALB fails, the other ALB will receive all the traffic. This way, although you might receive an event notification for the active ALB at the individual resource level if Shield detects a volumetric event, you might not receive it for the protection group because Shield Advanced will use CloudFront traffic as the baseline for determining the increase in volume. You can set one or more alarms and take corrective action according to your application’s use case.

Conclusion

In this blog post, we showed you how AWS Shield Advanced provides you with the capability to group resources in order to consider them a single logical entity for DDoS detection and mitigation. This can help reduce the number of false positives and accelerate the time to mitigation for your protected applications.

A Shield Advanced subscription provides additional capabilities, beyond those discussed in this post, that supplement your perimeter protection. It provides integration with AWS WAF for level 7 DDoS detection, health-based detection for reducing false positives, enhanced visibility into DDoS events, assistance from the Shield Response team, custom mitigations, and cost-protection safeguards. You can learn more about Shield Advanced capabilities in the AWS Shield Advanced User Guide.

 
If you have feedback about this blog post, submit comments in the Comments section below. You can also start a new thread on AWS Shield re:Post to get answers from the community.

Want more AWS Security news? Follow us on Twitter.

Joe Viggiano

Joe Viggiano

Joe is a Sr. Solutions Architect helping media and entertainment companies accelerate their adoptions of cloud-based solutions.

Deepak Garg

Deepak Garg

Deepak is a Solutions Architect at AWS. He loves diving deep into AWS services and sharing his knowledge with customers. Deepak has background in Content Delivery Networks and Telecommunications.

Metasploit Weekly Wrap-Up

Post Syndicated from Navya Harika Karaka original https://blog.rapid7.com/2022/09/09/metasploit-weekly-wrap-up-175/

Authenticated command injection vulnerability of Cisco ASA-X with FirePOWER Services:

Metasploit Weekly Wrap-Up

[jbaines-r7] (https://github.com/jbaines-r7) added a new module that exploits an authenticated command injection vulnerability CVE-2022-20828 of Cisco ASA-X with FirePOWER Services. This vulnerability affects all Cisco ASA appliances that support ASA FirePOWER module. Note that, although a patch has been added to most recent ASA FirePOWER module versions such as 6.2.3.19, 6.4.0.15, 6.6.7, and 7.0.21, some versions such as 6.2.2 and earlier, 6.3, 6.5, and 6.7 will not receive the patch. This exploit could allow the attacker to get root access and pivot to the inside network along with the outside network. This exploit takes advantage of the FirePower Services SFR module’s Linux virtual machine via ASA’s ASDM web server which also runs snort on the traffic. Therefore, an attacker can have access to the diverted traffic as well. Check out the (https://www.youtube.com/watch?v=_4FEU4GDtB8) of the exploit for more information!

Remote code execution vulnerability of Apache Spark:

KostyaKortchinsky and [h00die-gr3y] (https://github.com/h00die-gr3y) introduced a new module that exploits a remote code execution vulnerability CVE-2022-33891 in Apache Spark. This exploit affects several Apache Spark versions such as 3.0.3 and earlier, 3.1.1 to 3.1.2 and versions 3.2.0 to 3.2.1. Apache Spark allows its users to enable Access Control Lists (ACLs) via the configuration option spark.acls.enable. This was introduced in order to improve the security access within Apache Spark application but the code that’s triggered by this configuration option leads to a malicious shell command injection vulnerability. Check out this post by [HuskyHacks] (https://github.com/HuskyHacks) who provided more information along with great examples!

New module content (2)

Enhancements and features (7)

  • #16901 from bcoles – The post/windows/manage/killav.rb script has been updated to support shell and PowerShell sessions and has undergone some code cleanup. Additionally, documentation has now been created to explain its operations and how to use it.
  • #16934 from bcoles – This adds support for dumping process memory by name in the post/windows/gather/memory_dump module.
  • #16947 from ILightThings – This adds support for formatting buffers for golang.
  • #16948 from gwillcox-r7 – This adds arguments for specifying the username, password and database to the #run_sql post method.
  • #16952 from bcoles – This PR improves the domain_controller? method to allow lower-priv users to invoke it, extends it to support shell sessions, and adds additional useful domain controller enumeration methods to the library.
  • #16973 from HuskyHacks – This adds support for formatting buffers for nim.
  • #16983 from bcoles – This PR adds documentation, references and a more complete description for the firefox_xpi_bootstrapped_addon module.

Bugs fixed (5)

  • #16861 from adfoster-r7 – Fixes a bug in cmd/unix/reverse_ssh that stopped reverse SSH sessions from opening.
  • #16926 from jmartin-r7 – Fixes a bug when using RPC service with the analyze command and specifying a workspace, i.e. within Metasploit RPC client – rpc.call('db.analyze_host', { host: '<metasploitable3 ip>', workspace: 'other' } ).
  • #16968 from luisfso – This PR adds support for the new syntax of the find command’s perm parameter while also maintaining support for the deprecated syntax.
  • #16972 from cgranleese-r7 – Updates msfconsole’s tables to support word wrapping when colors are present.
  • #16974 from jbaines-r7 – Updates Rex::Proto::Http::Client to rely on Ruby’s built in string comparison.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
binary installers (which also include the commercial edition).

New additions to line charts in Amazon QuickSight

Post Syndicated from Bhupinder Chadha original https://aws.amazon.com/blogs/big-data/new-additions-to-line-charts-in-amazon-quicksight/

Amazon QuickSight is a fully-managed, cloud-native business intelligence (BI) service that makes it easy to create and deliver insights to everyone in your organization or even with your customers and partners. You can make your data come to life with rich interactive charts and create beautiful dashboards to be shared with thousands of users, either directly within the QuickSight application, or embedded in web apps and portals.

Line charts are ubiquitous to the world of data visualization and are used to visualize change in data over a dimension. They are a great way to analyze trends and patterns where data points are connected with a straight line to visualize the overall trend. In this post, we look at some of the new improvements to our line charts:

  • Support for missing data points for line and area charts
  • Improved performance and increased data limit to 10,000 data points

Missing data points

Line charts in QuickSight expect you to have data for each X axis item. If data is missing for any X axis item, it can lead to broken lines (default behavior) because there is no line drawn connecting the missing data points.

Drawing lines with points of missing data could be misleading because it would represent incorrect data, and there are valid use cases to do so. For example, imagine a scenario of a retail sales report for a given time period where data is recorded during days of operation (Monday through Saturday). In such cases, instead of displaying a broken line chart that skips Sunday, you may want to show a continuous trend by directly connecting Saturday to Monday, hiding the fact that Sunday isn’t operational. Alternatively, you may want to view store traffic for Sunday as 0 instead of displaying a broken line.

Previously, line charts only supported treating missing data for date/time fields. Now, we have added support for categorical data for both line and area charts. The following are the different line treatment options:

  1. Continuous line – Display continuous lines by directly connecting the line to the next available data point in the series
  2. Show as zero – Interpolate the missing values with zero and display a continuous line
  3. Broken line – Retain the default experience to display disjointed lines over missing values

The following diagram illustrates a line chart using each option.

This new feature applies for both categorical and time series data on area charts as well, as shown in the following graphs.

Authors can also configure different data treatments for the left and right Y axis for dual axis charts, as shown in the following example.

Increased data limit for line charts

With the recent update, we have improved line chart performance to support a maximum of 10,000 data points instead of the previous 2,500 data point limit. This also increases the limit for more line series created by the Color by field, which is also bound by the total data limit. For example, if the line chart has 1,000 data points for each series, you could display up to 10 unique colored series.

This update enables use cases where authors want to show a higher number of data points, such as hourly trends or daily trends for a year (365 data points) for multiple groups. This update doesn’t change the default limits of the Color by field (25) and X axis data point limit (100) that exist today to be compatible with existing dashboards and analysis, until authors choose to customize the limits.

Summary

In this post, we looked at how to treat missing data for line charts, where instead of viewing broken lines, you can display continuous lines. This helps you customize how you want to visualize overall trends and variations depending on the business context. Additionally, we looked at the new data handling limit for line charts, which supports 10,000 data points—four times more data than before. To learn more refer customizing missing data control.

Try out the new feature and share your feedback and questions in the comments section.

About the author

Bhupinder Chadha is a senior product manager for Amazon QuickSight focused on visualization and front end experiences. He is passionate about BI, data visualization and low-code/no-code experiences. Prior to QuickSight he was the lead product manager for Inforiver, responsible for building a enterprise BI product from ground up. Bhupinder started his career in presales, followed by a small gig in consulting and then PM for xViz, an add on visualization product.

Integrating Cloud Security With DevOps and CI/CD Tools

Post Syndicated from Clint Merrill original https://blog.rapid7.com/2022/09/09/integrating-cloud-security-with-devops-and-ci-cd-tools/

Integrating Cloud Security With DevOps and CI/CD Tools

This is the latest post in our blog series on shifting left in cloud security. In our last post, we kicked off the series with a high-level overview about Rapid7’s approach to shifting cloud security into the application development lifecycle. For this post, we’ll dive into a key aspect of our approach: integrating cloud security with developer and DevOps tooling.

Incentivizing adoption by reducing friction

When integrating security into any part of the development lifecycle there are some important factors to consider, including the security tools you’ll integrate, the processes you’ll ask developers to follow, and how aggressively you intend to enforce certain policies. When making these decisions, it’s important to consider the goals of adopting DevOps practices and infrastructure as code (IaC) respectively: to improve the velocity of application development and delivery, and to empower development teams to provision cloud infrastructure resources on a self-service basis.  

Infusing security into these goals requires guardrails and routine checks to make sure the need for speed doesn’t create vulnerabilities or potentially exploitable misconfigurations. For IaC development, this is accomplished by having individual developers scan templates and plans as early as possible, and at key points in the CI/CD pipeline, before they’re considered for use in staging or production deployment. This is much easier said than done, as it relies on organizational buy-in, particularly from the developers who are typically laser-focused on bringing new products and features to market as fast as possible with the highest quality possible.

As with anything that relies on multiple teams collaborating in a process, the goal is to make it as easy as possible to adopt and demonstrate tangible value to all involved. Shifting security left into the software development lifecycle (SDLC) via developers and CI/CD tool integrations is a perfect application of this. One common example is allowing developers to execute scans on IaC templates or plans prior to a push or pull request, using a local command-line interface (CLI) tool.

The comfort of the CLI

In this context, a CLI tool allows a developer to interact with IaC security scanning features via a terminal prompt for familiarity and convenience. This comfortable experience will encourage adoption by using the CLI rather than engaging with a security product interface or API directly. In late 2021, we released our first CLI tool to initiate IaC scans in InsightCloudSec (ICS): mimics.

mimics has many intended uses that will expand over the time, but for now, the primary goals are:

  1. Enabling developers to execute on-demand security scans of their IaC plans and templates with results delivered directly in the CLI, thereby shortening the discovery and feedback loop for security and compliance issues to the point of immediate remediation
  2. Enabling DevOps teams to easily integrate IaC security scans at any point in the CI/CD workflow, thereby standardizing the process and enforcing security compliance checks and remediation as needed before progressing to the next integration or deployment step

In all cases, the mimics CLI simplifies integration and doesn’t require more costly script-based integration with the ICS API.  In some cases, unique IaC security capabilities are exclusively available via mimics.

Introducing GitHub Actions integration

InsightCloudSec recently launched a GitHub Action to facilitate a bidirectional integration with our IaC scanning feature. Our goal is to streamline the incorporation of IaC security scans into your cloud application CI/CD process governed by GitHub. If you’re not familiar with GitHub Actions, they allow you to automate, customize, and execute workflow steps, including security and compliance checks. In doing so, users can discover, create, and share Actions with other community members.

A great use of the mimics CLI is to integrate with GitHub using our Action to trigger an ICS IaC scan at defined points in your workflow. Upon completion of the scan, you’ll receive an overall pass/fail result in reply, as well as detailed findings, if any, in SARIF format for display in the GitHub Advanced Security module as security alerts. If you don’t subscribe to the GitHub Advance Security module, you can still trigger IaC security scans and receive an overall pass/fail result to govern the workflow step, plus a detailed findings report in one of various readable formats.

More DevOps tool integrations on the way

As you can see, Rapid7’s InsightCloudSec is meeting developers and DevOps teams where they are today and expanding in the near future. We want to make integrating security controls by development teams easier. And we aren’t stopping there. We have a deep roadmap of additional integrations that will be coming soon. However, it’s important to note that you’re not limited by our formal integrations. The mimics CLI makes your custom integrations a snap, and we have examples in our product documents.

We understand the profound impact shifting security left can have on organizational buy-in, overall team efficiency, and of course, cloud security outcomes. Keep an eye out for upcoming enhancements that will further help you seamlessly integrate security throughout the entire SDLC.

If you’re interested in learning more about how InsightCloudSec helps your team get contextualized insight into your cloud security and risk posture, be sure to check out our bi-weekly demo series Gaining Layered Context in Cloud Security, which goes live every other Wednesday at 1pm EST.

Additional reading:

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

[$] Compiling Rust with GCC: an update

Post Syndicated from original https://lwn.net/Articles/907405/

While the Rust language has appeal for kernel development, many developers are
concerned by the fact that there is only one compiler available; there are
many reasons why a second implementation would be desirable. At the 2022
Kangrejos gathering, three developers
described projects to build Rust
programs with GCC in two different ways. A fully featured, GCC-based Rust
implementation is still going to take some time, but rapid progress is
being made.

Responsible Disclosure for Cryptocurrency Security

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/09/responsible-disclosure-for-cryptocurrency-security.html

Stewart Baker discusses why the industry-norm responsible disclosure for software vulnerabilities fails for cryptocurrency software.

Why can’t the cryptocurrency industry solve the problem the way the software and hardware industries do, by patching and updating security as flaws are found? Two reasons: First, many customers don’t have an ongoing relationship with the hardware and software providers that protect their funds­—nor do they have an incentive to update security on a regular basis. Turning to a new security provider or using updated software creates risks; leaving everything the way it was feels safer. So users won’t be rushing to pay for and install new security patches.

Second, cryptocurrency is famously and deliberately decentralized, anonymized, and low friction. That means that the company responsible for hardware or software security may have no way to identify who used its product, or to get the patch to those users. It also means that many wallets with security flaws will be publicly accessible, protected only by an elaborate password. Once word of the flaw leaks, the password can be reverse engineered by anyone, and the legitimate owners are likely to find themselves in a race to move their assets before the thieves do. Even in the software industry, hackers routinely reverse engineer Microsoft’s patches to find the security flaws they fix and then try to exploit them before the patches have been fully installed.

He doesn’t have any good ideas to fix this. I don’t either. Just add it to the pile of blockchain’s many problems.

Setup a high availability design for Oracle Data Guard (Fast-Start Failover) using Amazon Route 53

Post Syndicated from Viqash Adwani original https://aws.amazon.com/blogs/architecture/setup-a-high-availability-design-for-oracle-data-guard-fast-start-failover-using-amazon-route-53/

Many customers use Oracle Database deployed on Amazon Elastic Compute Cloud (Amazon EC2) to run their Oracle E-Business Suite applications. They rely on Oracle Data Guard for high availability databases, with a standby database running in a different availability zone. Oracle Data Guard can switch a standby database to the primary role in case a production database becomes unavailable due to planned/unplanned outage.

Oracle E-Business Suite has AutoConfig Database Context files that points to Domain Name System (DNS), like a private IP DNS name or IP address on Amazon EC2. In case of switchover/failover, Database Context files in Oracle E-Business Suite need to be updated. With this solution, database context files do not need to be updated in case of switchover/failover. This is achieved by providing a single DNS name hosted on Amazon Route 53, always pointing to a primary database irrespective of running on any node.

This post demonstrates setting up a Route 53 hosted zone that points to primary and standby databases and will route requests to the database having a primary role. We will setup Route 53 health checks to monitor Amazon CloudWatch alarms, based on Oracle Data Guard Fast-Start Failover (FSFO) logs pushed from Oracle Database using CloudWatch agent.

Prerequisites

Before getting started, you must have the following:

  • Oracle databases running on two separate EC2 instances for 1\Primary and 2\Standby node
  • EC2 instance for 3\Observer node, with either Oracle Client Administrator software or the full Oracle Database software stack
  • Oracle Data Guard configured to maintain standby databases as transactional consistent copy of the primary database
  • Oracle Data Guard Command-Line Interface (DGMGRL) configured with observer process to facilitate FSFO
  • FSFO enabled with observer configuration

Solution overview

Figure 1 depicts AWS services that are used build an architecture using a single Domain Name System (DNS) and Route 53 to have requests route to the primary database for Oracle Data Guard deployed on EC2 instances.

Using a single DNS and Amazon Route 53 to route requests

Figure 1. Using a single DNS and Amazon Route 53 to route requests

We encourage you to explore these articles prior to launching this architecture:

Architecture components

  • Primary node: An Oracle Data Guard configuration contains one production database (primary database) that functions in the primary role. This is the database that is accessed by most of your applications.
  • Standby node: A standby database is a transactional consistent copy of the primary database. Once created, Oracle Data Guard automatically maintains each standby database by transmitting redo data from the primary database and applying it to the standby database.
  • Observer node: A component of DGMGRL that is configured on a separate server with Oracle Client Interface outside the systems running the Oracle Data Guard configuration, which monitors the availability of the primary database. We recommend it is in a separate availability zone than the primary and standby databases. Should it detect that the primary database is unavailable or the connection cannot be made, it will issue a failover after waiting for the 30 seconds or specified by the FastStartFailoverThreshold property.

Note: This solution was tested on Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 and Oracle Enterprise Linux OL7.5-x86_64. Select the required combination of operating system and database by referring to E-Business Suite Database Certifications. Active Oracle Data Guard configuration is not used in this solution; therefore, the database stays in mount state and unavailable for customer reads. However, this solution can be used with an active Oracle Data Guard configuration as well.

Infrastructure setup

This table details the lab environment and the database instance names used throughout this post.

Database role IP address Instance name Database unique name Database open mode Database port
Primary node 172.31.xx.xx ORCLVIQ orclviq Read write 1522
Secondary node 172.31.xx.xx ORCLVIQ orclviq Mounted 1522
FSFO observer node 172.31.xx.xx – – – –
Route 53 hosted zone Dataguardviq.com Dataguardviq.com – – –

Solution implementation

1. IAM policy

To configure the CloudWatch agent on an EC2 instance, create an IAM role via the console, AWS Command Line Interface, or AWS API. By default, an IAM role does not have any privileges and cannot access AWS resources. Before creating an IAM role, create an IAM policy with the permissions required to access CloudWatch logs, specifying the CloudWatch resources you want to monitor or *, which will allow access to CloudWatch logs.

Create the IAM policy using the following JSON and give it a name, like dgpolicy, which grants specific permissions. In this case, the permissions include creating log groups and log streams, inputting log events, and describing log stream for all logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

2. IAM role

Create the IAM role (for example, dgrole) and attach the policy created in Step 1.

3. Associate IAM role with Amazon EC2 having FSFO observer node running on it

Associate the IAM role with Amazon EC2 from AWS console:

  1. Open the Amazon EC2 console.
  2. In the navigation pane, choose Instances.
  3. Select the instance, and choose Action, Security, Modify IAM role.
  4. Select the IAM role to attach to your instance; choose Save.

You also can use the associate-iam-instance-profile command to attach the IAM role to the instance by specifying the instance profile:

aws ec2 associate-iam-instance-profile \
    --instance-id i-1234567890lmnopq1 \
    --iam-instance-profile Name=" dgrole"

4. Install CloudWatch log agent on observer node

Install CloudWatch Logs agent on the observer node that already has FSFO log configuration setup on it. After installation is complete, logs will automatically flow from the instance to the log stream you created while installing the agent, as depicted in Figures 2 and 3.

$curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
$sudo python ./awslogs-agent-setup.py --region ap-southeast-2

Steps to install Amazon CloudWatch agent

Figure 2. Steps to install Amazon CloudWatch agent

Example of output

Figure 3. Example of output

At this point, FSFO logs are visible in CloudWatch logs. Perform a database switchover to confirm that logs are being published to CloudWatch logs.

5. Create CloudWatch metric filter

Create primary metric filter

  • Create metric filter on “Standby database has changed to orclviq” and set value to “1”.
  • Open the CloudWatch log group and search for “Standby database has changed to orclviq“. Once results are displayed, “Create Metric Filter” button will appear at top right (as demonstrated in Figure 4).
Amazon CloudWatch log group

Figure 4. Amazon CloudWatch log group

  • “Filter name” and “Filter pattern” are filled automatically, as we already filtered that while accessing the CloudWatch log. Create a new name space in the metric, which we will later use in second metric filter. Specify “Metric namespace”, which will also be the same for the second metric filter. “Metric value” is “1”, as detailed in Figure 5.
Amazon CloudWatch metric filter

Figure 5. Amazon CloudWatch metric filter

Create secondary metric filter

  • To create a second metric filter, follow steps to create the primary metric filter but the filter pattern will be “Standby database has changed to orclstd“. The only difference in this step will be setting the “Metric value” to “0”.
  • Verify the metric filter is working and able to find desired entries in the FSFO logs that will decide which instance is the primary database by:
    • Click on one of the metric filters
    • Select log data to test, and choose and EC2 instance ID
    • Testing the pattern to verify if metric filters are working properly
    • Select “Next”
    • Save changes

6. Create a CloudWatch alarm

Create a CloudWatch alarm that will monitor CloudWatch metrics and perform actions based on the value of the metric.

  • Within CloudWatch Alarms section (Step 1), select “Create alarm” and then select “Metric”, as in Figure 6.
  • In Step 2, you can create a new topic and specify the email address where you would like to receive notifications regarding changes in primary or standby database.
  • In Step 3, you can specify the alarm name and optional alarm description.
Amazon CloudWatch alarm

Figure 6. Amazon CloudWatch alarm

  • To configure Conditions, as detailed in Figure 7, be sure the threshold type is set to “Static”, the alarm condition is “Greater/Equal”, and the threshold value is “1”.
Amazon CloudWatch alarm condition

Figure 7. Amazon CloudWatch alarm condition

  • Figure 8 demonstrates the summary of the ready CloudWatch metrics and configured alarm.  At this point, “orclviq” is the primary database, so the alarm state will be “Insufficient data”. Try to do a switchover and the alarm state should be changed to “In alarm”.
  • Switch it back to original primary before proceeding further
Final view of both metric filters combined

Figure 8. Final view of both metric filters combined

7. Create Route 53 health checks

Creating Route 53 health checks based on the CloudWatch alarm identifies DNS failover. Health checks can be created on public IP directly without creating above metric filters and alarms, but it is unlikely that customers will have a public IP on database servers. Route 53 cannot check the health of an IP address endpoint despite if it is local, private, non-routable, or multi-cast ranges. Therefore, health checks will have to be created on the state of CloudWatch alarm (Figure 9):

  • Within the Route 53 console, select “Configure health check”.
  • Select “State of CloudWatch alarm”, then select the region of your CloudWatch metrics and choose the CloudWatch alarm that was created earlier from the dropdown menu.
  • Skip creating an alarm for health check, as the alarm is already configured for CloudWatch metric filter.
Amazon Route 53 health check parameters

Figure 9. Amazon Route 53 health check parameters

  • The health check is now ready and status will be “Unknown”. It will change to “Healthy”, as it is currently having primary orclviq, which means it’s satisfying “Standby database has changed to orclstd“.
  • At this point, try doing a failover and observe if health check status changes to “Unhealthy”. Switch it back to the original primary before proceeding.
  • The health check should return to “Healthy” state.
  • Check the status of the alarm:
    • OK: the status is healthy
    • ALARM: the status is unhealthy
    • INSUFFICIENT: use last known status

8. Create Route 53 hosted zone

A Route 53 hosted zone in this solution will have failover routing based on records pointing to IP addresses of primary and standby database server. When the Route 53 health check is in an “Unhealthy” state, failover routing kicks in and the hosted zone starts routing requests to the IP address of the standby database server.

  • A Route 53 hosted zone will have records of your primary and standby database instance private IP. Because it is a private hosted zone, it will use the same VPC as the observer node on which we deployed the CloudWatch agent (Figure 10).
Amazon Route 53 hosted zone

Figure 10. Amazon Route 53 hosted zone

  • Once the Route 53 hosted zone is ready, the last step is to create records by adding private IP addresses of primary and secondary database servers (Figures 11 and 12). Routing policy is set as a failover. That means, if the Route 53 health check fails, the hosted zone does a DNS failover to standby server IP.
Amazon Route 53 hosted zone record for primary servers

Figure 11. Amazon Route 53 hosted zone record for primary servers

Amazon Route 53 hosted zone record for secondary servers

Figure 12. Amazon Route 53 hosted zone record for secondary servers

Once the records are created, the hosted zone will have an IP address of primary and secondary database servers, as demonstrated in Figure 13.

Final view of Amazon Route 53 hosted zone

Figure 13. Final view of Amazon Route 53 hosted zone

Solution testing

To test the solution, do a telnet utility test to evaluate network connectivity on the Route 53 hosted zone that was created. It should display the IP address of the primary database server: 172.31.10.89.

[ec2-user@ip-172-xx-xx-xx ~]$ telnet dataguardviq.com 1522
Trying 172.31.10.89…
Connected to dataguardviq.com.
Escape character is ‘^]’.

Next, perform a switchover or failover of the primary to secondary database server. This can be done with the DGMGRL command “switchover to orclstd”. Output will be:

DGMGRL> switchover to orclstd;
Performing switchover NOW, please wait…
Operation requires a connection to database “orclstd”
Connecting …
Connected to “orclstd”
Connected as SYSDBA.
New primary database “orclstd” is opening…
Oracle Clusterware is restarting database “orclviq” …
Connected to an idle instance.
Connected to an idle instance.
Connected to an idle instance.
Connected to an idle instance.
Connected to “orclviq”
Connected to “orclviq”
Switchover succeeded, new primary is “orclstd”

Now, the new primary is “orclstd”. In the backend, the Route 53 health check will be initiated and cause a failover. This means the telnet Route 53 hosted zone will give the IP address of the new primary (old secondary), which is 172.31.36.241.

[ec2-user@ip-172-xx-xx-xx ~]$ telnet dataguardviq.com 1522
Trying 172.31.36.241…
Connected to dataguardviq.com.
Escape character is ‘^]’.

Cleanup

To cleanup, remove:

  • Route 53 host zone
  • CloudWatch metric filter
  • CloudWatch alarm
  • CloudWatch agent from observer node
  • IAM role and policy created specifically for this solution

Conclusion

This solution demonstrated how to use a Route 53 hosted zone to route requests to the active primary node for Oracle Data Guard on Amazon EC2.

What’s Up, Home? – Zabbix the Climate Remote

Post Syndicated from Janne Pikkarainen original https://blog.zabbix.com/whats-up-home-zabbix-the-climate-remote/23207/

Can you control your air conditioner using Zabbix? Of course, you can!

By day, I am a monitoring technical lead in a global cyber security company. By night, I monitor my home with Zabbix and Grafana and do some weird experiments with them. Welcome to my weekly blog about this project.

A few months ago I added our recently bought air conditioner to Zabbix monitoring by utilizing MQTT. Reading values is nice and mandatory for monitoring; but can we also send commands to the air conditioner using Zabbix? Yes.

Testing, testing

MQTT is a little unknown area for me, so please don’t hit me too hard if I’m doing this completely wrong. Anyway, this seems to work! I know that Zabbix supports MQTT for reading data, but I’m not sure if it can do writing, so that’s why I am using a command line and external scripts for my example.

At first, I tested if my whole idea would work just by using the command line:

The mosquitto_pub command might sound like a bar where some summer-time insects spend their free time, but no, this one instead publishes an MQTT message with value 22 to that aircon topic. In other words, I tried to change the temperature to 22 degrees Celsius.

And it worked!

From here, the rest was easy. I created three shell one-liners:

  • One for setting AC mode (off, cooling, etc.)
  • One for setting AC fan speed
  • One for setting AC temperature

These then receive some value from Zabbix by taking the command line argument. Yeah, I know, I didn’t add any input validation here.

Adding it to Zabbix

Now that I have my scripts, I then went to Zabbix Administration –> Scripts and added some scripts there.

I gave these scripts a menu tree structure, so from now on I can control my AC from Zabbix and it looks like this.

Setting fan speed:

Setting AC mode:

Setting temperature:

… and whenever I execute any of these, I get mosquitto_pub command output back, here setting the temperature to 22 C.

Of course, the scripts could be hooked with triggers, so if for example, our living room would be too hot, Zabbix could power on the AC automatically. For now, I’m not taking the automatic route as we might not be at home all the time.

A Grafana dashboard

And, like for so many other monitored items, I also have a separate Grafana dashboard for our air conditioning, with values being read from Zabbix. From the dashboard, I can easily see whenever our AC has been on or off, what’s the fan speed and so on.

And btw, those of you using Zabbix 6.2 and Grafana — please make sure you upgrade your Grafana Zabbix plugin to at least version 4.2.9 (released on July 12th, 2022), as the previous versions did not work with Zabbix 6.2 at all. I found this out the hard way at home, but I patiently waited for the update.

I have been working at Forcepoint since 2014 and it never has been cooler to work from home. — Janne Pikkarainen

This post was originally published on the author’s LinkedIn account.

The post What’s Up, Home? – Zabbix the Climate Remote appeared first on Zabbix Blog.

Елегия за пасти и избиратели

Post Syndicated from Емилия Милчева original https://toest.bg/elegiya-za-pasti-izbirateli/

Предизборната кампания e времето, в което българските политици се запознават с избирателите – с тези, които ще гласуват, и с тези, които няма да гласуват. Обещават им пасти, а гражданите не им вярват, защото са зрели хора и са наясно, че пастите ще си ги изядат политиците. Също и първото, и второто преди пастите, след като сложат табелата „Забранено за външни лица“.

Политиците наричат това предизборни платформи, гражданите – реалност, иначе си е политически цинизъм. Медиите дежурно критикуват, че партиите (пак) не представят идеи и политики, макар и те, и аудиториите им да са наясно, че повечето избиратели не четат предизборни програми, дори да ги има, а политическите сили не ги изпълняват, когато спечелят властта.

Парадоксът на обещанията

Защо политиците нарушават обещанията си, които са техният залог да бъдат избирани от гражданите? Лъжат съзнателно, неискрени са, знаейки, че могат да се справят само с някои проблеми, или наистина си вярват? Разбира се, обещанията се изпълняват по-често, когато една политическа формация не трябва да споделя властта с други, както е в коалиционните правителства. Макар че и първият самостоятелен кабинет на ГЕРБ не изпълни най-значимото обещание, с което Бойко Борисов дойде на власт – да накаже за корупцията на тройната коалиция. Впоследствие този Батман на политическия ни Готъм се утвърди чрез други митологеми – като Строителя например.

Коалиционното управление вече е норма за българската политика, което означава политики на компромиса. В българския модел коалицията се разбира не като споделена, а разпарчетосана власт на принципа „всяка коза за свой крак“ – тоест „концесиониране“ на министерства. Въпреки това някои червени линии в бившата четворна коалиция се оказаха непреодолими. Например онази на БСП да не се изпраща оръжие за Украйна. Други, като получервената линия за „депутинизация“ на страната ни, заявена от „Демократична България“, си остана на ниво декларация. „Продължаваме промяната“ пък преди изборите през ноември м.г. предлагаше закриване на Българския енергиен холдинг, но впоследствие от тази идея не остана и помен, след като БСП се възпротивиха на масата на преговорите. Така че няма гаранция за изпълнение на нито едно предизборно обещание, дадено от политическите сили с шанс за представителство в 48-мия парламент.

Освен коалиционните компромиси, фактори за изпълнение на обещанията са и икономическата ситуация, бизнес лобитата зад съответните политически сили, ефективното лидерство и също така предишен управленски опит. Последното се оказа съществен проблем за управлявалата половин година коалиция, тъй като, с изключение на БСП, нито една от останалите три политически сили нямаше такъв опит, а в добавка – и кадрови капацитет. В контекста на войната в Украйна обаче някои компромиси са просто неприемливи – като например кремълският уклон на „Позитано“ 20, който ПП и ДБ се правеха, че не забелязват преди 24 февруари.

Битка, ама не съвсем

Предизборната кампания в България щеше да придобие смисъл и добавена стойност, ако имаше поредица дебати между политическите сили по проблеми, които най-силно вълнуват избирателите. Но не в познатия формат по БНР и БНТ, където журналистите са секунданти, а кандидат-депутатите – високоговорители. В настоящия си вид кампанията е скучна като слово на чиновник от обреден дом, движейки се по пунктира на опорките: „говорете това“, „не говорете онова“, „повтаряйте и потретяйте следното“, изливайки се в официални или скрити платени форми. Преглед на сайтовете на политическите сили и профилите им в YouTube и социалните мрежи показва нейната вялост и еднообразие.

Отваряте например сайта на ГЕРБ и Бойко Борисов е на преден – и всякакъв – план. Борисов тук, Борисов там, Борисов каза, Борисов на трибуна, на пресконференция, сред мъже, жени и деца, награбил баби и гражданки, че и Лили Иванова в Стара Загора. Шестнайсети поредни избори „Граждани за европейско развитие на България“ все още е равно на Борисов. Лидерът иска да вдъхне оптимизъм, защото „държавата е счупена“, да въведе ред, изобщо – да я поправи. И както му отива на такъв ред, от ГЕРБ извадиха и антикризисна програма – единствени. Година и половина откакто партията на Борисов загуби властта, бездействието на прокуратурата по множество сигнали в комбинация с войната в Украйна притъпи спомените за „наследството“ от ГЕРБ – корупцията и безправовата държава.

Иначе в обкръжението на граждани Борисов е естествен, благ, насмешлив, даже костюмиран и внушава онова, което в криза всички ожидат – ста-бил-ност. Но въпреки тия феромони ГЕРБ не регистрира ръст на подкрепящите я. Само Борисов вече не е достатъчен.

А обиколката на автобуса на Промяната, срещан радушно от хората по места в кампанията миналата есен, този път не проработи. Затова липсват видеа, в които Кирил Петков, Асен Василев и други от ПП слизат по стъпалата и ги понася потокът от екзалтирани граждани. Визуалните материали от обиколките са доста селектирани, повечето видеа показват говорещи глави, а коментарите под тях са впечатляващо еднотипни – „Успех“ и „С вас сме“. Срещите с хора са сведени до възторжени изблици или ръкуване и кратък диалог – като този от Монтана.

От изброените три успеха – увеличението на пенсиите, безплатните детски градини и спрения канал на ГКПП „Капитан Андреево“ – видно този път ПП са се прицелили в пенсионерите. И не обелват дума срещу президента дори когато той ги напада с откровено неверни твърдения – че огромният скок в цената на газа се дължи на два танкера за втечнен газ, договорени през юли. От изявления на енергийни експерти и мениджъри на „Булгаргаз“ стана ясно, че втечненият газ, закупен от американската компания „Шениер“, е по-евтин от руския газ. Освен това танкерите дойдоха през юни, а „Булгаргаз“ предложи с 13% по-ниска цена на природния газ от 1 юни.

И все пак от ПП замълчаха. Ще им се да си върнат ония гласове, които получиха през ноември м.г. – и не говорят за геополитика, Русия и газови зависимости.

Самураи, физзарядка, платна, гъби… и „кални бани“

ГЕРБ и ПП, на които известните до момента социологически проучвания отреждат съответно първо и второ място, влизат в леки схватки – като спаринг партньори. Кирил Петков подхвърля нещо за пенсиониране на Бойко Борисов, който пък му отвръща, че е най-скоростно пенсионираният премиер – и толкоз. Като цяло до момента между политическите сили липсват сериозни атаки, както беше с „изчегъртването“ на ГЕРБ миналата година. Сега работата опря май само до изчегъртване на Борисов. Всички в момента ходят на пръсти и не преиграват повече от минимума, нужен за предизборна кампания. Дори в парламента бяха по-настървени.

Този стереотип бива разчупен от „криейтив“, който особено забавлява публиката. Като самураите с мечове от предизборния клип на „Има такъв народ“ или „физзарядката“ на ПП при откриването на кампанията в Пловдив, на която партийците изобразяват българския трикольор с цветни листове в ръце.

Кампанията на „Демократична България“ – коалицията между „Да, България“, ДСБ и Зеленото движение, също се отличи с проблясък. Лидерът на ДСБ ген. Атанас Атанасов, изпратен да води във Варна (ДСБ за първи път няма водач в нито една от трите софийски листи), пусна видео от платноходка с призив да „вдигаме платната“. Чудесно, но вдигането на платната означава само едно – да отплаваме. Видеото вече е свалено, оставени са само снимки от откриването на кампанията.

„Атака“ пробват с анимирано видео, озаглавено „Ти отказа ли ментетата“, в което главният герой се разхожда в гора, натъква се на две гъби (не халюциногенни, а като от детска книжка за Гъбарко) и се колебае коя от двете да избере. Избира отровния двойник… Развръзката – в образа на Волен Сидеров.

В кампанията бе употребена и човешката трагедия – калният потоп, унищожил домове, покъщнина, дърва за огрев и зимнина на хората от няколко карловски села. Селфи с текст „На кални бани сме. Честит празник!“ пуснаха двама кандидати на „Демократична България“ – Теодор Михайлов и Елисавета Белобрадова, широко усмихнати. По-късно неуместната снимка бе изтрита, останаха фотоси как помагат за разчистването на терена. Последваха ги и от „Продължаваме промяната“ – Кирил Петков, който се бори с лопата, при това по тениска, брандирана с логото на ПП, и Никола Минчев, който се преоблича в защитен костюм, за да направи същото. Щеше да е по-полезно вместо този изблик на хуманизъм да бяха купили дърва за колкото семейства могат – мнозина загубиха огрева си за зимата в потопа. Е, нямаше как да има снимки в социалните мрежи, но и от тези имиджовият ефект едва ли ще е значим.

Иначе кампанията си бълбука – като за начало. До 2 октомври може и да се ожълти. А „Български възход“ на Стефан Янев я стартира в Бургас, от откритата сцена „Охлюва“. Чудесно място да вдигнат платната.

Заглавна снимка: © Севда Семер

Източник

Introducing message data protection for Amazon SNS

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/introducing-message-data-protection-for-amazon-sns/

This post is written by Otavio Ferreira, Senior Software Development Manager, Marc Pinaud, Senior Product Manager, Usman Nisar, Senior Software Engineer, Hardik Vasa, Senior Solutions Architect, and Mithun Mallick, Senior Specialist Solution Architect.

Today, we are announcing the public preview release of new data protection capabilities for Amazon Simple Notification Service (SNS), message data protection. This is a new way to discover and protect sensitive data in motion at scale, without writing custom code.

SNS is a fully managed serverless messaging service. It provides topics for push-based, many-to-many pub/sub messaging for decoupling distributed systems, microservices, and event-driven serverless applications. As applications grow, so does the amount of data transmitted and the number of systems sending and receiving data. When moving data between different applications, guardrails can help you comply with data privacy regulations that require you to safeguard sensitive personally identifiable information (PII) or protected health information (PHI).

With message data protection for SNS, you can scan messages in real time for PII/PHI data and receive audit reports containing scan results. You can also prevent applications from receiving sensitive data by blocking inbound messages to an SNS topic or outbound messages to an SNS subscription. Message data protection for SNS supports a repository of over 25 unique PII/PHI data identifiers. These include people’s names, addresses, social security numbers, credit card numbers, and prescription drug codes.

These capabilities can help you adhere to a variety of compliance regulations, including HIPAA, FedRAMP, GDPR, and PCI. For more information, including the complete list of supported data identifiers, see message data protection in the SNS Developer Guide.

Overview

SNS topics enable you to integrate distributed applications more easily. As applications become more complex, it can become challenging for topic owners to manage the data flowing through their topics. Developers that publish messages to a topic may inadvertently send sensitive data, increasing regulatory risk. Message data protection enables SNS topic owners to protect sensitive application data with built-in, no-code, scalable capabilities.

To discover and protect data flowing through SNS topics with message data protection, topic owners associate data protection policies to their topics. Within these policies, you can write statements that define which types of sensitive data you want to discover and protect. As part of this, you can define whether you want to act on data flowing inbound to a topic or outbound to a subscription, which AWS accounts or specific AWS Identity and Access Management (AWS IAM) principals the policy is applicable to, and the actions you want to take on the data.

Message data protection provides two actions to help you protect your data. Auditing, to report on the amount of PII/PHI found, and blocking, to prevent the publishing or delivery of payloads that contain PII/PHI data. Once the data protection policy is set, message data protection uses pattern matching and machine learning models to scan your messages in real time for PII/PHI data identifiers and enforce the data protection policy.

For auditing, you can choose to send audit reports to Amazon Simple Storage Service (S3) for archival, Amazon Kinesis Data Firehose for analytics, or Amazon CloudWatch for logging and alarming. Message data protection does not interfere with the topic owner’s ability to use message data encryption at rest, nor with the subscriber’s ability to filter out unwanted messages using message filtering.

Applying message data protection in a use case

Consider an application that processes a variety of transactions for a set of health clinics, an organization that operates in a regulated environment. Compliance frameworks require that the organization take measures to protect both sensitive health records and financial information.

Reference architecture

The application is based on an event-driven serverless architecture. It has a data protection policy attached to the topic to audit for sensitive data and prevent downstream systems from processing certain data types.

The application publishes an event to an SNS topic every time a patient schedules a visit or sees a doctor at a clinic. The SNS topic fans out the event to two subscribed systems, billing and scheduling. Each system stores events in an Amazon SQS queue, which is processed using an AWS Lambda function.

Setting a data protection policy to an SNS topic

You can apply a data protection policy to an SNS topic using the AWS Management Console, the AWS CLI, or the AWS SDKs. You can also use AWS CloudFormation to automate the provisioning of the data protection policy.

This example uses CloudFormation to provision the infrastructure. You have two options for deploying the resources:

  • Deploy the resources by using the message data protection deploy script within the aws-sns-samples repository in GitHub.
  • Alternatively, use the following four CloudFormation templates in order. Allow time for each stack to complete before deploying the next stack, to create the following resources:

1. Prerequisites template

  • Two IAM roles with a managed policy that allows access to receive messages from the SNS topic, one for the billing and another for scheduling system, respectively.

2. Topic owner template

  • SNS topic that delivers events to two distinct systems.
  • A data protection policy that defines both auditing and blocking actions for specific types of PII and PHI.
  • S3 bucket to archive audit findings.
  • CloudWatch log group to monitor audit findings.
  • Kinesis Data Firehose to deliver audit findings to other destinations.

3. Scheduling subscriber template

  • SQS queue for the Scheduling system.
  • Lambda function for the Scheduling system.

4. Billing subscriber template

  • SQS queue for the Billing system.
  • Lambda function for the Billing system.

CloudFormation creates the following data protection policy as part of the topic owner template:

  ClinicSNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      TopicName: SampleClinic
      DataProtectionPolicy:
        Name: data-protection-example-policy
        Description: Policy Description
        Version: 2021-06-01
        Statement:
          - Sid: audit
            DataDirection: Inbound
            Principal:
             - '*'
            DataIdentifier:
              - 'arn:aws:dataprotection::aws:data-identifier/Address'
              - 'arn:aws:dataprotection::aws:data-identifier/AwsSecretKey'
              - 'arn:aws:dataprotection::aws:data-identifier/DriversLicense-US'
              - 'arn:aws:dataprotection::aws:data-identifier/EmailAddress'
              - 'arn:aws:dataprotection::aws:data-identifier/IpAddress'
              - 'arn:aws:dataprotection::aws:data-identifier/NationalDrugCode-US'
              - 'arn:aws:dataprotection::aws:data-identifier/PassportNumber-US'
              - 'arn:aws:dataprotection::aws:data-identifier/Ssn-US'
            Operation:
              Audit:
                SampleRate: 99
                FindingsDestination:
                  CloudWatchLogs:
                    LogGroup: !Ref AuditCWLLogs
                  Firehose:
                    DeliveryStream: !Ref AuditFirehose
                NoFindingsDestination:
                  S3:
                    Bucket: !Ref AuditS3Bucket
          - Sid: deny-inbound
            DataDirection: Inbound
            Principal:
              - '*'
            DataIdentifier:
              - 'arn:aws:dataprotection::aws:data-identifier/PassportNumber-US'
              - 'arn:aws:dataprotection::aws:data-identifier/Ssn-US'
            Operation:
              Deny: {}
          - Sid: deny-outbound-billing
            DataDirection: Outbound
            Principal:
              - !ImportValue "BillingRoleExportDataProtectionDemo"
            DataIdentifier:
              - 'arn:aws:dataprotection::aws:data-identifier/NationalDrugCode-US'
            Operation:
              Deny: {}
          - Sid: deny-outbound-scheduling
            DataDirection: Outbound
            Principal:
              - !ImportValue "SchedulingRoleExportDataProtectionDemo"
            DataIdentifier:
              - 'arn:aws:dataprotection::aws:data-identifier/Address'
              - 'arn:aws:dataprotection::aws:data-identifier/CreditCardNumber'
            Operation:
              Deny: {}

This data protection policy defines:

  • Metadata about the data protection policy, for example name, description, version, and statement IDs (sid).
  • The first statement (sid: audit) scans inbound messages from all principals for addresses, social security numbers, driver’s license, email addresses, IP addresses, national drug codes, passport numbers, and AWS secret keys.
    • The sampling rate is set to 99% so almost all messages are scanned for the defined PII/PHI.
    • Audit results with findings are delivered to CloudWatch Logs and Kinesis Data Firehose for analytics. Audit results without findings are archived to S3.
  • The second statement (sid: deny-inbound) blocks inbound messages to the topic coming from any principal, if the payload includes either a social security number or passport number.
  • The third statement (sid: deny-outbound-billing) blocks the delivery of messages to subscriptions created by the BillingRole, if the messages include any national drug codes.
  • The fourth statement (sid: deny-outbound-scheduling) blocks the delivery of messages to subscriptions created by the SchedulingRole, if the messages include any credit card numbers or addresses.

Testing the capabilities

Test the message data protection capabilities using the following steps:

  1. Publish a message without PII/PHI data to the Clinic Topic. In the CloudWatch console, navigate to the log streams of the respective Lambda functions to confirm that the message is delivered to both subscribers. Both messages are delivered because the payload contains no sensitive data for the data protection policy to deny. The log message looks as follows:
    "This is a demo! received from queue arn:aws:sqs:us-east-1:111222333444:Scheduling-SchedulingQueue"
  2. Publish a message with a social security number (try ‘SSN: 123-12-1234’) to the Clinic Topic. The request is denied, and an audit log is delivered to your CloudWatch Logs log group and Firehose delivery stream.
  3. Navigate to the CloudWatch log console and confirm that the audit log is visible in the /aws/vendedlogs/clinicaudit CloudWatch log group. The following example shows that the data protection policy (sid: deny-inbound) denied the inbound message as the payload contains a US social security number (SSN) between the 5th and the 15th character. 
    {
    "messageId": "77ec5f0c-5129-5429-b01d-0457b965c0ac",
    "auditTimestamp": "2022-07-28T01:27:40Z",
    "callerPrincipal": "arn:aws:iam::111222333444:role/Admin",
    "resourceArn": "arn:aws:sns:us-east-1:111222333444:SampleClinic",
    "dataIdentifiers": [
        {
            "name": "Ssn-US",
            "count": 1,
            "detections": [
                {
                    "start": 5,
                    "end": 15
                }
            ]
        }
    ]
}
  4. You can use the CloudWatch metrics, MessageWithFindings and MessageWithNoFindings, to track how frequently PII/PHI data is published to an SNS topic. Here’s an example of what the CloudWatch metric graph looks like as the amount of sensitive data published to a topic varies over time:
    CloudWatch metric graph
  5. Publish a message with an address (try ‘410 Terry Ave N, Seattle 98109, WA’). The request is only delivered to the Billing subscription. The data protection policy (sid: deny-outbound-scheduling) denies the outbound message to the Scheduling subscription as the payload contains an address.
  6. Confirm that the message is only delivered to the Billing Lambda function by navigating to the CloudWatch console and inspecting the logs of the two respective Lambda functions. The CloudWatch log of the Billing Lambda function contains the sensitive message that was delivered to it as it was an authorized subscriber. Here’s an example of what the log contains:410 Terry Ave N, Seattle 98109, WA received from queue arn:aws:sqs:us-east-1:111222333444:Billing-BillingQueue
  7. Publish a message with a drug code (try ‘NDC: 0777-3105-02’). The request is only delivered to the Scheduling subscription. The data protection policy (sid: deny-outbound-billing) denies the outbound message to the Billing subscription as the payload contains a drug code.
  8. Confirm that the message is only delivered to the Scheduling Lambda function by navigating to the CloudWatch console and inspecting the logs of the two respective Lambda functions. The CloudWatch log of the Scheduling Lambda function contains the sensitive message that was delivered to it as it was an authorized subscriber. Here’s an example of what the log contains:
    NDC: 0777-3105-02 received from queue arn:aws:sqs:us-east-1:111222333444:Scheduling-SchedulingQueue

Cleaning up

After testing, avoid incurring usage charges by deleting the resources that you created. Navigate to the CloudFormation console and delete the four CloudFormation stacks that you created during the walkthrough. Remember, you must delete all the objects from the S3 bucket before deleting the stack.

Conclusion

This post shows how message data protection enables a topic owner to discover and protect sensitive data that is exchanged through SNS topics. The example shows how to create a data protection policy that generates audit reports for sensitive data and blocks messages from delivery to specific subscribers if the payload contains sensitive data.

Get started with SNS and message data protection by using the AWS Management Console, AWS Command Line Interface (CLI), AWS SDKs, or CloudFormation.

For more details, see message data protection in the SNS Developer Guide. For information on pricing, see SNS pricing.

For more serverless learning resources, visit Serverless Land.

The collective thoughts of the interwebz