AWS re:Invent 2021 Security Track Recap

Post Syndicated from Marta Taggart original https://aws.amazon.com/blogs/security/aws-reinvent-2021-security-track-recap/

Another AWS re:Invent is in the books! We were so pleased to be able to host live in Las Vegas again this year. And we were also thrilled to be able to host a large virtual audience. If you weren’t able to participate live, you can now view some of the security sessions by visiting the AWS Events Channel on YouTube and checking out the AWS re:Invent 2021 Breakout Sessions for Security playlist. The following is a list of some of the security-focused sessions you’ll find on the playlist:

Security Leadership Session: Continuous security improvement: Strategies and tactics – SEC219
Steve Schmidt, Sarah Cecchetti, and Thomas Avant

In this session, Stephen Schmidt, Chief Information Security Officer at AWS, addresses best practices for security in the cloud, feature updates, and how AWS handles security internally. Discover the potential future of tooling for security, identity, privacy, and compliance.

AWS Security Reference Architecture: Visualize your security – SEC203
Neal Rothleder and Andy Wickersham

How do AWS security services work together and how do you deploy them? The new AWS Security Reference Architecture (AWS SRA) provides prescriptive guidance for deploying the full complement of AWS security services in a multi-account environment. AWS SRA describes and demonstrates how security services should be deployed and managed, the security objectives they serve, and how they interact with one another. In this session, learn about these assets, the AWS SRA team’s design decisions, and guidelines for how to use AWS SRA for your security designs. Discover an authoritative reference to help you design and implement your own security architecture on AWS.

Introverts and extroverts collide: Build an inclusive workforce – SEC204
Jenny Brinkley and Eric Brandwine

In this session, hear from the odd couple of AWS security on how to build diverse teams and develop proactive security cultures. Jenny Brinkley, Director of AWS Security, and Eric Brandwine, VP and Distinguished Engineer for AWS Security, provide best practices for and insights into the mechanisms applied to the AWS Security organization. Learn how to not only scale your programs, but also drive real business change.

Security posture monitoring with AWS Security Hub at Panasonic Avionics – SEC205
Himanshu Verma and Anand Desikan

In this session, learn to proactively monitor, identify, and protect data to help maintain security and compliance with low operational investment. Panasonic Avionics shares their robust security solution for migrating to Amazon S3 to reduce data center costs by more than 85 percent while remaining secure and compliant with comprehensive industry regulations. Discover best practices for deploying layered security to monitor data using Amazon Macie, learn how to detect threats using Amazon GuardDuty, and consider how automating responses can help protect your data and meet your compliance requirements. Explore how you can use AWS Security Hub as a central monitoring and posture-management control point.

[New Launch] AWS Shield: Automated layer 7 DDoS mitigation – SEC226
Kevin Lee and Chido Chemambo

In this session, learn how you can use automated application layer 7 (L7) DDoS mitigations with AWS Shield Advanced to protect your web applications. You can now use on AWS Shield Advanced to automatically recommend AWS WAF rules in response to an L7 DDoS event instead of manually crafting an AWS WAF rule to isolate the malicious traffic, evaluating the rule’s effectiveness, and then deploying it throughout your environment. AWS Shield Advanced is effective at alerting application owners when spikes in traffic may impact the availability of applications. Now, AWS Shield Advanced can automatically detect and mitigate L7 traffic anomalies that risk impacting application availability and responsiveness.

Locks without keys: AWS and confidentiality – SEC301
Colm MacCárthaigh

Every day AWS works with organizations and regulators to host some of the most sensitive workloads in industry and government. In this session, hear how AWS secures data even from trusted AWS operators and services. Learn about the AWS Nitro System and how it provides confidential computing and a trusted execution environment. Also, learn about the cryptographic chains of custody that are built into the AWS Identity and Access Management service, including how encryption is used to provide defense in depth and why AWS focuses on verified isolation and customer transparency.

Use AWS to improve your security posture against ransomware – SEC308
Merritt Baer and Megan O’Neil

Ransomware is not specific to the cloud—in fact, AWS can provide increased visibility and control over your security posture against malware. In this session, learn ways that enterprises can empower and even inoculate themselves against malware, including ransomware. From IAM policies and the principle of least privilege to AWS services like Amazon GuardDuty, AWS Security Hub for actionable insights, and CloudEndure Disaster Recovery and AWS Backup for retention and recovery, this session provides clarity into tools and approaches that can help you feel confident in your security posture against current malware.

Securing your data perimeter with VPC endpoints – SEC318
Becky Weiss

In this session, learn how to use your network perimeter as a straightforward defensive perimeter around your data in the cloud. VPC endpoints were first introduced for Amazon S3 in 2015 and have since incorporated many improvements, enhancements, and expansions. They enable you to lock your data into your networks as well as assert network-wide security invariants. This session provides practical guidance on what you can do with VPC endpoints and details how to configure them as part of your data perimeter strategy.

A least privilege journey: AWS IAM policies and Access Analyzer – SEC324
Brigid Johnson

Are you looking for tips and tools for applying least privilege permissions for your users and workloads? Love demonstrations and useful examples? In this session, explore advanced skills to use on your journey to apply least privilege permissions in AWS Identity and Access Management (IAM) by granting the right access to the right identities under the right conditions. For each stage of the permissions lifecycle, learn how to look at IAM policy specifics and use IAM Access Analyzer to set, verify, and refine fine-grained permissions. Get a review of the foundations of permissions in AWS and dive into conditions, tags, and cross-account access.

If watching these sessions has you thinking about your next hands-on learning opportunity with AWS Security, we invite you to save the date for AWS re:Inforce 2022. AWS re:Inforce, our learning conference focused on cloud security, compliance, identity, and privacy, will be held June 28-29, 2022 in Houston, Texas. We hope to see you there!

 
If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security news? Follow us on Twitter.

Author

Marta Taggart

Marta is a Seattle-native and Senior Product Marketing Manager in AWS Security Product Marketing, where she focuses on data protection services. Outside of work you’ll find her trying to convince Jack, her rescue dog, not to chase squirrels and crows (with limited success).

2022 Cybersecurity Predictions: The Experts Clear Off the Crystal Ball

Post Syndicated from Jesse Mack original https://blog.rapid7.com/2022/01/06/2022-cybersecurity-predictions-the-experts-clear-off-the-crystal-ball/

2022 Cybersecurity Predictions: The Experts Clear Off the Crystal Ball

As we walk through the doorway of 2022, it’s hard not to wish at least some among us had the gift of cosmic foresight. Many (most?) of the questions we thought in 2021 that we’d have answered by this point — chief among them, when will COVID finally leave us alone??? — still seem to elude us.

In keeping with our yearly tradition, we sat down with some experts at Rapid7 and across the industry to get their 2022 cybersecurity predictions. Here’s a look at what those in the know — some of them under the guise of clever fortune-teller names — think we’ll be talking about in the year to come.

Rob la Mystique (a.k.a. Robert Graham, CEO of Errata Security)

My third eye tells me that ransomware will become state-sponsored. Governments will notice the successful actors in their countries, and rather than shut them down, they’ll seek to co-opt their activities. In other words, pirates will be coopted into privateers.

Fahmida Y. Rashid, award-winning infosec journalist

I think we will see some surprising consolidation — some giant merger that’s going to dwarf even the ones we’ve seen so far. There’s still going to be insane venture funding rounds (like Transmit Security’s Series A) for security startups. But I think my prediction is that we are going to see the pendulum swing back from tools that do one thing well to large suites/integrated platforms that do all kinds of things, so the whole buying landscape is going to get even more murky and confusing.

Tod Beardsley, Director of Research at Rapid7

In 2022, managed service providers (MSPs) will continue to be in the hot seat as intermediary targets for ransomware gangs. The efficacy of hitting MSPs was proven out in 2021, and even small, regional MSPs will need to stay on their toes with patches and two-factor authentication everywhere to avoid getting exploited and phished by attackers who are targeting their downstream customers.

As cryptocurrency valuations continue to separate themselves from any realistic evidence of value, we will see more and more exchanges and clearinghouses get compromised, resulting in heists of millions of dollars’ worth of crypto — especially among off-shore exchanges.

Cyber-Zoltar the Blockchain Seer (a.k.a. Philip Amann, Head of Strategy at the European Cybercrime Center)

Ransomware will continue to dominate and proliferate with cybercriminals further moving toward a more calculated target selection. As is evidenced by several high-profile ransomware attacks, this has created a global cybersecurity risk that goes beyond the financial impact of these attacks. This will continue to be supported by a professional underground economy that provides the necessary tools and services.

We also expect investment fraud, BEC and CEO fraud to continue to cause disruptive losses and also a significant increase in mobile malware. The response to these threats will require us to further strengthen collaboration among law enforcement, industry, the CSIRT community, and academia globally with a view to collectively increasing cybersecurity, safety, and resilience.

Bob Rudis, Chief Security Data Scientist at Rapid7

The 2022 US election season will drive multiple (some impactful) cyberattacks on candidate/party technical and campaign logistics infrastructure and data from US-based sources.

Meanwhile, as companies accelerate toward a higher office-vs.-remote work ratio, initial access brokers will take advantage of the mobility (and weaknesses) in BYOD endpoints to gain footholds and refresh credentials and PII data stores. Multiple, major breaches will be reported.

In addition, the adoption of Software Bill of Materials (SBOM) will be astonishingly fast (in the US) toward the latter half of the year, heralding a new era of better third-party risk management and overall organizational safety and resilience.

Erick Galinkin, Principal Artificial Intelligence Researcher at Rapid7

Ransomware will continue to be a huge threat and will draw even more attention in 2022. While we should keep an eye out for potential attempts to disrupt a major US government agency, the revenue lost from ransomware will still be an order of magnitude less than business email compromise.

The media world and the security world will do their gnashing of teeth and rending of garments over deepfakes ahead of the 2022 midterms, but AI-powered disinformation will continue to be a mostly hypothetical threat.

Madame Bell LaPadula (a.k.a. Wendy Nather, Head of Advisory CISOs at Cisco)

On the heels of more visibility in supply chain security, and against the backdrop of steady disruption from ransomware, the security industry will have to face another maturity touchstone. It’s not enough simply to provide more transparency and share more data: what else do we owe one another in this broad ecosystem? SBOMs are the new shiny, but we will have to take many more steps together to improve our common, global defense.

Harley Geiger, Senior Director of Public Policy at Rapid7

State and federal agencies will step up their enforcement of existing cybersecurity regulations. This includes the SEC’s enforcement of required disclosures related to cybersecurity, DOJ’s enforcement of federal contractor cybersecurity requirements, and California’s enforcement of the CCPA.

But while regulators may issue new cybersecurity rules for the private sector under existing authorities, Congress will delay creating new federal authorities due to the midterm election year and the recent passage of large spending and incident reporting bills. Divisive items like federal privacy legislation are unlikely to pass. However, there will be plenty of hearings, press releases, and tweets expressing concern for ongoing cybersecurity threats!

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

More Hacky Holidays blogs

Developing a Platform for Software-defined Vehicles with Continental Automotive Edge (CAEdge)

Post Syndicated from Martin Stamm original https://aws.amazon.com/blogs/architecture/developing-a-platform-for-software-defined-vehicles-with-continental-automotive-edge-caedge/

This post was co-written by Martin Stamm, Principal Expert SW Architecture at Continental Automotive, Andreas Falkenberg, Senior Consultant at AWS Professional Services, Daniel Krumpholz, Engagement Manager at AWS Professional Services, David Crescence, Sr. Engagement Manager at AWS, and Junjie Tang, Principal Consultant at AWS Professional Services.

Automakers are embarking on a digital transformation journey to become more agile, efficient, and innovative. As part of this transformation, Continental created Continental Automotive Edge (CAEdge) – a modular multi-tenant hardware and software framework that connects the vehicle to the cloud. Continental collaborated with Amazon Web Services (AWS) to develop and scale this framework.

At this AWS re:Invent session, Continental and AWS demonstrated the new and transformative vehicle architectures and software built with CAEdge. These will provide future vehicle manufacturers, Original equipment manufacturers (OEMs) and partners with a multi-tenant development environment for software-intensive vehicle architectures. These can be used to implement software, sensor and big data solutions in a fraction of the development time needed before. As a result, vehicle software can be developed and tested more efficiently, then securely and rolled out directly to vehicles. The framework is already being tested in an automotive manufacturer’s series development.

Addressing core automotive industry pain points

Continental, OEMs and other major Tier 1 companies are required to quickly adapt to new technology requirements without knowing capacity or scaling needs, while at the same time staying ahead of the market. Developers are facing several challenges, in particular the processing of huge amounts of data. For example, a single test vehicle for AV/ADAS generates 20 – 100 TB of data per day. The handling of these data sets and the time to availability in distributed sites can cause major delays in development cycles. Delays are also experienced by developers due to the high numbers of test cases in simulation and validation. In an on-premises environment, this poses significant costs and scaling challenges to provide the required capacity.

The pace of the required transformation to becoming a software-centric organization is creating new challenges and opportunities like:

  • Current electronic architectures are decentralized, expensive, and complex to develop therefore difficult to maintain and extend.
  • Vehicle and cloud converge require new software (SW)-defined architectures, integration and operations competencies.
  • Digital Lifecycle Management enables new business models, go- to-market strategies and partnerships.

In addition to the distribution of huge datasets and distributed work setups is a need for cutting edge security technologies. Encryption at transfer/rest, data residency laws, and secure developer access are common security challenges and are addressed using CAEdge technology.

In this blog post, we describe how to build a secure multi-tenant AWS environment that forms the foundation for CAEdge. We discuss how AWS is helping Continental build the base infrastructure that allows for fast onboarding of OEMs, partners and suppliers through a highly automated process. Development activities can start within hours, instead of days or weeks; with a bootstrapped development environment for software-intensive vehicle architectures. This is all while meeting the strictest security and compliance requirements.

Overview of the CAEdge Framework

The following diagram gives an overview of the CAEdge Framework:

Architecture Diagram showing the CAEdge Platform

Figure 1 – Architecture Diagram showing the CAEdge Framework

The framework is based on the following modular building blocks:

  • Scalable Compute Platform – High Performance, embedded computer with automotive software stack and connection to the AWS cloud.
  • Cloud – Cloud services for developers and end-users.
  • DevOps Workbench – Toolchain for software development and maintenance covering the entire software lifecycle.

The building blocks of the framework are defined by clear API operations and can be integrated easily for various use cases, such as different middleware or CI / CD pipelines.

Overview of the CAEdge Multi-Tenant Framework

Continentals’ core architecture and terminology for a vehicle software development framework include:

  • CAEdge Framework as an Isolated AWS Organization – Continental’s CAEdge framework runs in a dedicated AWS Organization. Only CAEdge-related workloads are hosted in this AWS Organization. This ensures separation from any other workloads outside of the CAEdge context. The CAEdge framework provides multiple central security, access management, and orchestration services to its users.
  • Isolated Tenants – The framework is fully tenant-aware. A tenant is an isolated entity that represents an OEM, OEM sub-division, partner, or supplier. A key feature of this system is to ensure complete isolation from one tenant to another. We use a defense-in-depth security approach to ensure tenant separation.
  • Tenant-Owned Resources and Services – Each tenant has a dedicated set of resources and services that can be consumed and used by all tenant users and services. Tenant-owned resources and services include, but are not limited to:
    • Dedicated, tenant-specific data lake,
    • Tenant specific logging, monitoring, and operations,
    • Tenant-specific UI.
  • Projects – Each tenant can host an arbitrary number of projects with 1-N users assigned to them. A project is a high-level construct with the goal to create a unique product or service, such as a new “Door Lock” system software. The term project is used in a broad scope. Anything can be classified as a project.
  • Workbenches – A project consists of 1-N well-defined workbenches. A workbench represents a fully configured development environment of a specific “Workbench Type”. For example, a workbench of type “Simulation” allows for configuration and execution of Simulation Jobs based on drive data. Each workbench is implemented via a well-defined number of AWS Accounts, which is called an AWS Account Set.
    • An AWS Account Set always includes at least a Toolchain Account, Dev Account, QA Account and Prod Account. All AWS Accounts are baselined with IAM resources, security services and potentially workbench specific blueprints so development can start quickly for the end-user.

The following diagram illustrates the high-level architecture:

Figure 2 – High-level architecture diagram

Figure 2 – High-level architecture diagram

The CAEdge framework uses a data mesh architecture using AWS Lake Formation and Glue to create the tenant-level data lake. The Reference Architecture for Autonomous Driving Data Lake is used to design the Simulation workbench.

Implementation Details

With the core architecture and terminology defined, let’s look at the implementation details of the architecture that was described in the preceding image.

Isolated Tenants – Achieving a High Degree of Separation

To achieve a multi-tenant environment, we followed a multi-layered security hardening approach:

  • Tenant Separation on AWS Account Level: Starting at the AWS Account level, we used individual AWS Accounts where possible. An AWS account can never be assigned to more than one tenant. The functional scope of an AWS Account is kept as small as possible. This increases the number of total AWS Accounts, but significantly reduces the blast radius in case of any breach or incident. Just to give an example:
    • Each Dev, QA, and Prod Stage of a Workbench is its own AWS Account. No AWS Account ever combines multiple stages at once.
    • Each CAEdge tenant-owned data lake consists of multiple AWS Accounts. A data lake also requires updates as time passes. To allow for side-effect free and well tested updates of the data lake-infrastructure, each tenant comes with a Dev, QA, and Prod data lake.
  • Tenant Separation via a well-defined Organizational Unit (OU) structure and Service Control Policies (SCP): Each Tenant gets assigned a dedicated OU structure with multiple sub-OUs. This allows for tenant-specific security hardening on SCP-level and potential custom security hardening, in case dedicated tenants have specific security requirements. The SCPs are designed in such a way to allow for a maximum degree of freedom for the individual AWS Account user; while at the same time protecting the integrity of CAEdge and while staying compliant and secure according to specific requirements.
  • Tenant Separation through an AWS Account Metadata-Layer and automated IAM assignments: The CAedge framework uses a central Amazon DynamoDB database that maps AWS Accounts to Tenants and stores any other Metadata in the Context of an AWS Account. This includes including the Account Owner, Account Type, and Cost-related information. With this database, we can query AWS Accounts based on specific Tenants, Projects, and Workbenches. Furthermore, this forms the foundation of a fully automated permission and AWS Account access-management capability that enforces any Tenant, Project and Workbench boundary.
  • Tenant Separation Security Controls via AWS Security Services: On top of the standard AWS security services, such as AWS GuardDuty, AWS Config, AWS Inspector and AWS SecurityHub, we use IAM Access Analyzer in combination with our DynamoDB Account Metadata Store to detect the creation of any cross-account permissions that span outside of the AWS Organization, or that may have Cross-Tenant implications.

Automated creation and management of Tenant-Owned Resources and Services, Projects and Workbenches

CAEdge follows the “Everything-as-an-API Approach” and is designed as a fully open platform on the internet. All key features are exposed via a secured, public API. This includes the creation of Projects, Workbenches, and AWS Accounts including the management of access rights in a self-service manner for authorized users, as well as any updates affecting subsequent long-term management. This can only be achieved through a very high degree of automation.

We architect the following services to achieve this high degree of automation:

  • AWS Control Tower – An AWS managed service for account creation and OU assignment.
  • AWS Deployment Framework (AWS ADF) – an extensive and flexible framework to manage and deploy resources across multiple AWS Accounts and Regions within an AWS Organization. We use ADF to baseline all accounts with the resources required. This includes all security services, default IAM Roles, network related resources, such as VPCs and DNS and any other resources specific to the AWS Account type.
  • AWS Single Sign-On (AWS SSO) – A central IAM solution to control access to AWS Accounts. AWS SSO assignments are fully automated based on our defined access patterns using our custom Dispatch application and an extended version of the AWS SSO Enterprise solution.
  • AWS DynamoDB – A fully managed NoSQL database service for storing tenant, project and AWS Account data. Including information related to ownership, cost management, access management.
  • Dispatch CAEdge Web Application – A fully serverless web application that exposes functionality to end-users via API calls. It handles authentication, authorization, and provides business logic in the form of AWS Lambda functions to orchestrate all of the aforementioned services.

The following diagram provides a high-level overview of the automation mechanism at the platform level:

Figure 3 – High-level overview of the automation mechanism

Figure 3 – High-level overview of the automation mechanism

With this solution in place, Continental enables OEMs, suppliers, and other partners to spin up developer workbenches in a tenant context within minutes; thereby reducing the setup time from weeks to minutes using a self-service approach.

Conclusion

In this post, we showed how Continental built a secure multi-tenant platform that serves as the scalable foundation for software-intensive, vehicle-related workloads. For other organizations experiencing challenges when transforming into a software-centric organization, this solution eases the onboarding process so developers can start building within hours instead of months.

Validating addresses with AWS Lambda and the Amazon Location Service

Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/validating-addresses-with-aws-lambda-and-the-amazon-location-service/

This post is written by Matthew Nightingale, Associate Solutions Architect.

Traditional methods of performing address validation on geospatial datasets can be expensive and time consuming. Using Amazon Location Service with AWS Lambda in a serverless data processing pipeline, you may achieve significant performance improvements and cost savings on address validation jobs that use geospatial data.

This blog contains a deployable AWS Serverless Application Model (AWS SAM) template. It also uses sample data sourced from publicly available datasets that you can deploy and use to test the application. This blog offers a starting point to build out a serverless address validation pipeline in your own AWS account.

Overview

This application implements a serverless scatter/gather architecture using Lambda and Amazon S3, performing address validation with the Amazon Location Service. An S3 PUT event triggers each Lambda function to run data processing jobs along each step of the pipeline.

To test the application, a user uploads a .CSV file to S3. This dataset is labeled with fields that are recognized by the 2waygeocoder Lambda function. The application returns a processed dataset to S3 appended with location information from the Amazon Location Places API.

Solution overview

  1. The Scatter Lambda function takes a dataset from the S3 bucket labeled input and splits it into equally sized shards.
  2. The Process Lambda function takes each shard from the pre-processed bucket. It performs address validation in parallel with a 2waygeocoder function calling the Amazon Location Service Places API.
  3. The Gather Lambda function takes each shard from the post-processed bucket. It appends the data into a complete dataset with additional address information.

Amazon Location Service

Amazon Location Service sources high-quality geospatial data from HERE and ESRI to support searches by using a place index resource.

With the Amazon Locations Places API, you can convert addresses and other textual queries into geographic coordinates (also known as geocoding). You can also convert geographic positions into addresses and place descriptions (known as reverse geocoding).

The example application includes a 2waygeocoder capable of both geocoding and reverse geocoding. The next section shows examples of the call and response from the Amazon Location Places API for both geocoding and reverse geocoding.

Geocoding with Amazon Location Service

Here is an example of calling the Amazon Location Service Places API using the AWS SDK for Python (Boto3). This uses the search_place_index_for_text method:

Response = location.search_place_index_for_text(
	IndexName = ‘explore.place’ 
###index is created using Amazon Location service
	Text = “Boston, MA”)
location_response = Reponse[“Results”]
print(location_response)

Example response:

Response

Example reverse-geocoding with Amazon Location Service

Here is another example of calling the Amazon Location Service Places API using the AWS SDK for Python (boto3). This uses the search_place_index_for_position method:

Response = location.search_place_index_for_position(
	IndexName = ‘explore.place’ 
###index is created using Amazon Location service
	Position = “-71.056739, 42.358660”))
location_response = Reponse[“Results”]
print(location_response)

Example response:

Response

Design considerations

Processing data with Lambda in parallel using a serverless scatter/gather pipeline helps provide performance efficiency at lower cost. To provide even greater performance, you can optimize your Lambda configuration for higher throughput. There are several strategies you can implement to do this and a key few topics to keep in mind.

Increase the allocated memory for your Lambda function

The simplest way to increase throughput is to increase the allocated memory of the Lambda function.

Faster Lambda functions can process more data and increase throughput. This works even if a Lambda function’s memory utilization is low. This is because increasing memory also increases vCPUs in proportion to the amount configured. Each function supports up to 10 GB of memory and you can access up to six vCPUs per function.

To see the average cost and execution speed for each memory configuration, the Lambda Power Tuning tool helps to visualize the tradeoffs.

Optimize shard size

Another method for increasing performance in a serverless scatter/gather architecture is to optimize the total number of shards created by the scatter function. Increasing the total number of shards consequently reduces the size of any single shard, allowing Lambda to process each shard faster.

When scaling with Lambda, one instance of a function handles one request at a time. When the number of requests increases, Lambda creates more instances of the function to process traffic. Because S3 invokes Lambda asynchronously, there is an internal queue buffering requests between the event source and the Lambda service.

In a serverless scatter/gather architecture, having more shards results in more concurrent invocations of the process Lambda function. For more information about scaling and concurrency with Lambda, see this blog post. Increasing concurrency with Lambda can lead to API request throttling.

Consider API request throttling with your concurrent Lambda functions

In a serverless scatter/gather architecture, the rate at which your code calls APIs increases by a factor equal to the number of concurrent Lambda functions. This means API request limits can quickly be exceeded. You must consider Service Quotas and API request limits when trying to increase the performance of your serverless scatter/gather architecture.

For example, the Amazon Location Places APIs called in the processing function of this application has a default limit of 50 API requests per second. The 2waygeocoder calls on average about 12 APIs per second. Splitting the application into more than four shards may cause API throttling exception errors in this case. Requests to increase Service Quotas can be made through your AWS account.

Deploying the solution

You need the following perquisites to deploy the example application:

Deploy the example application:

  1. Clone the repository and download the sample source code to your environment where AWS SAM is installed:
    git clone https://github.com/aws-samples/amazon-location-service-serverless-address-validation
  2. Change into the project directory containing the template.yaml file:
    cd ~/environment/amazon-location-service-serverless-address-validation
  3. Build the application using AWS SAM:
    sam build
    Terminal output
  4. Deploy the application to your account using AWS SAM. Be sure to follow proper S3 naming conventions providing globally unique names for S3 buckets:
    sam deploy --guided
    Deployment output

Testing the application

Testing geocoding

To test the application, download the dataset that is linked in Testing the Application section of the GitHub repository. These tests demonstrate both the geocoding and reverse-geocoding capabilities of the application.

First, test the geocoding capabilities. You perform address validation on the City of Hartford Business Listing dataset linked in the GitHub repository. The dataset contains a listing of all the active businesses registered in the city Hartford, CT, and each business address. The GitHub repo links to an external website where you can download the dataset.

  1. Download the .csv version of the City of Hartford Business Listing dataset. The link is found in the Testing the Application section of the README file on GitHub.
  2. Open the file locally to explore its contents.
  3. Ensure that the .csv file contains columns labeled as “Address”, “City”, and “State”. The 2waygeocoder deployed as part of the AWS SAM template recognizes these columns to perform geocoding.
  4. Before testing the application’s geocoding capabilities, explore the pricing of Amazon Location Service. In order to save money, you can trim the length of the dataset for testing by removing rows. Once the dataset is trimmed to a desired length, navigate to S3 in the AWS Management Console.
  5. Upload the dataset to the S3 bucket labeled “input”. This triggers the scatter function.
  6. Navigate to the S3 bucket labeled “raw” to view the shards of your dataset created by the scatter function.
  7. Navigate to Lambda and select the 2waygeocoder function to view the CloudWatch Logs to see any information that is returned by the function code in near-real-time.
  8. Once the data is processed, navigate to the S3 bucket labeled “destination” to view the complete processed dataset that is created by the gather function. It may take several minutes for your dataset to finish processing.

Congratulations! You have successfully geocoded a dataset using Amazon Location Service with a serverless address validation pipeline.

Testing reverse-geocoding

Next, test the reverse-geocoding capabilities of the application. You perform address validation on the Miami Housing Dataset linked in the GitHub repository. This dataset contains information on 13,932 single-family homes sold in Miami. The repo links to an external website where you can download the dataset.

Before testing, explore the pricing of Amazon Location Service. To start the test:

  1. Download the zip file containing the .csv version of the dataset from . The link is found in the Testing the Application section of the README file on GitHub.
  2. Open the file locally to explore its contents.
  3. Ensure the .csv file contains columns A and B labeled “Latitude” and “Longitude”. You must edit these column headers to match the correct format that is recognized by the 2waygeocoder to perform reverse-geocoding. Only the “L” should be capitalized.
  4. To minimize cost, trim the length of the dataset for testing by removing rows. At the full size of ~13,933 rows, the dataset takes approx. 5 minutes to process.
  5. Once the dataset is trimmed to a desired length and both column A and B are labeled as “Latitude” and “Longitude” respectively, navigate to S3 in the AWS Management Console, and upload the dataset to your S3 bucket labeled “Input”.
  6. Navigate to the S3 bucket labeled “raw” to view the shards of your dataset.
  7. Navigate to Lambda and select the 2waygeocoder function to view the CloudWatch Logs to see any information that is returned by the function code in near-real-time.
  8. Navigate to the S3 bucket labeled “destination” to view the complete processed dataset that is created by the gather function. It may take several minutes for your dataset to finish processing.

Congratulations! You have successfully reverse-geocoded a dataset with Amazon Location Service using a serverless scatter/gather pipeline. You can move on to the conclusion, or continue to test the geocoding capabilities of the application with additional datasets.

Next steps

To get started testing your own datasets, use the AWS SAM template from GitHub deployed as part of this blog. Ensure that the labels in your dataset are labeled to match the constructs used in this blog post. The 2waygeocoder recognizes columns labeled “Latitude” and “Longitude” to perform reverse-geocoding, and “Address”, “City”, and “State” to perform geocoding.

Now that the data has been geocoded by Amazon Location Service and is in S3, you can use Amazon QuickSight geospatial charts to quickly and easily create interactive charts. For information on how to create a Dataset in QuickSight using Amazon S3 Files, check out the QuickSight User Guide.

Below is an example using QuickSight Geospatial charts to map the Miami housing dataset. The map shows average sale price by zipcode:

QuickSight map

This example uses QuickSight geospatial charts to map the City of Hartford Business dataset. The map shows DBA (doing business as) by latitude and longitude:

Dataset visualization

Conclusion

This blog post performs address validation with the Amazon Location Service, demonstrating both geocoding and reverse geocoding capabilities.

Using a serverless architecture with S3 and Lambda, you can achieve both cost optimization and performance improvement compared with traditional methods of address validation. Using this application, your organization can better understand and harness geospatial data.

For more serverless learning resources, visit Serverless Land.

Security updates for Thursday

Post Syndicated from original https://lwn.net/Articles/880564/rss

Security updates have been issued by Fedora (log4j and quaternion), Mageia (gnome-shell and singularity), SUSE (libsndfile, libvirt, net-snmp, and python-Babel), and Ubuntu (linux, linux-aws, linux-aws-5.11, linux-azure, linux-azure-5.11, linux-gcp, linux-gcp-5.11, linux-hwe-5.11, linux-kvm, linux-oracle, linux-oracle-5.11, linux-raspi, linux, linux-aws, linux-aws-5.4, linux-azure, linux-azure-5.4, linux-gcp, linux-gcp-5.4, linux-gke, linux-gke-5.4, linux-gkeop, linux-hwe-5.4, linux-ibm, linux-kvm, linux-oracle, linux-oracle-5.4, linux-raspi, linux, linux-aws, linux-aws-hwe, linux-azure-4.15, linux-dell300x, linux-gcp, linux-gcp-4.15, linux-hwe, linux-kvm, linux-oracle, linux-raspi2, linux-snapdragon, linux, linux-aws, linux-kvm, linux-lts-xenial, linux-oem-5.10, and linux-oem-5.14).

Handy Tips #19: Preventing alert storms with trigger dependencies

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/handy-tips-19-preventing-alert-storms-with-trigger-dependencies/18696/

Prevent receiving a flood of unwanted alerts and receive only the most critical notifications by defining trigger dependencies.

Every IT infrastructure has multiple elements, failure of which can cause a cascading set of problems across the particular infrastructure segment. It is important to prevent an unwanted alert storm and highlight only the root cause problem within the problem chain.

Define trigger dependencies to prevent alert storms:

  • Only the most critical problems will be displayed in Zabbix
  • Dependent triggers will not generate problems until the parent trigger is in an OK state

  • Each trigger can have multiple trigger dependencies
  • Trigger dependencies can be defined between triggers on different hosts

Check out the video to learn how to define trigger dependencies.

How to define a trigger dependency:
 
  1. Navigate to Configuration → Hosts
  2. Find the host for which you will define the trigger dependency
  3. Click on the triggers button next to the host
  4. Open the trigger for which you will define the dependency
  5. Click on the Dependencies tab
  6. Click add and select the host containing the parent trigger
  7. Select the trigger on which the current trigger will depend on
  8. If required, add more trigger dependencies
  9. Click the Update button
  10. Simulate a problem to test the dependency
  11. Navigate to Monitoring → Problems and observe the trigger dependency behavior

Tips and best practices:
  • The dependent trigger will only be re-evaluated once the related item receives new metrics
  • Trigger dependency may be added between host triggers as long as it wouldn’t result in a circular dependency
  • A trigger dependency chain between multiple hosts can be created
  • Zabbix will not execute actions for the dependent trigger if the parent trigger is in a problem state

The post Handy Tips #19: Preventing alert storms with trigger dependencies appeared first on Zabbix Blog.

People Are Increasingly Choosing Private Web Search

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/01/people-are-increasingly-choosing-private-web-search.html

DuckDuckGo has had a banner year:

And yet, DuckDuckGo. The privacy-oriented search engine netted more than 35 billion search queries in 2021, a 46.4% jump over 2020 (23.6 billion). That’s big. Even so, the company, which bills itself as the “Internet privacy company,” offering a search engine and other products designed to “empower you to seamlessly take control of your personal information online without any tradeoffs,” remains a rounding error compared to Google in search.

I use it. It’s not as a good a search engine as Google. Or, at least, Google often gets me what I want faster than DuckDuckGo does. To solve that, I use use the feature that allows me to use Google’s search engine through DuckDuckGo: prepend “!Google” to searches. Basically, DuckDuckGo launders my search.

EDITED TO ADD (1/12): I was wrong. DuckDuckGo does not provide privacy protections when searching using Google.

Къщи за тъщи: Делата зациклят, Прокуратурата бездейства, Карадайъ е недосегаем

Post Syndicated from Гешо Иванов original https://bivol.bg/guesthouses-2022.html

четвъртък 6 януари 2022


Преди близо 6 години БИВОЛЪ пръв писа за скандалните схеми при Програма за Развитие на Селските Райони (ПРСР) към Държавен Фонд Земеделие – при които, чрез европейско финансиране, стотици тарикати…

[$] Restricting SSH agent keys

Post Syndicated from original https://lwn.net/Articles/880458/rss

The OpenSSH suite of tools for
secure remote logins is used widely within our communities; it also
underlies things like remote Git repository access.
A recent experimental feature for the upcoming OpenSSH 8.9 release
will help close a security hole
that can be exploited by attacker-controlled SSH servers (e.g. sshd) when the user is forwarding
authentication to a local ssh-agent. Instead
of allowing the keys held in the agent to be used for authenticating to any
host where they might work, SSH agent
restriction will allow users to specify where and how those keys can be
used.

Set up cross-account audit logging for your Amazon Redshift cluster

Post Syndicated from Milind Oke original https://aws.amazon.com/blogs/big-data/set-up-cross-account-audit-logging-for-your-amazon-redshift-cluster/

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. With Amazon Redshift, you can analyze all your data to derive holistic insights about your business and your customers. One of the best practices of modern application design is to have centralized logging. Troubleshooting application problems is easy when you can correlate all your data together.

When you enable audit logging, Amazon Redshift logs information about connections and user activities in the database. These logs help you monitor the database for security and troubleshooting purposes, a process called database auditing. The logs are stored in Amazon Simple Storage Service (Amazon S3) buckets. These provide convenient access with data security features for users who are responsible for monitoring activities in the database.

If you want to establish a central audit logging account to capture audit logs generated by Amazon Redshift clusters located in separated AWS accounts, you can use the solution in this post to achieve cross-account audit logging for Amazon Redshift. As of this writing, the Amazon Redshift console only lists S3 buckets from the same account (in which the Amazon Redshift cluster is located) while enabling audit logging, so you can’t set up cross-account audit logging using the Amazon Redshift console. In this post, we demonstrate how to configure cross-account audit logging using the AWS Command Line Interface (AWS CLI).

Prerequisites

For this walkthrough, you must have the following prerequisites:

  • Two AWS accounts: one for analytics and one for centralized logging
  • A provisioned Amazon Redshift cluster in the analytics AWS account
  • An S3 bucket in the centralized logging AWS account
  • Access to the AWS CLI

Overview of solution

As a general security best practice, we recommend making sure that Amazon Redshift audit logs are sent to the correct S3 buckets. The Amazon Redshift service team has introduced additional security controls in the event that the destination S3 bucket resides in a different account from the Amazon Redshift cluster owner account. For more information, see Bucket permissions for Amazon Redshift audit logging.

This post uses the AWS CLI to establish cross-account audit logging for Amazon Redshift, as illustrated in the following architecture diagram.

For this post, we established an Amazon Redshift cluster named redshift-analytics-cluster-01 in the analytics account in Region us-east-2.

We also set up an S3 bucket named redshift-cluster-audit-logging-xxxxxxxxxxxx in the centralized logging account for capturing audit logs in Region us-east-1.

Now you’re ready to complete the following steps to set up the cross-account audit logging:

  1. Create AWS Identity and Access Management (IAM) policies in the analytics AWS account.
  2. Create an IAM user and attach the policies you created.
  3. Create an S3 bucket policy in the centralized logging account to allow Amazon Redshift to write audit logs to the S3 bucket, and allow the IAM user to enable audit logging for the S3 bucket.
  4. Configure the AWS CLI.
  5. Enable audit logging in the centralized logging account.

Create IAM policies in the analytics account

Create two IAM policies in the analytics account that has the Amazon Redshift cluster.

The first policy is the Amazon Redshift access policy (we named the policy redshift-audit-logging-redshift-policy). This policy allows the principal to whom it’s attached to enable, disable, or describe Amazon Redshift logs. It also allows the principal to describe the Amazon Redshift cluster. See the following code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "redshift:EnableLogging",
                "redshift:DisableLogging",
                "redshift:DescribeLoggingStatus"
            ],            
"Resource": "arn:aws:redshift:us-east-2:xxxxxxxxxxxx:cluster: redshift-analytics-cluster-01"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "redshift:DescribeClusters",
            "Resource": "*"
        }
    ]
}

The second policy is the Amazon S3 access policy (we named the policy redshift-audit-logging-s3-policy). This policy allows the principal to whom it’s attached to write to the S3 bucket in the centralized logging account. See the following code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx",
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx/*"
            ]
        }
    ]
}

Create an IAM user and attach the policies

Create an IAM user (we named it redshift-audit-logging-user) with programmatic access in the analytics account and attach the policies you created to it.

Save the generated AWS secret key and secret access key credentials for this user securely. We use these credentials in the next step.

Create an S3 bucket policy for the S3 bucket in the centralized logging AWS account

Add the following bucket policy to the audit logging S3 bucket redshift-cluster-audit-logging-xxxxxxxxxxxx in the centralized logging account. This policy serves two purposes: it allows Amazon Redshift to write audit logs to the S3 bucket, and it allows the IAM user to enable audit logging for the S3 bucket. See the following code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Put bucket policy needed for audit logging",
            "Effect": "Allow",
            "Principal": {
                "Service": "redshift.amazonaws.com"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetBucketAcl"
            ],
            "Resource": [
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx",
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx/*"
            ]
        },
        {
            "Sid": "Put IAM User bucket policy needed for audit logging",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::xxxxxxxxxxxx:user/redshift-audit-logging-user"
            },
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx",
                "arn:aws:s3:::redshift-cluster-audit-logging-xxxxxxxxxxxx/*"
            ]
        }
    ]
}

Note that you have to modify the service name redshift.amazonaws.com to look like redshift.region.amazonaws.com if the cluster is in one of the opt-in Regions.

Configure the AWS CLI

As part of this step, you need to install and configure the AWS CLI. After you install the AWS CLI, configure it to use the IAM user credentials that we generated earlier. We perform the next steps based on the permissions attached to the IAM user we created.

Enable audit logging in the centralized logging account

Run the AWS CLI command to enable audit logging for the Amazon Redshift cluster in an S3 bucket in the centralized logging AWS account. In the following code, provide the Amazon Redshift cluster ID, S3 bucket name, and the prefix applied to the log file names:

aws redshift enable-logging --cluster-identifier <ClusterName> --bucket-name <BucketName> --s3-key-prefix <value>

The following screenshot shows that the cross-account Amazon Redshift audit logging is successfully set up.

A test file is also created by AWS to ensure that the log files can be successfully written into the S3 bucket. The following screenshot shows the test file was created successfully in the S3 bucket under the rsauditlog1 prefix.

After some time, we started seeing the audit logs created in the S3 bucket. By default, Amazon Redshift organizes the log files in the S3 bucket using the following bucket and object structure:

AWSLogs/AccountID/ServiceName/Region/Year/Month/Day/AccountID_ServiceName_Region_ClusterName_LogType_Timestamp.gz

Amazon Redshift logs information in the following log files:

  • Connection log – Logs authentication attempts, connections, and disconnections
  • User log – Logs information about changes to database user definitions
  • User activity log – Logs each query before it’s run on the database

The following screenshot shows that log files, such as connection logs and user activity logs, are now being created in the centralized logging account in us-east-1 from the Amazon Redshift cluster in the analytics account in us-east-2.

For more details on analyzing Amazon Redshift audit logs, refer to below mentioned blogs

  1. Visualize Amazon Redshift audit logs using Amazon Athena and Amazon QuickSight
  2. How do I analyze my audit logs using Amazon Redshift Spectrum?

Clean up

To avoid incurring future charges, you can delete all the resources you created while following the steps in this post.

Conclusion

In this post, we demonstrated how to accomplish cross-account audit logging for an Amazon Redshift cluster in one account to an Amazon S3 bucket in another account. Using this solution, you can establish a central audit logging account to capture audit logs generated by Amazon Redshift clusters located in separated AWS accounts.

Try this solution to achieve cross-account audit logging for Amazon Redshift and leave a comment.

About the Authors

Milind Oke is a Data Warehouse Specialist Solutions Architect based out of New York. He has been building data warehouse solutions for over 15 years and specializes in Amazon Redshift.

Dipankar Kushari is a Sr. Analytics Solutions Architect with AWS.

Pankaj Pattewar is a Cloud Application Architect at Amazon Web Services. He specializes in architecting and building cloud-native applications and enables customers with best practices in their cloud journey.

Sudharshan Veerabatheran is a Cloud Support Engineer based out of Portland.

Rapid7 2021 Wrap-Up: Highlights From a Year of Empowering the Protectors

Post Syndicated from Rapid7 original https://blog.rapid7.com/2022/01/05/rapid7-2021-wrap-up-highlights-from-a-year-of-empowering-the-protectors/

Rapid7 2021 Wrap-Up: Highlights From a Year of Empowering the Protectors

Now that 2022 is fully underway, it’s time to wrap up some of the milestones that Rapid7 achieved in 2021. We worked harder than ever last year to help protectors keep their organization’s infrastructure secure — even in the face of some of the most difficult threats the security community has dealt with in recent memory. Here’s a rundown of some of our biggest moments in that effort from 2021.

Emergent threats and vulnerability disclosures

As always, our Research and Emergent Threat Response teams spent countless hours this year tirelessly bringing you need-to-know information about the most impactful late-breaking security exploits and vulnerabilities. Let’s revisit some of the highlights.

Emergent threat reports

Vulnerability disclosures

Research and policy highlights

That’s not all our Research team was up to in 2021. They also churned out a wealth of content and resources weighing in on issues of industry-wide, national, and international importance.

The Rapid7 family keeps growing

Throughout 2021, we made some strategic acquisitions to broaden the solutions we offer and help make the Insight Platform the one-stop shop for your security program.

Industry accolades

We’re always thrilled to get industry recognition for the work we do helping protectors secure their organizations — and we had a few big nods to celebrate in 2021.

Keeping in touch

Clearly, we had a pretty busy 2021 — and we have even more planned for 2022. If you need the latest and greatest in security content to tide you over throughout the last few weeks of the year, we have a few ideas for you.

Stay tuned for more great content, research, and much more in 2022!

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

Internet shut down in Kazakhstan amid unrest

Post Syndicated from João Tomé original https://blog.cloudflare.com/internet-shut-down-in-kazakhstan-amid-unrest/

Internet shut down in Kazakhstan amid unrest

In Kazakhstan, the year had barely got going when yesterday disruptions of Internet access ended up in a nationwide Internet shutdown from today, January 5, 2022. The disruptions and subsequent shutdown happened amid mass protests against sudden energy price rises.

Cloudflare Radar shows that the full shutdown happened after 10:30 UTC (16:30 local time). But it was preceded by restrictions to mobile Internet access yesterday.

Internet shut down in Kazakhstan amid unrest

Our data confirm that Kazakhstan’s ASNs were affected after that time (around 18:30 local time). That’s particularly evident with the largest telecommunication company in the country, Kaz Telecom, as the next chart shows.

Internet shut down in Kazakhstan amid unrest

The first disruptions reported affected mobile services, and we can see that at around 14:30 UTC yesterday, January 4, 2022, there was significantly less mobile devices traffic than the day before around the same time. Kazakhstan is a country where mobile represents something like 75% of Internet traffic (shown on Radar), a usual trend in the region. So mobile disruption has a big impact on the country’s Internet, even before the shutdown that affected almost all connectivity.

When we focus on other ASNs besides Kaz Telecom such as the leading mobile Internet services Tele2 or Kcell we can see a big drop in traffic yesterday after 16:00 UTC, confirming local reports. Mobile traffic did not drop to zero which may indicate throttling rather than a full shutdown. Today, however, the Internet, mobile or not, is shut down.

Internet shut down in Kazakhstan amid unrest

Looking at BGP (Border Gateway Protocol) updates from Kazakhstan’s ASNs around the time of the shutdown, we see a clear spike at exactly the same time the bigger ASNs were affected ~10:45 UTC, January 5, 2022. These update messages are BGP signaling that Kazakhstan’s ASNs are no longer routable, something similar to what we saw happening in The Gambia yesterday but for very different reasons.

Internet shut down in Kazakhstan amid unrest

The Kazakhstan case is similar to other state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams. There are similarities with the Sudan 25-day shutdown that we reported at the end of 2021, the Sudanese prime minister resigned this week in the aftermath of those shutdowns, but it’s very different from the Internet outage in The Gambia that we reported today.

You can keep an eye on Cloudflare Radar to monitor how we see Internet traffic globally and in every country.

Building a Cloud-Native File Transfer Platform Using AWS Transfer Family Workflows

Post Syndicated from Shoeb Bustani original https://aws.amazon.com/blogs/architecture/building-a-cloud-native-file-transfer-platform-using-aws-transfer-family-workflows/

File-based transfers are one of the most prevalent mechanisms for organizations to exchange data over various interfaces with their partners and consumers. There are specialized third-party managed file transfer (MFT) products available in the market that provide rich workflows for managing these transfers.

A typical MFT platform provides features to perform a series of linked pre- and post-file upload processing steps. The new managed workflows feature within AWS Transfer Family allows you to define a lightweight workflow that is invoked in response to file uploads. This feature, combined with the core SFTP, FTPS, and FTP functionality, enables you to build a cloud-native MFT platform for your organization. The workflows are also integrated with Amazon CloudWatch to provide complete traceability.

Before this feature was released, the MFT architecture based on Transfer Family involved responding to Amazon Simple Storage Service (Amazon S3) events within AWS Lambda functions. There was no overarching orchestration layer. With the new managed workflows feature, the sequencing of steps and error handling is greatly simplified.

In this blog, I show you how to architect common MFT scenarios using the new Transfer Family managed workflows feature. This will help you build a robust and well-integrated cloud-native MFT platform.

Scenario 1: Inbound Flow – file push by external providers

In this scenario, a file is supplied by an external data provider. It must be decrypted, checked for errors, and transferred to an internal application area (Amazon S3 bucket) for further processing by an application.

The internal application that processes the file could be an in-house Java application, an Enterprise Resourcing Planning system that processes payments, telecommunication billing system that consumes call data, or even financial regulatory organization that scans daily share trading data for anomalies.

The architecture for this scenario is presented in Figure 1. Here’s how it works:

  1. The external data provider connects to the organization’s public Transfer Family endpoint and provides the authentication credentials.
  2. The service authenticates the user via the pre-configured authentication mechanism. This could be a custom identity provider, AWS Directory Service, or service managed.
  3. Once authenticated, the data provider uploads the file to a logical folder. This results in the file being stored in the underlying Upload S3 bucket.
  4. Transfer Family initiates the configured workflow once the file has been uploaded to the S3 bucket. The workflow performs the required pre-processing steps, including:
    • Invoking a Lambda function to decrypt the file.
    • Invoking a Lambda function to ensure the file data is valid.
    • Copying the file to the Application S3 bucket.
    • Deleting or archiving the file by copying it to another S3 bucket or storing it with a different S3 prefix.
    • If an error occurs, the workflow exception handler moves (copy and delete) the file to the Quarantine S3 bucket or stores it with a different S3 prefix.
MFT inbound flow – push by data provider

Figure 1. MFT inbound flow – push by data provider

Scenario 2: Outbound flow – file push to external consumers

In this scenario, an internal application generates files that are to be provided to external parties. Examples include submissions to credit check agencies, direct debits or payment files to banking institutions. These files must be re-formatted, encrypted, and transferred to an external SFTP site or an API endpoint.

The architecture for implementing this scenario is presented in Figure 2. Here’s how it works:

  1. An internal application connects to the organization’s Transfer Family’s private SFTP endpoint hosted within Amazon Virtual Private Cloud (Amazon VPC) and provides the authentication information.
  2. The service authenticates the application using the pre-configured authentication mechanism. This could be a custom identity provider, Directory Service, or service managed.
  3. Once authenticated, the application uploads the file to a logical folder. This results in the file being stored in the underlying Upload S3 bucket.
  4. Transfer Family initiates the configured workflow once the file has been uploaded to the S3 bucket. The workflow performs the required processing steps, including:
    • Invoking a Lambda function to reformat and encrypt the file.
    • Invoking a Lambda function to transfer the file to the external SFTP site or API endpoint.
    • Copying the transferred file to the Processed S3 bucket or storing the file with a different Amazon S3 prefix.
    • Emptying the internal upload folder by deleting the file.
    • In case of errors, the workflow exception handler moves (copy and delete) the file to Error S3 bucket or stores with a different S3 prefix.
MFT outbound flow – push to data consumer

Figure 2. MFT outbound flow – push to data consumer

Scenario 3: Outbound Flow – file pull by external consumers

In this scenario, an internal application generates files that are to be provided to external parties. However, in this case, the files are downloaded or “pulled” from the external facing SFTP download folder by the consumers.

Examples include scenarios wherein external parties have a pre-defined schedule to download files or the consumers need to download the files manually in absence of an SFTP endpoint on their side.

The architecture for this scenario is presented in Figure 3. In this case, two instances of Transfer Family are created:

  1. Internal facing private instance from Scenario 2.
  2. External facing public instance to be used by the consumer for file downloads.

Here’s how it works:

Steps A through D. Flow remains the same as Scenario 2, except the internal workflow task uploads files to the S3 bucket underneath the external facing instance of Transfer Family.
E. The external consumer connects to the organization’s public Transfer Family endpoint and provides the authentication credentials.
F. The external facing Transfer Family service instance authenticates the consumer using the pre-configured authentication mechanism. This could be a custom identity provider, AWS Directory Service, or service managed.
G. Once authenticated, the data consumer downloads the file from the external Transfer Family SFTP server instance.

MFT outbound flow – pull by data consumer

Figure 3. MFT outbound flow – pull by data consumer

Conclusion

The new managed workflow feature within Transfer Family provides a simple mechanism to create file transfer flows. In this blog post, I showed you some of the common use cases you can implement using this new feature. You can combine this architecture approach with additional AWS services to build a robust and well-integrated cloud native managed file transfer platform.

Related information

Looking for more architecture content? AWS Architecture Center provides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!

How The Gambia lost access to the Internet for more than 8 hours

Post Syndicated from David Belson original https://blog.cloudflare.com/the-gambia-without-internet/

How The Gambia lost access to the Internet for more than 8 hours

How The Gambia lost access to the Internet for more than 8 hours

Internet outages are more common than most people think, and may be caused by misconfigurations, power outages, extreme weather, or infrastructure damage. Note that such outages are distinct from state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams.

On the morning of January 4, 2022, citizens of The Gambia woke up to a country-wide Internet outage. Gamtel (the main state-owned telecommunications company of the West Africa country), announced that it happened due to “technical issues on the backup links” — we elaborate more on this below.

Cloudflare Radar shows that the outage had a significant impact on Internet traffic in the country and started after 01:00 UTC (which is the same local time), lasting until ~09:45 — a disruption of over 8 hours.

How The Gambia lost access to the Internet for more than 8 hours

Looking at  BGP (Border Gateway Protocol) updates from Gambian ASNs around the time of the outage, we see a clear spike at 01:10 UTC. These update messages are BGP signaling that the Gambian ASNs are no longer routable.

How The Gambia lost access to the Internet for more than 8 hours

It is important to know that BGP is a mechanism to exchange routing information between autonomous systems (networks) on the Internet. The routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work. As we saw in our blog post in 2021 about how Facebook disappeared from the Internet, the Internet is literally a network of networks, and it’s bound together by BGP.

The Gambia’s Internet access is solely dependent on a single provider, Gamtel. Because The Gambia’s international Internet connectivity via the ACE submarine cable was unavailable, it was reliant on the “backup links” referenced above – terrestrial connectivity via Senegal and the provider Sonatel. This is visible in BGP data. If we look at the ASNs that are allocated to networks in The Gambia (AS25250, AS37309, AS37503, AS37552, AS37524, AS37323, AS328488, AS328140), and put those into a regular expression on BGP routing tools like route-views as so:

route-views>show ip bgp regexp .*_(25250|37309|37503|37552|37524|37323|328488|328140)

We are able to see all the possible upstream ASN paths from these networks to the rest of the Internet.

Looking at the “Path” results, we see that AS8346 (Sonatel) and AS25250 (Gamtel) are in the path for all the Gambian networks.

How The Gambia lost access to the Internet for more than 8 hours

Visualized, you can see the dependency on this network path for The Gambia’s Internet access.

How The Gambia lost access to the Internet for more than 8 hours

No interruptions were seen in Sonatel (AS8346), so this indicates that the single network path between Sonatel and Gamtel (AS25250) is a critical point for connectivity. A failure in either of these networks could result in The Gambia going offline again.

Yesterday’s outage in The Gambia outage illustrates something we frequently reference here in the blog: the Internet is literally a network of networks. A significant amount of  Internet traffic is carried by a complex network of undersea fiber-optic cables that connect countries and continents — all the cable systems used have landing points in two or more countries. So a problem in one country can easily affect others.

Going back to The Gambia, Gamtel explained in a January 5, 2022, press release that there was “a primary link failure at ACE” — the cable system that serves 24 countries, from Europe to Africa. “The ACE cable repair is expected to be completed in mid-January, 2022,” explained the company.

How The Gambia lost access to the Internet for more than 8 hours
The full ACE (Africa Coast to Europe) submarine cable system. From NSRC

The “backup failure” here was “due to a faulty card at Toubakota, in Senegal”. That problem affects “both the Karang and Seleti links [points of cable connections from Senegal to The Gambia] as both North and South links converges there”. “Thus, the reason for the complete isolation on the Sonatel link”, concludes Gamtel.

Recognizing the critical importance of reliable Internet connectivity, The Gambia Public Utilities Regulatory Authority also issued a statement noting “The Authority, operators, MOICI, and the Government are exploring other options of making sure that the Gambia has a second fibre cable backup considering the impact that these failures are having on our national security, economy, and social activities.”

Metasploit 2021 Annual Wrap-Up

Post Syndicated from Spencer McIntyre original https://blog.rapid7.com/2022/01/05/metasploit-2021-annual-wrapup/

Metasploit 2021 Annual Wrap-Up

As 2022 kicks off, we now have another year in the books. Like years past, 2021 brought some surprises and had its share of celebrity vulnerabilities and recurring trends. Let’s highlight some statistics!

Quick stats

  • 651 merged pull requests from 113 users
  • 184 new modules
    • 102 exploits, 45 post, 32 auxiliary, 3 payload, and 2 evasion
  • 1 Metasploit Community CTF hosted
    • 1,501 users registered across 727 teams
    • 18 total challenges
    • 1,264 correct challenge submissions

URI support

As of Metasploit 6.1.4, users can now supply URI strings as arguments to the run command to specify RHOST values and option values at once:

use exploit/linux/postgres/postgres_payload
run postgres://administrator:[email protected] lhost=192.168.123.1 lport=5000

This new workflow will not only make it easier to use reverse-i-search with CTRL+R in Metasploit’s console — it will also make it easier to share cheat sheets among pentesters. Support includes HTTP, MySQL, PostgreSQL, SMB, SSH, and more; check out the full announcement post.

Sessions without payloads

Metasploit 2021 Annual Wrap-Up

AV evasion is a hard problem that’s not going to be solved in the foreseeable future. Payloads are caught in a variety of ways by a variety of AVs. One sustainable approach Metasploit is attempting to take is to enable users to leverage sessions that don’t require payload code to be running on the target. While not always a feasible solution, when it is, it can be quite reliable.

Earlier in 2021, community member smashery took on a large effort to enable Metasploit users to obtain interactive command shell sessions using Microsoft’s WinRM. The result is an improvement that enables the scanner/winrm/winrm_login module to open a command shell session without having uploaded a payload to the target. This session can then of course be used with post modules that are compatible with shell payloads.

msf6 auxiliary(scanner/winrm/winrm_login) > run username=Administrator password=pass rhost=192.168.123.15

[!] No active DB -- Credential data will not be saved!
[+] 192.168.123.15:5985 - Login Successful: WORKSTATION\Administator:pass
[*] Command shell session 4 opened (192.168.123.1:50321 -> 192.168.123.15:5985 ) at 2021-12-17 14:14:25 +0000
[*] Scanned 1 of 1 hosts (100% complete)
[*] Auxiliary module execution completed

In a similar vein, Metasploit has for a while now had the ability to open command shell sessions from the scanner/ssh/ssh_login module. These command shell sessions could also be used with post modules that didn’t require full Meterpreter sessions. However, one notable feature that SSH servers did not support until 2021 was the ability to port-forward over these connections. Last year saw improvements to Metasploit’s handling of SSH sessions that enable both standard port forwarding (for client connections) and reverse port forwarding (for server connections). Being fully wired into Metasploit, so to speak, means users can forward connections over them using the route command in the same way they can with Meterpreter sessions.

We hope these new capabilities provide users with more options to perform their testing from Metasploit while keeping payloads entirely out of memory.

Evasion modules

Evasion modules are one of Metasploit’s most infrequently added types, but they are certainly noteworthy when they are added. Last year saw two such modules added, both targeting Windows executables. The first module, based on Johnny Shaw’s work, implemented Process Herpaderping. This novel technique obfuscates the payload’s main logic from security products. This technique was effective for a few months but was ultimately added as a detection to Windows Defender.

Another evasion module added this year was kensh1ro’s syscall module. Using direct system calls is a popular technique to evade user-mode analysis hooks, and this module brings the capability to Metasploit, too.

RDLL exploit improvements

Last year, the post exploit library used by quite a few Windows local exploits saw a great improvement that reduced code reuse and laid the foundation to randomize the target process used to host the injected DLL. Prior to this, most exploits would start notepad using a piece of template code that would then load the RDLL and, when successful, execute the payload. This often led to the notepad process making network calls, which is pretty easily identified as malicious behavior. Instead, these modules will now randomly select a binary from a list and automatically start a process of the correct architecture. No more notepad instances making network calls from exploits. Currently, the new implementation will randomly select between msiexec and netsh, both of which are widely available across Windows versions and are less likely to be identified when making network connections.

Kubernetes support

It’s safe to say that cloud computing is here to stay. Metasploit added the first modules that target the Kubernetes platform. The first module is an auxiliary module that is capable of enumerating namespace, pod, and secret information. Following up on that is an exploit module that, when provided the necessary credentials, can execute a payload within a pod. In a similar vein to the previously mentioned payload-less post-exploitation capabilities, this module can also open a direct command shell session using a new, native WebSocket implementation. We hope these modules help Metasploit users who are testing these environments and look forward to expanding on the capabilities in 2022.

Session validation

Being a framework, Metasploit offers a variety of payloads and session types. Unfortunately, not every payload yields a session type with the same capabilities (e.g. the PHP Meterpreter does not offer Kiwi). This can be very confusing for users as they’re attempting to use various post modules and Meterpreter commands. Last year, Metasploit improved the way this is handled and now offers concise error messages when certain capabilities are missing or can’t be performed with a particular session type. Now running a Meterpreter command that’s either unsupported or provided by an extension that hasn’t been loaded will be reported as such.

meterpreter > creds_all
[-] The "creds_all" command requires the "kiwi" extension to be loaded (run: `load kiwi`)
meterpreter > load kiwi
Loading extension kiwi...
[-] Failed to load extension: The "kiwi" extension is not supported by this Meterpreter type (python/osx)
[-] The "kiwi" extension is supported by the following Meterpreter payloads:
[-]   - windows/x64/meterpreter*
[-]   - windows/meterpreter*

Improved SMB capture server

SMB1 has not been enabled by default in Windows 10 since 2017. Last year, Metasploit began the long process of updating the SMB server capabilities to work with the modern SMB 2 and SMB 3 versions. The first milestone allowed the capture server (auxiliary/server/capture/smb) that collects authentication information from incoming client connections to be upgraded to support incoming connections from SMB 2 and SMB 3 clients. Today, the capture server can be used with modern versions for Windows, in their default configuration.

New module highlights

  • exploits/windows/http/exchange_proxylogon_rce – This was the first of two high-profile Exchange RCEs added to Metasploit and highlighted the need for administrators to stay on top of patching their on premises Exchange servers or migrate.
  • exploit/multi/http/git_lfs_clone_command_exec – This exploit brought along with it new capabilities for Metasploit to act as a malicious Git server. This opens the door for future modules to exploit similar vulnerabilities.
  • [exploits/linux/local/cve_2021_3490_ebpf_alu32_bounds_check_lpe])(https://github.com/rapid7/metasploit-framework/pull/15567) eBPF has been a popular target for Linux LPEs this year. This particular exploit, based on @chompie1337’s original research was particularly valuable due to the number of platforms it affected as well as its reliability. Speaking of reliability…
  • exploits/linux/local/sudo_baron_samedit – Being January 2022, this particular celebrity vulnerability seems like old news. At the time, however, it gained quite a bit of attention, as it was in the ever-prevalent sudo utility. One quality that made this exploit particularly valuable was that there is no risk of system instability while exploiting it. This will likely remain a go-to exploit for users needing to escalate on Linux systems in years to come.
    auxiliary/gather/windows_secrets_dump – While not technically a new module, this particular entry saw a massive improvement in its addition of support for targeting Domain Controllers. This was a monumental effort that included a foundation that also makes it easier for modules to run attacks over DCERPC (think PrintNightmare and ZeroLogon).
  • exploit/multi/http/cve_2021_35464_forgerock_openam – Any unauthenticated RCE in an application that’s intended to be an IAM solution is worth calling out.
  • post/windows/gather/credentials/windows_sam_hivenightmare – This was another highly reliable privilege escalation technique that could be used to recover sensitive files on Windows systems. The module’s implementation performs the entire operation in memory using Meterpreter with spawning new processes or dropping artifacts to disk, making it a very stealthy approach.

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

The collective thoughts of the interwebz