Welcome to AWS Storage Day 2022

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/welcome-to-aws-storage-day-2022/

We are on the fourth year of our annual AWS Storage Day! Do you remember our first Storage Day 2019 and the subsequent Storage Day 2020? I watched Storage Day 2021, which was streamed live from downtown Seattle. We continue to hear from our customers about how powerful the Storage Day announcements and educational sessions were. With this year’s lineup, we aim to share our insights on how to protect your data and put it to work. The free Storage Day 2022 virtual event is happening now on the AWS Twitch channel. Tune in to hear from experts about new announcements, leadership insights, and educational content related to the broad portfolio of AWS Storage services.

Our customers are looking to reduce and optimize storage costs, while building the cloud storage skills they need for themselves and for their organizations. Furthermore, our customers want to protect their data for resiliency and put their data to work. In this blog post, you will find our insights and announcements that address all these needs and more.

Let’s get into it…

Protect Your Data
Data protection has become an operational model to deliver the resiliency of applications and the data they rely on. Organizations use the National Institute of Standards and Technology (NIST) cybersecurity framework and its Identify->Protect->Detect->Respond->Recover process to approach data protection overall. It’s necessary to consider data resiliency and recovery upfront in the Identify and Protect functions, so there is a plan in place for the later Respond and Recover functions.

AWS is making data resiliency, including malware-type recovery, table stakes for our customers. Many of our customers use Amazon Elastic Block Store (Amazon EBS) for mission-critical applications. If you already use Amazon EBS and you regularly back up EBS volumes using EBS multi-volume snapshots, I have an announcement that you will find very exciting.

Amazon EBS
Amazon EBS scales fast for the most demanding, high-performance workloads, and this is why our customers trust Amazon EBS for critical applications such as SAP, Oracle, and Microsoft. Currently, Amazon EBS enables you to back up volumes at any time using EBS Snapshots. Snapshots retain the data from all completed I/O operations, allowing you to restore the volume to its exact state at the moment before backup.

Many of our customers use snapshots in their backup and disaster recovery plans. A common use case for snapshots is to create a backup of a critical workload such as a large database or file system. You can choose to create snapshots of each EBS volume individually or choose to create multi-volume snapshots of the EBS volumes attached to a single Amazon Elastic Compute Cloud (EC2) instance. Our customers love the simplicity and peace of mind that comes with regularly backing up EBS volumes attached to a single EC2 instance using EBS multi-volume snapshots, and today we’re announcing a new feature—crash consistent snapshots for a subset of EBS volumes.

Previously, when you wanted to create multi-volume snapshots of EBS volumes attached to a single Amazon EC2 instance, if you only wanted to include some—but not all—attached EBS volumes, you had to make multiple API calls to keep only the snapshots you wanted. Now, you can choose specific volumes you want to exclude in the create-snapshots process using a single API call or by using the Amazon EC2 console, resulting in significant cost savings. Crash consistent snapshots for a subset of EBS volumes is also supported by Amazon Data Lifecycle Manager policies to automate the lifecycle of your multi-volume snapshots.

This feature is now available to you at no additional cost. To learn more, please visit the EBS Snapshots user guide.

Put Your Data to Work
We give you controls and tools to get the greatest value from your data—at an organizational level down to the individual data worker and scientist. Decisions you make today will have a long-lasting impact on your ability to put your data to work. Consider your own pace of innovation and make sure you have a cloud provider that will be there for you no matter what the future brings. AWS Storage provides the best cloud for your traditional and modern applications. We support data lakes in AWS Storage, analytics, machine learning (ML), and streaming on top of that data, and we also make cloud benefits available at the edge.

Amazon File Cache (Coming Soon)
Today we are also announcing Amazon File Cache, an upcoming new service on AWS that accelerates and simplifies hybrid cloud workloads. Amazon File Cache provides a high-speed cache on AWS that makes it easier for you to process file data, regardless of where the data is stored. Amazon File Cache serves as a temporary, high-performance storage location for your data stored in on-premises file servers or in file systems or object stores in AWS.

This new service enables you to make dispersed data sets available to file-based applications on AWS with a unified view and at high speeds with sub-millisecond latencies and up to hundreds of GB/s of throughput. Amazon File Cache is designed to enable a wide variety of cloud bursting workloads and hybrid workflows, ranging from media rendering and transcoding, to electronic design automation (EDA), to big data analytics.

Amazon File Cache will be generally available later this year. If you are interested in learning more about this service, please sign up for more information.

AWS Transfer Family
During Storage Day 2020, we announced that customers could deploy AWS Transfer Family server endpoints in Amazon Virtual Private Clouds (Amazon VPCs). AWS Transfer Family helps our customers easily manage and share data with simple, secure, and scalable file transfers. With Transfer Family, you can seamlessly migrate, automate, and monitor your file transfer workflows into and out of Amazon S3 and Amazon Elastic File System (Amazon EFS) using the SFTP, FTPS, and FTP protocols. Exchanged data is natively accessible in AWS for processing, analysis, and machine learning, as well as for integrations with business applications running on AWS.

On July 26th of this year, Transfer Family launched support for the Applicability Statement 2 (AS2) protocol. Customers across verticals such as healthcare and life sciences, retail, financial services, and insurance that rely on AS2 for exchanging business-critical data can now use AWS Transfer Family’s highly available, scalable, and globally available AS2 endpoints to more cost-effectively and securely exchange transactional data with their trading partners.

With a focus on helping you work with partners of your choice, we are excited to announce the AWS Transfer Family Delivery Program as part of the AWS Partner Network (APN) Service Delivery Program (SDP). Partners that deliver cloud-native Managed File Transfer (MFT) and business-to-business (B2B) file exchange solutions using AWS Transfer Family are welcome to join the program. Partners in this program meet a high bar, with deep technical knowledge, experience, and proven success in delivering Transfer Family solutions to our customers.

Five New AWS Storage Learning Badges
Earlier I talked about how our customers are looking to add the cloud storage skills they need for themselves and for their organizations. Currently, storage administrators and practitioners don’t have an easy way of externally demonstrating their AWS storage knowledge and skills. Organizations seeking skilled talent also lack an easy way of validating these skills for prospective employees.

In February 2022, we announced digital badges aligned to Learning Plans for Block Storage and Object Storage on AWS Skill Builder. Today, we’re announcing five additional storage learning badges. Three of these digital badges align to the Skill Builder Learning Plans in English for File, Data Protection & Disaster Recovery (DPDR), and Data Migration. Two of these badges—Core and Technologist—are tiered badges that are awarded to individuals who earn a series of Learning Plan-related badges in the following progression:

Image showing badge progression. To get the Storage Core badge users must first get Block, File, and Object badges. To get the Storage Technologist Badge users must first get the Core, Data Protection & Disaster Recovery, and Data Migration badges.

To learn more, please visit the AWS Learning Badges page.

Well, That’s It!
As I’m sure you’ve picked up on the pattern already, today’s announcements focused on continuous innovation and AWS’s ongoing commitment to providing the cloud storage training that your teams are looking for. Best of all, this AWS training is free. These announcements also focused on simplifying your data migration to the cloud, protecting your data, putting your data to work, and cost-optimization.

Now Join Us Online
Register for free and join us for the AWS Storage Day 2022 virtual event on the AWS channel on Twitch. The event will be live from 9:00 AM Pacific Time (12:00 PM Eastern Time) on August 10. All sessions will be available on demand approximately 2 days after Storage Day.

We look forward to seeing you on Twitch!

– Veliswa x

AWS co-announces release of the Open Cybersecurity Schema Framework (OCSF) project

Post Syndicated from Mark Ryland original https://aws.amazon.com/blogs/security/aws-co-announces-release-of-the-open-cybersecurity-schema-framework-ocsf-project/

In today’s fast-changing security environment, security professionals must continuously monitor, detect, respond to, and mitigate new and existing security issues. To do so, security teams must be able to analyze security-relevant telemetry and log data by using multiple tools, technologies, and vendors. The complex and heterogeneous nature of this task drives up costs and may slow down detection and response times. Our mission is to innovate on behalf of our customers so they can more quickly analyze and protect their environment when the need arises.

With that goal in mind, alongside a number of partner organizations, we’re pleased to announce the release of the Open Cybersecurity Schema Framework (OCSF) project, which includes an open specification for the normalization of security telemetry across a wide range of security products and services, as well as open-source tools that support and accelerate the use of the OCSF schema. As a co-founder of the OCSF effort, we’ve helped create the specifications and tools that are available to all industry vendors, partners, customers, and practitioners. Joining us in this announcement is an array of key security vendors, beginning with Splunk, the co-founder with AWS of the OCSF project, and also including Broadcom, Salesforce, Rapid7, Tanium, Cloudflare, Palo Alto Networks, DTEX, CrowdStrike, IBM Security, JupiterOne, Zscaler, Sumo Logic, IronNet, Securonix, and Trend Micro. Going forward, anyone can participate in the evolution of the specification and tooling at https://github.com/ocsf.

Our customers have told us that interoperability and data normalization between security products is a challenge for them. Security teams have to correlate and unify data across multiple products from different vendors in a range of proprietary formats; that work has a growing cost associated with it. Instead of focusing primarily on detecting and responding to events, security teams spend time normalizing this data as a prerequisite to understanding and response. We believe that use of the OCSF schema will make it easier for security teams to ingest and correlate security log data from different sources, allowing for greater detection accuracy and faster response to security events. We see value in contributing our engineering efforts and also projects, tools, training, and guidelines to help standardize security telemetry across the industry. These efforts benefit our customers and the broader security community.

Although we as an industry can’t directly control the behavior of threat actors, we can improve our collective defenses by making it easier for security teams to do their jobs more efficiently. At AWS, we are excited to see the industry come together to use the OCSF project to make it easier for security professionals to focus on the things that are important to their business: identifying and responding to events, then using that data to proactively improve their security posture.

To learn more about the OCSF project, visit https://github.com/ocsf.

Want more AWS Security news? Follow us on Twitter.

Mark Ryland

Mark Ryland

Mark is the director of the Office of the CISO for AWS. He has over 30 years of experience in the technology industry and has served in leadership roles in cybersecurity, software engineering, distributed systems, technology standardization and public policy. Previously, he served as the Director of Solution Architecture and Professional Services for the AWS World Public Sector team.

Classroom activities to discuss machine learning accuracy and ethics | Hello World #18

Post Syndicated from Janina Ander original https://www.raspberrypi.org/blog/classroom-activity-machine-learning-accuracy-ethics-hello-world-18/

In Hello World issue 18, available as a free PDF download, teacher Michael Jones shares how to use Teachable Machine with learners aged 13–14 in your classroom to investigate issues of accuracy and ethics in machine learning models.

Cover of Hello World issue 18.
Michael Jones.

Machine learning: Accuracy and ethics

The landscape for working with machine learning/AI/deep learning has grown considerably over the last couple of years. Students are now able to develop their understanding from the hard-coded end via resources such as Machine Learning for Kids, get their hands dirty using relatively inexpensive hardware such as the Nvidia Jetson Nano, and build a classification machine using the Google-driven Teachable Machine resources. I have used all three of the above with my students, and this article focuses on Teachable Machine.

For this module, I’m more concerned with the fuzzy end of AI, including how credible AI decisions are, and the elephant-in-the-room aspect of bias and potential for harm.

Michael Jones

For the worried, there is absolutely no coding involved in this resource; the ‘machine’ behind the portal does the hard work for you. For my Year 9 classes (students aged 13 to 14) undertaking a short, three-week module, this was ideal. The coding is important, but was not my focus. For this module, I’m more concerned with the fuzzy end of AI, including how credible AI decisions are, and the elephant-in-the-room aspect of bias and potential for harm.

Getting started with Teachable Machine activities

There are three possible routes to use in Teachable Machine, and my focus is the ‘Image Project’, and within this, the ‘Standard image model’. From there, you are presented with a basic training scenario template — see Hello World issue 16 (pages 84–86) for a step-by-step set-up and training guide. For this part of the project, my students trained the machine to recognise different breeds of dog, with border collie, labrador, saluki, and so on as classes. Any AI system devoted to recognition requires a substantial set of training data. Fortunately, there are a number of freely available data sets online (for example, download a folder of dog photos separated by breed by accessing helloworld.cc/dogdata). Be warned, these can be large, consisting of thousands of images. If you have more time, you may want to set students off to collect data to upload using a camera (just be aware that this can present safeguarding considerations). This is a key learning point with your students and an opportunity to discuss the time it takes to gather such data, and variations in the data (for example, images of dogs from the front, side, or top).

Drawing of a machine learning ars rover trying to decide whether it is seeing an alien or a rock.
Image recognition is a common application of machine learning technology.

Once you have downloaded your folders, upload the images to your Teachable Machine project. It is unlikely that you will be able to upload a whole subfolder at once — my students have found that the optimum number of images seems to be twelve. Remember to build this time for downloading and uploading into your lesson plan. This is a good opportunity to discuss the need for balance in the training data. Ask questions such as, “How likely would the model be to identify a saluki if the training set contained 10 salukis and 30 of the other dogs?” This is a left-field way of dropping the idea of bias into the exploration of AI — more on that later!

Accuracy issues in machine learning models

If you have got this far, the heavy lifting is complete and Google’s training engine will now do the work for you. Once you have set your model on its training, leave the system to complete its work — it takes seconds, even on large sets of data. Once it’s done, you should be ready to test you model. If all has gone well and a webcam is attached to your computer, the Output window will give a prediction of what is being viewed. Again, the article in Hello World issue 16 takes you through the exact steps of this process. Make sure you have several images ready to test. See Figure 1a for the response to an image of a saluki presented to the model. As you might expect, it is showing as a 100 percent prediction.

Screenshots from Teachable Machine showing photos of dogs classified as specific breeds with different degrees of confidence by a machine learning model.
Figure 1: Outputs of a Teachable Machine model classifying photos of dog breeds. 1a (left): Photo of a saluki. 1b (right): Photo of a Samoyed and two people.

It will spark an interesting discussion if you now try the same operation with an image with items other than the one you’re testing in it. For example see Figure 1b, in which two people are in the image along with the Samoyed dog. The model is undecided, as the people are affecting the outcome. This raises the question of accuracy. Which features are being used to identify the dogs as border collie and saluki? Why are the humans in the image throwing the model off the scent?

Getting closer to home, training a model on human faces provides an opportunity to explore AI accuracy through the question of what might differentiate a female from a male face. You can find a model at helloworld.cc/maleorfemale that contains 5418 images almost evenly spread across male and female faces (see Figure 2). Note that this model will take a little longer to train.

Screenshot from Teachable Machine showing two datasets of photos of faces labeled either male or female.
Figure 2: Two photo sets of faces labeled either male or female, uploaded to Teachable Machine.

Once trained, try the model out. Props really help — a top hat, wig, and beard give the model a testing time (pun intended). In this test (see Figure 3), I presented myself to the model face-on and, unsurprisingly, I came out as 100 percent male. However, adding a judge’s wig forces the model into a rethink, and a beard produces a variety of results, but leaves the model unsure. It might be reasonable to assume that our model uses hair length as a strong feature. Adding a top hat to the ensemble brings the model back to a 100 percent prediction that the image is of a male.

Screenshots from Teachable Machine showing two datasets of a model classifying photos of the same face as either male or female with different degrees of confidence, based on the face is wearing a wig, a fake beard, or a tophat.
Figure 3: Outputs of a Teachable Machine model classifying photos of the author’s face as male or female with different degrees of confidence. Click to enlarge.

Machine learning uses a best-fit principle. The outputs, in this case whether I am male or female, have a greater certainty of male (65 percent) versus a lesser certainty of female (35 percent) if I wear a beard (Figure 3, second image from the right). Remove the beard and the likelihood of me being female increases by 2 percent (Figure 3, second image from the left).

Bias in machine learning models

Within a fairly small set of parameters, most human faces are similar. However, when you start digging, the research points to there being bias in AI (whether this is conscious or unconscious is a debate for another day!). You can exemplify this by firstly creating classes with labels such as ‘young smart’, ‘old smart’, ‘young not smart’, and ‘old not smart’. Select images that you think would fit the classes, and train them in Teachable Machine. You can then test the model by asking your students to find images they think fit each category. Run them against the model and ask students to debate whether the AI is acting fairly, and if not, why they think that is. Who is training these models? What images are they receiving? Similarly, you could create classes of images of known past criminals and heroes. Train the model before putting yourself in front of it. How far up the percentage scale are you towards being a criminal? It soon becomes frighteningly worrying that unless you are white and seemingly middle class, AI may prove problematic to you, from decisions on financial products such as mortgages through to mistaken arrest and identification.

It soon becomes frighteningly worrying that unless you are white and seemingly middle class, AI may prove problematic to you, from decisions on financial products such as mortgages through to mistaken arrest and identification.

Michael Jones

Encourage your students to discuss how they could influence this issue of race, class, and gender bias — for example, what rules would they use for identifying suitable images for a data set? There are some interesting articles on this issue that you can share with your students at helloworld.cc/aibias1 and helloworld.cc/aibias2.

Where next with your learners?

In the classroom, you could then follow the route of building models that identify letters for words, for example. One of my students built a model that could identify a range of spoons and forks. You may notice that Teachable Machine can also be run on Arduino boards, which adds an extra dimension. Why not get your students to create their own AI assistant that responds to commands? The possibilities are there to be explored. If you’re using webcams to collect photos yourself, why not create a system that will identify students? If you are lucky enough to have a set of identical twins in your class, that adds just a little more flavour! Teachable Machine offers a hands-on way to demonstrate the issues of AI accuracy and bias, and gives students a healthy opportunity for debate.

Michael Jones is director of Computer Science at Northfleet Technology College in the UK. He is a Specialist Leader of Education and a CS Champion for the National Centre for Computing Education.

More resources for AI and data science education

At the Foundation, AI education is one of our focus areas. Here is how we are supporting you and your learners in this area already:

  • Hello World issue 12 focuses on AI and machine learning education, with many practical resources, insightful interviews, and inspiring features from computer science educators. Download your free copy of issue 12 now.
  • In Hello World issue 16, the focus is on all things data science and data literacy for your learners. As always, you can download a free copy of the issue.
  • On our Hello World podcast, we’ve got episodes where we talk with practicing computing educators about how they bring AI, AI ethics, machine learning, and data science to the young people they teach.
  • If you’d like a practical introduction to the basics of machine learning and how to use it, take our free online course.
An image demonstrating that AI systems for object recognition do not distinguish between a real banana on a desk and the photo of a banana on a laptop screen.
  • Computing education researchers are working to answer the many open questions about what good AI and data science education looks like for young people. To learn more, you can watch the recordings from our research seminar series focused on this. We ourselves are working on research projects in this area and will share the results freely with the computing education community.
  • You can find a list of free educational resources about these topics that we’ve collated based on our research seminars, seminar participants’ recommendations, and our own work.

The post Classroom activities to discuss machine learning accuracy and ethics | Hello World #18 appeared first on Raspberry Pi.

Virtual Production — A Validation Framework For Unreal Engine

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/virtual-production-a-validation-framework-for-unreal-engine-aab780b2f8c8

Virtual Production — A Validation Framework For Unreal Engine

By Adam Davis, Jimmy Fusil, Bhanu Srikanth and Girish Balakrishnan

Game Engines in Virtual Production

The use of Virtual Production and real time technologies has markedly accelerated in the past few years. At Netflix, we are always thrilled to see technology enable new ways of telling stories, and the use of these techniques on some of our shows like 1899 and Super Giant Robot Brothers has given us a front row seat to this exciting evolution in filmmaking. Each production that deploys these methods is an opportunity for the crew, tech manufacturers and us–the Netflix Production Innovation team–to learn, innovate and collaborate towards a common goal: universally accessible workflows that will enable creative opportunities and technical success for all filmmakers regardless of the size, location or scope of their project.

Game engines have been a major part of driving the democratization of advanced virtual production techniques such as pre-visualization and in-camera visual effects. While these platforms were initially developed to build games, their open and versatile design and development environment offer other capabilities. In the hands of film and TV artists and technicians–combined with techniques and technologies such as LED Panels and tracking systems–they’ve become powerful creation tools for filmmakers.

The incredible depth and flexibility of game engines also means there are infinite permutations and combinations of configurations and settings available to the operator. But not getting the configuration right can have a significant impact on the results. Could we harness the benefits of this depth and flexibility while also supporting the predictability and repeatability expected from high pressure shooting environments? Or put more simply: could we offer productions a shortcut to achieving the right configuration for their scenario every time?

The Netflix Production Innovation Team is always asking itself versions of this question: “How do we ensure a high level of creative opportunity and flexibility, while achieving technical consistency and quality? How do we reduce the potential for accidents and errors, while making sure artists and technicians still have the agency to stay true to their vision and their practice?” One past approach to enabling excellence while mitigating risk has been creating knowledge resources for our productions and partners, while also working with the industry to develop standards and best practices. But in this instance, the open development framework of the game engine itself, as well as its central place in Virtual Production workflows, offered another compelling solution.

The Validation Framework as Helper, Teacher and Enabler

Many large scale operations use quality control methods and validation frameworks to ensure successful outcomes. Validation frameworks can act as a flexible and unobtrusive assistant, while offering a safety net for the operator. We’ve had success with this type of validation in the past with Photon, which acted as a library, reference implementation and validation tool for IMF packages, the delivery format for all of our Netflix original programming.

An excellent validation framework is designed with the following characteristics:

  1. It is informative and instructive, providing useful information as to why an issue has arisen, what problems will be caused if not resolved, and what can be done to resolve it.
  2. It acts as dynamic documentation, a checklist, and a co-pilot to allow the operator to focus on the creative aspects.
  3. It promotes standards and consistency across individuals and teams, regardless of how bespoke the workflow is, and allows them to operate along a common and agreed set of expectations and checks.
  4. It enhances efficiency in complex interconnected workflows by looking across multiple components, roles and responsibilities and reporting on the system as a whole.

With such a validation framework in place, we can avoid:

  • Locking users behind layers of confining procedures and tooling, which can limit the evolution of new ways of working by constraining the options available to the operator and creator to adapt to new scenarios, as well as preventing them from learning what’s really going on under the hood.
  • Creating piles of documentation, sticky notes, runbooks… While all of that material is often useful as learning aids or during prep, it’s rarely practical to be flicking through pages of documentation to solve a problem during production.

Furthermore, by incorporating the validation framework within the game engine itself, we could ensure that everything is set up for success, maintaining flexibility while minimizing the risk of error. Because when the camera rolls, creators don’t want to assume everything is ok; they need to know that all the possible steps were taken to prevent errors.

Netflix’s Unreal Engine Validation Framework

Image of the Netflix Unreal Engine Validation Framework User Interface

Functionality

Our Validation Framework, which we have developed as a plugin to Unreal Engine, is extensible and customizable. It hosts and manages automated validation checks and fixes, which help identify and address technical problems within a given workflow.

The Validation Framework builds upon Unreal’s EditorUtility functionality and provides a simple base–the Validation Base–from which all other validations are built. The core also provides a registration to find all validations built atop this base within the Unreal project itself. This allows us to execute the validations from either blueprints or from C++, serving a variety of users: from developers extending the execution into a CI/CD from the C++ side, to an artist executing via a widget or the UI in the editor.

Regardless of how many validations there are or where in the project they live, they are always accessible and can be run from either entry point thanks to the registration mechanics. This gives teams a lot of freedom and flexibility as to how they extend validations into their workflows. For example, a core library of validations can be shared across projects within a company, or across a production–including with other vendors. Or project-specific validations can be deployed, encapsulated within an Unreal project until it is delivered.

While we aim to keep as much exposed in blueprints as possible, having a thin underlying C++ layer is a convenient way to grant the framework access to some of the objects, settings and parameters which are inaccessible via blueprinting alone.

All of the Validations provide two simple hooks: the Validation itself, which checks something, and a Fix, which can apply a correction. What the validation framework checks for, and what and how failures are fixed is entirely up to the artists and developers!

Validations can be grouped to a dedicated Scope, specifically either Level or Project. This enables a hierarchy in the assignment of validations, as some will be applied on Project settings and configurations, while others will be inspecting the content of the Levels.

A second layer of organizational tagging can then be applied to create Workflows. While the validations can only apply to a single Scope (either Level or Project), they can belong to multiple workflows, each with its own set of validations. Several workflow “presets” are available in the initial release, along with the capability for users to define their own workflows.

Users can also define new validations, but the tool requires that they provide a description of what they are checking for, as well as a description of what and how non-conformities (invalid settings or configurations) can be corrected. While we can’t police it, we encourage users to describe not only what the new validation checks, but also why, as well as how failing to resolve non-conformities will impact the workflow results. This will help ensure that the validation framework is useful not only as a risk management tool but also as an educational resource.

The UI is built almost entirely in blueprints, and utilizes a few helper blueprint nodes implemented in C++. Again, this allows users to use what’s there, replace it with their own, or integrate it into their existing UIs.

Finally, the framework generates validation reports, in either JSON or CSV format. Each time it runs, it writes a validation report for the project/level that was just inspected. This allows users to share results with others, such as support teams, or create a record of configurations for reporting or archival purposes.

We focused our initial development on a validation tool within Epic’s Unreal Engine because of its rapid adoption across our global slate of movies and shows, and the vendors that support them. This plug-in approach is also well aligned with Epic Games’ toolkit philosophy: being open to productions of all shapes, sizes and experience levels. Given this wide applicability, we saw a lot of benefit in an easy-to-implement utility that addresses the most prevalent “gotchas” in Unreal project set-ups for Virtual Production projects.

In-Camera VFX Validations

The plugin ships with a set of validations for the most commonly required checks and fixes for inspecting configurations aimed at executing ICVFX techniques. This set of validations is the result of a collaborative effort between the Netflix Production Innovation team and Epic Games; it is built from our collective knowledge and experience on ICVFX productions.

We initially focused only on LED In-Camera VFX (ICVFX) applications. The goal is to help production teams catch and resolve common issues which can cause unexpected render results, data outputs, and potential performance or resource allocation challenges when working with Unreal Engine. Many of the issues this tool mitigates can sometimes be hard to spot, particularly in the complex and high pressure environment of a virtual production set.

That said, the Validation Framework is not solely for use on-set; it can also be utilized during the prep and content build phases, ensuring issues are identified and handled early, before moving further along the production pipeline. The framework can also easily be extended by users to encompass bespoke workflows with custom validations and fixes, allowing teams to create and perform their own unique checks.

Current Usage and Next Steps

The Validation Framework is currently being used on a number of Netflix shows, and has already significantly reduced the amount of time needed for system setup and troubleshooting. We have also received great feedback from our crews, which has led us to add new checks and fixes. Some of our partners have even begun adapting and extending this framework to better suit their proprietary workflows.

In the future, we would like to see such frameworks become commonly used in virtual production environments in two ways:

  1. Continued implementation and extension of this particular validation framework by Virtual Production crews and service providers;
  2. Adaptation of this automated validation approach to assist with the entire stage environment, even where Unreal is not part of the workflow: checking the settings on media servers, LED processors, workstations, tracking systems… all of which play a critical part in a Virtual Production system. In order for that to happen, we need to ensure that this wider ecosystem of devices can also be queried and monitored procedurally, which in turn will free crews to focus on the creative work, and not worry so much about the tech.

Availability

The validation framework can be found at: https://github.com/Netflix-Skunkworks/UnrealValidationFramework

The system can be integrated into pipelines and additional tooling around CI/CD to generate validation reports.

We look forward to receiving your feedback and suggestions via the “Issues” tab on our git repo (above).

Virtual Production — A Validation Framework For Unreal Engine was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Security updates for Wednesday

Post Syndicated from original https://lwn.net/Articles/904374/

Security updates have been issued by Debian (gst-plugins-good1.0), Fedora (firefox and ghostscript), Gentoo (consul, firefox, libass, libraw, lxml, mdbtools, pam_u2f, spice, and thunderbird), Oracle (kernel, kernel-container, and vim), Red Hat (galera, mariadb, and mysql-selinux, kernel, and kernel-rt), Scientific Linux (kernel), SUSE (bind, java-11-openjdk, kernel, mokutil, ncurses, and u-boot), and Ubuntu (epiphany-browser, libcdio, linux, linux-aws, linux-azure-4.15, linux-dell300x, linux-gcp-4.15,
linux-kvm, linux-oracle, linux-raspi2, linux-snapdragon, linux, linux-aws, linux-kvm, linux-lts-xenial, and linux-hwe, linux-aws-hwe, linux-azure, linux-gcp, linux-oracle).

Navigating the Evolving Patchwork of Incident Reporting Requirements

Post Syndicated from Peter Woolverton original https://blog.rapid7.com/2022/08/10/navigating-the-evolving-patchwork-of-incident-reporting-requirements/

Navigating the Evolving Patchwork of Incident Reporting Requirements

In March 2022, President Biden signed into law the Cyber Incident Reporting for Critical Infrastructure Act (CIRCIA), a bipartisan initiative that empowers CISA to require cyber incident reporting from critical infrastructure owners and operators. Rapid7 is supportive of CIRCIA and cyber incident reporting in general, but we also encourage regulators to ensure reporting rules are streamlined and do not impose unnecessary burdens on companies that are actively recovering from cyber intrusions.

Although a landmark legislative change, CIRCIA is just one highly visible example of a broader trend. Incident reporting has emerged as a predominant cybersecurity regulatory strategy across government. Numerous federal and state agencies are implementing their own cyber incident reporting requirements under their respective rulemaking authorities – such as SEC, FTC, the Federal Reserve, OCC, NCUA, NERC, TSA, NYDFS, and others. Several such rules are already in force in US law, with at least three more likely to become effective within the next year.

The trend is not limited to the US. Several international governing bodies have proposed similar cyber incident reporting rules, such as the European Union’s (EU) NIS-2 Directive.

Raising the bar for security transparency through incident reporting is a productive step in a positive direction. Incident reporting requirements can help the government to manage sectoral risk, encourage a higher level of private-sector cyber hygiene, and enhance intrusion remediation and prevention capabilities. But the rapid embrace of this new legal paradigm may have created too much of a good thing, with the emerging regulatory environment risks becoming unmanageable.

Current state

Cyber incident reporting rules that enforce overlapping or contradictory requirements can impose undue compliance burdens on organizations that are actively responding to cyberattacks. To illustrate the problem, consider the potential experience of a hypothetical company – let’s call it Energy1. Energy1 is a US-based, publicly traded utility company that owns and operates energy generation plants, electrical transmission systems, and natural gas distribution lines. If Energy1 experiences a significant cyber attack, it may be required to submit the following reports:

  • Within one hour, provide to NERC – under NERC CIP rules – a report with preliminary details about the incident and its functional impact on operations.
  • Within 24 hours, provide to TSA – under the pipeline security directive – a report with a complete description of the incident, its functional impact on business operations, and the details of remediation steps.
  • Within 72 hours, provide to CISA – under CIRCIA – a complete description of the incident, details of remediation steps, and threat intelligence information that may identify the perpetrator.
  • Within 96 hours, provide to SEC – under the SEC’s proposed rule – a complete description of the incident and its impact, including whether customer data was compromised.

In our hypothetical scenario, Energy1 may need to rapidly compile the necessary information to comply with each different reporting rule or statute, all while balancing the urgent need to remediate and recover from a cyber intrusion. Furthermore, if Energy1 operates in non-US markets as well, it may be subject to several more reporting requirements, such as those proposed under the draft NIS-2 Directive in the EU or the CERT-IN rule in India. Many of these regulations would also require subsequent status updates after the initial report.

The example above demonstrates the complexity of the emerging patchwork of incident reporting requirements. Legal compliance in this new environment creates a number of challenges for the private sector and the government. For example:

  • Redundant requirements: Unnecessarily duplicative compliance requirements imposed in the wake of a cyber incident can draw critical resources away from incident remediation, potentially leading to lower-quality data submitted in the reports.
  • Public vs. private disclosure: Most reports are held privately by regulators, but the SEC’s proposed rule would require companies to file public reports within 96 hours of determining that an incident is significant. Public disclosure before the incident is contained or mitigated may expose the affected company to further risk of cyberattack. In addition, premature public reporting of incidents prior to mitigation may not provide an accurate reflection of the affected company’s cyber incident response capabilities.
  • Inconsistent requirements: The definition of what is reportable is not consistent across agency rules. For example, the SEC requires reporting of cyber incidents that are “material” to a reasonable investor, whereas NERC requires reporting of almost any cyber incident, including failed “attempts” at cyber intrusion. The lack of a uniform definition of reportability adds another layer of complexity to the compliance process.
  • Process inconsistencies: As demonstrated in the Energy1 example, all incident reporting rules and proposed rules have different deadlines. In addition, each rule and proposed rule has different required reporting formats and methods of submission. These process inconsistencies add friction to the compliance process.

Recommendations

The key issues outlined above may be addressed by the Cyber Incident Reporting Council (CIRC), an interagency working group led by the Department of Homeland Security (DHS). This Council was established under CIRCIA and is tasked with harmonizing existing incident reporting requirements into a more unified regulatory regime. A readout of the Council’s first meeting, convened on July 25, stated CIRC’s intent to “reduce [the] burden on industry by advancing common standards for incident reporting.”

In addition to DHS, CIRC includes representatives from across government, including from the Departments of Justice, Commerce, Treasury, and Energy among others. It is not yet clear from the Council’s initial meeting how exactly CIRC will reshape cyber incident reporting regulations, or whether such changes will be achievable through executive action or whether new legislation will be needed. The Council will release a report with recommendations by the end of 2022.

Rapid7 urges CIRC to consider several harmonization strategies intended to streamline compliance while maintaining the benefits of cyber incident reporting, such as:

  • Unified process: When practically possible, develop a single intake point for all incident reporting submissions with a universal format accepted by multiple agencies. This would help eliminate the need for organizations to submit several reports to different agencies with different formats and on different timetables.
  • Deconflicted requirements: Agree on a more unified definition of what constitutes a reportable cyber incident, and build toward more consistent reporting requirements that satisfy the needs of multiple agency rules.
  • Public disclosure delay: Releasing incident reports publicly before affected organizations have time to contain the breach may put the security of the company and its customers at unnecessary risk. Requirements that involve public disclosure, such as proposed rules from the SEC and FTC, should consider delaying and coordinating disclosure timing with the affected company.

Some agencies in the Federal government are already designing incident reporting rules with harmonization in mind. The Federal Reserve, FDIC, and OCC, rather than building out three separate rules for each agency, designed a single universal incident reporting requirement for all three agencies. The rule requires only one report be submitted to whichever of the three agencies is the affected company’s “primary regulator.” The sharing of reports between agencies is handled internally, removing from companies the burden of submitting multiple reports to multiple agencies. Rapid7 supports this approach and would encourage the CIRC to pursue a similarly streamlined strategy in its harmonization efforts where possible.

Striking the right balance

Rapid7 supports the growing adoption of cyber incident reporting. Greater cybersecurity transparency between government and industry can deliver considerable benefits. However, unnecessarily overlapping or contradictory reporting requirements may cause harm by detracting from the critical work of incident response and recovery. We encourage regulators to streamline and simplify the process in order to capture the full benefits of incident reporting without exposing organizations to unnecessary burden or risk in the process.

Additional reading:

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

[$] Adding auditing to pip

Post Syndicated from original https://lwn.net/Articles/904197/

A tool to discover known security vulnerabilities in the Python packages installed on
a system or required by a project, called pip-audit, was recently
discussed on the Python discussion
forum. The developers of pip-audit raised
the idea of adding the functionality directly into the pip package installer, rather than
keeping it as a separately installable tool. While the functionality provided by
pip-audit was seen as a clear benefit to the ecosystem, moving it
inside the pip “tent” was not as overwhelmingly popular. It is not obvious
that auditing is part of the role that the package
installer should play.

Patch Tuesday – August 2022

Post Syndicated from Greg Wiseman original https://blog.rapid7.com/2022/08/09/patch-tuesday-august-2022/

Patch Tuesday - August 2022

It’s the week of Hacker Summer Camp in Las Vegas, and Microsoft has published fixes for 141 separate vulnerabilities in their swath of August updates. This is a new monthly record by raw CVE count, but from a patching perspective, the numbers are slightly less dire. 20 CVEs affect their Chromium-based Edge browser, and 34 affect Azure Site Recovery (up from 32 CVEs affecting that product last month). As usual, OS-level updates will address a lot of these, but note that some extra configuration is required to fully protect Exchange Server this month.

There is one 0-day being patched this month. CVE-2022-34713 is a remote code execution (RCE) vulnerability affecting the Microsoft Windows Support Diagnostic Tool (MSDT) – it carries a CVSSv3 base score of 7.8, as it requires convincing a potential victim to open a malicious file. The advisory indicates that this CVE is a variant of the “Dogwalk” vulnerability, which made news alongside Follina (CVE-2022-30190) back in May.

Publicly disclosed, but not (yet) exploited is CVE-2022-30134, an Information Disclosure vulnerability affecting Exchange Server. In this case, simply patching is not sufficient to protect against attackers being able to read targeted email messages. Administrators should enable Extended Protection in order to fully remediate this vulnerability, as well as the five other vulnerabilities affecting Exchange this month. Details about how to accomplish this are available via the Exchange Blog.

Microsoft also patched several flaws affecting Remote Access Server (RAS). The most severe of these (CVE-2022-30133 and CVE-2022-35744) are related to Windows Point-to-Point Tunneling Protocol and could allow RCE simply by sending a malicious connection request to a server. Seven CVEs affecting the Windows Secure Socket Tunneling Protocol (SSTP) on RAS were also fixed this month: six RCEs and one Denial of Service. If you have RAS in your environment but are unable to patch immediately, consider blocking traffic on port 1723 from your network.

Vulnerabilities affecting Windows Network File System (NFS) have been trending in recent months, and today sees Microsoft patching CVE-2022-34715 (RCE, CVSS 9.8) affecting NFSv4.1 on Windows Server 2022.

This is the worst of it. One last vulnerability to highlight: CVE-2022-35797 is a Security Feature Bypass in Windows Hello – Microsoft’s biometric authentication mechanism for Windows 10. Successful exploitation requires physical access to a system, but would allow an attacker to bypass a facial recognition check.

Summary charts

Patch Tuesday - August 2022
Patch Tuesday - August 2022
Patch Tuesday - August 2022
Patch Tuesday - August 2022

Summary tables

Azure vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-35802 Azure Site Recovery Elevation of Privilege Vulnerability No No 8.1 Yes
CVE-2022-30175 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-30176 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-34687 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35773 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35779 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35806 Azure RTOS GUIX Studio Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35772 Azure Site Recovery Remote Code Execution Vulnerability No No 7.2 Yes
CVE-2022-35824 Azure Site Recovery Remote Code Execution Vulnerability No No 7.2 Yes
CVE-2022-33646 Azure Batch Node Agent Elevation of Privilege Vulnerability No No 7 Yes
CVE-2022-35780 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35781 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35799 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35775 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35801 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35807 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35808 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35782 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35809 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35784 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35810 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35811 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35785 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35786 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35813 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35788 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35814 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35789 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35815 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35790 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35816 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35817 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35791 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35818 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35819 Azure Site Recovery Elevation of Privilege Vulnerability No No 6.5 Yes
CVE-2022-35776 Azure Site Recovery Denial of Service Vulnerability No No 6.2 Yes
CVE-2022-34685 Azure RTOS GUIX Studio Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34686 Azure RTOS GUIX Studio Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-35774 Azure Site Recovery Elevation of Privilege Vulnerability No No 4.9 Yes
CVE-2022-35800 Azure Site Recovery Elevation of Privilege Vulnerability No No 4.9 Yes
CVE-2022-35787 Azure Site Recovery Elevation of Privilege Vulnerability No No 4.9 Yes
CVE-2022-35821 Azure Sphere Information Disclosure Vulnerability No No 4.4 Yes
CVE-2022-35783 Azure Site Recovery Elevation of Privilege Vulnerability No No 4.4 Yes
CVE-2022-35812 Azure Site Recovery Elevation of Privilege Vulnerability No No 4.4 Yes

Browser vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-33649 Microsoft Edge (Chromium-based) Security Feature Bypass Vulnerability No No 9.6 Yes
CVE-2022-33636 Microsoft Edge (Chromium-based) Remote Code Execution Vulnerability No No 8.3 Yes
CVE-2022-35796 Microsoft Edge (Chromium-based) Elevation of Privilege Vulnerability No No 7.5 Yes
CVE-2022-2624 Chromium: CVE-2022-2624 Heap buffer overflow in PDF No No N/A Yes
CVE-2022-2623 Chromium: CVE-2022-2623 Use after free in Offline No No N/A Yes
CVE-2022-2622 Chromium: CVE-2022-2622 Insufficient validation of untrusted input in Safe Browsing No No N/A Yes
CVE-2022-2621 Chromium: CVE-2022-2621 Use after free in Extensions No No N/A Yes
CVE-2022-2619 Chromium: CVE-2022-2619 Insufficient validation of untrusted input in Settings No No N/A Yes
CVE-2022-2618 Chromium: CVE-2022-2618 Insufficient validation of untrusted input in Internals No No N/A Yes
CVE-2022-2617 Chromium: CVE-2022-2617 Use after free in Extensions API No No N/A Yes
CVE-2022-2616 Chromium: CVE-2022-2616 Inappropriate implementation in Extensions API No No N/A Yes
CVE-2022-2615 Chromium: CVE-2022-2615 Insufficient policy enforcement in Cookies No No N/A Yes
CVE-2022-2614 Chromium: CVE-2022-2614 Use after free in Sign-In Flow No No N/A Yes
CVE-2022-2612 Chromium: CVE-2022-2612 Side-channel information leakage in Keyboard input No No N/A Yes
CVE-2022-2611 Chromium: CVE-2022-2611 Inappropriate implementation in Fullscreen API No No N/A Yes
CVE-2022-2610 Chromium: CVE-2022-2610 Insufficient policy enforcement in Background Fetch No No N/A Yes
CVE-2022-2606 Chromium: CVE-2022-2606 Use after free in Managed devices API No No N/A Yes
CVE-2022-2605 Chromium: CVE-2022-2605 Out of bounds read in Dawn No No N/A Yes
CVE-2022-2604 Chromium: CVE-2022-2604 Use after free in Safe Browsing No No N/A Yes
CVE-2022-2603 Chromium: CVE-2022-2603 Use after free in Omnibox No No N/A Yes

Developer Tools vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-35777 Visual Studio Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-35825 Visual Studio Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-35826 Visual Studio Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-35827 Visual Studio Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-34716 .NET Spoofing Vulnerability No No 5.9 Yes

ESU Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-30133 Windows Point-to-Point Protocol (PPP) Remote Code Execution Vulnerability No No 9.8 Yes
CVE-2022-35744 Windows Point-to-Point Protocol (PPP) Remote Code Execution Vulnerability No No 9.8 Yes
CVE-2022-34691 Active Directory Domain Services Elevation of Privilege Vulnerability No No 8.8 Yes
CVE-2022-34714 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-35745 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-35752 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-35753 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-34702 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-35767 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-34706 Windows Local Security Authority (LSA) Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-34707 Windows Kernel Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35768 Windows Kernel Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35756 Windows Kerberos Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35751 Windows Hyper-V Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35795 Windows Error Reporting Service Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35820 Windows Bluetooth Driver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35750 Win32k Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-34713 Microsoft Windows Support Diagnostic Tool (MSDT) Remote Code Execution Vulnerability Yes Yes 7.8 Yes
CVE-2022-35743 Microsoft Windows Support Diagnostic Tool (MSDT) Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35760 Microsoft ATA Port Driver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-30194 Windows WebBrowser Control Remote Code Execution Vulnerability No No 7.5 Yes
CVE-2022-35769 Windows Point-to-Point Protocol (PPP) Denial of Service Vulnerability No No 7.5 No
CVE-2022-35793 Windows Print Spooler Elevation of Privilege Vulnerability No No 7.3 Yes
CVE-2022-34690 Windows Fax Service Elevation of Privilege Vulnerability No No 7.1 Yes
CVE-2022-35759 Windows Local Security Authority (LSA) Denial of Service Vulnerability No No 6.5 No
CVE-2022-35747 Windows Point-to-Point Protocol (PPP) Denial of Service Vulnerability No No 5.9 Yes
CVE-2022-35758 Windows Kernel Memory Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34708 Windows Kernel Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34701 Windows Secure Socket Tunneling Protocol (SSTP) Denial of Service Vulnerability No No 5.3 No

Exchange Server vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-21980 Microsoft Exchange Server Elevation of Privilege Vulnerability No No 8 Yes
CVE-2022-24516 Microsoft Exchange Server Elevation of Privilege Vulnerability No No 8 Yes
CVE-2022-24477 Microsoft Exchange Server Elevation of Privilege Vulnerability No No 8 Yes
CVE-2022-30134 Microsoft Exchange Information Disclosure Vulnerability No Yes 7.6 Yes
CVE-2022-34692 Microsoft Exchange Information Disclosure Vulnerability No No 5.3 Yes
CVE-2022-21979 Microsoft Exchange Information Disclosure Vulnerability No No 4.8 Yes

Microsoft Office vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-34717 Microsoft Office Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-33648 Microsoft Excel Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35742 Microsoft Outlook Denial of Service Vulnerability No No 7.5 Yes
CVE-2022-33631 Microsoft Excel Security Feature Bypass Vulnerability No No 7.3 Yes

System Center Azure vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-33640 System Center Operations Manager: Open Management Infrastructure (OMI) Elevation of Privilege Vulnerability No No 7.8 Yes

Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score Has FAQ?
CVE-2022-34715 Windows Network File System Remote Code Execution Vulnerability No No 9.8 Yes
CVE-2022-35804 SMB Client and Server Remote Code Execution Vulnerability No No 8.8 Yes
CVE-2022-35761 Windows Kernel Elevation of Privilege Vulnerability No No 8.4 Yes
CVE-2022-35766 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-35794 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1 Yes
CVE-2022-34699 Windows Win32k Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-33670 Windows Partition Management Driver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-34703 Windows Partition Management Driver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-34696 Windows Hyper-V Remote Code Execution Vulnerability No No 7.8 Yes
CVE-2022-35746 Windows Digital Media Receiver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35749 Windows Digital Media Receiver Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-34705 Windows Defender Credential Guard Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35771 Windows Defender Credential Guard Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35762 Storage Spaces Direct Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35763 Storage Spaces Direct Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35764 Storage Spaces Direct Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35765 Storage Spaces Direct Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-35792 Storage Spaces Direct Elevation of Privilege Vulnerability No No 7.8 Yes
CVE-2022-30144 Windows Bluetooth Service Remote Code Execution Vulnerability No No 7.5 Yes
CVE-2022-35748 HTTP.sys Denial of Service Vulnerability No No 7.5 Yes
CVE-2022-35755 Windows Print Spooler Elevation of Privilege Vulnerability No No 7.3 Yes
CVE-2022-35757 Windows Cloud Files Mini Filter Driver Elevation of Privilege Vulnerability No No 7.3 Yes
CVE-2022-35754 Unified Write Filter Elevation of Privilege Vulnerability No No 6.7 Yes
CVE-2022-35797 Windows Hello Security Feature Bypass Vulnerability No No 6.1 Yes
CVE-2022-34709 Windows Defender Credential Guard Security Feature Bypass Vulnerability No No 6 Yes
CVE-2022-30197 Windows Kernel Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34710 Windows Defender Credential Guard Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34712 Windows Defender Credential Guard Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34704 Windows Defender Credential Guard Information Disclosure Vulnerability No No 5.5 Yes
CVE-2022-34303 CERT/CC: CVE-20220-34303 Crypto Pro Boot Loader Bypass No No N/A Yes
CVE-2022-34302 CERT/CC: CVE-2022-34302 New Horizon Data Systems Inc Boot Loader Bypass No No N/A Yes
CVE-2022-34301 CERT/CC: CVE-2022-34301 Eurosoft Boot Loader Bypass No No N/A Yes

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

How NerdWallet uses AWS and Apache Hudi to build a serverless, real-time analytics platform

Post Syndicated from Kevin Chun original https://aws.amazon.com/blogs/big-data/how-nerdwallet-uses-aws-and-apache-hudi-to-build-a-serverless-real-time-analytics-platform/

This is a guest post by Kevin Chun, Staff Software Engineer in Core Engineering at NerdWallet.

NerdWallet’s mission is to provide clarity for all of life’s financial decisions. This covers a diverse set of topics: from choosing the right credit card, to managing your spending, to finding the best personal loan, to refinancing your mortgage. As a result, NerdWallet offers powerful capabilities that span across numerous domains, such as credit monitoring and alerting, dashboards for tracking net worth and cash flow, machine learning (ML)-driven recommendations, and many more for millions of users.

To build a cohesive and performant experience for our users, we need to be able to use large volumes of varying user data sourced by multiple independent teams. This requires a strong data culture along with a set of data infrastructure and self-serve tooling that enables creativity and collaboration.

In this post, we outline a use case that demonstrates how NerdWallet is scaling its data ecosystem by building a serverless pipeline that enables streaming data from across the company. We iterated on two different architectures. We explain the challenges we ran into with the initial design and the benefits we achieved by using Apache Hudi and additional AWS services in the second design.

Problem statement

NerdWallet captures a sizable amount of spending data. This data is used to build helpful dashboards and actionable insights for users. The data is stored in an Amazon Aurora cluster. Even though the Aurora cluster works well as an Online Transaction Processing (OLTP) engine, it’s not suitable for large, complex Online Analytical Processing (OLAP) queries. As a result, we can’t expose direct database access to analysts and data engineers. The data owners have to solve requests with new data derivations on read replicas. As the data volume and the diversity of data consumers and requests grow, this process gets more difficult to maintain. In addition, data scientists mostly require data files access from an object store like Amazon Simple Storage Service (Amazon S3).

We decided to explore alternatives where all consumers can independently fulfill their own data requests safely and scalably using open-standard tooling and protocols. Drawing inspiration from the data mesh paradigm, we designed a data lake based on Amazon S3 that decouples data producers from consumers while providing a self-serve, security-compliant, and scalable set of tooling that is easy to provision.

Initial design

The following diagram illustrates the architecture of the initial design.

The design included the following key components:

  1. We chose AWS Data Migration Service (AWS DMS) because it’s a managed service that facilitates the movement of data from various data stores such as relational and NoSQL databases into Amazon S3. AWS DMS allows one-time migration and ongoing replication with change data capture (CDC) to keep the source and target data stores in sync.
  2. We chose Amazon S3 as the foundation for our data lake because of its scalability, durability, and flexibility. You can seamlessly increase storage from gigabytes to petabytes, paying only for what you use. It’s designed to provide 11 9s of durability. It supports structured, semi-structured, and unstructured data, and has native integration with a broad portfolio of AWS services.
  3. AWS Glue is a fully managed data integration service. AWS Glue makes it easier to categorize, clean, transform, and reliably transfer data between different data stores.
  4. Amazon Athena is a serverless interactive query engine that makes it easy to analyze data directly in Amazon S3 using standard SQL. Athena scales automatically—running queries in parallel—so results are fast, even with large datasets, high concurrency, and complex queries.

This architecture works fine with small testing datasets. However, the team quickly ran into complications with the production datasets at scale.

Challenges

The team encountered the following challenges:

  • Long batch processing time and complexed transformation logic – A single run of the Spark batch job took 2–3 hours to complete, and we ended up getting a fairly large AWS bill when testing against billions of records. The core problem was that we had to reconstruct the latest state and rewrite the entire set of records per partition for every job run, even if the incremental changes were a single record of the partition. When we scaled that to thousands of unique transactions per second, we quickly saw the degradation in transformation performance.
  • Increased complexity with a large number of clients – This workload contained millions of clients, and one common query pattern was to filter by single client ID. There were numerous optimizations that we were forced to tack on, such as predicate pushdowns, tuning the Parquet file size, using a bucketed partition scheme, and more. As more data owners adopted this architecture, we would have to customize each of these optimizations for their data models and consumer query patterns.
  • Limited extendibility for real-time use cases – This batch extract, transform, and load (ETL) architecture wasn’t going to scale to handle hourly updates of thousands of records upserts per second. In addition, it would be challenging for the data platform team to keep up with the diverse real-time analytical needs. Incremental queries, time-travel queries, improved latency, and so on would require heavy investment over a long period of time. Improving on this issue would open up possibilities like near-real-time ML inference and event-based alerting.

With all these limitations of the initial design, we decided to go all-in on a real incremental processing framework.

Solution

The following diagram illustrates our updated design. To support real-time use cases, we added Amazon Kinesis Data Streams, AWS Lambda, Amazon Kinesis Data Firehose and Amazon Simple Notification Service (Amazon SNS) into the architecture.

The updated components are as follows:

  1. Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams. We set up a Kinesis data stream as a target for AWS DMS. The data stream collects the CDC logs.
  2. We use a Lambda function to transform the CDC records. We apply schema validation and data enrichment at the record level in the Lambda function. The transformed results are published to a second Kinesis data stream for the data lake consumption and an Amazon SNS topic so that changes can be fanned out to various downstream systems.
  3. Downstream systems can subscribe to the Amazon SNS topic and take real-time actions (within seconds) based on the CDC logs. This can support use cases like anomaly detection and event-based alerting.
  4. To solve the problem of long batch processing time, we use Apache Hudi file format to store the data and perform streaming ETL using AWS Glue streaming jobs. Apache Hudi is an open-source transactional data lake framework that greatly simplifies incremental data processing and data pipeline development. Hudi allows you to build streaming data lakes with incremental data pipelines, with support for transactions, record-level updates, and deletes on data stored in data lakes. Hudi integrates well with various AWS analytics services such as AWS Glue, Amazon EMR, and Athena, which makes it a straightforward extension of our previous architecture. While Apache Hudi solves the record-level update and delete challenges, AWS Glue streaming jobs convert the long-running batch transformations into low-latency micro-batch transformations. We use the AWS Glue Connector for Apache Hudi to import the Apache Hudi dependencies in the AWS Glue streaming job and write transformed data to Amazon S3 continuously. Hudi does all the heavy lifting of record-level upserts, while we simply configure the writer and transform the data into Hudi Copy-on-Write table type. With Hudi on AWS Glue streaming jobs, we reduce the data freshness latency for our core datasets from hours to under 15 minutes.
  5. To solve the partition challenges for high cardinality UUIDs, we use the bucketing technique. Bucketing groups data based on specific columns together within a single partition. These columns are known as bucket keys. When you group related data together into a single bucket (a file within a partition), you significantly reduce the amount of data scanned by Athena, thereby improving query performance and reducing cost. Our existing queries are filtered on the user ID already, so we significantly improve the performance of our Athena usage without having to rewrite queries by using bucketed user IDs as the partition scheme. For example, the following code shows total spending per user in specific categories: 
    SELECT ID, SUM(AMOUNT) SPENDING
FROM "{{DATABASE}}"."{{TABLE}}"
WHERE CATEGORY IN (
'ENTERTAINMENT',
'SOME_OTHER_CATEGORY')
AND ID_BUCKET ='{{ID_BUCKET}}'
GROUP BY ID;

  1. Our data scientist team can access the dataset and perform ML model training using Amazon SageMaker.
  2. We maintain a copy of the raw CDC logs in Amazon S3 via Amazon Kinesis Data Firehose.

Conclusion

In the end, we landed on a serverless stream processing architecture that can scale to thousands of writes per second within minutes of freshness on our data lakes. We’ve rolled out to our first high-volume team! At our current scale, the Hudi job is processing roughly 1.75 MiB per second per AWS Glue worker, which can automatically scale up and down (thanks to AWS Glue auto scaling). We’ve also observed an outstanding improvement of end-to-end freshness at less than 5 minutes due to Hudi’s incremental upserts vs. our first attempt.

With Hudi on Amazon S3, we’ve built a high-leverage foundation to personalize our users’ experiences. Teams that own data can now share their data across the organization with reliability and performance characteristics built into a cookie-cutter solution. This enables our data consumers to build more sophisticated signals to provide clarity for all of life’s financial decisions.

We hope that this post will inspire your organization to build a real-time analytics platform using serverless technologies to accelerate your business goals.

About the authors

Kevin Chun is a Staff Software Engineer in Core Engineering at NerdWallet. He builds data infrastructure and tooling to help NerdWallet provide clarity for all of life’s financial decisions.

Dylan Qu is a Specialist Solutions Architect focused on big data and analytics with Amazon Web Services. He helps customers architect and build highly scalable, performant, and secure cloud-based solutions on AWS.

The mechanics of a sophisticated phishing scam and how we stopped it

Post Syndicated from Matthew Prince original https://blog.cloudflare.com/2022-07-sms-phishing-attacks/

The mechanics of a sophisticated phishing scam and how we stopped it

The mechanics of a sophisticated phishing scam and how we stopped it

Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees. While individual employees did fall for the phishing messages, we were able to thwart the attack through our own use of Cloudflare One products, and physical security keys issued to every employee that are required to access all our applications.

We have confirmed that no Cloudflare systems were compromised. Our Cloudforce One threat intelligence team was able to perform additional analysis to further dissect the mechanism of the attack and gather critical evidence to assist in tracking down the attacker.

This was a sophisticated attack targeting employees and systems in such a way that we believe most organizations would be likely to be breached. Given that the attacker is targeting multiple organizations, we wanted to share here a rundown of exactly what we saw in order to help other companies recognize and mitigate this attack.

Targeted Text Messages

On July 20, 2022, the Cloudflare Security team received reports of employees receiving legitimate-looking text messages pointing to what appeared to be a Cloudflare Okta login page. The messages began at 2022-07-22 22:50 UTC. Over the course of less than 1 minute, at least 76 employees received text messages on their personal and work phones. Some messages were also sent to the employee’s family members. We have not yet been able to determine how the attacker assembled the list of employee’s phone numbers but have reviewed access logs to our employee directory services and have found no sign of compromise.

Cloudflare runs a 24×7 Security Incident Response Team (SIRT). Every Cloudflare employee is trained to report anything that is suspicious to the SIRT. More than 90 percent of the reports to SIRT turn out to not be threats. Employees are encouraged to report anything and never discouraged from over-reporting. In this case, however, the reports to SIRT were a real threat.

The text messages received by employees looked like this:

The mechanics of a sophisticated phishing scam and how we stopped it

They came from four phone numbers associated with T-Mobile-issued SIM cards: (754) 268-9387, (205) 946-7573, (754) 364-6683 and (561) 524-5989. They pointed to an official-looking domain: cloudflare-okta.com. That domain had been registered via Porkbun, a domain registrar, at 2022-07-20 22:13:04 UTC — less than 40 minutes before the phishing campaign began.

Cloudflare built our secure registrar product in part to be able to monitor when domains using the Cloudflare brand were registered and get them shut down. However, because this domain was registered so recently, it had not yet been published as a new .com registration, so our systems did not detect its registration and our team had not yet moved to terminate it.

If you clicked on the link it took you to a phishing page. The phishing page was hosted on DigitalOcean and looked like this:

The mechanics of a sophisticated phishing scam and how we stopped it

Cloudflare uses Okta as our identity provider. The phishing page was designed to look identical to a legitimate Okta login page. The phishing page prompted anyone who visited it for their username and password.

Real-Time Phishing

We were able to analyze the payload of the phishing attack based on what our employees received as well as its content being posted to services like VirusTotal by other companies that had been attacked. When the phishing page was completed by a victim, the credentials were immediately relayed to the attacker via the messaging service Telegram. This real-time relay was important because the phishing page would also prompt for a Time-based One Time Password (TOTP) code.

Presumably, the attacker would receive the credentials in real-time, enter them in a victim company’s actual login page, and, for many organizations that would generate a code sent to the employee via SMS or displayed on a password generator. The employee would then enter the TOTP code on the phishing site, and it too would be relayed to the attacker. The attacker could then, before the TOTP code expired, use it to access the company’s actual login page — defeating most two-factor authentication implementations.

The mechanics of a sophisticated phishing scam and how we stopped it

Protected Even If Not Perfect

We confirmed that three Cloudflare employees fell for the phishing message and entered their credentials. However, Cloudflare does not use TOTP codes. Instead, every employee at the company is issued a FIDO2-compliant security key from a vendor like YubiKey. Since the hard keys are tied to users and implement origin binding, even a sophisticated, real-time phishing operation like this cannot gather the information necessary to log in to any of our systems. While the attacker attempted to log in to our systems with the compromised username and password credentials, they could not get past the hard key requirement.

But this phishing page was not simply after credentials and TOTP codes. If someone made it past those steps, the phishing page then initiated the download of a phishing payload which included AnyDesk’s remote access software. That software, if installed, would allow an attacker to control the victim’s machine remotely. We confirmed that none of our team members got to this step. If they had, however, our endpoint security would have stopped the installation of the remote access software.

How Did We Respond?

The main response actions we took for this incident were:

1. Block the phishing domain using Cloudflare Gateway

Cloudflare Gateway is a Secure Web Gateway solution providing threat and data protection with DNS / HTTP filtering and natively-integrated Zero Trust. We use this  solution internally to proactively identify malicious domains and block them. Our team added the malicious domain to Cloudflare Gateway to block all employees from accessing it.

Gateway’s automatic detection of malicious domains also identified the domain and blocked it, but the fact that it was registered and messages were sent within such a short interval of time meant that the system hadn’t automatically taken action before some employees had clicked on the links. Given this incident we are working to speed up how quickly malicious domains are identified and blocked. We’re also implementing controls on access to newly registered domains which we offer to customers but had not implemented ourselves.

2. Identify all impacted Cloudflare employees and reset compromised credentials

We were able to compare recipients of the phishing texts to login activity and identify threat-actor attempts to authenticate to our employee accounts. We identified login attempts blocked due to the hard key (U2F) requirements indicating that the correct password was used, but the second factor could not be verified. For the three of our employees’ credentials were leaked, we reset their credentials and any active sessions and initiated scans of their devices.

3. Identify and take down threat-actor infrastructure

The threat actor’s phishing domain was newly registered via Porkbun, and hosted on DigitalOcean. The phishing domain used to target Cloudflare was set up less than an hour before the initial phishing wave. The site had a Nuxt.js frontend, and a Django backend. We worked with DigitalOcean to shut down the attacker’s server. We also worked with Porkbun to seize control of the malicious domain.

From the failed sign-in attempts we were able to determine that the threat actor was leveraging Mullvad VPN software and distinctively using the Google Chrome browser on a Windows 10 machine. The VPN IP addresses used by the attacker were 198.54.132.88, and 198.54.135.222. Those IPs are assigned to Tzulo, a US-based dedicated server provider whose website claims they have servers located in Los Angeles and Chicago. It appears, actually, that the first was actually running on a server in the Toronto area and the latter on a server in the Washington, DC area. We blocked these IPs from accessing any of our services.

4. Update detections to identify any subsequent attack attempts

With what we were able to uncover about this attack, we incorporated additional signals to our already existing detections to specifically identify this threat-actor. At the time of writing we have not observed any additional waves targeting our employees. However, intelligence from the server indicated the attacker was targeting other organizations, including Twilio. We reached out to these other organizations and shared intelligence on the attack.

5. Audit service access logs for any additional indications of attack

Following the attack, we screened all our system logs for any additional fingerprints from this particular attacker. Given Cloudflare Access serves as the central control point for all Cloudflare applications, we can search the logs for any indication the attacker may have breached any systems. Given employees’ phones were targeted, we also carefully reviewed the logs of our employee directory providers. We did not find any evidence of compromise.

Lessons Learned and Additional Steps We’re Taking

We learn from every attack. Even though the attacker was not successful, we are making additional adjustments from what we’ve learned. We’re adjusting the settings for Cloudflare Gateway to restrict or sandbox access to sites running on domains that were registered within the last 24 hours. We will also run any non-whitelisted sites containing terms such as “cloudflare” “okta” “sso” and “2fa” through our browser isolation technology. We are also increasingly using Area 1’s phish-identification technology to scan the web and look for any pages that are designed to target Cloudflare. Finally, we’re tightening up our Access implementation to prevent any logins from unknown VPNs, residential proxies, and infrastructure providers. All of these are standard features of the same products we offer to customers.

The attack also reinforced the importance of three things we’re doing well. First, requiring hard keys for access to all applications. Like Google, we have not seen any successful phishing attacks since rolling hard keys out. Tools like Cloudflare Access made it easy to support hard keys even across legacy applications. If you’re an organization interested in how we rolled out hard keys, reach out to [email protected] and our security team would be happy to share the best practices we learned through this process.

Second, using Cloudflare’s own technology to protect our employees and systems. Cloudflare One’s solutions like Access and Gateway were critical to staying ahead of this attack. We configured our Access implementation to require hard keys for every application. It also creates a central logging location for all application authentications. And, if ever necessary, a place from which we can kill the sessions of a potentially compromised employee. Gateway allows us the ability to shut down malicious sites like this one quickly and understand what employees may have fallen for the attack. These are all functionalities that we make available to Cloudflare customers as part of our Cloudflare One suite and this attack demonstrates how effective they can.

Third, having a paranoid but blame-free culture is critical for security. The three employees who fell for the phishing scam were not reprimanded. We’re all human and we make mistakes. It’s critically important that when we do, we report them and don’t cover them up. This incident provided another example of why security is part of every team member at Cloudflare’s job.

Detailed Timeline of Events

2022-07-20 22:49 UTC Attacker sends out 100+ SMS messages to Cloudflare employees and their families.
2022-07-20 22:50 UTC Employees begin reporting SMS messages to Cloudflare Security team.
2022-07-20 22:52 UTC Verify that the attacker’s domain is blocked in Cloudflare Gateway for corporate devices.
2022-07-20 22:58 UTC Warning communication sent to all employees across chat and email.
2022-07-20 22:50 UTC to
2022-07-20 23:26 UTC		 Monitor telemetry in the Okta System log & Cloudflare Gateway HTTP logs to locate credential compromise. Clear login sessions and suspend accounts on discovery.
2022-07-20 23:26 UTC Phishing site is taken down by the hosting provider.
2022-07-20 23:37 UTC Reset leaked employee credentials.
2022-07-21 00:15 UTC Deep dive into attacker infrastructure and capabilities.

Indicators of compromise

Value Type Context and MITRE Mapping
cloudflare-okta[.]com hosted on 147[.]182[.]132[.]52 Phishing URL T1566.002: Phishing: Spear Phishing Link sent to users.
64547b7a4a9de8af79ff0eefadde2aed10c17f9d8f9a2465c0110c848d85317a SHA-256 T1219: Remote Access Software being distributed by the threat actor

What You Can Do

If you are similar attacks in your environment, please don’t hesitate to reach out to [email protected], and we’re happy to share best practices on how to keep your business secure. Finally, if you want to work on detecting and mitigating the next attacks with us? We’re hiring on our Detection and Response team, come join us!

Building AWS Lambda governance and guardrails

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/building-aws-lambda-governance-and-guardrails/

When building serverless applications using AWS Lambda, there are a number of considerations regarding security, governance, and compliance. This post highlights how Lambda, as a serverless service, simplifies cloud security and compliance so you can concentrate on your business logic. It covers controls that you can implement for your Lambda workloads to ensure that your applications conform to your organizational requirements.

The Shared Responsibility Model

The AWS Shared Responsibility Model distinguishes between what AWS is responsible for and what customers are responsible for with cloud workloads. AWS is responsible for “Security of the Cloud” where AWS protects the infrastructure that runs all the services offered in the AWS Cloud. Customers are responsible for “Security in the Cloud”, managing and securing their workloads. When building traditional applications, you take on responsibility for many infrastructure services, including operating systems and network configuration.

Traditional application shared responsibility

Traditional application shared responsibility

One major benefit when building serverless applications is shifting more responsibility to AWS so you can concentrate on your business applications. AWS handles managing and patching the underlying servers, operating systems, and networking as part of running the services.

Serverless application shared responsibility

Serverless application shared responsibility

For Lambda, AWS manages the application platform where your code runs, which includes patching and updating the managed language runtimes. This reduces the attack surface while making cloud security simpler. You are responsible for the security of your code and AWS Identity and Access Management (IAM) to the Lambda service and within your function.

Lambda is SOC, HIPAA, PCI, and ISO-compliant. For more information, see Compliance validation for AWS Lambda and the latest Lambda certification and compliance readiness services in scope.

Lambda isolation

Lambda functions run in separate isolated AWS accounts that are dedicated to the Lambda service. Lambda invokes your code in a secure and isolated runtime environment within the Lambda service account. A runtime environment is a collection of resources running in a dedicated hardware-virtualized Micro Virtual Machines (MVM) on a Lambda worker node.

Lambda workers are bare metalEC2 Nitro instances, which are managed and patched by the Lambda service team. They have a maximum lease lifetime of 14 hours to keep the underlying infrastructure secure and fresh. MVMs are created by Firecracker, an open source virtual machine monitor (VMM) that uses Linux’s Kernel-based Virtual Machine (KVM) to create and manage MVMs securely at scale.

MVMs maintain a strong separation between runtime environments at the virtual machine hardware level, which increases security. Runtime environments are never reused across functions, function versions, or AWS accounts.

Isolation model for AWS Lambda workers

Isolation model for AWS Lambda workers

Network security

Lambda functions always run inside secure Amazon Virtual Private Cloud (Amazon VPCs) owned by the Lambda service. This gives the Lambda function access to AWS services and the public internet. There is no direct network inbound access to Lambda workers, runtime environments, or Lambda functions. All inbound access to a Lambda function only comes via the Lambda Invoke API, which sends the event object to the function handler.

You can configure a Lambda function to connect to private subnets in a VPC in your account if necessary, which you can control with IAM condition keys . The Lambda function still runs inside the Lambda service VPC but sends all network traffic through your VPC. Function outbound traffic comes from your own network address space.

AWS Lambda service VPC with VPC-to-VPC NAT to customer VPC

AWS Lambda service VPC with VPC-to-VPC NAT to customer VPC

To give your VPC-connected function access to the internet, route outbound traffic to a NAT gateway in a public subnet. Connecting a function to a public subnet doesn’t give it internet access or a public IP address, as the function is still running in the Lambda service VPC and then routing network traffic into your VPC.

All internal AWS traffic uses the AWS Global Backbone rather than traversing the internet. You do not need to connect your functions to a VPC to avoid connectivity to AWS services over the internet. VPC connected functions allow you to control and audit outbound network access.

You can use security groups to control outbound traffic for VPC-connected functions and network ACLs to block access to CIDR IP ranges or ports. VPC endpoints allow you to enable private communications with supported AWS services without internet access.

You can use VPC Flow Logs to audit traffic going to and from network interfaces in your VPC.

Runtime environment re-use

Each runtime environment processes a single request at a time. After Lambda finishes processing the request, the runtime environment is ready to process an additional request for the same function version. For more information on how Lambda manages runtime environments, see Understanding AWS Lambda scaling and throughput.

Data can persist in the local temporary filesystem path, in globally scoped variables, and in environment variables across subsequent invocations of the same function version. Ensure that you only handle sensitive information within individual invocations of the function by processing it in the function handler, or using local variables. Do not re-use files in the local temporary filesystem to process unencrypted sensitive data. Do not put sensitive or confidential information into Lambda environment variables, tags, or other freeform fields such as Name fields.

For more Lambda security information, see the Lambda security whitepaper.

Multiple accounts

AWS recommends using multiple accounts to isolate your resources because they provide natural boundaries for security, access, and billing. Use AWS Organizations to manage and govern individual member accounts centrally. You can use AWS Control Tower to automate many of the account build steps and apply managed guardrails to govern your environment. These include preventative guardrails to limit actions and detective guardrails to detect and alert on non-compliance resources for remediation.

Lambda access controls

Lambda permissions define what a Lambda function can do, and who or what can invoke the function. Consider the following areas when applying access controls to your Lambda functions to ensure least privilege:

Execution role

Lambda functions have permission to access other AWS resources using execution roles. This is an AWS principal that the Lambda service assumes which grants permissions using identity policy statements assigned to the role. The Lambda service uses this role to fetch and cache temporary security credentials, which are then available as environment variables during a function’s invocation. It may re-use them across different runtime environments that use the same execution role.

Ensure that each function has its own unique role with the minimum set of permissions..

Identity/user policies

IAM identity policies are attached to IAM users, groups, or roles. These policies allow users or callers to perform operations on Lambda functions. You can restrict who can create functions, or control what functions particular users can manage.

Resource policies

Resource policies define what identities have fine-grained inbound access to managed services. For example, you can restrict which Lambda function versions can add events to a specific Amazon EventBridge event bus. You can use resource-based policies on Lambda resources to control what AWS IAM identities and event sources can invoke a specific version or alias of your function. You also use a resource-based policy to allow an AWS service to invoke your function on your behalf. To see which services support resource-based policies, see “AWS services that work with IAM”.

Attribute-based access control (ABAC)

With attribute-based access control (ABAC), you can use tags to control access to your Lambda functions. With ABAC, you can scale an access control strategy by setting granular permissions with tags without requiring permissions updates for every new user or resource as your organization scales. You can also use tag policies with AWS Organizations to standardize tags across resources.

Permissions boundaries

Permissions boundaries are a way to delegate permission management safely. The boundary places a limit on the maximum permissions that a policy can grant. For example, you can use boundary permissions to limit the scope of the execution role to allow only read access to databases. A builder with permission to manage a function or with write access to the applications code repository cannot escalate the permissions beyond the boundary to allow write access.

Service control policies

When using AWS Organizations, you can use Service control policies (SCPs) to manage permissions in your organization. These provide guardrails for what actions IAM users and roles within the organization root or OUs can do. For more information, see the AWS Organizations documentation, which includes example service control policies.

Code signing

As you are responsible for the code that runs in your Lambda functions, you can ensure that only trusted code runs by using code signing with the AWS Signer service. AWS Signer digitally signs your code packages and Lambda validates the code package before accepting the deployment, which can be part of your automated software deployment process.

Auditing Lambda configuration, permissions and access

You should audit access and permissions regularly to ensure that your workloads are secure. Use the IAM console to view when an IAM role was last used.

IAM last used

IAM last used

IAM access advisor

Use IAM access advisor on the Access Advisor tab in the IAM console to review when was the last time an AWS service was used from a specific IAM user or role. You can use this to remove IAM policies and access from your IAM roles.

IAM access advisor

IAM access advisor

AWS CloudTrail

AWS CloudTrail helps you monitor, log, and retain account activity to provide a complete event history of actions across your AWS infrastructure. You can monitor Lambda API actions to ensure that only appropriate actions are made against your Lambda functions. These include CreateFunction, DeleteFunction, CreateEventSourceMapping, AddPermission, UpdateEventSourceMapping,  UpdateFunctionConfiguration, and UpdateFunctionCode.

AWS CloudTrail

AWS CloudTrail

IAM Access Analyzer

You can validate policies using IAM Access Analyzer, which provides over 100 policy checks with security warnings for overly permissive policies. To learn more about policy checks provided by IAM Access Analyzer, see “IAM Access Analyzer policy validation”.

You can also generate IAM policies based on access activity from CloudTrail logs, which contain the permissions that the role used in your specified date range.

IAM Access Analyzer

IAM Access Analyzer

AWS Config

AWS Config provides you with a record of the configuration history of your AWS resources. AWS Config monitors the resource configuration and includes rules to alert when they fall into a non-compliant state.

For Lambda, you can track and alert on changes to your function configuration, along with the IAM execution role. This allows you to gather Lambda function lifecycle data for potential audit and compliance requirements. For more information, see the Lambda Operators Guide.

AWS Config includes Lambda managed config rules such as lambda-concurrency-check, lambda-dlq-check, lambda-function-public-access-prohibited, lambda-function-settings-check, and lambda-inside-vpc. You can also write your own rules.

There are a number of other AWS services to help with security compliance.

  1. AWS Audit Manager: Collect evidence to help you audit your use of cloud services.
  2. Amazon GuardDuty: Detect unexpected and potentially unauthorized activity in your AWS environment.
  3. Amazon Macie: Evaluates your content to identify business-critical or potentially confidential data.
  4. AWS Trusted Advisor: Identify opportunities to improve stability, save money, or help close security gaps.
  5. AWS Security Hub: Provides security checks and recommendations across your organization.

Conclusion

Lambda makes cloud security simpler by taking on more responsibility using the AWS Shared Responsibility Model. Lambda implements strict workload security at scale to isolate your code and prevent network intrusion to your functions. This post provides guidance on assessing and implementing best practices and tools for Lambda to improve your security, governance, and compliance controls. These include permissions, access controls, multiple accounts, and code security. Learn how to audit your function permissions, configuration, and access to ensure that your applications conform to your organizational requirements.

For more serverless learning resources, visit Serverless Land.

6 Reasons Managed Detection and Response Is Hitting Its Stride

Post Syndicated from Mikayla Wyman original https://blog.rapid7.com/2022/08/09/6-reasons-managed-detection-and-response-is-hitting-its-stride/

6 Reasons Managed Detection and Response Is Hitting Its Stride

Cyber threats have risen to the #1 concern of CEOs, which means security teams — in the hot seat for years — are really feeling it now. Files and data live in the cloud. Work is hybrid or remote. There’s turmoil around the world. Cyberattacks are not just a distant boogieman – they’re here and happening every day.

As companies try to make sure their existing security infrastructure can keep up, they confront the skills gap, a 0% industry unemployment rate, and no room for mistakes. Managed Detection and Response (MDR) is having a moment.

According to a recent ESG study, MDR is one of the fastest growing areas of cybersecurity today. A whopping 85% of surveyed organizations currently use or plan to use managed services for their security operations. And 88% say they will increase their use of managed services in the next 1-2 years.

What’s driving this move to MDR? Let’s take a look at six main factors.

1. Focus

Augmenting an internal security team means internal security personnel can focus on more strategic security initiatives rather than day-to-day operational tasks. In fact, 55% of surveyed organizations want to focus their internal security teams on more strategic initiatives rather than spend time on daily basics, the ESG study found.

By partnering with an MDR provider, alert triaging and investigations are generally taken care of by the external team. Of course, your organization still has some things you’ll need to do – partnership is the name of the game. But by working with a MDR service, security teams suddenly have more time and bandwidth to work strategically.

2. Services

ESG reports that 52% of companies surveyed believe managed service providers can do a better job with security operations than they can.

What you would once have to train your detection and response team to do, MDR providers take over. That means they’re able to detect active attackers within your environment and contain threats. Analyze incidents and provide recommendations for remediation, and apply learnings from other environments they manage to your environment to make sure you’re protected from the latest attacker behaviors. Finally, good MDR providers are able to pivot into breach response if an attacker is live within your network.

To learn more about how to evaluate MDR providers on eight core capabilities, read the MDR Buyers Guide here.

3. Augmentation

About half of organizations (49%) believe a service provider can augment their security operations center (SOC) team with additional support.

Most companies that are able to build internal SOCs are generally well-funded, can afford roughly 10-12 full-time personnel, have a large array of security tools at their disposal, and have extensive processes already outlined. Sound doable? Great! If not, augmentation by way of an MDR provider is your tall glass of water.

Sign on with an MDR provider, get deployed, and your team is instantly extended. Benefits include time savings, cost savings, and experience level that most companies can’t afford to hire at scale.

4. Skills

No surprise, 42% of surveyed organizations in the ESG study believe they don’t have adequate skills for security operations in-house.

MDR is more than outsourcing 24x7x365 monitoring. It’s a partnership that helps you move towards a more secure stature with guidance and expertise.

This type of partnership allows teams to contextualize metrics and reports, get a better understanding of investigations that take place within their environment, and have someone to walk through processes should an attack take place. You also have an expert in your corner during CISO, board, or executive meetings.

5. Price

40% of surveyed organizations did a cost analysis and found that it would cost less to use a service provider than to do it themselves.

We won’t sugar-coat it – partnering with an MDR service provider is expensive. But so is building out an internal team that can actually monitor and investigate within an organization’s environment round the clock.

The cost of partnering with an MDR provider pales in comparison to the cost of employing 10-12 security personnel that operate an around-the-clock SOC, and it can offer ROI much more quickly.

Check out this recent Forrester study to learn more about cost-saving outcomes of partnering with Rapid7’s MDR team.

6. Staff

Finally, ESG tells us that 35% of surveyed organizations don’t have an adequately sized staff for security operations.

Even with unlimited budget to hire a full team, it would be an incredibly labor-intensive and time-consuming process. It would be nearly impossible for most organizations to accomplish. Not only is finding qualified candidates and hiring a huge pain point, but the resources needed to onboard and train staff often aren’t there.

Of course, all MDR services are not the same

Keep these three things in mind:

  • Forrester found Rapid7 MDR reduced breaches by 90%
  • Forrester found Rapid7 MDR delivered 549% ROI
  • In the event of a breach, Rapid7 MDR pivots to full-on digital forensics and incident response, no delay, no limits

Check out our full MDR Buyer’s Guide for 2022 here.

Additional reading:

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

Introducing new Cloudflare for SaaS documentation

Post Syndicated from Mia Malden original https://blog.cloudflare.com/introducing-new-cloudflare-for-saas-documentation/

Introducing new Cloudflare for SaaS documentation

Introducing new Cloudflare for SaaS documentation

As a SaaS provider, you’re juggling many challenges while building your application, whether it’s custom domain support, protection from attacks, or maintaining an origin server. In 2021, we were proud to announce Cloudflare for SaaS for Everyone, which allows anyone to use Cloudflare to cover those challenges, so they can focus on other aspects of their business. This product has a variety of potential implementations; now, we are excited to announce a new section in our Developer Docs specifically devoted to Cloudflare for SaaS documentation to allow you take full advantage of its product suite.

Cloudflare for SaaS solution

You may remember, from our October 2021 blog post, all the ways that Cloudflare provides solutions for SaaS providers:

  • Set up an origin server
  • Encrypt your customers’ traffic
  • Keep your customers online
  • Boost the performance of global customers
  • Support custom domains
  • Protect against attacks and bots
  • Scale for growth
  • Provide insights and analytics
Introducing new Cloudflare for SaaS documentation

However, we received feedback from customers indicating confusion around actually using the capabilities of Cloudflare for SaaS because there are so many features! With the existing documentation, it wasn’t 100% clear how to enhance security and performance, or how to support custom domains. Now, we want to show customers how to use Cloudflare for SaaS to its full potential by including more product integrations in the docs, as opposed to only focusing on the SSL/TLS piece.

Bridging the gap

Cloudflare for SaaS can be overwhelming with so many possible add-ons and configurations. That’s why the new docs are organized into six main categories, housing a number of new, detailed guides (for example, WAF for SaaS and Regional Services for SaaS):

Introducing new Cloudflare for SaaS documentation

Once you get your SaaS application up and running with the Get Started page, you can find which configurations are best suited to your needs based on your priorities as a provider. Even if you aren’t sure what your goals are, this setup outlines the possibilities much more clearly through a number of new documents and product guides such as:

Instead of pondering over vague subsection titles, you can peruse with purpose in mind. The advantages and possibilities of Cloudflare for SaaS are highlighted instead of hidden.

Possible configurations

This setup facilitates configurations much more easily to meet your goals as a SaaS provider.

For example, consider performance. Previously, there was no documentation surrounding reduced latency for SaaS providers. Now, the Performance section explains the automatic benefits to your performance by onboarding with Cloudflare for SaaS. Additionally, it offers three options of how to reduce latency even further through brand-new docs:

Similarly, the new organization offers WAF for SaaS as a previously hidden security solution, extending providers the ability to enable automatic protection from vulnerabilities and the flexibility to create custom rules. This is conveniently accompanied by a step-by-step tutorial using Cloudflare Managed Rulesets.

What’s next

While this transition represents an improvement in the Cloudflare for SaaS docs, we’re going to expand its accessibility even more. Some tutorials, such as our Managed Ruleset Tutorial, are already live within the tile. However, more step-by-step guides for Cloudflare for SaaS products and add-ons will further enable our customers to take full advantage of the available product suite. In particular, keep an eye out for expanding documentation around using Workers for Platforms.

Check it out

Visit the new Cloudflare for SaaS tile to see the updates. If you are a SaaS provider interested in extending Cloudflare benefits to your customers through Cloudflare for SaaS, visit our Cloudflare for SaaS overview and our Plans page.

The collective thoughts of the interwebz