Amazon OpenSearch Service Under the Hood: Multi-AZ with standby

Post Syndicated from Rohin Bhargava original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-under-the-hood-multi-az-with-standby/

Amazon OpenSearch Service recently announced Multi-AZ with standby, a new deployment option for managed clusters that enables 99.99% availability and consistent performance for business-critical workloads. With Multi-AZ with standby, clusters are resilient to infrastructure failures like hardware or networking failure. This option provides improved reliability and the added benefit of simplifying cluster configuration and management by enforcing best practices and reducing complexity.

In this post, we share how Multi-AZ with standby works under the hood to achieve high resiliency and consistent performance to meet the four 9s.

Background

One of the principles in designing highly available systems is that they need to be ready for impairments before they happen. OpenSearch is a distributed system, which runs on a cluster of instances that have different roles. In OpenSearch Service, you can deploy data nodes to store your data and respond to indexing and search requests, you can also deploy dedicated cluster manager nodes to manage and orchestrate the cluster. To provide high availability, one common approach for the cloud is to deploy infrastructure across multiple AWS Availability Zones. Even in the rare case that a full zone becomes unavailable, the available zones continue to serve traffic with replicas.

When you use OpenSearch Service, you create indexes to hold your data and specify partitioning and replication for those indexes. Each index is comprised of a set of primary shards and zero to many replicas of those shards. When you additionally use the Multi-AZ feature, OpenSearch Service ensures that primary shards and replica shards are distributed so that they’re in different Availability Zones.

When there is an impairment in an Availability Zone, the service would scale up in other Availability Zones and redistribute shards to spread out the load evenly. This approach was reactive at best. Additionally, shard redistribution during failure events causes increased resource utilization, leading to increased latencies and overloaded nodes, further impacting availability and effectively defeating the purpose of fault-tolerant, multi-AZ clusters. A more effective, statically stable cluster configuration requires provisioning infrastructure to the point where it can continue operating correctly without having to launch any new capacity or redistribute any shards even if an Availability Zone becomes impaired.

Designing for high availability

OpenSearch Service manages tens of thousands of OpenSearch clusters. We’ve gained insights into which cluster configurations like hardware (data or cluster-manager instance types) or storage (EBS volume types), shard sizes, and so on are more resilient to failures and can meet the demands of common customer workloads. Some of these configurations have been included in Multi-AZ with standby to simplify configuring the clusters. However, this alone is not enough. A key ingredient in achieving high availability is maintaining data redundancy.

When you configure a single replica (two copies) for your indexes, the cluster can tolerate the loss of one shard (primary or replica) and still recover by copying the remaining shard. A two-replica (three copies) configuration can tolerate failure of two copies. In the case of a single replica with two copies, you can still sustain data loss. For example, you could lose data if there is a catastrophic failure in one Availability Zone for a prolonged duration, and at the same time, a node in a second zone fails. To ensure data redundancy at all times, the cluster enforces a minimum of two replicas (three copies) across all its indexes. The following diagram illustrates this architecture.

The Multi-AZ with standby feature deploys infrastructure in three Availability Zones, while keeping two zones as active and one zone as standby. The standby zone offers consistent performance even during zonal failures by ensuring same capacity at all times and by using a statically stable design without any capacity provisioning or data movements during failure. During normal operations, the active zone serves coordinator traffic for read and write requests and shard query traffic, and only replication traffic goes to the standby zone. OpenSearch uses synchronous replication protocol for write requests, which by design has zero replication lag, enabling the service to instantaneously promote a standby zone to active in the event of any failure in an active zone. This event is referred to as a zonal failover. The previously active zone is demoted to the standby mode and recovery operations to bring the state back to healthy begin.

Why zonal failover is critical but hard to do right

One or more nodes in an Availability Zone can fail due to a wide variety of reasons, like hardware failures, infrastructure failures like fiber cuts, power or thermal issues, or inter-zone or intra-zone networking problems. Read requests can be served by any of the active zones, whereas write requests need to be synchronously replicated to all copies across multiple Availability Zones. OpenSearch Service orchestrates two modes of failovers: read failovers and the write failovers.

The primarily goals of read failovers are high availability and consistent performance. This requires the system to constantly monitor for faults and shift traffic away from the unhealthy nodes in the impacted zone. The system takes care of handling the failovers gracefully, allowing all in-flight requests to finish while simultaneously shifting new incoming traffic to a healthy zone. However, it’s also possible for multiple shard copies across both active zones to be unavailable in cases of two node failures or one zone plus one node failure (often referred to as double faults), which poses a risk to availability. To solve this challenge, the system uses a fail-open mechanism to serve traffic off the third zone while it may still be in a standby mode to ensure the system remains highly available. The following diagram illustrates this architecture.

An impaired network device impacting inter-zone communication can cause write requests to significantly slow down, owing to the synchronous nature of replication. In such an event, the system orchestrates a write failover to isolate the impaired zone, cutting off all ingress and egress traffic. Although with write failovers the recovery is immediate, it results in all nodes along with its shards being taken offline. However, after the impacted zone is brought back after network recovery, shard recovery should still be able to use unchanged data from its local disk, avoiding full segment copy. Because the write failover results in the shard copy to be unavailable, we exercise write failovers with extreme caution, neither too frequently nor during transient failures.

The following graph depicts that during a zonal failure, automatic read failover prevents any impact to availability.

The following depicts that during a networking slowdown in a zone, write failover helps recover availability.

To ensure that the zonal failover mechanism is predictable (able to seamlessly shift traffic during an actual failure event), we regularly exercise failovers and keep rotating active and standby zones even during steady state. This not only verifies all network paths, ensuring we don’t hit surprises like clock skews, stale credentials, or networking issues during failover, but it also keeps gradually shifting caches to avoid cold starts on failovers, ensuring we deliver consistent performance at all times.

Improving the resiliency of the service

OpenSearch Service uses several principles and best practices to increase reliability, like automatic detection and faster recovery from failure, throttling excess requests, fail fast strategies, limiting queue sizes, quickly adapting to meet workload demands, implementing loosely coupled dependencies, continuously testing for failures, and more. We discuss a few of these methods in this section.

Automatic failure detection and recovery

All faults get monitored at a minutely granularity, across multiple sub-minutely metrics data points. Once detected, the system automatically triggers a recovery action on the impacted node. Although most classes of failures discussed so far in this post refer to binary failures where the failure is definitive, there is another kind of failure: non-binary failures, termed gray failures, whose manifestations are subtle and usually defy quick detection. Slow disk I/O is one example, which causes performance to be adversely impacted. The monitoring system detects anomalies in I/O wait times, latencies, and throughput, to detect and replace a node with slow I/O. Faster and effective detection and quick recovery is our best bet for a wide variety of infrastructure failures beyond our control.

Effective workload management in a dynamic environment

We’ve studied workload patterns that cause the system either to be overloaded with too many requests, maxing out CPU/memory, or a few rogue queries that can that either allocate huge chunks of memory or runaway queries that can exhaust multiple cores, either degrading the latencies of other critical requests or causing multiple nodes to fail due to the system’s resources running low. Some of the improvements in this direction are being done as a part of search backpressure initiatives, starting with tracking the request footprint at various checkpoints that prevents accommodating more requests and cancels the ones already running if they breach the resource limits for a sustained duration. To supplement backpressure in traffic shaping, we use admission control, which provides capabilities to reject a request at the entry point to avoid doing non-productive work (requests either time out or get cancelled) when the system is already run high on CPU and memory. Most of the workload management mechanisms have configurable knobs. No one size fits all workloads, therefore we use Auto-Tune to control them more granularly.

The cluster manager performs critical coordination tasks like metadata management and cluster formation, and orchestrates a few background operations like snapshot and shard placement. We added a task throttler to control the rate of dynamic mapping updates, snapshot tasks, and so on to prevent overwhelming it and to let critical operations run deterministically all the time. But what happens when there is no cluster manager in the cluster? The next section covers how we solved this.

Decoupling critical dependencies

In the event of cluster manager failure, searches continue as usual, but all write requests start to fail. We concluded that allowing writes in this state should still be safe as long as it doesn’t need to update the cluster metadata. This change further improves the write availability without compromising data consistency. Other service dependencies were evaluated to ensure downstream dependencies can scale as the cluster grows.

Failure mode testing

Although it’s hard to mimic all kinds of failures, we rely on AWS Fault Injection Simulator (AWS FIS) to inject common faults in the system like node failures, disk impairment, or network impairment. Testing with AWS FIS regularly in our pipelines helps us improve our detection, monitoring, and recovery times.

Contributing to open source

OpenSearch is an open-source, community-driven software. Most of the changes including the high availability design to support active and standby zones have been contributed to open source; in fact, we follow an open-source first development model. The fundamental primitive that enables zonal traffic shift and failover is based on a weighted traffic routing policy (active zones are assigned weights as 1 and standby zones are assigned weights as 0). Write failovers use the zonal decommission action, which evacuates all traffic from a given zone. Resiliency improvements for search backpressure and cluster manager task throttling are some of the ongoing efforts. If you’re excited to contribute to OpenSearch, open up a GitHub issue and let us know your thoughts.

Summary

Efforts to improve reliability is a never-ending cycle as we continue to learn and improve. With the Multi-AZ with standby feature, OpenSearch Service has integrated best practices for cluster configuration, improved workload management, and achieved four 9s of availability and consistent performance. OpenSearch Service also raised the bar by continuously verifying availability with zonal traffic rotations and automated tests via AWS FIS.

We are excited to continue our efforts into improving the reliability and fault tolerance even further and to see what new and existing solutions builders can create using OpenSearch Service. We hope this leads to a deeper understanding of the right level of availability based on the needs of your business and how this offering achieves the availability SLA. We would love to hear from you, especially about your success stories achieving high levels of availability on AWS. If you have other questions, please leave a comment.

About the authors

Bukhtawar Khan is a Principal Engineer working on Amazon OpenSearch Service. He is interested in building distributed and autonomous systems. He is a maintainer and an active contributor to OpenSearch.

Gaurav Bafna is a Senior Software Engineer working on OpenSearch at Amazon Web Services. He is fascinated about solving problems in distributed systems. He is a maintainer and an active contributor to OpenSearch.

Murali Krishna is a Senior Principal Engineer at AWS OpenSearch Service. He has built AWS OpenSearch Service and AWS CloudSearch. His areas of expertise include Information Retrieval, Large scale distributed computing, low latency real time serving systems etc. He has vast experience in designing and building web scale systems for crawling, processing, indexing and serving text and multimedia content. Prior to Amazon, he was part of Yahoo!, building crawling and indexing systems for their search products.

Ranjith Ramachandra is a Senior Engineering Manager working on Amazon OpenSearch Service. He is passionate about highly scalable distributed systems, high performance and resilient systems.

Rohin Bhargava is a Sr. Product Manager with the Amazon OpenSearch Service team. His passion at AWS is to help customers find the correct mix of AWS services to achieve success for their business goals.

FBI Disables Russian Malware

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/05/fbi-disables-russian-malware.html

Reuters is reporting that the FBI “had identified and disabled malware wielded by Russia’s FSB security service against an undisclosed number of American computers, a move they hoped would deal a death blow to one of Russia’s leading cyber spying programs.”

The headline says that the FBI “sabotaged” the malware, which seems to be wrong.

Presumably we will learn more soon.

EDITED TO ADD: New York Times story.

EDITED TO ADD: Maybe “sabotaged” is the right word. The FBI hacked the malware so that it disabled itself.

Despite the bravado of its developers, Snake is among the most sophisticated pieces of malware ever found, the FBI said. The modular design, custom encryption layers, and high-caliber quality of the code base have made it hard if not impossible for antivirus software to detect. As FBI agents continued to monitor Snake, however, they slowly uncovered some surprising weaknesses. For one, there was a critical cryptographic key with a prime length of just 128 bits, making it vulnerable to factoring attacks that expose the secret key. This weak key was used in Diffie-Hellman key exchanges that allowed each infected machine to have a unique key when communicating with another machine.

Perform secure database write-backs with Amazon QuickSight

Post Syndicated from Srikanth Baheti original https://aws.amazon.com/blogs/big-data/perform-secure-database-write-backs-with-amazon-quicksight/

Amazon QuickSight is a scalable, serverless, machine learning (ML)-powered business intelligence (BI) solution that makes it easy to connect to your data, create interactive dashboards, get access to ML-enabled insights, and share visuals and dashboards with tens of thousands of internal and external users, either within QuickSight itself or embedded into any application.

A write-back is the ability to update a data mart, data warehouse, or any other database backend from within BI dashboards and analyze the updated data in near-real time within the dashboard itself. In this post, we show how to perform secure database write-backs with QuickSight.

Use case overview

To demonstrate how to enable a write-back capability with QuickSight, let’s consider a fictional company, AnyCompany Inc. AnyCompany is a professional services firm that specializes in providing workforce solutions to their customers. AnyCompany determined that running workloads in the cloud to support its growing global business needs is a competitive advantage and uses the cloud to host all its workloads. AnyCompany decided to enhance the way its branches provide quotes to its customers. Currently, the branches generate customer quotes manually, and as a first step in this innovation journey, AnyCompany is looking to develop an enterprise solution for customer quote generation with the capability to dynamically apply local pricing data at the time of quote generation.

AnyCompany currently uses Amazon Redshift as their enterprise data warehouse platform and QuickSight as their BI solution.

Building a new solution comes with the following challenges:

  • AnyCompany wants a solution that is easy to build and maintain, and they don’t want to invest in building a separate user interface.
  • AnyCompany wants to extend the capabilities of their existing QuickSight BI dashboard to also enable quote generation and quote acceptance. This will simplify feature rollouts because their employees already use QuickSight dashboards and enjoy the easy-to-use interface that QuickSight provides.
  • AnyCompany wants to store the quote negotiation history that includes generated, reviewed, and accepted quotes.
  • AnyCompany wants to build a new dashboard with quote history data for analysis and business insights.

This post goes through the steps to enable write-back functionality to Amazon Redshift from QuickSight. Note that the traditional BI tools are read-only with little to no options to update source data.

Solution overview

This solution uses the following AWS services:

  • Amazon API Gateway – Hosts and secures the write-back REST API that will be invoked by QuickSight
  • AWS Lambda – Runs the compute function required to generate the hash and a second function to securely perform the write-back
  • Amazon QuickSight – Offers BI dashboards and quote generation capabilities
  • Amazon Redshift – Stores quotes, prices, and other relevant datasets
  • AWS Secrets Manager – Stores and manages keys to sign hashes (message digest)

Although this solution uses Amazon Redshift as the data store, a similar approach can be implemented with any database that supports creating user-defined functions (UDFs) that can invoke Lambda.

The following figure shows the workflow to perform write-backs from QuickSight.

The first step in the solution is to generate a hash or a message digest of the set of attributes in Amazon Redshift by invoking a Lambda function. This step prevents request tampering. To generate a hash, Amazon Redshift invokes a scalar Lambda UDF. The hashing mechanism used here is the popular BLAKE2 function (available in the Python library hashlib). To further secure the hash, keyed hashing is used, which is a faster and simpler alternative to hash-based message authentication code (HMAC). This key is generated and stored by Secrets Manager and should be accessible only to allowed applications. After the secure hash is generated, it’s returned to Amazon Redshift and combined in an Amazon Redshift view.

Writing the generated quote back to Amazon Redshift is performed by the write-back Lambda function, and an API Gateway REST API endpoint is created to secure and pass requests to the write-back function. The write-back function performs the following actions:

  1. Generate the hash based on the API input parameters received from QuickSight.
  2. Sign the hash by applying the key from Secrets Manager.
  3. Compare the generated hash with the hash received from the input parameters using the compare_digest method available in the HMAC module.
  4. Upon successful validation, write the record to the quote submission table in Amazon Redshift.

The following section provide detailed steps with sample payloads and code snippets.

Generate the hash

The hash is generated using a Lambda UDF in Amazon Redshift. Additionally, a Secrets Manager key is used to sign the hash. To create the hash, complete the following steps:

  1. Create the Secrets Manager key from the AWS Command Line Interface (AWS CLI):
aws secretsmanager create-secret --name “name_of_secret” --description "Secret key to sign hash" --secret-string '{" name_of_key ":"value"}' --region us-east-1
  1. Create a Lambda UDF to generate a hash for encryption:
import boto3	
import base64
import json
from hashlib import blake2b
from botocore.exceptions import ClientError

def get_secret(): 	#This key is used by the Lambda function to further secure the hash.

    secret_name = "<name_of_secret>"
    region_name = "<aws_region_name>"

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(service_name='secretsmanager', region_name=<aws_region_name>    )

    # In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
    # See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
    # We rethrow the exception by default.

    try:
        get_secret_value_response = client.get_secret_value(SecretId=secret_name)
    except Exception as e:
            raise e

   if "SecretString" in get_secret_value_response:
       access_token = get_secret_value_response["SecretString"]
   else:
       access_token = get_secret_value_response["SecretBinary"]

   return json.loads(access_token)[<token key name>]

SECRET_KEY = get_secret()
AUTH_SIZE = 16 

def sign(payload):
    h = blake2b(digest_size=AUTH_SIZE, key=SECRET_KEY)
    h.update(payload)
    return h.hexdigest().encode('utf-8')

def lambda_handler(event, context):
ret = dict()
 try:
  res = []
  for argument in event['arguments']:
   
   try:
     msg = json.dumps(argument)
     signed_key = sign(str.encode(msg))
     res.append(signed_key.decode('utf-8'))
     
   except:
   res.append(None)     
   ret['success'] = True
   ret['results'] = res
    
except Exception as e:
  ret['success'] = False
  ret['error_msg'] = str(e)
  
 return json.dumps(ret)
  1. Define an Amazon Redshift UDF to call the Lambda function to create a hash:
CREATE OR REPLACE EXTERNAL FUNCTION udf_get_digest (par1 varchar)
RETURNS varchar STABLE
LAMBDA 'redshift_get_digest'
IAM_ROLE 'arn:aws:iam::<AWSACCOUNTID>role/service-role/<role_name>';

The AWS Identity and Access Management (IAM) role in the preceding step should have the following policy attached to be able to invoke the Lambda function:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:us-east-1:<AWSACCOUNTID>1:function:redshift_get_digest"
        }
}
  1. Fetch the key from Secrets Manager.

This key is used by the Lambda function to further secure the hash. This is indicated in the get_secret function in Step 2.

Set up Amazon Redshift datasets in QuickSight

The quote generation dashboard uses the following Amazon Redshift view.

Create an Amazon Redshift view that uses all the preceding columns along with the hash column:

create view quote_gen_vw as select *, udf_get_digest 
( customername || BGCheckRequired || Skill|| Shift ||State ||Cost ) from billing_input_tbl

The records will look like the following screenshot.

The preceding view will be used as the QuickSight dataset to generate quotes. A QuickSight analysis will be created using the dataset. For near-real-time analysis, you can use QuickSight direct query mode.

Create API Gateway resources

The write-back operation is initiated by QuickSight invoking an API Gateway resource, which invokes the Lambda write-back function. As a prerequisite for creating the calculated field in QuickSight to call the write-back API, you must first create these resources.

API Gateway secures and invokes the write-back Lambda function with the parameters created as URL query string parameters with mapping templates. The mapping parameters can be avoided by using the Lambda proxy integration.

Create a REST API resource of method type GET that uses Lambda functions (created in the next step) as the integration type. For instructions, refer to Creating a REST API in Amazon API Gateway and Set up Lambda integrations in API Gateway.

The following screenshot shows the details for creating a query string parameter for each parameter passed to API Gateway.

The following screenshot shows the details for creating a mapping template parameter for each parameter passed to API Gateway.

Create the Lambda function

Create a new Lambda function for the API Gateway to invoke. The Lambda function performs the following steps:

  1. Receive parameters from QuickSight through API Gateway and hash the concatenated parameters.

The following code example retrieves parameters from the API Gateway call using the event object of the Lambda function:

   customer= event['customer’])
    bgc = event['bgc']

The function performs the hashing logic as shown in the create hash step earlier using the concatenated parameters passed by QuickSight.

  1. Compare the hashed output with the hash parameter.

If these don’t match, the write-back won’t happen.

  1. If the hashes match, perform a write-back. Check for the presence of a record in the quote generation table by generating a query from the table using the parameters passed from QuickSight:
query_str = "select * From tbquote where cust = '" + cust + "' and bgc = '" + bgc +"'" +" and skilledtrades = '" + skilledtrades + "'  and shift = '" +shift + "' and jobdutydescription ='" + jobdutydescription + "'"
  1. Complete the following action based on the results of the query:
    1. If no record exists for the preceding combination, generate and run an insert query using all parameters with the status as generated.
    2. If a record exists for the preceding combination, generate and run an insert query with the status as in review. The quote_Id for the existing combination will be reused.

Create a QuickSight visual

This step involves creating a table visual that uses a calculated field to pass parameters to API Gateway and invoke the preceding Lambda function.

  1. Add a QuickSight calculated field named Generate Quote to hold the API Gateway hosted URL that will be triggered to write back the quote history into Amazon Redshift:
concat("https://xxxxx.execute-api.us-east-1.amazonaws.com/stage_name/apiresourcename/?cust=",customername,"&bgc=",bgcheckrequired,"&billrate=",toString(billrate),"&skilledtrades=",skilledtrades,"&shift=",shift,"&jobdutydescription=",jobdutydescription,"&hash=",hashvalue)
  1. Create a QuickSight table visual.
  2. Add required fields such as Customer, Skill, and Cost.
  3. Add the Generate Quote calculated field and style this as a hyperlink.

Choosing this link will write the record into Amazon Redshift. This is incumbent on the same hash value returning when the Lambda function performs the hash on the parameters.

The following screenshot shows a sample table visual.

Write to the Amazon Redshift database

The Secrets Manager key is fetched and used by the Lambda function to generate the hash for comparison. The write-back will be performed only if the hash matches with the hash passed in the parameter.

The following Amazon Redshift table will capture the quote history as populated by the Lambda function. Records in green represent the most recent records for the quote.

Considerations and next steps

Using secure hashes prevents the tampering of payload parameters that are visible in the browser window when the write-back URL is invoked. To further secure the write-back URL, you can employ the following techniques:

  • Deploy the REST API in a private VPC that is accessible only to QuickSight users.
  • To prevent replay attacks, a timestamp can be generated alongside the hashing function and passed as an additional parameter in the write-back URL. The backend Lambda function can then be modified to only allow write-backs within a certain time-based threshold.
  • Follow the API Gateway access control and security best practices.
  • Mitigate potential Denial of Service for public-facing APIs.

You can further enhance this solution to render a web-based form when the write-back URL is opened. This could be implemented by dynamically generating an HTML form in the backend Lambda function to support the input of additional information. If your workload requires a high number of write-backs that require higher throughput or concurrency, a purpose-built data store like Amazon Aurora PostgreSQL-Compatible Edition might be a better choice. For more information, refer to Invoking an AWS Lambda function from an Aurora PostgreSQL DB cluster. These updates can then be synchronized into Amazon Redshift tables using federated queries.

Conclusion

This post showed how to use QuickSight along with Lambda, API Gateway, Secrets Manager, and Amazon Redshift to capture user input data and securely update your Amazon Redshift data warehouse without leaving your QuickSight BI environment. This solution eliminates the need to create an external application or user interface for database update or insert operations, and reduces related development and maintenance overhead. The API Gateway call can also be secured using a key or token to ensure only calls originating from QuickSight are accepted by the API Gateway. This will be covered in subsequent posts.

About the Authors

Srikanth Baheti is a Specialized World Wide Principal Solutions Architect for Amazon QuickSight. He started his career as a consultant and worked for multiple private and government organizations. Later he worked for PerkinElmer Health and Sciences & eResearch Technology Inc, where he was responsible for designing and developing high traffic web applications, highly scalable and maintainable data pipelines for reporting platforms using AWS services and Serverless computing.

Raji Sivasubramaniam is a Sr. Solutions Architect at AWS, focusing on Analytics. Raji is specialized in architecting end-to-end Enterprise Data Management, Business Intelligence and Analytics solutions for Fortune 500 and Fortune 100 companies across the globe. She has in-depth experience in integrated healthcare data and analytics with wide variety of healthcare datasets including managed market, physician targeting and patient analytics.

The Velociraptor 2023 Annual Community Survey

Post Syndicated from Rapid7 original https://blog.rapid7.com/2023/05/10/the-velociraptor-2023-annual-community-survey/

The Velociraptor 2023 Annual Community Survey

By Dr. Mike Cohen & Carlos Canto

Velociraptor is an open-source project led and shaped by the community. Over the years, Velociraptor has become a real force in the field of DFIR, making it an obvious choice for many operational situations. Rapid7 is committed to continue making Velociraptor the premier open-source DFIR and security tool.

To learn more about how the tool is used in the community and what the community expectations are with regard to capabilities, features, and use cases the Velociraptor team distributed our first community survey in early 2023. We are using this information in order to shape future development direction, set priorities and develop our road map. We are grateful to the community members who took the time to respond.

As an open-source project, we depend on our community to contribute. There are many ways contributors can help the project, from developing code, to filing bugs, to improving documentation. One of the most important ways users can contribute is by providing valuable feedback through channels such as this survey, which helps to shape the future road map and new features.

We’re excited to share some of the responses we received in this blog post.

Who is the Velociraptor community?

Of the 213 survey respondents, the majority were analysts (57%) and managers (26%), indicating that most of the respondents are people who know and use Velociraptor frequently.

We also wanted to get a feel for the type of companies using Velociraptor. Users fell pretty evenly into company sizes, with about 30% of responses from small companies (less than 100 employees) and 20% of responses from very large companies of 10,000 employees or more.

These companies also came from a wide range of industries. While many were primarily in the information security fields such as managed security service providers (MSSPs), consultants, and cybersecurity businesses, we also saw a large number of responses from the government sector, the aerospace industries, education, banking/finance, healthcare, etc.

With such a wide range of users, we were interested in how often they use Velociraptor. About a third said they use Velociraptor frequently, another third use it occasionally, and the final third are in the process of evaluating and learning about the tool.

Velociraptor use cases

Velociraptor is a powerful tool with a wide feature set. We wanted to glimpse an idea of what features were most popular and how users prioritize these features. Specifically, we asked about the following main use cases:

Client monitoring and alerts (detection)
Velociraptor can collect client event queries focused on detection. This allows the client to autonomously monitor the endpoint and send back prioritized alerts when certain conditions are met.

→ 12% of users were actively using this feature to monitor endpoints.

Proactively hunting for indicators (threat intelligence)
Velociraptor’s unique ability to collect artifacts at scale from many systems can be combined with threat-intelligence information (such as hashes, etc.) to proactively hunt for compromises by known actors. This question was specifically related to hunting for threat-feed indicators, such as hashes, IP addresses, etc.

→ 16% of users were utilizing this feature.

Ongoing forwarding of events to another system
Velociraptor’s client monitoring queries can be used to simply forward events (such as ETW feeds).

→ 6% of users were utilizing this feature.

Collecting bulk files for analysis on another system (digital forensics)
Velociraptor can be used to collect bulk files from the endpoint for later analysis by other tools (for example, using the Windows.Collection.KapeFiles artifact).

→ 20% of users were using this feature regularly.

Parsing for indicators on the endpoint (digital forensics)
Velociraptor’s artifacts are used to directly parse files on the endpoint, quickly returning actionable high-value information without the need for lengthy post processing.

→ 21% of users use these types of queries.

Proactive hunting for indicators across many systems (incident response)
Velociraptor can hunt for artifacts from many endpoints at once.

→ 21% of users benefit from this capability.

We further asked for the relative importance of these features. Users most valued the ability to collect bulk files and hunt for artifacts across many systems, followed by the ability to directly parse artifacts on the endpoints.

Backwards compatibility

Some users deployed Velociraptor for limited-time engagements so they did not need backwards compatibility for stored data, as they wouldn’t be upgrading to major versions within the same deployment.

Other users required more stable data migration but were generally happy with removing backwards data compatibility, if necessary. For example, one response stated “I would rather you prioritize improvements over compatibility even if it breaks things.”

Another user explained: “In a typical Incident Response scenario, Digital Forensics data has a shelf life of a few weeks or months at best and I am comfortable with the convertibility and portability of much of the data that Velociraptor collects such that archival data can still be worked with even if newer versions of the server no longer support a deprecated format/archive. I think there will be workarounds if this becomes an issue for folks with mountains of legacy data that hasn’t been exported somewhere more meaningful for longer term storage and historical data analytic/intelligence purposes.”

Generally, most users indicated they rarely or never needed to go back to archived data and reanalyze.

Version compatibility

The Velociraptor support policy officially only supports clients and servers on the same release version. However, in reality it usually takes longer to upgrade clients than servers. While some users are able to upgrade clients promptly, many users estimate between 10-50% of deployed clients are a version (or more) older than the server. Therefore, the Velociraptor team needs to maintain some compatibility with older clients to allow time for users to upgrade their endpoints.

The offline collector

The offline collector gives users a way to use Velociraptor’s artifacts without needing to deploy a server. This feature is used exclusively by about 10% of users, while a further 30% of users employ it frequently.

Most users of the offline collection deploy it manually (50%). Deploying via another EDR tool or via Group Policy are also robust options. Some users have created custom wrappers to deploy the offline collector in the field. The offline collection supports directly uploading the collection to a cloud server using a number of methods.

The most popular upload method is to an AWS S3 bucket (30%) while the SFTP connector in the cloud or a custom SFTP server on a VM are also popular options (20% and 23%, respectively). Uploading directly to Google Cloud Storage is the least popular option at about 5%.

Manual copy methods were also popular, ranging from EDR-based copying to Zoom file copy.

Azure blob storage was a common request that Velociraptor currently does not support. Many responses indicate that SFTP is currently a workaround to the lack of direct Azure support. The Velociraptor team should prioritize supporting Azure blob storage.

Data analysis

Velociraptor supports collecting raw files (e.g. Event log files, $MFT etc.) for analysis in other tools. Alternatively, Velociraptor already contains extensive parsers for most forensic artifacts that can be used directly on the endpoint.

Most users do use the built-in forensic parsing and analysis artifacts (55%) but many users also collect raw files (e.g. via the Windows.Collection.KapeFiles artifact).

VQL artifacts

Velociraptor uses the Velociraptor Query Language to perform collections and analysis. The VQL is usually shared with the community via an artifact. Most users utilize the built-in artifacts as well as the artifact exchange. However, over 60% of users report they develop their own artifacts, as well. For those users who develop their own artifacts, we asked about limitations and difficulties in this process.

A common theme that arose was around debugging artifacts and the lack of a VQL debugger and better error reporting. Training and documentation were also pointed out as needing improvement. A suggestion was made to enhance documentation with more examples of how each VQL plugin can be used in practice.

In a related note, the Velociraptor team is running a training course at BlackHat 2023. Developers will impart detailed information on how to deploy Velociraptor and write effective custom VQL.

Role-based access controls

Velociraptor has a role-based access control (RBAC) mechanism where users can be assigned roles from administrator, to investigator, to read-only access provided by the reader role. Users generally found this feature useful—40% found it “moderately useful,” 20% “very useful” and 15% “extremely useful”.The main suggestions for improvements include:

  • Easier management through the GUI (as of version 0.6.8 all user ACLs are managed through the GUI)
  • Custom roles with more granular permissions
  • Better logging and auditing
  • The ability to allow a specific role to only run a pre-approved subset of artifacts
  • A way to only run signed/hashed VQL / prevent a malicious artifact being dropped on the server
  • Making it clearer what each permission grants the user

Multi-tenant support

Velociraptor offers a fully multi-tenanted mode, where organizations can be created or decommissioned quickly with minimal resource overhead. This feature is used by 25% of respondents, who are mainly consultants and service providers using it to support multiple customers. Some companies use multi-tenancy to separate different divisions or subsidiaries of the business.

Client monitoring and alerting

Velociraptor can run event queries on a client. These VQL queries run continuously and stream results to the server when certain conditions are met. Common use cases for these are to generate alerts and enhanced detection.

Some users deploy client monitoring artifacts frequently while others see it as an alternative to EDR tools, when these are available. The primary use-case breakdown was:

  • Detection (e.g. alert when an anomalous event occurs): 27% of users
  • Collection of client events (e.g. forward process event logs to an external system): 18% of users
  • Remediation (e.g. quarantine or remove files automatically): 15% of users

→ 30% of users do not use client monitoring at all.

The most common pain point with client monitoring is the lack of integrated alerting capability (an issue currently being worked on). Some useful feedback on this feature included:

  • Better support for integration with business tools (e.g., Teams, Slack, etc.)
  • Easier to manage event data
  • Not having to build a server side artifact for each client_event artifact
  • A dashboard that lists all alerts
  • An easier way to forward alerts based on severity
  • Lack of pre-built detection rules/packs—in other words, it would be easier to tune down, than to build up

The Quarantine feature

Velociraptor can quarantine an endpoint by collecting the Windows.Remediation.Quarantine artifact. This artifact tunes the firewall rules on the endpoint to block all external network communication while maintaining connectivity to the Velociraptor host. This allows for an endpoint to be isolated during investigation.

The feature is fairly popular—it was “sometimes used” by about 30% of users and “always used” by another 12%.

How is Velociraptor deployed?

Velociraptor is a very lightweight solution, typically taking a few minutes to provision a new deployment. For many of our users, Velociraptor is used in an incident response context on an as-needed basis (46%). Other users prefer a more permanent deployment (25%).

For larger environments, Velociraptor also supports multi-server configuration (13% of users), as well as the more traditional single-server deployment option (70% of users). While some users leverage very short-lived deployments of several days or less (13%), most users keep their deployment for several weeks (27%) to months or permanently (44%).

Velociraptor is designed to work efficiently with many endpoints. We recommend a maximum of 15-20k endpoints on a single server before switching to a multi-server architecture (although users reported success with larger deployment sizes on a single server). This level of performance is adequate in practice for the majority of users.

Many users run deployments of less than 250 endpoints (44%) while a further 40% of users deploy to less than 5,000 endpoints.

Approximately 10% of users have deployment sizes larger than 25,000 endpoints, with 2% of users over 100,000 endpoints.

Popular operating systems

Among Velociraptor’s supported operating systems, Windows 64-bit is the most popular (with 82% of users ranking it the most-deployed OS type), while Linux is the next most popular deployed endpoint OS. Mac is the third popular choice for Velociraptor’s users. Finally, 32-bit Windows systems are still prevalent, as well.

Resources and references

Velociraptor’s website at https://docs.velociraptor.app/ contains a wealth of reference material, training courses, and presentations. We also have an active YouTube channel with many instructional videos.

While some users ranked the website as “extremely useful” (25%), there is clearly room for improvement. 42% of users rated it as only “very useful” or “moderately useful” (28%).Suggestions for improvements included:

  • More in-depth YouTube videos breaking down the tool’s features with workflows
  • More detailed “how to” with practical examples
  • Improved documentation about functions and plugins, with a slightly more detailed explanation and a small example
  • Updates to the documentation to reflect the new versions and features

Testimonials

Finally, I wanted to share with you some of the testimonials that users wrote in the survey. We are humbled with the encouraging and positive words we read, and are excited to be making an impact on the DFIR field:

  • "I have to congratulate you and thank you for developing such an amazing tool. It’s the future of DFIR."
  • "Awesome product, can’t wait to use it in prod!"
  • "This is a game-changer for the DFIR industry. Keep up the great work."
  • "Keep the file system based backend, its simplicity makes chain of custody/court submissions possible."
  • "I thoroughly love Velociraptor. The team and community are absolutely fantastic. I would go as far as to say that Mike and Matthew Green are my favorite infosec gentlemen in the industry."
  • "Y’all are awesome. I feel like I was pretty critical, but that’s because this is an amazing software, and I want to see it continue to grow and improve."
  • "We have been deploying Velociraptor to client environments almost since it was released. Our DFIR business model is entirely centered around it and it works very well for us. It is a great solution that just keeps getting better and better."

Conclusions

This is our first Velociraptor community survey, and it has proven to be extremely useful. Since Velociraptor is a community-led, open-source project, we need an open feedback loop to our users. This helps us understand where things need improvement and which features should be prioritized.

At the same time, since Velociraptor is an open-source project, I hope this survey will inspire contributions from the community. We value all contributions, from code to documentation, testing, and bug reports.

Finally, for all of our US-based users, we hope to see you all in person this year at BlackHat 2023! Join us for an in-depth Velociraptor training and to geek out with VQL for 4 days, learning practical, actionable skills and supporting this open-source project.

Keep Digging!

Security updates for Wednesday

Post Syndicated from original https://lwn.net/Articles/931488/

Security updates have been issued by Debian (emacs), Fedora (chromium, community-mysql, and LibRaw), Red Hat (nodejs nodejs-nodemon, nodejs:18, and webkit2gtk3), Slackware (mozilla), SUSE (amazon-ssm-agent, conmon, distribution, docker-distribution, google-cloud-sap-agent, ignition, kernel, ntp, prometheus-ha_cluster_exporter, protobuf-c, python-cryptography, runc, and shim), and Ubuntu (ceph, freetype, and node-css-what).

Introducing Object Lifecycle Management for Cloudflare R2

Post Syndicated from Harshal Brahmbhatt original http://blog.cloudflare.com/introducing-object-lifecycle-management-for-cloudflare-r2/

Introducing Object Lifecycle Management for Cloudflare R2

Introducing Object Lifecycle Management for Cloudflare R2

Last year, R2 made its debut, providing developers with object storage while eliminating the burden of egress fees. (For many, egress costs account for over half of their object storage bills!) Since R2’s launch, tens of thousands of developers have chosen it to store data for many different types of applications.

But for some applications, data stored in R2 doesn’t need to be retained forever. Over time, as this data grows, it can unnecessarily lead to higher storage costs. Today, we’re excited to announce that Object Lifecycle Management for R2 is generally available, allowing you to effectively manage object expiration, all from the R2 dashboard or via our API.

Object Lifecycle Management

Object lifecycles give you the ability to define rules (up to 1,000) that determine how long objects uploaded to your bucket are kept. For example, by implementing an object lifecycle rule that deletes objects after 30 days, you could automatically delete outdated logs or temporary files. You can also define rules to abort unfinished multipart uploads that are sitting around and contributing to storage costs.

Getting started with object lifecycles in R2

Cloudflare dashboard

Introducing Object Lifecycle Management for Cloudflare R2
  1. From the Cloudflare dashboard, select R2.
  2. Select your R2 bucket.
  3. Navigate to the Settings tab and find the Object lifecycle rules section.
  4. Select Add rule to define the name, relevant prefix, and lifecycle actions: delete uploaded objects or abort incomplete multipart uploads.
  5. Select Add rule to complete the process.

S3 Compatible API

With R2’s S3-compatible API, it’s easy to apply any existing object lifecycle rules to your R2 buckets.

Here’s an example of how to configure your R2 bucket’s lifecycle policy using the AWS SDK for JavaScript. To try this out, you’ll need to generate an Access Key.

import S3 from "aws-sdk/clients/s3.js";

const client = new S3({
  endpoint: `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`,
  credentials: {
    accessKeyId: ACCESS_KEY_ID, //  fill in your own
    secretAccessKey: SECRET_ACCESS_KEY, // fill in your own
  },
  region: "auto",
});

await client
  .putBucketLifecycleConfiguration({
    LifecycleConfiguration: {
      Bucket: "testBucket",
      Rules: [
        // Example: deleting objects by age
        // Delete logs older than 90 days
        {
          ID: "Delete logs after 90 days",
          Filter: {
            Prefix: "logs/",
          },
          Expiration: {
            Days: 90,
          },
        },
        // Example: abort all incomplete multipart uploads after a week
        {
          ID: "Abort incomplete multipart uploads",
          AbortIncompleteMultipartUpload: {
            DaysAfterInitiation: 7,
          },
        },
      ],
    },
  })
  .promise();

For more information on how object lifecycle policies work and how to configure them in the dashboard or API, see the documentation here.

Speaking of documentation, if you’d like to provide feedback for R2’s documentation, fill out our documentation survey!

What’s next?

Creating object lifecycle rules to delete ephemeral objects is a great way to reduce storage costs, but what if you need to keep objects around to access in the future? We’re working on new, lower cost ways to store objects in R2 that aren’t frequently accessed, like long tail user-generated content, archive data, and more. If you’re interested in providing feedback and gaining early access, let us know by joining the waitlist here.

Join the conversation: share your feedback and experiences

If you have any questions or feedback relating to R2, we encourage you to join our Discord community to share! Stay tuned for more exciting R2 updates in the future.

Let’s Architect! Designing serverless solutions

Post Syndicated from Luca Mezzalira original https://aws.amazon.com/blogs/architecture/lets-architect-designing-serverless-solutions/

During his re:Invent 2022 keynote, Werner Vogels, AWS Vice President and Chief Technology Officer, emphasized the asynchronous nature of our world and the challenges associated with incorporating asynchronicity into our architectures. AWS serverless services can help users concentrate on the asynchronous aspects of their workloads, easing the execution of event-driven architectures and enabling the adoption of effective integration patterns for communication both within and beyond a bounded context.

In this edition of Let’s Architect!, we offer an in-depth exploration of the architecture of serverless AWS services, such as AWS Lambda. We also present a new workshop centered on design patterns employing serverless AWS services, which ultimately delivers valuable insights on implementing event-driven architectures within systems.

A closer look at AWS Lambda

This video is the perfect companion for those seeking to learn and master a Lambda architecture, empowering you to effectively leverage its capabilities in your workloads.

With the knowledge gained from this video, you will be well-equipped to design your functions’ code in a highly optimized manner, ensuring efficient performance and resource utilization. Furthermore, a comprehensive understanding of Lambda functions can help identify and apply the most suitable approach to cloud workloads, resulting in an agile and robust cloud infrastructure that meets a project’s unique requirements.

Take me to this video!

Discover how AWS Lambda functions work under the hood

Discover how AWS Lambda functions work under the hood

Implementing an event-driven serverless story generation application with ChatGPT and DALL-E

This example of an event-driven serverless architecture showcases the power of leveraging AWS services and AI technologies to develop innovative solutions. Built upon a foundation of serverless services, including Amazon EventBridge, Amazon DynamoDB, Lambda, Amazon Simple Storage Service, and managed artificial intelligence (AI ) services like Amazon Polly, this architecture demonstrates the seamless capacity to create daily stories with a scheduled launch. By utilizing EventBridge scheduler, an Lambda function is initiated every night to generate new content. The integration of AI services, like ChatGPT and DALL-E, further elevates the solution, as their compatibility with the serverless model enables efficient and dynamic content creation. This case serves as a testament to the potential of combining event-driven serverless architectures, with cutting-edge AI technologies for inventive and impactful applications.

Take me to this Compute Blog post!

How to build an event-driven architecture with serverless AWS services integrating ChatGPT and DALL-E

How to build an event-driven architecture with serverless AWS services integrating ChatGPT and DALL-E

AWS Workshop Studio: Serverless Patterns

The AWS Serverless Patterns workshop offers a comprehensive learning experience to enhance your understanding of architectural patterns applicable to serverless projects. Throughout the workshop, participants will delve into various patterns, such as synchronous and asynchronous implementations, tailored to meet the demands of modern serverless applications. This hands-on approach ensures a production-ready understanding, encompassing crucial topics like testing serverless workloads, establishing automation pipelines, and more. Take this workshop to elevate your serverless architecture knowledge!

Take me to the serverless workshop!

The high-level architecture of the workshops modules

The high-level architecture of the workshops modules

Building Serverlesspresso: Creating event-driven architectures

Serverlesspresso is an event-driven, serverless workload that uses EventBridge and AWS Step Functions to coordinate events across microservices and support thousands of orders per day. This comprehensive session delves into design considerations, development processes, and valuable lessons learned from creating a production-ready solution. Discover practical patterns and extensibility options that contribute to a robust, scalable, and cost-effective application. Gain insights into combining EventBridge and Step Functions to address complex architectural challenges in larger applications.

Take me to this video!

How to leverage AWS Step Functions for orchestrating your workflows

How to leverage AWS Step Functions for orchestrating your workflows

See you next time!

Thanks for joining our conversation on serverless solutions! We’ll see you next time when we talk about AWS microservices.

Can’t get enough of the Let’s Architect! series? Visit the Let’s Architect! page of the AWS Architecture Blog!

На второ четене: „Барнабо от планината“

Post Syndicated from Севда Семер original https://www.toest.bg/na-vtoro-chetene-barnabo-ot-planinata/

„Барнабо от планината“ от Дино Будзати

На второ четене: „Барнабо от планината“

превод от италиански Нева Мичева, изд. „Лист“, 2020

В планината винаги е сегашно време. С всяка крачка от похода по нещо излишно се отхлабва и изпада от съзнанието, докато не останат единствено усещанията от настоящия момент. Тялото разпознава горския пейзаж като място, в което сетивата имат най-важната информация. Вниманието към това, което сме способни да видим, чуем и изпитаме, е далеч по-голямо.

Романът на Дино Будзати е разположен в миналото, но е написан в сегашно време. Всичко се случва пред очите ни, но вече е било. Също както първите пролетни листа всъщност са резултат от случилото се в края на миналото лято, преди растенията да заспят за цяла зима. И пак както сезоните имат и бъдещето в себе си с неизбежната им и вечна промяна, така и

историята има в себе си третото време.

Когато ни се представя младият герой от заглавието, ни се казва, че той е, който „после ще стане Барнабо от планината“. Какво означава това, не ни е дадено – може би той ще стане известен горски, прекарал целия си живот из планините, и това е прякорът му?

Още първите страници на книгата носят поезия.

Но не в смисъла на нещо отнесено, в което някои я обвиняват, а напротив – поезия според истинската ѝ дефиниция, съсредоточеното и чувствително наблюдаване на онова, което се случва. Точно каквото се случва с нас в гората. До самия край не знаех, че Будзати е бил алпинист; впоследствие го научих от текста на преводачката Нева Мичева. Не се изненадах – толкова са силни описанията на звуците, гледките, вятъра по кожата, че е ясно – идват от дълбоко почитание и любов.

А книгата разчита и на собствените спомени на читателя. На първата страница гората се описва „със сухите дънери, рухнали на земята, и всичките престари горски неща“. Въображението ти дорисува онова, което няма нужда да се описва, и по този начин повествованието прави здрава спойка с твоите лични преживявания. И ето как връзката ти с разказа е силна още от първата страница.

На второ четене: „Барнабо от планината“

Сюжетът е толкова прост, че прилича на притча. Също като нея има неочаквана дълбочина. Именно това прави така, че историята не може да се обобщи, всъщност предизвиква точно обратното – да мислим и усещаме дълго след като е свършила.

Дел Коле е старшият на горските стражари. Той пази толкова истории, че „ако тръгне да ги разправя всичките, ще се мръкне, ще се съмне и пак няма да имат свършек“. От тях опознаваме ситуацията, преди да започне същинското действие. Планират да направят път, за който да взривят скала с барут, но този план се проваля – някой задига инструментите. Властите решават да запазят барута в планината, докато дойде пак пролет, но тогава пък няма пари за довършване на пътя. После военните решават, че това е добро място за съхранение на боеприпаси – ето как от години дванайсет стражари охраняват пещерата на смени. Сега са плъзнали слухове, че в гората има разбойници, които са набелязали склада.

Началото на историята лъкатуши между спомените на Дел Коле. Той отдавна живее в гората, и – казва ни се – тя също в него. Фокусът ни се обостря, когато той е нападнат и убит от разбойниците. Научили сме достатъчно за него, че смъртта му да ни бодне – и веднага да поискаме историята да продължи, докато не се стигне до справедливост.

Младият Барнабо открива Дел Коле вече безжизнен – вероятно това, че пръв вижда и осъзнава какво се случва, го ангажира дори повече от другарите му. Известно време и жителите на близкото градче са нащрек, даже свалят от пироните своите пушки, но с времето ги връщат пак на същото място – гората е труднопроходима, пък и къде ще тръгнеш по чукарите да търсиш разбойници, опасно е. Стражарите обаче живеят там.

Образът на Барнабо все повече се избистря. Той притежава деликатност и свенливост. Става дума не само за това, че докато другите танцуват с момичета на празненство, той стои настрана – и чак на последните две песни се престрашава. За същото говори и решението му как да действа, когато се сдърпва с друг стражар:

И при все че е по-силен, Моло наистина е на път да клекне, то е все по-явно – как само криви уста от болка. Голям срам ще е за него. Всички го гледат. Барнабо усеща, че съперникът му се измъчва, преструва се, че се подхлъзва, отпуска хватката си и отскача назад. Моло се изправя запъхтян и свъсен.
– Видя ли, че не можеш?

Барнабо, както се вижда тук, е особено чувствителен към болката. Всъщност според мен в тези няколко реда е цялата му личност. По-нататък в историята ранена птица ще го доведе до това, което той вижда като своето падение (и което е решаващ инцидент в историята). Той не е способен да се гордее, че спестява унижението на друг или пък че се вълнува от съдбата на една птица. Вместо това се проявява силно другата му чувствителност, която също се вижда в откъса – страхът му от срама.

Точно за това е цялата история според мен. И точно срамът я превръща от разказ за разбойници, стражи и гора в

разказ за битката да приемеш собствената си сила и личност.

Барнабо знае, че е различен от другарите си. Именно от напрежението между това, което е, и онова, което те са – и което сигурно му изглежда, че трябва да бъде и той самият, – се поражда силният му срам, движещ го след инцидента в сърцето на романа. Той е герой, който усеща, че „дори канарите могат да го видят“ да прави нещо срамно. Макар че срамът му е от история, която никой друг не знае, пред другарите си „завинаги ще е длъжен да свежда очи“.

Барнабо обаче не е страхлив, той е човечен. Дали ще успее да го види? Или ще продължи да живее, измъчван от срам?

До каква степен сме способни да защитаваме собствените си принципи – особено когато те са в разрез с всичко друго, което виждаме?

Това са големите въпроси на романа и в крайна сметка заради тях препусках през страниците – за мен те вдигат залога. Впрочем това са и теми, толкова сериозни и страшни, преди да се разкрият на светло, че са и изключително рядко срещани в литературата.

Темите за страха, срама, дълга, чувствителността са общочовешки,

но никак не е случайно според мен, че тази книга е писана от мъж, както и че главният герой е мъж.

Не е случайно, че има толкова оръжия, нито че действието се случва в гората, извън обществото, в изолация. Срамът и страхът лесно се превръщат в убийствена агресия. Тези проблеми са решаващи за поколения млади мъже, виждаме го и днес. За мен Будзати прави много повече от далеч по-известни съвременни писатели, които напътстват тези хора. Много рядко ми се случва да видя някой да е писал за всичко това с подобна чувствителност. Затова ми се струва изключително важно да продължаваме да четем този роман и днес, точно сто години след като е писан.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталога на „Лист“, както и на няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.

Никой от нас не чете единствено най-новите книги. Тогава защо само за тях се пише? „На второ четене“ е рубрика, в която отваряме списъците с книги, публикувани преди поне година, четем ги и препоръчваме любимите си от тях. Рубриката е част от партньорската програма Читателски клуб „Тоест“. Изборът на заглавия обаче е единствено на авторите – Стефан Иванов и Севда Семер, които биха ви препоръчали тези книги и ако имаше как веднъж на две седмици да се разходите с тях в книжарницата.

Customer checklist for eIDAS regulation now available

Post Syndicated from Borja Larrumbide original https://aws.amazon.com/blogs/security/customer-checklist-for-eidas-regulation-now-available/

AWS is pleased to announce the publication of a checklist to help customers align with the requirements of the European Union’s electronic identification, authentication, and trust services (eIDAS) regulation regarding the use of electronic identities and trust services. The eIDAS regulation covers electronic identification and trust services for electronic transactions in the European single market.

This checklist is intended as a reference and supporting document to help institutions align with the requirements of eIDAS and the European Telecommunications Standards Institute (ETSI). Where applicable, under the AWS Shared Responsibility Model, this checklist provides supporting details and references in relation to AWS to assist institutions when adopting eIDAS and ETSI for their workloads on AWS services.

For the controls that AWS is fully or partially responsible for, the checklist compares the eIDAS and ETSI requirements to the following:

This checklist is valid until the current eIDAS EU regulation 910/2014, published July 23rd, 2014, ceases to be in force. The checklist is available upon request.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Borja Larrumbide

Borja Larrumbide

Borja is a Security Assurance Manager for AWS in Spain and Portugal. Previously, he worked at companies such as Microsoft and BBVA in different roles and sectors. Borja is a seasoned security assurance practitioner with years of experience engaging key stakeholders at national and international levels. His areas of interest include security, privacy, risk management, and compliance.

Daniel Fuertes

Daniel Fuertes

Daniel is a Security Audit Program Manager at AWS based in Madrid, Spain. Daniel leads multiple security audits, attestations, and certification programs in Spain and other EMEA countries. Daniel has nine years of experience in security assurance, including previous experience as an auditor for the PCI DSS security framework.

Thunderbird 2022 financial report

Post Syndicated from original https://lwn.net/Articles/931425/

The Thunderbird email-client project has put out a
report describing its financial situation in 2022.

The breakout growth we enjoyed last year means hiring even more
talented people to vastly improve the Thunderbird desktop
experience. This past year we expended significant effort to
dramatically improve Thunderbird’s UX and bring it in-line with
modern expectations and standards. In 2022 we also laid the
groundwork for large architectural changes for Thunderbird on the
desktop. These changes address many years of technical debt that
has limited our ability to add new features at a brisk pace. This
work will largely pay off in our 2024 release, however it does
power some of the improvements in the 115 “Supernova” release this
summer.

New Storage-Optimized Amazon EC2 I4g Instances: Graviton Processors and AWS Nitro SSDs

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-storage-optimized-amazon-ec2-i4g-instances-graviton-processors-and-aws-nitro-ssds/

Today we are launching I4g instances powered by AWS Graviton2 processors that deliver up to 15% better compute performance than our other storage-optimized instances.

With up to 64 vCPUs, 512 GiB of memory, and 15 TB of NVMe storage, one of the six instance sizes is bound to be a great fit for your storage-intensive workloads: relational and non-relational databases, search engines, file systems, in-memory analytics, batch processing, streaming, and so forth. These workloads are generally very sensitive to I/O latency, and require plenty of random read/write IOPS along with high CPU performance.

Here are the specs:

Instance Name vCPUs
 Memory
 Storage
 Network Bandwidth
 EBS Bandwidth
i4g.large 2 16 GiB 468 GB up to 10 Gbps up to 40 Gbps
i4g.xlarge 4 32GiB 937 GB up to 10 Gbps up to 40 Gbps
i4g.2xlarge 8 64 GiB 1.875 TB up to 12 Gbps up to 40 Gbps
i4g.4xlarge 16 128 GiB 3.750 TB up to 25 Gbps up to 40 Gbps
i4g.8xlarge 32 256 GiB 7.500 TB
(2 x 3.750 TB)		 18.750 Gbps 40 Gbps
i4g.16xlarge 64 512 GiB 15.000 TB
(4 x 3.750 TB)		 37.500 Gbps 80 Gbps

The I4g instances make use of AWS Nitro SSDs (read AWS Nitro SSD – High Performance Storage for your I/O-Intensive Applications to learn more) for NVMe storage. Each storage volume can deliver the following performance (all measured using 4 KiB blocks):

  • Up to 800K random write IOPS
  • Up to 1 million random read IOPS
  • Up to 5600 MB/second of sequential writes
  • Up to 8000 MB/second of sequential reads

Torn Write Protection is supported for 4 KiB, 8 KiB, and 16 KiB blocks.

Available Now
I4g instances are available today in the US East (Ohio, N. Virginia), US West (Oregon), and Europe (Ireland) AWS Regions in On-Demand, Spot, Reserved Instance, and Savings Plan form.

— Jeff;

Patch Tuesday – May 2023

Post Syndicated from Adam Barnett original https://blog.rapid7.com/2023/05/09/patch-tuesday-may-2023/

Patch Tuesday - May 2023

A less crowded Patch Tuesday for May 2023: Microsoft is offering fixes for just 49 vulnerabilities this month. There are no fixes this month for printer drivers, DNS, or .NET, three components which have featured heavily in recent months. Three zero-day vulnerabilities are patched, alongside a further five critical Remote Code Execution (RCE) vulnerabilities. None of the three zero-day vulnerabilities have a particularly high CVSSv3 base score, but timely patching is always indicated.

Zero-day vulnerability: BlackLotus malware Secure Boot bypass

First up: a zero-day Secure Boot Security Feature Bypass vulnerability which is actively exploited by the BlackLotus bootkit malware. Microsoft warns that an attacker who already has Administrator access to an unpatched asset could exploit CVE-2023-24932 without necessarily having physical access. The relatively low CVSSv3 base score of 6.7 isn’t necessarily a reliable metric in this case.

Microsoft has provided a supplementary guidance article specifically calling out the threat posed by BlackLotus malware, which loads ahead of the operating system on compromised assets, and provides attackers with an array of powerful evasion, persistence, and Command & Control (C2) techniques, including deploying malicious kernel drivers, and disabling Microsoft Defender or Bitlocker.

Administrators should be aware that additional actions are required for remediation of CVE-2023-24932 beyond simply applying the patches. The patch enables the configuration options necessary for protection, but administrators must apply changes to UEFI config after patching. Attack surface is not limited to physical assets, either; Windows assets running on some VMs, including Azure assets with Secure Boot enabled, also require these extra remediation steps for protection. Rapid7 has noted in the past that enabling Secure Boot is a foundational protection against driver-based attacks. Defenders ignore this vulnerability at their peril.

Zero-day vulnerability: RTF OLE RCE

The second of this month’s zero-day trio is an RCE vulnerability targeting Outlook users, as well as Windows Explorer. The vulnerability is in the proprietary Microsoft Object Linking and Embedding (OLE) layer, which allows embedding and linking to documents and other objects, and the Microsoft bulletin for CVE-2023-29336 suggests that the attack is likely conducted via a specially-crafted Rich Text File (RTF). All current versions of Windows are vulnerable, and viewing the malicious file via the Preview pane is one route to exploitation; however, successful exploitation requires an attacker to win a race condition and to otherwise prepare the target environment. This should significantly reduce the real-world impact of this vulnerability. Mitigations include disabling the Preview Pane, as well as configuring Outlook to read all emails in plain text mode. Microsoft is not aware of public disclosure, but has detected in-the-wild exploitation.

Zero-day vulnerability: Win32k LPE to SYSTEM

Rounding out this month’s trio of zero-day vulnerabilities is a Win32k Local Privilege Escalation (LPE) vulnerability. Successful exploitation will result in SYSTEM privileges. Win32k is a kernel-space driver responsible for aspects of the Windows GUI. As Rapid7 has noted in the past, the Win32k sub-system offers reliable attack surface that is not configuration-dependent. Although LPE vulnerabilities may seem less immediately concerning than a remote exploit, attackers frequently chain them together with other vulnerabilities to achieve full control over remote resources. Microsoft assesses attack complexity as low, and is aware of in-the-wild exploitation.

Critical RCE: NFS, MSQS, SharePoint Server, SSTP, LDAP

The remaining five RCE vulnerabilities this month include two with high CVSSv3 base scores of 9.8.

Although Microsoft is not aware of public disclosure or in-the-wild exploitation, Network File System (NFS) RCE vulnerability CVE-2023-24941 is a network attack with low complexity affecting Windows assets running NFS v4.1. As a mitigation prior to patching, Microsoft recommends disabling NFSv4.1 and then re-enabling it once the patch is applied, although this may impact functionality. OIder versions of NFS (NFSv3 and NFSv2) are not affected by this vulnerability. Microsoft warns that assets which haven’t been patched for over a year would be vulnerable to CVE-2022-26937 which is a Critical vulnerability in NFSV2.0 and NFSV3.0. In other words: applying today’s mitigation to an asset missing the May 2022 patches would effectively cause a downgrade attack.

CVE-2023-24943 describes a vulnerability in Windows Pragmatic General Multicast (PGM), and is a concern only for assets running Windows Message Queuing Service (MSQS) in a PGM environment. Microsoft recommends newer alternatives to PGM in the advisory. A further two critical RCE for MSQS were patched last month, and the continued flow of vulnerabilities suggests that MSQS will continue to be an area of interest for security researchers. Although MSQS is not installed by default, some software, including some versions of Microsoft Exchange Server, will helpfully enable it as part of their own installation routine.

Another candidate for inclusion in an exploit chain is SharePoint RCE CVE-2023-24955, which requires the attacker to authenticate as Site Owner to run code on the SharePoint Server host. Microsoft assesses this one as Exploitation More Likely, due in part to the low attack complexity. SharePoint Server 2016, 2019, and Subscription Edition are all vulnerable until patched. Anyone still running SharePoint Server 2013 should upgrade immediately, as May 2023 is the first Patch Tuesday after the end of ESU; absence of evidence of vulnerability is by no means evidence of absence.

Long-standing Patch Tuesday entrant Windows Secure Socket Tunneling Protocol (SSTP) provides CVE-2023-24903 this month, which is a critical RCE involving sending a specially crafted SSTP packet to an SSTP server and winning a race condition. This qualifies as high attack complexity, and Microsoft considers exploitation less likely.

The final Critical RCE this month is CVE-2023-28283, which is also a high-complexity network-vector attack involving a race condition. In this case, the attack is conducted via a specially-crafted set of LDAP calls.

Summary Charts

Patch Tuesday - May 2023
Several of the usual suspects are notable by their absence this month.
Patch Tuesday - May 2023
It’s hard to imagine Patch Tuesday without Remote Code Execution vulnerabilities.
Patch Tuesday - May 2023
It would be surprising if the CVSSv3 base score chart for almost any random sample of vulnerabilities didn’t look similar to this.
Patch Tuesday - May 2023
Perhaps a coincidence, but two of the three most prominent cells in this heatmap include zero-day vulnerabilities.

Summary Tables

Browser vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-29350 Microsoft Edge (Chromium-based) Elevation of Privilege Vulnerability No No 7.5
CVE-2023-29354 Microsoft Edge (Chromium-based) Security Feature Bypass Vulnerability No No 4.7
CVE-2023-2468 Chromium: CVE-2023-2468 Inappropriate implementation in PictureInPicture No No N/A
CVE-2023-2467 Chromium: CVE-2023-2467 Inappropriate implementation in Prompts No No N/A
CVE-2023-2466 Chromium: CVE-2023-2466 Inappropriate implementation in Prompts No No N/A
CVE-2023-2465 Chromium: CVE-2023-2465 Inappropriate implementation in CORS No No N/A
CVE-2023-2464 Chromium: CVE-2023-2464 Inappropriate implementation in PictureInPicture No No N/A
CVE-2023-2463 Chromium: CVE-2023-2463 Inappropriate implementation in Full Screen Mode No No N/A
CVE-2023-2462 Chromium: CVE-2023-2462 Inappropriate implementation in Prompts No No N/A
CVE-2023-2460 Chromium: CVE-2023-2460 Insufficient validation of untrusted input in Extensions No No N/A
CVE-2023-2459 Chromium: CVE-2023-2459 Inappropriate implementation in Prompts No No N/A

Developer Tools vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-29343 SysInternals Sysmon for Windows Elevation of Privilege Vulnerability No No 7.8
CVE-2023-29338 Visual Studio Code Information Disclosure Vulnerability No No 5

ESU vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-24904 Windows Installer Elevation of Privilege Vulnerability No No 7.1

ESU Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-24943 Windows Pragmatic General Multicast (PGM) Remote Code Execution Vulnerability No No 9.8
CVE-2023-24903 Windows Secure Socket Tunneling Protocol (SSTP) Remote Code Execution Vulnerability No No 8.1
CVE-2023-29325 Windows OLE Remote Code Execution Vulnerability No Yes 8.1
CVE-2023-28283 Windows Lightweight Directory Access Protocol (LDAP) Remote Code Execution Vulnerability No No 8.1
CVE-2023-24946 Windows Backup Service Elevation of Privilege Vulnerability No No 7.8
CVE-2023-29336 Win32k Elevation of Privilege Vulnerability Yes No 7.8
CVE-2023-24940 Windows Pragmatic General Multicast (PGM) Denial of Service Vulnerability No No 7.5
CVE-2023-24942 Remote Procedure Call Runtime Denial of Service Vulnerability No No 7.5
CVE-2023-24932 Secure Boot Security Feature Bypass Vulnerability Yes Yes 6.7
CVE-2023-29324 Windows MSHTML Platform Security Feature Bypass Vulnerability No No 6.5
CVE-2023-24900 Windows NTLM Security Support Provider Information Disclosure Vulnerability No No 5.9
CVE-2023-24945 Windows iSCSI Target Service Information Disclosure Vulnerability No No 5.5
CVE-2023-28251 Windows Driver Revocation List Security Feature Bypass Vulnerability No No 5.5

Microsoft Office vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-29344 Microsoft Office Remote Code Execution Vulnerability No No 7.8
CVE-2023-24953 Microsoft Excel Remote Code Execution Vulnerability No No 7.8
CVE-2023-29335 Microsoft Word Security Feature Bypass Vulnerability No No 7.5
CVE-2023-24955 Microsoft SharePoint Server Remote Code Execution Vulnerability No No 7.2
CVE-2023-24881 Microsoft Teams Information Disclosure Vulnerability No No 6.5
CVE-2023-24950 Microsoft SharePoint Server Spoofing Vulnerability No No 6.5
CVE-2023-24954 Microsoft SharePoint Server Information Disclosure Vulnerability No No 6.5
CVE-2023-29333 Microsoft Access Denial of Service Vulnerability No No 3.3

Windows vulnerabilities

CVE Title Exploited? Publicly disclosed? CVSSv3 base score
CVE-2023-24941 Windows Network File System Remote Code Execution Vulnerability No No 9.8
CVE-2023-24947 Windows Bluetooth Driver Remote Code Execution Vulnerability No No 8.8
CVE-2023-24949 Windows Kernel Elevation of Privilege Vulnerability No No 7.8
CVE-2023-24902 Win32k Elevation of Privilege Vulnerability No No 7.8
CVE-2023-24905 Remote Desktop Client Remote Code Execution Vulnerability No No 7.8
CVE-2023-29340 AV1 Video Extension Remote Code Execution Vulnerability No No 7.8
CVE-2023-29341 AV1 Video Extension Remote Code Execution Vulnerability No No 7.8
CVE-2023-24898 Windows SMB Denial of Service Vulnerability No No 7.5
CVE-2023-24901 Windows NFS Portmapper Information Disclosure Vulnerability No No 7.5
CVE-2023-24939 Server for NFS Denial of Service Vulnerability No No 7.5
CVE-2023-24948 Windows Bluetooth Driver Elevation of Privilege Vulnerability No No 7.4
CVE-2023-24899 Windows Graphics Component Elevation of Privilege Vulnerability No No 7
CVE-2023-24944 Windows Bluetooth Driver Information Disclosure Vulnerability No No 6.5
CVE-2023-28290 Microsoft Remote Desktop app for Windows Information Disclosure Vulnerability No No 5.3

The collective thoughts of the interwebz