Dutch political websites hit by cyber attacks as EU voting starts

Post Syndicated from João Tomé original https://blog.cloudflare.com/dutch-political-websites-hit-by-cyber-attacks-as-eu-voting-starts


The 2024 European Parliament election started in the Netherlands today, June 6, 2024, and will continue through June 9 in the other 26 countries that are part of the European Union. Cloudflare observed DDoS attacks targeting multiple election or politically-related Internet properties on election day in the Netherlands, as well as the preceding day.

These elections are highly anticipated. It’s also the first European election without the UK after Brexit.

According to news reports, several websites of political parties in the Netherlands suffered cyberattacks on Thursday, with a pro-Russian hacker group called HackNeT claiming responsibility.

On June 5 and 6, 2024, Cloudflare systems automatically detected and mitigated DDoS attacks that targeted at least three politically-related Dutch websites. Significant attack activity targeted two of them, and is described below.

A DDoS attack, short for Distributed Denial of Service attack, is a type of cyber attack that aims to take down or disrupt Internet services such as websites or mobile apps and make them unavailable for users. DDoS attacks are usually done by flooding the victim’s server with more traffic than it can handle. To learn more about DDoS attacks and other types of attacks, visit our Learning Center.

Attackers typically use DDoS attacks but also exploit other vulnerabilities and types of attacks simultaneously.

Daily DDoS mitigations on June 5 reached over 1 billion HTTP requests in the Netherlands, most of which targeted two election or political party websites. The attack continued on June 6. Attacks on one website peaked on June 5 at 14:00 UTC (16:00 local time) with 115 million requests per hour, with the attack lasting around four hours. Attacks on another politically-related website peaked at the same time at 65 million requests per hour.

On June 6, the first politically-related site with the highest peak on June 5 referenced above was attacked again for several hours. The main attack peak occurred at 11:00 UTC (13:00 local time), with 44 million requests per hour.

The main June 5 DDoS attack on one of the websites peaked at 14:13 UTC (16:13 local time), reaching 73,000 requests per second (rps) in an attack that lasted for a few hours. This attack is illustrated by the blue line in the graph below, which shows that it ramped slowly over the first half of the day, and then appeared to abruptly stop at 18:06. And on June 6, the main attack on the second website peaked at 11:01 UTC (13:01 local time) with 52,000 rps.

Geopolitical motivations

Elections, geopolitical changes, and disputes also impact the online world and cyberattacks. Our DDoS threat report for Q1 2024 gives a few recent examples. One notable case was the 466% surge in DDoS attacks on Sweden after its acceptance into the NATO alliance, mirroring the pattern observed during Finland’s NATO accession in 2023.

As we’ve seen in recent years, real-world conflicts, disputed and highly anticipated elections, and wars are always accompanied by cyberattacks. We reported (1, 2) on an increase in cyberattacks following the start of the Israel-Hamas war on October 7, 2023. We’ve put together a list of recommendations to optimize your defenses against DDoS attacks, and you can also follow our step-by-step wizards to secure your applications and prevent DDoS attacks.

If you want to follow more trends and insights about the Internet and elections in particular, you can check Cloudflare Radar, and more specifically our new 2024 Elections Insights report, that we’re keeping up to date as national elections take place throughout the year.

Simplify risk and compliance assessments with the new common control library in AWS Audit Manager

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/simplify-risk-and-compliance-assessments-with-the-new-common-control-library-in-aws-audit-manager/

With AWS Audit Manager, you can map your compliance requirements to AWS usage data and continually audit your AWS usage as part of your risk and compliance assessment. Today, Audit Manager introduces a common control library that provides common controls with predefined and pre-mapped AWS data sources.

The common control library is based on extensive mapping and reviews conducted by AWS certified auditors, verifying that the appropriate data sources are identified for evidence collection. Governance, Risk and Compliance (GRC) teams can use the common control library to save time time when mapping enterprise controls into Audit Manager for evidence collection, reducing their dependence on information technology (IT) teams.

Using the common control library, you can view the compliance requirements for multiple frameworks (such as PCI or HIPAA) associated with the same common control in one place, making it easier to understand your audit readiness across multiple frameworks simultaneously. In this way, you don’t need to implement different compliance standard requirements individually and then review the resulting data multiple times for different compliance regimes.

Additionally, by using controls from this library, you automatically inherit improvements as Audit Manager updates or adds new data sources, such as additional AWS CloudTrail events, AWS API calls, AWS Config rules, or maps additional compliance frameworks to common controls. This eliminates the efforts required by GRC and IT teams to constantly update and manage evidence sources and makes it easier to benefit from additional compliance frameworks that Audit Manager adds to its library.

Let’s see how this works in practice with an example.

Using AWS Audit Manager common control library
A common scenario for an airline is to implement a policy so that their customer payments, including in-flight meals and internet access, can only be taken via credit card. To implement this policy, the airline develops an enterprise control for IT operations that says that “customer transactions data is always available.” How can they monitor whether their applications on AWS meet this new control?

Acting as their compliance officer, I open the Audit Manager console and choose Control library from the navigation bar. The control library now includes the new Common category. Each common control maps to a group of core controls that collect evidence from AWS managed data sources and makes it easier to demonstrate compliance with a range of overlapping regulations and standards. I look through the common control library and search for “availability.” Here, I realize the airline’s expected requirements map to common control High availability architecture in the library.

Console screenshot.

I expand the High availability architecture common control to see the underlying core controls. There, I notice this control doesn’t adequately meet all the company’s needs because Amazon DynamoDB is not in this list. DynamoDB is a fully managed database, but given extensive usage of DynamoDB in their application architecture, they definitely want their DynamoDB tables to be available when their workload grows or shrinks. This might not be the case if they configured a fixed throughput for a DynamoDB table.

I look again through the common control library and search for “redundancy.” I expand the Fault tolerance and redundancy common control to see how it maps to core controls. There, I see the Enable Auto Scaling for Amazon DynamoDB tables core control. This core control is relevant for the architecture that the airline has implemented but the whole common control is not needed.

Console screenshot.

Additionally, common control High availability architecture already includes a couple of core controls that check that Multi-AZ replication on Amazon Relational Database Service (RDS) is enabled, but these core controls rely on an AWS Config rule. This rule doesn’t work for this use case because the airline does not use AWS Config. One of these two core controls also uses a CloudTrail event, but that event does not cover all scenarios.

Console screenshot.

As the compliance officer, I would like to collect the actual resource configuration. To collect this evidence, I briefly consult with an IT partner and create a custom control using a Customer managed source. I select the api-rds_describedbinstances API call and set a weekly collection frequency to optimize costs.

Console screenshot.

Implementing the custom control can be handled by the compliance team with minimal interaction needed from the IT team. If the compliance team has to reduce their reliance on IT, they can implement the entire second common control (Fault tolerance and redundancy) instead of only selecting the core control related to DynamoDB. It might be more than what they need based on their architecture, but the acceleration of velocity and reduction of time and effort for both the compliance and IT teams is often a bigger benefit than optimizing the controls in place.

I now choose Framework library in the navigation pane and create a custom framework that includes these controls. Then, I choose Assessments in the navigation pane and create an assessment that includes the custom framework. After I create the assessment, Audit Manager starts collecting evidence about the selected AWS accounts and their AWS usage.

By following these steps, a compliance team can precisely report on the enterprise control “customer transactions data is always available” using an implementation in line with their system design and their existing AWS services.

Things to know
The common control library is available today in all AWS Regions where AWS Audit Manager is offered. There is no additional cost for using the common control library. For more information, see AWS Audit Manager pricing.

This new capability streamlines the compliance and risk assessment process, reducing the workload for GRC teams and simplifying the way they can map enterprise controls into Audit Manager for evidence collection. To learn more, see the AWS Audit Manager User Guide.

— Danilo

Secure file sharing solutions in AWS: A security and cost analysis guide, Part 1

Post Syndicated from Sumit Bhati original https://aws.amazon.com/blogs/security/how-to-securely-transfer-files-with-presigned-urls/

July 28, 2025: This post has been updated and expanded into a comprehensive two-part series covering multiple AWS file sharing solutions. This new series provides in-depth analysis of security and cost considerations to help you make informed decisions based on your requirements.

Note: This is Part 1 of a two-part post. You can read Part 2 here.

Sharing files with an outside entity—to share data between business partners or facilitate customer access to files—is a common use case for Amazon Web Services (AWS) customers. Organizations must balance security, cost, and usability. In a business-to-business data sharing scenario, these challenges become even more complex because human interaction is often minimal or absent, requiring robust automated solutions. Many AWS services offer multiple options for granting access. The one that’s best for your use case depends on multiple factors.

This post helps you decide which AWS services to use to implement a file sharing approach that suits your business needs. We focus on security controls and cost implications, describe some of the trade-offs, and highlight key differences to help you make an informed decision based on your specific requirements. We go through each option, highlighting their strengths and limitations, and provide guidance on choosing the right solution for your use case.

Understand your needs first

The first step in designing an AWS file sharing solution is to develop a clear understanding of your requirements and constraints. Because there are several possible design patterns and a number of different AWS services to consider, you need to start by identifying and prioritizing the features that you need. Gather the following information to guide your approach:

Access patterns and scale

When planning for access patterns and scale, there are a few key factors to keep in mind. First, consider how files are shared—machine-to-machine, human-to-machine, or human-to-human—because that impacts security and performance. Then, think about transfer frequency—are files exchanged only once a day, or are thousands moving every hour? If download control matters, setting limits on how often a file can be accessed might be necessary. File sizes also play a role, from typical everyday transfers to the largest files you need to support. Finally, total data volume shapes how much information you’ll be transferring on a regular basis.

Technical requirements

Your choice of solution will be influenced by technical constraints and capabilities. Protocol requirements often drive initial decisions, such as whether you need SFTP, FTPS, or HTTPS access. Consider existing systems that must interface with your solution and how they’ll connect. Performance considerations span several dimensions: acceptable latency for file transfers, geographic distribution of your users, bandwidth requirements, and whether you need built-in retry mechanisms for failed transfers. Additionally, think about how many simultaneous transfers your solution needs to support.

Security and compliance

Security and compliance requirements will definitely influence your file sharing strategy. Consider who controls encryption keys—whether managed by AWS or your organization—and what key rotation policies are needed. Authentication needs often vary—you might be authenticating individual users, specific systems, or entire business entities, using methods ranging from passwords to API keys, multi-factor authentication, or certificates. Your audit requirements will influence your choices in logging and monitoring capabilities. You might have geographic considerations like data sovereignty requirements, storage location restrictions, and access controls that consider the recipient’s location. If your data is subject to a law, like GDPR in Europe or HIPAA in the United States, or if your data is regulated by a standard like the Payment Card Industry’s Data Security Standard (PCI-DSS), you will need to consult with your own legal and compliance advisors to see what is required. When assessing risk tolerance, consider the security triad of confidentiality, integrity, and availability—some use cases might tolerate brief periods of unavailability but cannot risk data exposure, while others prioritize continuous availability.

Operational requirements

Day-to-day operations bring their own set of considerations. File retention policies determine how long data needs to be kept, while auto-deletion capabilities might be necessary for managing storage and compliance. Consider what kind of reporting and monitoring of file transfer activities you need. Do you need monthly reports, daily reports or perhaps detailed real-time tracking of transfer activities. By adding handling and notification systems, you can help make sure that problems are caught and addressed promptly. Disaster recovery requirements, expressed through recovery point objectives (RPO) and recovery time objectives (RTO), help determine the resilience needed in your solution.

Business constraints

Your solution must operate within your business constraints, such as budget limitations, technical limitations, timelines, available expertise, and service level agreements (SLAs). Budget limitations include initial implementation costs and ongoing operational expenses. Consider other parties’ technical limitations—they might use specific protocols such as SFTP, require mobile device compatibility, or operate older systems that have limited cryptographic capabilities. Implementation timelines influence choices between managed services that can be deployed quickly and custom solutions that require more time and expertise. The expertise available for solution maintenance is also a consideration. SLAs for file transfers might specify availability and performance requirements that you’re obligated to meet. To meet these constraints, you must estimate how much your file sharing needs will grow over time and determine if you need a regional or a global solution.

By carefully considering these aspects, you’ll be better prepared to evaluate different AWS file sharing solutions and select the one that best fits your use case. Understanding your requirements for uploads and downloads will help determine if your use case can be supported through a single AWS service or needs a combination of services.

Solutions

Let’s start by looking at the various file sharing mechanisms that AWS supports. The following table identifies the key AWS services needed for each solution, describes the security and cost implications of the solutions, and describes their complexity and protocol support capabilities. The following table shows the solutions described in this post.

Solution AWS services Security features Cost* Region control
AWS Transfer Family Transfer Family, Amazon S3, API Gateway, and Lambda Managed security, encryption in transit and at rest, IAM integration, and custom authentication $0.30 per hour per protocol, data transfer fees, and storage costs Can deploy to specific AWS Regions, can only transfer files to and from S3 buckets in the same Region
Transfer Family web apps Transfer Family, S3, and CloudFront Browser-based access, IAM Identity Center integration, and S3 Access Grants Pay-per-file operation, CloudFront costs, and storage costs Uses CloudFront (global) for web access, but backend components can be Region-specific
Amazon S3 pre-signed URLs S3 Time-limited URLs, IAM controls for URL generation, and HTTPS S3 request and data transfer fees Can be restricted to specific Regions
Serverless application with Amazon S3 presigned URLs S3, AWS Lambda, and API Gateway Time-limited URLs, HTTPS, IAM controls, customizable authentication Pay per request and minimal infrastructure cost Components can be Region-specific

The following table shows the solutions described in Part 2.

Solution AWS services Security features Cost* Region control
CloudFront signed URLs CloudFront, Amazon S3, and Lambda Optional edge security using AWS Lambda@Edge, AWS WAF integration, SSL/TLS, geo restrictions, and AWS Shield Standard (included automatically) Content delivery network (CDN) costs, request pricing, and data transfer fees Global service by design; origin can be AWS Region-specific
Amazon VPC endpoint service PrivateLink, VPC, and NLB Complete network isolation, private connectivity, and multi-layer security Endpoint hourly charges, NLB costs, and data processing fees Service endpoints are strictly Region-specific; must create endpoints in each Region where access is needed
S3 Access Points S3, IAM, VPC (for VPC-specific access points)
  • Dedicated IAM policies per access point
  • VPC-only access restrictions available
  • Works with bucket policies for layered security
  • Supports AWS PrivateLink for private network access
  • Compatible with S3 Block Public Access settings
  • No additional charge for S3 Access Points
  • Standard S3 request pricing applies
  • Data transfer fees apply based on standard S3 rates
  • VPC endpoint charges apply when using VPC endpoints with access points
  • Access points are Region-specific
  • Each access point is created in the same Region as its S3 bucket
  • Cross-Region access requires separate access points in each Region
  • VPC-specific access points are limited to the VPC’s Region

* Pricing information provided is based on AWS service rates at the time of publication and is intended as an estimation only. Additional costs may be incurred depending on your specific implementation and usage patterns. For the most current and accurate pricing details, please consult the official AWS pricing pages for each service mentioned.

Let’s examine the solutions in detail.

AWS Transfer Family

AWS Transfer Family is a managed file transfer service for SFTP, FTPS, and AS2 protocols. It integrates directly with Amazon Simple Storage Service (Amazon S3) for storage and supports custom identity providers for authentication through Amazon API Gateway and AWS Lambda.

As shown in Figure 1, when a user initiates a file transfer, Transfer Family authenticates them through the configured identity provider using API Gateway and Lambda. After authentication succeeds, the service maps the user to an AWS Identity and Access Management (IAM) role that defines their S3 bucket access permissions. The service encrypts data in transit using TLS 1.2 and data at rest using S3 server-side encryption.

Figure 1: AWS Transfer Family architecture

Figure 1: AWS Transfer Family architecture

Transfer Family automatically handles scaling from zero to thousands of concurrent users, manages high availability across Availability Zones, and minimizes infrastructure management. It records detailed metrics and logs in Amazon CloudWatch for monitoring and auditing, supporting compliance requirements with activity tracking.

It’s important to note that Transfer Family also offers service-managed authentication. This simpler setup stores user credentials (passwords or SSH keys) directly in Transfer Family, minimizing the need for external identity providers. Service-managed authentication is best suited if you have a small number of users or no existing identity management system, or when you want to have a disconnected identity system and don’t want to give external partners an account in your identity provider system.

Pros

One of the biggest advantages of Transfer Family is how it provides the reliability and scalability of Amazon S3 for storing your data, while keeping that data available to existing client applications and workflows. The service integrates with existing authentication systems through custom identity providers, while maintaining security through IAM policies. Its auto-scaling capabilities handle variable workloads, from occasional transfers to high-volume scenarios.

Transfer Family also offers detailed CloudWatch logging and audit trails for file transfer activities, which should be sufficient for most logging and audit needs. It encrypts data in transit using TLS 1.2 and at rest using Amazon S3 server-side encryption. You can implement fine-grained access controls through IAM roles and integrate with AWS Organizations for multi-account management. The service supports VPC endpoints for secure internal access and custom domain names for branded endpoints.

Because data is stored in S3, some of your requirements will be fulfilled by configuring S3, not the Transfer Family services. Data retention (for example, avoiding deletion and scheduling deletion) is achieved through S3 Object Lock and S3 Lifecycle Events.

Cons

The pricing structure of Transfer Family includes $0.30 per hour for each protocol you enable and data transfer fees based on data volume. There can be additional charges for custom domain names. If you use VPC endpoints for secure internal access to Amazon S3, there will also be VPC data charges. If you have high-volume transfers or multiple endpoints across AWS Regions, you will face increased costs. Because the data ultimately lives in S3; S3 storage and request pricing applies as well.

Custom identity provider implementations (such as SAML or OAuth) add latency to authentication processes, affecting transfer initiation times. This authentication process requires additional configuration and introduces extra steps and latency during transfer initiation compared to service-managed authentication.

The Regional nature of Transfer Family means you must choose between deploying in a single Region (simpler management but potential latency for global users) or multiple Regions (better performance but higher costs at $0.30 per protocol per hour per Region). Multi-Region can serve as a disaster recovery strategy or when Regional data isolation is needed.

Transfer Family web apps

Transfer Family web apps provide browser-based access to Amazon S3, enabling users to upload and download files through a web interface. With the web apps, you can create a branded, secure, and highly available portal for your users to browse, upload, and download data in S3. Web apps are built using Storage Browser for S3 and offer the same user functionalities in a fully managed offering without having to write code or host your own application.

When a user accesses the web application, authentication occurs through AWS IAM Identity Center, and S3 Access Grants determine their permissions to specific S3 buckets or prefixes. The access grant permissions can be either read-only or read and write. After authentication succeeds, users can upload or download files directly through the web interface. The service uses Amazon CloudFront for content delivery and implements SSL/TLS encryption for data transfers, while S3 provides server-side encryption for data at rest. Figure 2 shows a simplified Transfer Family web app architecture.

Figure 2: Simplified Transfer Family web app architecture

Figure 2: Simplified Transfer Family web app architecture

The web application automatically scales to accommodate varying numbers of users and provides high availability through the CloudFront global edge network. It minimizes the need for custom web application development and provides logging through AWS CloudTrail and CloudWatch. You can customize the user experience by implementing custom domains through CloudFront distributions.

Transfer Family web apps support multiple authentication methods, with IAM Identity Center being one of the primary options. While Identity Center provides simplified user management and integration with existing identity providers. It also provides useful mechanisms such as multi-factor authentication (MFA), strong password policies, and resetting lost passwords. It’s not the only authentication method available; you can also use custom identity providers for authentication, providing flexibility in how you manage user access to the web application.

Pros

Transfer Family web apps minimize the need to build and maintain custom web interfaces for Amazon S3 file sharing. It provides seamless integration with IAM Identity Center for user management and authentication, enabling you to use existing identity providers. The service offers fine-grained access control through S3 Access Grants, allowing precise permission management at the bucket and prefix level. Its integration with CloudFront provides global availability and enhanced performance, while CloudTrail logging offers audit capabilities.

The service provides robust security features including SSL/TLS encryption, CORS policy management, and optional integration with AWS WAF for protection against bots, web scrapers, DDoS events, and more. You can implement custom domains for branded experiences and use CloudFront security features including DDoS protection using AWS Shield. The web interface offers intuitive file management capabilities without requiring client software or that users have technical expertise.

Cons

Transfer Family web apps require using IAM Identity Center, which might require additional setup and configuration if you’re not currently using this service. The web interface currently requires the Identity Center identities to live in the same AWS account as the S3 buckets. That might create design challenges if you want to keep identities in one AWS account and data storage in another. Implementation requires careful cross-origin resource sharing (CORS) configuration for each S3 bucket.

The service incurs costs for both Transfer Family and associated services, including CloudFront distribution and data transfer fees. Custom domain implementation requires additional configuration and SSL certificate management through AWS Certificate Manager (ACM). The web interface is well suited for humans to upload or download, but it’s not as good for automated workflows that transfer files from machine to machine. You must carefully manage user assignments and access grants to maintain security, adding administrative overhead.

S3 pre-signed URLs

Amazon S3 pre-signed URLs enable secure, time-limited access to objects in S3 without requiring the file recipient to have an identity in your identity systems. The URLs are generated using the AWS SDK or AWS Command Line Interface (AWS CLI), granting specific permissions (GET, PUT) that are valid for up to seven days. When accessing files, S3 validates the cryptographically signed parameters in these URLs before permitting access to objects. This provides a direct method for secure file sharing through HTTPS endpoints.

The solution requires only an S3 bucket and appropriate IAM permissions for URL generation. S3 handles the authentication of the pre-signed URL parameters and manages access to objects. File transfers occur directly between users and S3 through HTTPS endpoints, with the pre-signed URL controlling the access patterns.

Amazon S3 provides security features including server-side encryption, access logging, and CloudTrail integration. The security of pre-signed URLs is primarily managed through expiration times and specific operation permissions defined during URL generation.

Pros

Amazon S3 pre-signed URLs follow a straightforward pay-per-use pricing model, charging only for S3 storage, requests, and data transfers. For example, if you create pre-signed URLs but the object isn’t actually downloaded, you pay storage costs as usual, but you don’t pay transfer costs. The solution uses the native scalability of S3 to handle varying numbers of concurrent users without additional infrastructure. you can implement granular access controls through URL expiration times and specific operation permissions (GET, PUT, DELETE).

Access is controlled through URL expiration enforcement. Amazon S3 server access logging and CloudTrail integration enable audit capabilities. The solution’s simplicity makes it ideal for basic file sharing needs while maintaining security and scalability.

Cons

A pre-signed URL can be used by anyone who has access to the URL. That’s the goal of this design: You don’t need to have an identity for the user. Pre-signed URLs can be reused an unlimited number of times until they expire. To improve security, short expiration times can limit the potential for URL re-use. Shorter expiration times, however, require the recipient to download the file soon after the URL is created.

When implementing this solution, you should establish processes for secure URL generation and distribution. Set your URL expiration times based on realistic expectations about how quickly your recipients will download the files. A web or mobile app where the user selects a link to download something (such as a document, an image, a data file) and they expect the download to start immediately is a good candidate for this design.

The solution works with files up to 5 GB for single operations. To share a file larger than 5 GB, you must split the file into multiple parts, issue multiple pre-signed URLs, and then the recipient must download all the parts and join the parts together correctly. This isn’t a good solution for sharing large files. Also, distributing large files as a single download can be difficult if the recipient doesn’t have good connectivity. Amazon S3 can start an object download from the middle of the object, but selecting a pre-signed URL cannot. So, if the recipient transfers 1 GB out of a 2 GB download, and then their connection is disrupted, they cannot pick up where they left off. They will restart from the beginning, which is undesirable. Overall, this design is unsuitable for transmitting large files over unreliable internet connections.

You should enable appropriate monitoring through Amazon S3 access logs and CloudTrail to track usage patterns and meet security compliance.

This solution is particularly effective if you’re seeking straightforward, secure file sharing capabilities where the files are small enough to download in one request, and where you have a secure mechanism to share the download URLs.

Serverless web application with S3 presigned URLs

Amazon S3 presigned URLs combined with a custom web application enable secure, time-limited access to S3 objects. The application generates URLs that grant specific S3 permissions (GET, PUT) for between one minute and seven days. When requesting file access, the application authenticates users and generates presigned URLs using the AWS SDK with defined permissions and expiration times.

The web application uses API Gateway and Lambda functions for authentication and URL generation. Amazon S3 validates the cryptographically signed parameters in these URLs before permitting access to objects. File transfers occur directly between users and S3 through HTTPS endpoints, with the application controlling the access patterns. The architecture is shown in Figure 3.

Figure 3: Amazon S3 pre-signed URLs architecture

Figure 3: Amazon S3 pre-signed URLs architecture

The web application can implement security controls including request logging, rate limiting (requests per second), and authentication workflows. CloudWatch logs record API access patterns and Lambda execution metrics, while Amazon S3 access logging records object-level operations.

Pros

Amazon S3 presigned URLs follow a pay-per-use pricing model. This solution charges only for API Gateway requests, Lambda executions, and S3 operations performed. The serverless architecture scales automatically from zero to thousands of concurrent users without infrastructure management. You can implement custom security controls and business logic for specific access requirements through API Gateway authorizers (using custom identity solutions or Amazon Cognito) and Lambda functions.

The solution enforces security through URL expiration (maximum seven days), IAM policies restricting URL generation permissions, and HTTPS encryption for data transfers. Custom authentication workflows integrate with existing identity providers (SAML, OIDC). Additional security features include IP-based restrictions, required request headers, and request validation through AWS WAF. This solution would be good, for example, if you have a variety of files or a variety of buckets and you’re trying to build a unified front-end where people can download various files without knowing which bucket the files are stored in or what URL expiration time is appropriate. You can configure the frontend to look at tags on objects, tags on buckets, object names, or another attribute that fits your use case, and then choose a URL expiration time based on that attribute. For example, objects from buckets tagged Data Classification: Restricted might expire after 1 minute, whereas objects from buckets tagged Data Classification: Public might be valid for 7 days.

Cons

Building a custom web application requires developing and maintaining the code for URL generation, authentication, and error handling logic. The application must track URL expiration times and implement mechanisms that permit retries for failed transfers. Monitoring systems must track URL usage, detect abuse patterns, and send alerts for security violations through CloudWatch metrics and logs.

One limitation of this solution is the 10 MB size limit imposed by API Gateway. This affects how your application handles file uploads and downloads. For uploads, files under 10 MB can be uploaded directly through API Gateway. Larger files require implementing multipart uploads, where the client splits the file into chunks and sends each chunk separately. For downloads, files under 10 MB can be downloaded directly through API Gateway but for larger files, your application should generate a pre-signed URL for direct Amazon S3 access, bypassing API Gateway.

URL generation errors or misconfigured IAM permissions can expose objects to unauthorized access. The HTTPS-only protocol limits integration with SFTP and FTPS clients. Files larger than 5 GB require multipart upload implementation, and network interruptions need custom resume logic. This design will incur some extra charges if the number of file transfers are the millions. Lambda functions cost $0.20 per million requests, and API Gateway costs $1.00 per million requests. Analyze your expected access patterns to determine whether these extra costs will be significant and if they’re worth the additional flexibility of custom transfer logic.

Decision matrix: When to use each solution

The following table summarizes the characteristics of the solutions presented in the two parts of this post. See Part 2 for full descriptions of the solutions not covered in Part 1.

Characteristics Transfer Family Transfer Family web app S3 pre-signed URLs (Direct) Serverless web application with S3 pre-signed URL CloudFront signed URLs (Part 2) VPC endpoint service (Part 2) S3 Object Lambda (Part 2)
Protocol support SFTP, FTPS, and AS2 HTTPS (web-based) HTTPS HTTPS HTTPS with CDN A TCP-based protocol HTTPS
Global distribution Global endpoint support CloudFront integration Global S3 access Global S3 access Global edge network acceleration Direct AWS backbone access Global S3 access with Regional endpoints
Pricing model Hourly service rate and usage Pay per file operation Pay-per-request Pay-per-request and application costs Pay-per-request with caching savings Hourly endpoint rate and usage No additional charge for access points; standard S3 request pricing applies
Content processing Direct S3 integration Built-in web interface Direct S3 access Custom app processing Edge-based file processing Access files through private network Direct S3 access with customized permissions per access point
Authentication options Custom IdP and service-managed IAM Identity Center IAM Custom authentication possible IAM, custom authentication, and edge validation VPC security controls and custom authentication IAM policies, VPC endpoint policies, resource-based policies
Upload capabilities Unlimited file size Web interface upload Up to 5 GB direct and multipart for larger Up to 10 MB using API Gateway Optimized for global ingestion Unlimited file size over private connection Same as standard S3
Download capabilities Unlimited file size Browser-based downloads Up to 5 GB using a single URL Up to 10 MB using API Gateway Accelerated downloads using global edge locations Unlimited file size over private connection Same as standard S3 with customized access controls
Example use cases
  • Enterprise file transfer systems
  • B2B data exchange
  • Compliance-focused transfers
  • Browser-based file sharing
  • Internal document management
  • Client portals
  • Simple direct S3 access
  • Temporary file sharing
  • Mobile app backend
  • Custom file sharing systems
  • Integrated web applications
  • Enhanced S3 access control
  • Global content delivery
  • Media distribution
  • Web application assets
  • Private network transfers
  • Custom protocol support
  • Secure enterprise data exchange
  • Simplified data access management at scale
  • Multi-application access to shared datasets
  • VPC-restricted data access

The following list gives you a quick overview of the strengths of each solution presented in the two parts of this post.

  • Transfer Family is the optimal choice for organizations that require legacy file transfer protocols such as SFTP, FTPS, or AS2 protocols, and you must integrate with existing authentication systems. It’s ideal for scenarios with strict compliance and audit requirements, where operational overhead needs to be minimized. While the solution comes with higher costs because of its managed service nature, it’s often the lowest-friction option to support existing enterprise use cases that depend on these protocols.
  • Transfer Family web apps suit organizations that need browser-based file sharing without custom development. They integrate with IAM Identity Center for user authentication and uses Amazon S3 Access Grants for permission management. The solution works well for internal document sharing, client portals, and scenarios requiring a branded web interface. While limited to web browser access, they provide built-in features like MFA and password management without infrastructure maintenance.
  • Amazon S3 pre-signed URLs excel in scenarios where simplicity, cost-effectiveness, and temporary access are key requirements. This solution is ideal if you’re seeking a straightforward file sharing mechanism without the need for custom application development or additional infrastructure. This approach shines in environments that require a quick implementation of secure file sharing and cost-effective solutions with minimal overhead.
  • Serverless web application with S3 presigned URLs best serves scenarios where cost optimization is paramount and the HTTPS protocol meets your requirements. This solution shines in environments that need simple, direct file sharing capabilities with quick implementation timelines. It’s particularly effective for moderate usage patterns where serverless architecture can provide cost benefits. The solution’s simplicity makes it ideal for web applications and scenarios where complex file transfer protocols aren’t necessary, though careful consideration must be given to its 10 MB file size limitation for single operations using API Gateway.

In Part 2:

  • CloudFront signed URLs excel in situations that demand global content distribution with high performance requirements. This solution is the clear choice when your architecture needs built-in DDoS protection and performance optimization through caching. It’s particularly valuable when content delivery speed is crucial and you require security at edge locations. The solution’s global reach and caching capabilities make it cost-effective for large-scale content distribution, though it’s primarily optimized for download scenarios rather than uploads.
  • Amazon VPC endpoint service is the preferred choice if you require complete network isolation and maximum security. This solution is ideal when you need support for custom protocols while maintaining private network connectivity. It’s particularly suitable for scenarios with extremely high security requirements and when you have the necessary resources to managed networking configurations. While this solution requires significant expertise and investment, it provides the highest level of security and control for sensitive data transfers.
  • S3 Access Points are best suited for scenarios that require simplified data access management at scale. This solution excels when you need to provide different access patterns to the same underlying data for multiple applications or user groups. It’s ideal if you prefer a structured approach to permissions and need network-level access controls. While primarily focused on simplifying complex access scenarios without modifying bucket policies, it offers unique capabilities for VPC-restricted access and granular permissions management, though subject to certain service limits and configuration requirements.

Conclusion

In this first part of a two-part post, you’ve learned about multiple solutions for secure file sharing using AWS services and the pros and cons of each. You can find additional options in Part 2. The optimal solution depends on your specific organizational requirements, technical capabilities, and budget constraints. You don’t have to choose just one option, you can implement multiple solutions to address different use cases, creating a file sharing strategy that balances security, cost, and operational efficiency.

Additional resources:

If you have feedback about this post, submit comments in the Comments section below.

Swapnil Singh

Swapnil Singh

Swapnil is a Senior Solutions Architect for AWS World Wide Public Sector. As a Product Acceleration Solutions Architect at AWS, she currently works with GovTech customers to ideate, design, validate, and launch products using cloud-native technologies and modern development practices.

Sumit Bhati

Sumit Bhati

Sumit is a Senior Customer Solutions Manager at AWS, specializing in expediting the cloud journey for enterprise customers. Sumit is dedicated to assisting customers through every phase of their cloud adoption, from accelerating migrations to modernizing workloads and facilitating the integration of innovative practices.

Сънуват ли джихадистите девици?

Post Syndicated from Атанас Шиников original https://www.toest.bg/sanuvat-li-dzhihadistite-devici/

Сънуват ли джихадистите девици?

Тези дни зачитам едно от любимите ми възрожденски четива, гротесково-ироничната „Видрица“ на Минчо Кънчев, българския революционен юначен поп. Предполагам, че на турски може да го наречем нещо като папаз бабаит. Записките му от Диарбекир винаги са ми напомняли на нашенска версия на големите западни ориенталисти, пътуващи на изток. Веднъж, разказва той, заточен по Анадола,

не знам тези турски ходжи, молли, дервиши какъв сън сънували,

ама отишли в гробищата да питат техния войнстващ светец Гази Хамза ба̀ба какво да правят. Та явно починалият светец, от чиятo сабя капела кръв, им казал, че Пророкът бил сърдит, задето гяурите не били избити. 

И самият Минчо Кънчев, при всичките му бабаитлъци, обрисува себе си като сънуващ човек. Че и приписва сънища на други участници. Каймакаминът сънува дякона Паисий (онзи, грешния предател, да не се бърка с автора на „История славянобългарска“) като обращенец в правата вяра. Самият дякон Паисий се сънува като повишен в чин владика. Накрая тия сънища се схождат трагично в убийството на дякона, извършено от Димитър Общи. 

От камбанарията на скептика подозирам, че разказите носят белезите на типична реторическа хватка. Без значение от коя страна на религиозната бариера стоиш, като искаш да вмъкнеш нещо скандално или да си измислиш оправдание, го поднасяш, облечено във формулата „сънувах“. Едновременно е авторитетно и недоказуемо. Та затова и „какъв сън сънували“ може да значи просто „какво са си наумили“. Сънищата на каймакамина и дякона Паисий са потвърждение на една заслужена смъртна казън. Сънищата на самия поп Минчо свидетелстват за неговата героично-мъченическа съдба. 

Но всъщност „ходжите, моллите и дервишите“ сънуват, и то много.

Далеч преди поп Минчо Кънчев. Защото са част от огромна и непрекъсната традиция на разбиране за ролята на сънищата сред мюсюлманите, която започва с фигурата на самия Пророк. Като че ли в пясъците на Арабия от седмото столетие след Христа традициите на сънуване, тълкуване и съответно впрягане на тълкуванията в публична употреба придобиват нова съдържателност.

Коранът, подобно на Библията (че кой от нас не е чел за сънищата на Йосиф, за „седемте добри и седемте лоши години“, за „колоса на глинени крака“ от съня на пророк Даниил или за съня на жената на Пилат от Евангелието?), дава изобилен материал относно виденията по време на сън. Даже използва поне четири термина за това. Първият от тях е буквално „видение“ (ру’йа); на второ място срещаме думата манам (просто „сън“ в смисъл на „състояние на заспалост“ и „видение по време на сън“), а инцидентно срещаме и преносното бушра, означаващо първоначално „добра новина“, интерпретирано веднъж като видение в сън. И накрая, за „лоши сънища“ Коранът използва най-популярната днес в арабския език дума за сън – хулм.

Доколкото самият Пророк Мохамед твърди, че „стъпва в обувките“ на пророческата мисия на библейските пророци, често пъти разказите за сънища в Писанието на мюсюлманите са свързани с персонажи, заети от библейския разказ, като Ибрахим (библ. Авраам) или Юсуф (библ. Йосиф). А разказите за Мохамед, т.нар. Сунна – онова, което мюсюлманите, особено в суннизма, твърдят, че той е казал, направил, одобрил или порицал, – са истински трамплин към развиване на детайлна традиция за разбиране на тази мъглява, но неотменна част от човешкия живот. 

Да надзърнем например в т.нар. Достоверен (Сахих) сборник на Ал-Бухари от IX век. В него имаме огромен раздел под надслов „Тълкуванието на сънищата“, който съдържа няколко десетки предания (хадиси). Появяват се нови категории сънища (например „добри сънища“, мубашшират), обяснява се какво място заемат в живота на правоверните. Добрият, верен сън е от Аллах, а лошият сън е само от дявола. „Добрите сънища на праведните са една четирийсет и шеста част от пророчеството“; ако човек види сън, който му се харесва, значи той е от Аллах. И не е възможно човек да види самия Пророк в съня си и този сън да е от дявола. 

Сунната задава и рамката на нещо друго – появяват се конкретни напътствия как да се разбира това или онова, видяно по време на сън. Ако вярващият сънува дявола, трябва да плюе наляво и да помоли Аллах за убежище. Същото предписание следва да се изпълни при всеки лош сън. Дрехата, носена от съратниците на Мухаммад, в сън представя ислямската религия; зелената градина също представя правоверието; черната разчорлена жена е представена като чумата. 

Докато спях, ми връчиха чаша с мляко, което започнах да пия, докато млякото не започна да струи от ноктите ми,

казва Пророкът. После дал остатъка на неговия съратник и бъдещ халиф Умар. Запитан как тълкува това, Мохамед отговорил, че млякото е символ на религиозното познание. 

Конкретни събития от времето на Пророка се обясняват чрез сън.

Видях се насън да размахвам меч, който се счупи по средата. После го размахах пак и той се възстанови по-хубав от преди.

В първия случай, пояснява той, счупеният меч символизира падналите мюсюлмани в претърпяното поражение в битката при Ухуд през 625 г., докато във втория очевидно става въпрос за последвалата победа, дарена на правоверните. 

Въз основа на зададените рамки в първоначалната мюсюлманска общност се развива и огромна традиция за употреба на сънищата с най-разнообразна цел. В крайна сметка, ако нещо се появява с авторитета на самия Бог в Корана и после е утвърдено чрез думите и делата на Пророка, не е ли редно да породи след себе си традиция? Обикновено така работи религиозното мислене. Може да го премисляш, може да го променяш, може да го реинтерпретираш, но трудно може да го изхвърлиш и да се откажеш от него. Поне трябва да се потрудиш върху аргументацията защо го правиш. 

С времето се появяват такива емблематични съчинения, като „Тълкувания на сънищата“ на Ибн Сирин от VIII век, върху когото стъпват всички по-нататъшни усилия. Умението за разбиране на сънищата и тяхната връзка с реалността е засвидетелствана и от факта, че през XII век Ал-Халлал съставя биографичен речник на тълкувателите на сънища, а философът Ибн Сина или богословът Ал-Газзали от епохата на късните Абасиди посвещават значителни усилия на ситуирането на сънищата в живота на правоверните. Дори Ибн Халдун от XV столетие сл.Хр., иначе познат като трезв, повратлив дипломат и историк (и досаден кошмар за всеки студент в българската арабистика), отделя част от прословутото си „Встъпление“ на наречената от него „наука за тълкуване на сънищата“. Съновникът на богослова Ан-Набулси, живял на границата между XVII и XVIII век, надгражда този на Ибн Сирин и до днес се преиздава с голям успех. Сега по-ясно може да си представим защо мюсюлманите сънуват. Сънуват от времето на Мохамед до днес. 

Сънуват всякакви неща. Или поне казват, че ги сънуват.

През август 1068 г. например богословът Ибн ал-Банна в Багдад записва в личния си дневник, че един човек го посещава с молба за разтълкуването на „страшен, велик сън“. В него се виждал слон с две крила, телосложение и ръст на човек, с голяма мъжественост (фалос), който се спуска над река Тигър. Ибн ал-Банна записва веднага значението на съня. Няма начин слонът да не е султанът Алп Арслан, а двете му крила – неговите двама сина. Голямата му мъжественост, която впрочем в съня спадала и се връщала към обичайния си размер, била неговата огромна репутация и авторитет. Малко след това в дневника са отбелязани и други сънища в около двайсет различни разказа – например зелени скакалци с бисери в устата. Пак Ибн Банна научава в сън, че шейх от общността е преследван от еретици.

Сънуват ли джихадистите девици?
„Спящи приятели“, илюстрация към ръкопис Arabe 3929 на „Макамите“ от Ал-Харири от XIII век, съхраняван във Френската национална библиотека

Ако пък зачетем Ан-Набулси, може да видим и че в пространството на сънищата се появяват неща, които наяве не са много легитимни. Но са изпълнени със смисъл.

Ето например как се сънува прасето в култура, която му отрежда презряно ъгълче на творението и го възприема като греховно (харам).

Прасето, казва богословът, може да те навести насън под всякаква форма. Обикновено означава проклет, силен, лукав враг, който никак не държи на думата си. Но ако човек е яхнал прасе насън, щял да вземе пари, и то много. Ама няма да бъдат чисти пари, нали? Ако ядеш насън, и знаеш, че кусаш възбранената пържола печена или готвена по друг начин, пак ще вземеш чрез търговия много пари. Но по непозволен начин.

Дивото прасе подсказва идващ голям дъжд и студ, ако пътуваш по суша или плаваш по морето. За онзи, който има противоречие, някаква дрязга или вражда, показва, че врагът му е силен, злобен, с мръсен език. Ако жителите на селата сънуват прасе, значи, че идват усилни, трудни времена, а ако някой, който сади разсад, сънува прасе, този разсад не е както трябва. По подобен начин, ако някой, който иска да се жени, сънува прасе, значи, че не се жени за подходящата жена. Защото

прасето насън може да обозначава и жена. 

И месото на прасето насън се услажда. Затова, който сънува, че яде печено свинско, значи ще се сдобие с бърза полза. А който съзре насън прасе в постелята си, ще се сноши с юдейка. Малките прасенца също имат място в сънищата. Означават големи грижи за онзи, който ги притежава или ги вижда. Домашното прасе обаче може да значи урожай. Ако просто виждаш прасе насън, значи, че си повелител над народ от юдеи и християни. За онзи, който е решил да влезе в конфликт с жена си и види насън мъжко прасе или свиня, значи, че ще се разведе. Възможно е прасето да бъде разтълкувано и като човек измежду юдеите и християните, или пак ако го видиш насън, да означава зло, нещастие, недоволство и скръб, възбранена печалба. Но ако е свинка, може да значи и многобройно потомство. В случай че насън пострадаш от прасе, значи ще пострадаш от християнин. 

Който обаче насън порази прасе, ще получи подкрепа и възможност за влияние от човек с голям авторитет. Който владее много прасета, ще получи много пари накуп. Има и вероятност насън да се превърнеш в прасе. Тогава ще получиш пари и други облаги, но ще бъде заедно с унижения и злощастия във вярата. Ако се сражаваш с прасе, ще надделееш над враг, който угнетява. Ядене на свинско, освен придобиването на пари с мътен произход и по нечестен начин, може да значи и че ще извършиш нечестие. Ако малки прасенца влязат в къщата ти и из двора, значи при теб ще дойдат слугите на султана, тъй че трябва да внимаваш. И обратното – ако насън изгонваш прасенца от двора, ще се откажеш от султанската работа.

Ан-Набулси, когото чета в арабския оригинал, отскоро може да четете в частичен превод на английски от Ясмин Сийл, която през 2022 г. получи литературен грант от клуба „ПЕН“ за превода под поетичното заглавие If You See Them Fall to Earth. Не знам дали там може да откриете откъса за нечистото животно, но за мен е важно признанието за превод на арабски класици от османската епоха на езика на съвремието. При което арабските автори придобиват нова актуалност. 

А моето подозрение около разказите за сънища на поп Минчо Кънчев и ходжите се оказва нелишено от основания.

Пророкът може и да затваря вратата за по-нататъшно пророчество от начина, по който бива изпратен от Аллах на земята, и това да обезкуражава част от общността. За да ги насърчи обаче, той отбелязва, че остават „добрите новини“ (мубашшират), а те на свой ред се обясняват по-късно като „видения насън, разкрити на благочестиви мюсюлмани“ и като „част от пророчеството“. Това вдъхва увереност в сънищата като средство, което предоставя водителство за общността и в частност може да реши конкретни предизвикателства пред нея.

Много по-лесно е да се позовеш на сън при оправдаването на дадено действие или обяснение на събитие, отколкото да намериш предание от Пророка (хадис), което да го подкрепя. Защото механизмът за обявяване на хадисите за достоверни е предмет на твърде тежък критически поглед към гарантирането на достоверността на механизма на тяхното предаване (иснад) чрез авторитети, стигащи до времето на Мохамед¹. Оттук и логичното заключение, че онова, което авторът не може да каже, опирайки се на своя собствен авторитет, може да подкрепи чрез външен източник чрез разказването на сън и видение². 

1 Kinberg, Leah. “Dreams”, EI3, p. 97–98.

2 Ibid., p. 97.

Водещо изображение: Али Парники сънува шейх Сафи в компанията на Пророка Мохамед, биографичен ръкопис за живота на шейх Сафи ад-Дин Исхак Ардабили, илюстрация от XVI век

В рубриката „Ориент кафе“ Атанас Шиников поднася любопитни теми, свързани не толкова с горещата политика, колкото с историята и културата на Близкия изток. А той, древен и днешен, е по-близко до нас и съвремието ни, отколкото си представяме.

[$] A generic ring buffer for the kernel

Post Syndicated from corbet original https://lwn.net/Articles/976836/

The kernel’s user-space ABI does not lack for ring buffers; they have been
defined for subsystems like BPF, io_uring, perf,
and tracing, for
example. Naturally, each of those ring buffers is unique, with no common
interface between them. The natural response to this ABI proliferation is,
of course, to add yet another ring buffer as the generic option; that is
the intent of this
patch series from Kent Overstreet adding a new set of system calls for
ring buffers.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/977442/

Security updates have been issued by AlmaLinux (cockpit, kernel, kernel-rt, libxml2, ruby:3.1, and tomcat), Debian (libarchive, pillow, and tinyproxy), Fedora (apptainer), Mageia (amavisd-new and libxml2), Oracle (edk2), Red Hat (booth, cockpit, kernel-rt, less, libxml2, nghttp2, ruby:3.1, ruby:3.3, and tomcat), Slackware (kernel), and Ubuntu (atril, bluez, frr, gdk-pixbuf, openjdk-17, openjdk-21, openjdk-8, openjdk-lts, qemu, and unixodbc).

Backblaze Live Read: The Game Changer for Live Media Cloud Workflows

Post Syndicated from Elton Carneiro original https://backblaze.com/blog/announcing-b2-live-read/

A decorative image with the title Live Read.

Every sports fan knows that when something incredible happens on the field/ice/court, we want to see the replay right now. But many of us don’t know the impressive efforts that live media teams undertake to deliver clips in real time to all of us on whatever viewing platform we might prefer. Today, Backblaze is excited to make the work of live media production (and the end results) a lot easier with our latest innovation.

Announcing Backblaze B2 Live Read

Backblaze B2 Live Read is a patent-pending service that gives media production teams working on live events the ability to access, edit, and transform media content while it is being uploaded into Backblaze B2 Cloud Storage. This means that teams can start working on content far faster than they could before, without having to drastically change their workflows and tools, massively speeding up their time to engagement and revenue. 

This is a game changer for live media teams, who are passionate about bringing content to their audience as soon as possible. It means they don’t need to worry as screen resolutions continue to expand, ranging from 4K to 8K and beyond. It also reduces the need for having production teams on-site to minimize latency, which could be extremely costly depending on the venue. 

Previously, producers had to wait hours or days before they could access uploaded data, or they had to rely on cost-prohibitive and complicated options that often required on-premises storage. That’s no longer necessary. This innovation will make it faster and less expensive to:

  • Create near real-time highlight clips for news segments, in-app replays, and much more.
  • Tap into talent where they are versus trying to find local talent to produce events.
  • Promote content for on-demand sales within minutes of presentations at live events.
  • Distribute teasers for buzz on social media before talent has even left the venue.

For our customers, turnaround time is essential, and Live Read promises to speed up workflows and operations for producers across the industry. We’re incredibly excited to offer this innovative feature to boost performance and accelerate our customers’ business engagements.”

Richard Andes, VP, Product Management, Telestream

Coming soon inside your favorite tools

We designed Live Read to be easily accessible directly via the Backblaze S3 Compatible API and/or seamlessly within the user interface of launch partners including Telestream, Glookast, and Mimir. These platforms, along with CineDeck, Alteon, Hedge, Hiscale, MoovIT, and many others to come, are enabling Live Read within their platforms soon.   

If you want to use Live Read, you can join our private preview.  

How does it work?

Previously, media teams were forced to either wait for uploads to complete or use on-premises storage. Now, Live Read uniquely supports accessing parts of each growing file or growing object as it is uploaded so there’s no need to wait for the full file upload to complete. And, when the full upload is complete, it’s accessible like any other file in a Backblaze B2 Cloud Storage Bucket, with no middleware or proprietary software needed. 

Here’s a short video showing both how Live Read works on a conceptual level, as well as a live demo showing how one app can upload video data to Backblaze B2 using Live Read while a second app reads the uploaded video data:

For those of you who want to dig deeper into the code samples you saw in the video, here is some example code that uses the Amazon SDK for Python, Boto3, to start uploading data with Live Read. If you’re familiar with Amazon S3, you’ll recognize that this is a standard multipart upload apart from the add_custom_header handler function and the call to register it with Boto3’s event system:

def add_custom_header(params, **_kwargs):
    """
    Add the Live Read custom headers to the outgoing request.
    See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
    """
    params['headers']['x-backblaze-live-read-enabled'] = 'true'

client = boto3.client('s3')
client.meta.events.register('before-call.s3.CreateMultipartUpload', add_custom_header)

response = client.create_multipart_upload(Bucket='my-video-files', Key='liveread.mp4')

upload_id = response['UploadId']

# Now upload data as usual with repeated calls to client.upload_part()

As it processes the call to create_multipart_upload(), Boto3 calls the add_custom_header() handler function, which adds a custom HTTP header, x-backblaze-live-read-enabled, with the value true, to the S3 API request. The custom HTTP header signals to Backblaze B2 that this is a Live Read upload. As with standard multipart uploads, the data is uploaded in parts between 5MB and 5GB in size. To facilitate reading data efficiently, all parts except the last one must have the same size.

Since this is a Live Read upload, as soon as a part is uploaded, it is accessible for downloading.

An app that downloads the file needs to send the same custom HTTP header when it retrieves data. For example:

def add_custom_header(params, **_kwargs):
    """
    Add the Live Read custom headers to the outgoing request.
    See https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
    """
    params['headers']['x-backblaze-live-read-enabled'] = 'true'

client = boto3.client('s3')
client.meta.events.register('before-call.s3.GetObject', add_custom_header)

# Read the first 1 KiB of the file
response = client.get_object(
    Bucket='my-video-files',
    Key='liveread.mp4',
    Range='bytes=0-1023'
)

Note that you must supply either Range or PartNumber to specify a portion of the file when you download data using Live Read. If you request a range or part that does not exist, then Backblaze B2 responds with a 416 Range Not Satisfiable error, just as you might expect. On receiving this error, an app reading the file might repeatedly retry the request, waiting for a short interval after each unsuccessful request.

The source code for the applications is available as open source at https://github.com/backblaze-b2-samples/live-read-demo/.

How much does it cost?

Live Read upload capacity is offered in $15/TB increments—and the capacity is only consumed when an upload is marked for Live Read. Standard uploads are free, as usual. After uploading is complete, the data stored in Backblaze B2 is billed as normal. From a cost perspective, this represents significant savings versus the workflows that production teams must currently follow to achieve anything close to the functionality delivered by Live Read.

And it’s not just for live media

Beyond media, the Live Read API can support breakthroughs across development and IT workloads. For example, organizations maintaining large data logs or surveillance footage backups have often had to parse them into hundreds or thousands of small files each day in order to have quick access when needed—but with Live Read, they can now move to far more manageable single files per day or hour while preserving ability to access parts immediately after they are written.

What’s next

For those interested in Live Read, you can sign up for the private preview here. We’ll continue to report as we add more integrations and we’ll share stories as customers succeed with the new feature. Until then, feel free to ask any question you have in the comments below. 

Want to see more?

Join Pat Patterson, Chief Technical Evangelist, and Elton Carneiro, Senior Director of Partnerships, on January 26, 2024 at 10:00 a.m. PT to learn more in real time. Can’t make it live? Sign up anyway and we’ll send a recording straight to your inbox.

Join the Webinar 

The post Backblaze Live Read: The Game Changer for Live Media Cloud Workflows appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

#Бетономорие ПП-ДБ и Сглобката са прокарали и покровителстват незаконен строеж на огромно селище в залива на Корал

Post Syndicated from Екип на Биволъ original https://bivol.bg/grad_koral.html

четвъртък 6 юни 2024


Само два месеца преди Кирил Петков да подаде оставка като Mинистър-председател на Република България на 27 юни 2022 г., непосредствено зад плаж Корал, ударно започва да изниква огромен строеж на…

Zabbix 7.0 – Everything You Need to Know

Post Syndicated from Michael Kammer original https://blog.zabbix.com/zabbix-7-0-everything-you-need-to-know/28210/

After plenty of breathless anticipation, we’re proud to announce the release of the latest major Zabbix version – the new and improved Zabbix 7.0 LTS. This release is the direct result of user feedback and delivers a variety of improvements, including cloud-native Zabbix proxy scalability, website transaction monitoring, improved data collection speed and scalability, new dashboard widgets, major network discovery speed improvements, new templates and integrations, and more!

Without further ado, let’s take a whistle-stop tour of what you need to know:

Synthetic end-user web monitoring

Busy enterprises can now monitor multiple websites and applications by defining flexible multi-step browser-based scenarios. 7.0 LTS also makes it easy to capture screenshots of the current website state, collect and visualize website performance and availability metrics, extract, monitor, and analyze web application data, and get alerts when issues are discovered.

Zabbix proxy high availability and load balancing

When it’s time to expand, Zabbix 7.0 LTS makes it easy to scale a Zabbix environment, guaranteeing 100% availability with automatic proxy load balancing and high availability features, including the ability to assign hosts to load-balanced proxy groups and seamlessly scale a Zabbix environment by deploying additional proxies.

Faster, more efficient Zabbix proxies

Zabbix proxy now fully supports in-memory data storage for collected metrics. Users can choose from Disk, Memory, and Hybrid proxy buffer modes, all of which are ideal for embedded hardware. In addition, memory mode enables the support of edge computing use cases. Users can expect 10-100x better proxy performance by switching to memory or hybrid modes, depending on allocated hardware.

Centralized control of data collection timeouts

Centralizing control of data collection timeouts enables better support for metrics and custom checks, taking longer data collection time intervals. Data collection timeouts can be defined per item-type and overridden per proxy or on an individual item level. In addition, timeouts are now fully configurable in the Zabbix GUI or via Zabbix API.

Faster and more scalable data collection

Synchronous poller processes have been replaced with asynchronous pollers, which improves the speed and scalability of metric polling, particularly for agent, SNMP, and HTTP checks. The next metric can now be polled before waiting for a response from a previously requested metric, and up to 1,000 concurrent checks can now be supported per poller process.

New ways to visualize data

A variety of new dashboard widgets have been introduced, with the goal of giving users detailed information about their monitored metrics and infrastructure at a glance.

Dynamic dashboard widget navigation

Speaking of dashboard widgets, a new communication framework has also been introduced for dashboard widgets, enabling communication between widgets, allowing a widget to serve as a data source for other widgets, and dynamically updating information displayed in a dashboard widget based on the data source.

Faster network discovery

Discovering services and hosts has never been easier, thanks to support of parallelization while performing network discovery. Concurrency support allows for massive improvements in network discovery speed and simplifies host and service discovery while scanning large network segments.

Better security via enterprise-grade multi-factor authentication

Out-of-the box support of multi-factor authentication enables enterprise-grade security and added flexibility for configuring user authentication methods. Support MFA providers include time-based one-time Password (TOTP) and Duo Universal Prompt authentication.

More flexible resource discovery and management

Low-level discovery has received a variety of improvements, which enable enhanced host configuration and management flexibility when discovering hosts in complex environments, such as VMware or Kubernetes.

New templates and integrations

In response to user demand, Zabbix 7.0 LTS comes pre-packaged with a range of new templates for the most popular vendors and cloud providers.

Zabbix 7.0 training updates

All Zabbix training materials have been updated based on the new functionalities that have been added to the product since Zabbix 6.0.

Everyone is welcome to sharpen their skills, but if you’re a Zabbix 6.0 Certified Specialist or Certified Professional you can master Zabbix 7.0 LTS in just one day with our Upgrade Courses. As a 7.0 Specialist, you’ll be able to automate user provisioning with the Just-in-time (JIT) feature, monitor websites with new synthetic end-user monitoring, leverage new visualization features, and enhance the speed and performance of your data collection.

The 7.0 Certified Professional course covers proxy group configuration with high availability and load balancing, improved proxy data collection, new SNMP bulk monitoring, and enhanced host discovery for VMware, Kubernetes, and Cloud infrastructures.

We’re also happy to organize private trainings for organizations of any size, so don’t hesitate to get in touch!

Upcoming 7.0 events

If you’re looking for more information regarding Zabbix 7.0, you’re in luck! You can tune in to the “What’s new in Zabbix 7.0” webinar on June 11 at 12 PM CST or June 12 at 10 AM EEST. If you’d prefer a more hands-on approach, the following workshops are also available:

• “Zabbix Proxy High-availability and Load Balancing” (June 18, 6 PM EEST)
• “New Web Monitoring Features in Zabbix 7.0” (June 20, 6 PM EEST)

While you’re at it, feel free to explore Zabbix 7.0 LTS webinars and workshops in other languages. You can also check out worldwide events related to Zabbix 7.0 LTS, including our free in-person meetup in Riga on June 19 and Zabbix Summit 2024 this fall. 

Ready to upgrade or migrate?

With a brand-new version out, there’s never been a better time to take advantage of our upgrade or migration services. Let our team take the risk out of migrating or upgrading to 7.0, giving you the latest version at a lower cost and with minimal disruption to your organization.

Need a consultation about the latest version?

Not sure about how to get the most out of Zabbix 7.0? Our expert consultants can answer any questions related to the architecture of your infrastructure, the implementation of a back-up strategy, and your capacity planning, while providing strategic advice on which 7.0 services are right for you.

Make your contribution as a translator

The Documentation 7.0 translation project is now live, which means that you can help localize Zabbix 7.0 documentation in multiple languages. Your efforts will help make Zabbix accessible to users around the globe, and you’ll also receive a reward for your contributions. The guidelines, which contain essential information about the project, are available here.

Useful links

To see what else is in store for the future, have a look at the Zabbix roadmap.

You can find the instructions and download the new version on the Download page.

Detailed, step-by-step upgrade instructions are available on our Upgrade procedure page.

Learn about new features and changes introduced in Zabbix 7.0 LTS by visiting the What’s new in Zabbix 7.0 page.

The What’s new documentation section provides a detailed description of the new features.

Take a look at the release notes to see the full list of new features and improvements.

 

The post Zabbix 7.0 – Everything You Need to Know appeared first on Zabbix Blog.

Protecting vulnerable communities for 10 years with Project Galileo

Post Syndicated from Jocelyn Woolbright original https://blog.cloudflare.com/galileo10anniversaryradardashboard


In celebration of Project Galileo’s 10th anniversary, we want to give you a snapshot of what organizations that work in the public interest experience on an everyday basis when it comes to keeping their websites online. With this, we are publishing the Project Galileo 10th anniversary Radar dashboard with the aim of providing valuable insights to researchers, civil society members, and targeted organizations, equipping them with effective strategies for protecting both internal information and their public online presence.

Key Statistics

  • Under Project Galileo, we protect more than 2,600 Internet properties in 111 countries.
  • Between May 1, 2023, and March 31, 2024, Cloudflare blocked 31.93 billion cyber threats against organizations protected under Project Galileo. This is an average of nearly 95.89 million cyber attacks per day over the 11-month period.
  • When looking at the different organizational categories, journalism and media organizations were the most attacked, accounting for 34% of all attacks targeting the Internet properties protected under the Project in the last year, followed by human rights organizations at 17%.
  • On October 11, 2023, Cloudflare detected one of the largest attacks we’ve seen against an organization under Project Galileo, targeting a prominent independent journalism website covering stories in Russia and across Eastern Europe. We identified a DDoS attack that peaked at 7 million requests per second, with an attack duration of 7 minutes. In total, 1.9 billion DDoS requests targeting the attacked organization were mitigated that day.
  • We saw two attacks against an organization that manages vital Internet infrastructure in the Middle East. We mitigated 177 million DDoS requests targeting the organization over a three-hour period in October 2023. The second attack in December 2023 reached 42.6 million requests that were mitigated over a two-hour period.
  • We observed an attack targeting LGBT Foundation, a UK-based LGBTQ+ organization, during the beginning of Pride Month in June 2023. Cloudflare mitigated 144.7 million requests to this organization on June 2, 2023. In addition to this spike in June, we also saw another attack on August 26, 2023, which coincided with Manchester Pride. This second attack peaked at 1.46 million requests per second before finally subsiding on August 29.

This year, we broke down the dashboard into several sections:

  • Global civil society and human rights organizations
  • Global journalism and media organizations
  • Organizations based in Ukraine
  • Organizations in Israel and Palestine
  • Voting rights organizations based in the United States

Check out the full report here.

Highlights of the Report

Protecting free speech and a free press

The number of journalists imprisoned worldwide has grown in recent years. Reporters are increasingly at risk of being censored or shut down by governments or falling victim to cyberattacks. Project Galileo started as an initiative to protect free expression online. It’s grown to not only protect journalists, but also organizations working in the public interest such as voting rights groups, environmental activists, human rights defenders and more. We’ve seen journalists targeted on the Internet for various reasons, often stemming from the sensitive and impactful nature of their work. To that end, we’ve partnered with prominent organizations such as Internews, Center for International Media Assistance, International Press Institute, International Media Support, and many more to identify where our services are needed.

“Truth is the first casualty of war”

As the conflict in Ukraine continues, Cloudflare has been providing protection to journalists reporting on the conflict, human rights organizations helping refugees on the ground, and groups that have built mobile apps giving people early warnings of missile strikes.

Among them is Russian-born Galina Timchenko, co-founder, CEO, and owner of independent news outlet Meduza. A recent investigation by Access Now and the Citizen Lab reveals Timchenko had her iPhone infected with NSO Group’s Pegasus spyware during a trip to Berlin, Germany around February 10, 2023. This is the first documented case of Pegasus infection against a Russian journalist, which shows the growing suspicions among European Union governments regarding Russian civil society in exile. Labeled as an “undesirable organization” and blocked by the Russian government, Meduza operates out of Latvia to maintain editorial independence as it continues to publish news focused on covering stories in Russia and the former Soviet Union, including the conflict in Ukraine.

Meduza is an example of an important organization that lacks the resources to protect itself against intensive online attacks. On a single day in October 2023, Meduza came under DDoS attack peaking at 7 million requests per second and lasting 7 minutes—an onslaught which would have disabled the site under normal circumstances.

Protecting organizations in a time of conflict

We’ve reported on patterns of wartime violence coinciding with cyberattacks. Unfortunately, these trends have continued during the war between Israel and Hamas, and the humanitarian crisis in Gaza. Under Project Galileo, we protect a range of organizations based in the region that work to provide emergency response service, vital equipment for hospitals, crowdfunding platforms supporting the Muslim community worldwide, and more. We saw an increase in traffic after October 7, 2023, to both Israeli and Palestinian organizations, coinciding with the start of the Israel-Hamas war.

As we explored the data further, we saw an attack against a prominent organization based in the United Kingdom that works to secure Palestinian human rights, observing two dates on which there was an increase in mitigated traffic. The first, on October 15, 2023, coincided with the national demonstration in London in support of Palestine. We see in the first spike the requests go from 0 to 44,500 mitigated requests per second within two minutes. When we took a closer look, we identified that many of the requests were mitigated by Cloudflare’s Security Level, a product that uses the threat score (IP reputation) to decide whether to present a challenge to the visitor. The second spike, on February 21, 2024, coincided with UK lawmakers calling for cease-fire in the Israel-Hamas war. This peaked at 10,500 mitigations per second that lasted 40 minutes with an average of 6,638 requests per second.

As we reviewed the data, we saw two attacks against an organization that manages vital Internet infrastructure in the Middle East. Attacking infrastructure entities like domain name registries and registrars is not new, as we saw in Ukraine during the beginning of the war in March 2022, and follows an unsettling trend of targeting broad swaths of a country’s Internet infrastructure.

We saw two notable spikes in traffic, the first in October and second in December 2023. The first attack took place in three waves on October 18 and 19th, peaking around 78,500 requests per second. In total, the attack went from 2.48 million requests to 177.42 million requests mitigated per day.

On December 20-21, 2023, there was an attack that lasted more than 2 hours, averaging 8,600 requests per second throughout that period, reaching as high as 13,830 requests per second. In total, this attack saw 42.6 million daily requests mitigated.

And more…

Here we’ve provided just a snapshot of what organizations see on a daily basis when it comes to keeping their websites online. For more information on attacks against organizations protected under Project Galileo, check out the full Radar report.

If you are an organization looking for protection under Project Galileo, please visit our website: cloudflare.com/galileo.

Sabrent Apex X16 Rocket 5 Destroyer 64TB PCIe Gen5 Card Shown

Post Syndicated from Cliff Robinson original https://www.servethehome.com/sabrent-apex-x16-rocket-5-destroyer-64tb-pcie-gen5-card-shown/

The Sabrent Apex X16 Rocket 5 Destroyer is a 64TB card that uses a Microchip Switchtec PCIe Gen5 switch to provide over 50GB/s of throughput

The post Sabrent Apex X16 Rocket 5 Destroyer 64TB PCIe Gen5 Card Shown appeared first on ServeTheHome.

Modernize your data observability with Amazon OpenSearch Service zero-ETL integration with Amazon S3

Post Syndicated from Joshua Bright original https://aws.amazon.com/blogs/big-data/modernize-your-data-observability-with-amazon-opensearch-service-zero-etl-integration-with-amazon-s3/

We are excited to announce the general availability of Amazon OpenSearch Service zero-ETL integration with Amazon Simple Storage Service (Amazon S3) for domains running 2.13 and above. The integration is new way for customers to query operational logs in Amazon S3 and Amazon S3-based data lakes without needing to switch between tools to analyze operational data. By querying across OpenSearch Service and S3 datasets, you can evaluate multiple data sources to perform forensic analysis of operational and security events. The new integration with OpenSearch Service supports AWS’s zero-ETL vision to reduce the operational complexity of duplicating data or managing multiple analytics tools by enabling you to directly query your operational data, reducing costs and time to action.

OpenSearch is an open source, distributed search and analytics suite derived from Elasticsearch 7.10. OpenSearch Service currently has tens of thousands of active customers with hundreds of thousands of clusters under management processing hundreds of trillions of requests per month.

Amazon S3 is an object storage service offering industry-leading scalability, data availability, security, and performance. Organizations of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-centered applications, and mobile apps. With cost-effective storage classes and user-friendly management features, you can optimize costs, organize data, and configure fine-tuned access controls to meet specific business, organizational, and compliance requirements. Let’s dig into this exciting new feature for OpenSearch Service.

Benefits of using OpenSearch Service zero-ETL integration with Amazon S3

OpenSearch Service zero-ETL integration with Amazon S3 allows you to use the rich analytics capabilities of OpenSearch Service SQL and PPL directly on infrequently queried data stored outside of OpenSearch Service in Amazon S3. It also integrates with other OpenSearch integrations so you can install prepackaged queries and visualizations to analyze your data, making it straightforward to quickly get started.

The following diagram illustrates how OpenSearch Service unlocks value stored in infrequently queried logs from popular AWS log types.

You can use OpenSearch Service direct queries to query data in Amazon S3. OpenSearch Service provides a direct query integration with Amazon S3 as a way to analyze operational logs in Amazon S3 and data lakes based in Amazon S3 without having to switch between services. You can now analyze data in cloud object stores and simultaneously use the operational analytics and visualizations of OpenSearch Service.

Many customers currently use Amazon S3 to store event data for their solutions. For operational analytics, Amazon S3 is typically used as a destination for VPC Flow Logs, Amazon S3 Access Logs, AWS Load Balancer Logs, and other event sources from AWS services. Customers also store data directly from application events in Amazon S3 for compliance and auditing needs. The durability and scalability of Amazon S3 makes it an obvious data destination for many customers that want a longer-term storage or archival option at a cost-effective price point.

Bringing data from these sources into OpenSearch Service stored in hot and warm storage tiers may be prohibitive due to the size and volume of the events being generated. For some of these event sources that are stored into OpenSearch Service indexes, the volume of queries run against the data doesn’t justify the cost to continue to store them in their cluster. Previously, you would pick and choose which event sources you brought in for ingestion into OpenSearch Service based on the storage provisioned in your cluster. Access to other data meant using different tools such as Amazon Athena to view the data on Amazon S3.

For a real-world example, let’s see how using the new integration benefited Arcesium.

“Arcesium provides advanced cloud-native data, operations, and analytics capabilities for the financial services industry. Our software platform processes many millions of transactions a day, emitting large volumes of log and audit records along the way. The volume of log data we needed to process, store, and analyze was growing exponentially given our retention and compliance needs. Amazon OpenSearch Service’s new zero-ETL integration with Amazon S3 is helping our business scale by allowing us to analyze infrequently queried logs already stored in Amazon S3 instead of incurring the operational expense of maintaining large and costly online OpenSearch clusters or building ad hoc ingestion pipelines.”

– Kyle George, SVP & Global Head of Infrastructure at Arcesium.

With direct queries with Amazon S3, you no longer need to build complex extract, transform, and load (ETL) pipelines or incur the expense of duplicating data in both OpenSearch Service and Amazon S3 storage.

Fundamental concepts

After configuring a direct query connection, you’ll need to create tables in the AWS Glue Data Catalog using the OpenSearch Service Query Workbench. The direct query connection relies on the metadata in Glue Data Catalog tables to query data stored in Amazon S3. Note that tables created by AWS Glue crawlers or Athena are not currently supported.

By combining the structure of Data Catalog tables, SQL indexing techniques, and OpenSearch Service indexes, you can accelerate query performance, unlock advanced analytics capabilities, and contain querying costs. Below are a few examples of how you can accelerate your data:

  • Skipping indexes – You ingest and index only the metadata of the data stored in Amazon S3. When you query a table with a skipping index, the query planner references the index and rewrites the query to efficiently locate the data, instead of scanning all partitions and files. This allows the skipping index to quickly narrow down the specific location of the stored data that’s relevant to your analysis.
  • Materialized views – With materialized views, you can use complex queries, such as aggregations, to power dashboard visualizations. Materialized views ingest a small amount of your data into OpenSearch Service storage.
  • Covering indexes – With a covering index, you can ingest data from a specified column in a table. This is the most performant of the three indexing types. Because OpenSearch Service ingests all data from your desired column, you get better performance and can perform advanced analytics. OpenSearch Service creates a new index from the covering index data. You can use this new index for dashboard visualizations and other OpenSearch Service functionality, such as anomaly detection or geospatial capabilities.

As new data comes in to your S3 bucket, you can configure a refresh interval for your materialized views and covering indexes to provide local access to the most current data on Amazon S3.

Solution overview

Let’s take a test drive using VPC Flow Logs as your source! As mentioned before, many AWS services emit logs to Amazon S3. VPC Flow Logs is a feature of Amazon Virtual Private Cloud (Amazon VPC) that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. For this walkthrough, you perform the following steps:

  1. Create an S3 bucket if you don’t already have one available.
  2. Enable VPC Flow Logs using an existing VPC that can generate traffic and store the logs as Parquet on Amazon S3.
  3. Verify the logs exist in your S3 bucket.
  4. Set up a direct query connection to the Data Catalog and the S3 bucket that has your data.
  5. Install the integration for VPC Flow Logs.

Create an S3 bucket

If you have an existing S3 bucket, you can reuse that bucket by creating a new folder inside of the bucket. If you need to create a bucket, navigate to the Amazon S3 console and create an Amazon S3 bucket with a name that is suitable for your organization.

Enable VPC Flow Logs

Complete the following steps to enable VPC Flow Logs:

  1. On the Amazon VPC console, choose a VPC that has application traffic that can generate logs.
  2. On the Flow Logs tab, choose Create flow log.
  3. For Filter, choose ALL.
  4. Set Maximum aggregation interval to 1 minute.
  5. For Destination, choose Send to an Amazon S3 bucket and provide the S3 bucket ARN from the bucket you created earlier.
  6. For Log record format, choose Custom format and select Standard attributes.

For this post, we don’t select any of the Amazon Elastic Container Service (Amazon ECS) attributes because they’re not implemented with OpenSearch integrations as of this writing.

  1. For Log file format, choose Parquet.
  2. For Hive-compatible S3 prefix, choose Enable.
  3. Set Partition logs by time to every 1 hour (60 minutes).

Validate you are receiving logs in your S3 bucket

Navigate to the S3 bucket you created earlier to see that data is streaming into your S3 bucket. If you drill down and navigate the directory structure, you find that the logs are delivered in an hourly folder and emitted every minute.

Now that you have VPC Flow Logs flowing into an S3 bucket, you need to set up a connection between your data on Amazon S3 and your OpenSearch Service domain.

Set up a direct query data source

In this step, you create a direct query data source which uses Glue Data Catalog tables and your Amazon S3 data. The action creates all the necessary infrastructure to give you access to the Hive metastore (databases and tables in Glue Data Catalog and the data housed in Amazon S3 for the bucket and folder combination you want the data source to have access to. It will also wire in all the appropriate permissions with the Security plugin’s fine-grained access control so you don’t have to worry about permissions to get started.

Complete the following steps to set up your direct query data source:

  1. On the OpenSearch Service domain, choose Domains in the navigation pane.
  2. Choose your domain.
  3. On the Connections tab, choose Create new connection.
  4. For Name, enter a name without dashes, such as zero_etl_walkthrough.
  5. For Description, enter a descriptive name.
  6. For Data source type, choose Amazon S3 with AWS Glue Data Catalog.
  7. For IAM role, if this is your first time, let the direct query setup take care of the permissions by choosing Create a new role. You can edit it later based on your organization’s compliance and security needs. For this post, we name the role zero_etl_walkthrough.
  8. For S3 buckets, use the one you created.
  9. Do not select the check box to grant access to all new and existing buckets.
  10. For Checkpoint S3 bucket, use the same bucket you created. The checkpoint folders get created for you automatically.
  11. For AWS Glue tables, because you don’t have anything that you have created in the Data Catalog, enable Grant access to all existing and new tables.

The VPC Flow Logs OpenSearch integration will create resources in the Data Catalog, and you will need access to pick those resources up.

  1. Choose Create.

Now that the initial setup is complete, you can install the OpenSearch integration for VPC Flow Logs.

Install the OpenSearch integration for VPC Flow Logs

The integrations plugin contains a wide variety of prebuilt dashboards, visualizations, mapping templates, and other resources that make visualizing and working with data generated by your sources simpler. The integration for Amazon VPC installs a variety of resources to view your VPC Flow Logs data as it sits in Amazon S3.

In this section, we show you how to make sure you have the most up-to-date integration packages for installation. We then show you how to install the OpenSearch integration. In most cases, you will have the latest integrations such as VPC Flow Logs, NGINX, HA Proxy, or Amazon S3 (access logs) at the time of the release of a minor or major version. However, OpenSearch is an open source community-led project, and you can expect that there will be version changes and new integrations not yet included with your current deployment.

Verify the latest version of the OpenSearch integration for Amazon VPC

You may have upgraded from earlier versions of OpenSearch Service to OpenSearch Service version 2.13. Let’s confirm that your deployment matches what is present in this post.

On OpenSearch Dashboards, navigate to the Integrations tab and choose Amazon VPC. You will see a release version for the integration.

Confirm that you have version 1.1.0 or higher. If your deployment doesn’t have it, you can install the latest version of the integration from the OpenSearch catalog. Complete the following steps:

  1. Navigate to the OpenSearch catalog.
  2. Choose Amazon VPC Flow Logs.
  3. Download the 1.1.0 Amazon VPC Integration file from the repository folder labeled amazon_vpc_flow_1.1.0.
  4. In the OpenSearch Dashboard’s Dashboard Management plugin, choose Saved objects.
  5. Choose Import and browse your local folders.
  6. Import the downloaded file.

The file contains all the necessary objects to create an integration. After it’s installed, you can proceed to the steps to set up the Amazon VPC OpenSearch integration.

Set up the OpenSearch integration for Amazon VPC

Let’s jump in and install the integration:

  1. In OpenSearch Dashboards, navigate to the Integrations tab.
  2. Choose the Amazon VPC integration.
  3. Confirm the version is 1.1.0 or higher and choose Set Up.
  4. For Display Name, keep the default.
  5. For Connection Type, choose S3 Connection.
  6. For Data Source, choose the direct query connection alias you created in prior steps. In this post, we use zero_etl_walkthrough.
  7. For Spark Table Name, keep the prepopulated value of amazon_vpc_flow.
  8. For S3 Data Location, enter the S3 URI of your log folder created by VPC Flow Logs set up in the prior steps. In this post, we use s3://zero-etl-walkthrough/AWSLogs/.

S3 bucket names are globally unique, and you may want to consider using bucket names that conform to your company’s compliance guidance. UUIDs plus a descriptive name are good options to guarantee uniqueness.

  1. For S3 Checkpoint Location, enter the S3 URI of your checkpoint folder which you define. Checkpoints store metadata for the direct query feature. Make sure you pick any empty or unused path in the bucket you choose. In this post, we use s3://zero-etl-walkthrough/CP/, which is in the same bucket we created earlier.
  2. Select Queries (recommended) and Dashboards and Visualizations for Flint Integrations using live queries.

You get a message that states “Setting Up the Integration – this can take several minutes.” This particular integration sets up skipping indexes and materialized views on top of your data in Amazon S3. The materialized view aggregates the data into a backing index that occupies a significantly smaller data footprint in your cluster compared to ingesting all the data and building visualizations on top of it.

When the Amazon VPC integration installation is complete, you have a broad variety of assets to play with. If you navigate to the installed integrations, you will find queries, visualizations, and other assets that can help you jumpstart your data exploration using data sitting on Amazon S3. Let’s look at the dashboard that gets installed for this integration.

I love it! How much does it cost?

With OpenSearch Service direct queries, you only pay for the resources consumed by your workload. OpenSearch Service charges for only the compute needed to query your external data as well as maintain optional indexes in OpenSearch Service. The compute capacity is measured in OpenSearch Compute Units (OCUs). If no queries or indexing activities are active, no OCUs are consumed. The following table contains sample compute prices based on searching HTTP logs in IAD.

Data scanned per query (GB) OCU price per query (USD)
1-10 $0.026
100 $0.24
1000 $1.35

Because the price is based on the OCUs used per query, this solution is tailored for infrequently queried data. If your users query data often, it makes more sense to fully ingest into OpenSearch Service and take advantage of storage optimization techniques such as using OR1 instances or UltraWarm.

OCUs consumed by zero-ETL integrations will be populated in AWS Cost Explorer. This will be at the account level. You can account for OCU usage at the account level and set thresholds and alerts when thresholds have been crossed. The format of the usage type to filter on in Cost Explorer is RegionCode-DirectQueryOCU (OCU-hours). You can create a budget using AWS Budgets and configure an alert to be notified when DirectQueryOCU (OCU-Hours) usage meets the threshold you set. You can also optionally use an Amazon Simple Notification Service (Amazon SNS) topic with an AWS Lambda function as a target to turn off a data source when a threshold criterion is met.

Summary

Now that you have a high-level understanding of the direct query connection feature, OpenSearch integrations, and how the OpenSearch Service zero-ETL integration with Amazon S3 works, you should consider using the feature as part of your organization’s toolset. With OpenSearch Service zero-ETL integration with Amazon S3, you now have a new tool for event analysis. You can bring hot data into OpenSearch Service for near real-time analysis and alerting. For the infrequently queried, larger data, mainly used for post-event analysis and correlation, you can query that data on Amazon S3 without moving the data. The data stays in Amazon S3 for cost-effective storage, and you access that data as needed without building additional infrastructure to move the data into OpenSearch Service for analysis.

For more information, refer to Working with Amazon OpenSearch Service direct queries with Amazon S3.

About the authors

Joshua Bright is a Senior Product Manager at Amazon Web Services. Joshua leads data lake integration initiatives within the OpenSearch Service team. Outside of work, Joshua enjoys listening to birds while walking in nature.

Kevin Fallis is an Principal Specialist Search Solutions Architect at Amazon Web Services. His passion is to help customers leverage the correct mix of AWS services to achieve success for their business goals. His after-work activities include family, DIY projects, carpentry, playing drums, and all things music.


Sam Selvan is a Principal Specialist Solution Architect with Amazon OpenSearch Service.

[$] Measuring and improving buffered I/O

Post Syndicated from jake original https://lwn.net/Articles/976856/

There are two types of file I/O on Linux, buffered I/O, which goes through
the page cache, and direct I/O, which goes directly to the storage device.
The performance of buffered I/O was reported to be a lot worse than direct
I/O, especially for one specific test, in Luis Chamberlain’s
topic
proposal for a session at the 2024 Linux Storage,
Filesystem, Memory Management, and BPF Summit.
The proposal resulted in a lengthy mailing-list discussion,
which also came up in Paul McKenney’s RCU session the next
day; Chamberlain led a
combined storage and filesystem session to discuss those results with an
eye toward improving buffered I/O performance.

Kali Linux 2024.2 released

Post Syndicated from jzb original https://lwn.net/Articles/977303/

Version 2024.2 of the Kali Linux penetration testing distribution
has been released. This
release includes an update to GNOME
46, a high-resolution (HiDPI) mode for Xfce, as well as a number
of new packages such as the AutoRecon network
reconnaissance tool, pspy command-line utility for
snooping on Linux processes, and SploitScan tool for
fetching and displaying CVE information. Kali Linux is based on Debian
testing, and 2024.2 incorporates Debian’s work to transition to 64-bit
time_t to avoid year 2038 problems. Users with existing Kali
systems should be sure to follow the documentation
when upgrading.

The collective thoughts of the interwebz