Building a GPU workstation for visual effects with Amazon EC2: Part 3 – Windows

Post Syndicated from Andy Hayes original https://aws.amazon.com/blogs/compute/building-a-gpu-workstation-for-visual-effects-with-amazon-ec2-part-3-windows/

For many years Amazon Web Services (AWS) has been guiding customers in the deployment of powerful NVIDIA GPU-backed workstations for graphics-intensive workloads. With the recent expansion of our G-series Amazon Elastic Compute Cloud (Amazon EC2) graphics-based instances, we are excited to introduce new instance families that deliver enhanced performance and capabilities. We are also refreshing our comprehensive guidance on choosing and configuring these powerful virtual workstations.

This is the third post in a five-part series, focusing on the deployment of a Windows Server 2025 visual effects (VFX) workstation, using Amazon DCV as a performant desktop display protocol.

“Amazon DCV has unlocked previously unrealised playback performance, that has massively increased my productivity. Amazon DCV is performant on a range of systems and where other solutions have failed, I have been able to continue working and most importantly, continue delivering”

— Chris Coupe, CG Supervisor, Outpost VFX

Prerequisites

This post assumes that you have configured the following:

  1. An active AWS account with increased GPU Amazon EC2 service quota limits.
  2. A valid key pair in your AWS Region of choice.
  3. A configured virtual firewall security group, with open TCP/UDP ports 22, 443, 8443, and 3389 limited to only your own IP address.
  4. An AWS Identity and Access Management (IAM) profile with least-permissions Amazon Simple Storage Service (Amazon S3) read access.

We strongly recommend that you follow Part 1 of the series, Build a GPU workstation for visual effects on Amazon EC2: Part 1 – Setup, which goes through the preceding steps in detail.

Launching the Windows workstation

We use the latest AWS provided Windows Server 2025 Amazon Machine Image (AMI) for our deployment, and install the NVIDIA, Amazon DCV, and Wacom components to create a fully functioning VFX Windows desktop environment.

Follow these steps to launch a g6.4xlarge instance in your chosen AWS Region. If you want to use multiple GPU cards (for example, by using the 12xlarge, 24xlarge, or 48xlarge instance types), then Windows requires you to have that number of GPU cards present when setting up the GPU driver. In this case, we recommend starting the relevant multiple GPU instance size when making your AMI.

  1. Log in to the AWS Management Console and choose EC2 Dashboard in the top left.
  2. Choose Launch instances to start the prompt to launch a new instance.
  3. Give the instance a name, for example my_windows_workstation.
  4. Choose Windows under the Quick Start AMI and use the default Microsoft Windows Server Base Image 2025.
  5. Under Instance type, choose g6.4xlarge.
  6. Under Key pair name, choose the menu and choose your key pair. In the previous post we called it ec2_test_keypair to specify the password security credentials.
  7. Edit the Network settings to adjust security and network placement:
    1. Leave the Subnet to No preference.
    2. Make sure that Auto-assign public IP is set to Enable, so that you can access the instance through the internet.
  8. Under the Firewall (security groups) section of the Network Settings, choose Select existing security group, and specify your security group. In the previous post we named it ec2_test_sg to set up the firewall for the machine.
  9. In Configuration storage, expand the default root volume from 10 GiB to 50 GiB of gp3 storage.Important: Turn on encryption in the Advanced options by selecting the Encrypted option and selecting the default AWS managed key (aws/ebs) or a customer-managed KMS key for enhanced control. Amazon Elastic Block Store (Amazon EBS) encryption protects your VFX project files, renders, and application data at rest. Encryption uses industry-standard AES-256 encryption and is transparent to applications with minimal performance impact.
  10. Under Advanced details, specify the role that you created for workstation usage. In the previous post we called it basic_workstation_role through the IAM instance profile menu, so your workstation can access more services.
  11. Choose Launch instance on the right-hand side, as shown in the following figure.

Choose the link to the EC2 instance that has been launched, or view it through the Instances pane in the Amazon EC2 Service page. The following screen capture shows the newly launched workstation in the EC2 Instances list, where you can confirm it is ready to use. Wait until the my_windows_workstation instance shows an Instance state of Running and a Status check of 3/3 checks passed. Once both conditions are met, you can continue.

EC2 Instances page showing the my_windows_workstation instance with an Instance state of Running and a Status check of 3/3 checks passed

Figure 1: A screen capture of the EC2 Instances page, showing the my_windows_workstation in a ready state

Configuring the Windows workstation

In the last part of our series, Building a GPU workstation for visual effects with Amazon EC2: Part 4 – Optimizations, we show how to add scripting capabilities to the boot sequence of EC2 instances. This runs processes such as the NVIDIA GPU driver and Wacom tablet driver automatically during the startup of a machine. For the purposes of this post, we take a more manual approach, showing how you can remote into the machine to perform setup tasks.

You initially connect to the instance through the Windows Remote Desktop Protocol (RDP). In the Amazon EC2 Service page browser tab with the list of EC2 instances running, again choose the my_windows_workstation and choose Connect.

Choose the RDP client tab to connect to the machine through RDP with the following steps:

  1. Choose the Get password option to retrieve the Administrator password for the machine through your key pair credential.
  2. Upload the private ec2_test_keypair .pem key file that you downloaded previously and choose Decrypt password.
  3. The password is now displayed. Copy the password into the clipboard memory by choosing the copy icon, and choose OK, as shown in the following figure.
Get Windows Password page showing the instance ID, the private IP address, the username, and a redacted password

Figure 2: A screen capture of the Get Windows Password retrieval options available to EC2 instances

  1. Choose Download remote desktop file to download the .rdp file to your local machine.
  2. Run the .rdp file with your Remote Desktop Application of choice.
  3. When prompted, paste the previously copied Administrator password and choose OK.
  4. Choose Yes to trust the machine, and the session is authenticated and opened.

Security

We recommend setting a new, private password when you first start the fresh instance.

Open a Windows PowerShell to configure and make changes to the workstation. In the shell, configure that change and store your new password in a safe place.

net user Administrator <<NewPassword>>

Tools

To install the AWS Command Line Interface (AWS CLI) enter the following into the shell, and follow the on-screen prompts:

msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

Graphics

You must use AWS approved NVIDIA GPU drivers, issued through Amazon S3, not only for optimal compatibility and performance, but also to make sure that the necessary professional GRID drivers are automatically provisioned. You can always find the latest instructions in the Amazon EC2 Windows User Guide. At the time of writing, you can download the latest version to your desktop by running the following in a new Windows PowerShell:

$Bucket = "ec2-windows-nvidia-drivers"
$KeyPrefix = "latest"
$LocalPath = "$home\Desktop\NVIDIA"
$Objects = Get-S3Object -BucketName $Bucket -KeyPrefix $KeyPrefix -Region us-east-1
foreach ($Object in $Objects) {
    $LocalFileName = $Object.Key
    if ($LocalFileName -ne '' -and $Object.Size -ne 0) {
        $LocalFilePath = Join-Path $LocalPath $LocalFileName
        Copy-S3Object -BucketName $Bucket -Key $Object.Key -LocalFile $LocalFilePath -Region us-east-1
    }
}

When it is downloaded, navigate to your Desktop, open the NVIDIA folder, then the latest folder inside, and run the NVIDIA installer executable, following the instructions.

Wacom support and Wacom Bridge

To add Wacom stylus support on the workstation, open Microsoft Edge (or install your browser of choice) and download and install the latest drivers from the Wacom drivers page.

Wacom Bridge is now generally available on Windows and Mac, and offers new advantages for Wacom users on AWS:

  • Applications running locally and remotely on AWS transfer their Wacom-specific settings automatically.
  • Artists can switch from local to remote use of the pen by choosing the necessary application window.
  • Wacom Inkline draws an instant temporary line on your screen, to bridge any gap between the local pen tip and the remote drawn line.
  • Amazon DCV provides enhanced security through data transport encrypted through HTTPS/TLS connections.

You can purchase a Wacom Bridge license (or request a trial) from a Wacom Sales representative.

Amazon DCV setup

To use Amazon DCV, you need to download the server application for the Amazon EC2 workstation from https://www.amazondcv.com/.

On the Amazon EC2 workstation, open the preceding link in your browser of choice, expand the latest Amazon DCV Server entry (2026.0 at the time of this post), choose the Windows (x86_64) link, download the installer, and run it. The default options automatically configure firewall entries for the Amazon DCV server ports, and start the Amazon DCV server.

Instance store setup

Finally, consider setting up the instance store on the instance:

  1. In the search bar at the bottom of the screen, enter Computer Management and press Enter.
  2. Choose Disk Management in the left pane.
  3. If prompted, then choose OK to accept the requested formatting of the instance store. The following figure shows such a prompt.
  4. Open the context menu for the Unallocated drive and choose New Simple Volume.
  5. Accept the requests, making any wanted changes to the drive name, or assigning to a folder instead.
  6. Choose Finish and the new volume is available to use as a temporary drive.
Disk Management prompt requesting automatic formatting of the newly found instance store

Figure 3: The Disk Management tool prompting the automatic formatting of the instance store

Proceed to the next section to log in to your workstation to use Amazon DCV.

Logging into your workstation through Amazon DCV

To log in to your machine using Amazon DCV, you must first set up the client software on your local machine:

  1. On your local machine or laptop, open Amazon DCV.
  2. Expand the Amazon DCV Client entry that matches the Server version that you installed on your workstation.
  3. Choose the relevant OS for your machine or laptop, for example Windows (x86_64).
  4. Download the installer.
  5. Run the installer.

When it is installed, you must determine the public IP address of the workstation to connect to:

  1. In the Amazon EC2 Console, choose Amazon EC2 Dashboard in the top left.
  2. Choose Instances (running).
  3. Choose the my_windows_workstation EC2 instance, in the top pane of the screen.
  4. Choose the copy icon next to the Public IPv4 address in the lower pane to copy the IP address.

Finally, use the Amazon DCV client to view the workstation:

  1. Open the Amazon DCV client on your local machine.
  2. Paste the previous IP address and press Enter, then trust the connection.
  3. You will see a security warning about an untrusted certificate. This is expected behavior because Amazon DCV uses a self-signed certificate by default. For this initial setup, it is safe to accept the certificate and proceed with the connection. For a detailed explanation of why this warning appears and how to implement proper certificate security for production environments, see Amazon DCV Connection Security in Building a GPU workstation for visual effects with Amazon EC2: Part 5 – Security.
  4. In the next Username / Password entry field, choose Administrator, and the appropriate password to which you set that user.

After the connection is established, Amazon DCV opens and displays the remote workstation desktop. The following screen capture shows the running workstation, confirming that your remote session is active and the environment is ready to use. In this example, the Windows Server 2025 desktop is shown with SideFX Houdini open, displaying a rendered scene in the viewport above the Windows taskbar.

Windows Server 2025 workstation running SideFX Houdini, with a rendered scene shown above the Windows taskbar

Figure 4: A screen capture of a Windows Server 2025 based workstation

With Windows, you automatically get brush sensitivity with Wacom and Amazon DCV connectivity. You don’t need to configure additional steps for your brushes to dynamically use the tablet’s controls.

Note: If you have need for a managed, non-persistent Windows desktop where several users share one GPU instance, consider Amazon WorkSpaces Applications multi-session fleets. They’re best for lighter graphics users such as reviewers, CAD viewers and editors, rather than artists working heavily in the viewport.

Cleaning up

Finally, you must remember to turn off your instances when not in use, either through the AWS Management Console, or through other programmatic methods such as AWS CLI. You incur costs associated with a running machine, and so deactivating machines when not in use reduces your expenditure.

This can be done by choosing Terminate instance from the Instance state options of the workstation in the Amazon EC2 console. The instance shuts down and reports a Terminated state on the console, but it automatically disappears after a short time.

However, if you want to learn more about the various methods to deactivate an EC2 instance, and create a persistent machine image of your Windows instance, then refer to Part 4 of the series, Building a GPU workstation for visual effects with Amazon EC2: Part 4 – Optimizations.

Conclusion

This post has shown how Amazon EC2 instances running Windows Server 2025 can create a GPU-backed VFX workstation. Your artists can use the latest technologies to unlock efficiencies, unhampered by the lengthy hardware renewal cycles typically found in on-premises environments.

We recommend that you continue to Part 4 of our series, Building a GPU workstation for visual effects with Amazon EC2: Part 4 – Optimizations, to learn how to implement GPU optimizations and infrastructure to streamline the provisioning of workstations. Part 4 also covers essential information on security practices for Windows EC2 instances, including build-time hardening and automated patch management to protect your workstation infrastructure from evolving threats.

For more information, contact us at .

Further reading

If you want to augment a VFX workstation environment with a dedicated render farm, then consider the managed service AWS Deadline Cloud.

EC2 instance metrics, such as CPU and memory usage, are sent to Amazon CloudWatch logs, so you can access this data through the console and AWS CLI. This information helps you discern whether an EC2 instance is over- or under-provisioned for the work at hand. For example, an instance might have excessive CPU, memory, and networking resources for the current tasks and could be optimized for price. Alternatively, it might not have enough resources to cope. For more information, visit the Amazon EC2 User Guide.

Windows instances can be lengthy to boot. Amazon EC2 Fast Launch for Windows is an AWS feature for faster launching of Windows machines through smart snapshots of the AMI.

Another post shows how to programmatically configure a Windows EC2 instance as part of a domain. This is particularly useful when the instances are used ephemerally, for example as part of a fleet of instances that spool up and down for rendering, and you need methods to bind and remove instances quickly.

More advanced examples of using Amazon DCV can be found in the aws-samples dcv-samples GitHub repo. This includes methods to broker multiple users and instances. Amazon DCV Connection Gateway provides a single entry point for users to connect to multiple Amazon DCV sessions across different instances, streamlining access management and improving security through centralized authentication.

Commercial options also exist to perform artist/instance brokering, using Amazon DCV and Amazon EC2 launch templates. An example of such an offering is Leostream.

For reference, examples of setting up Amazon DCV-capable instances and using AWS CloudFormation templates as a deployment method can be found in the amazon-ec2-nice-dcv-samples GitHub repo.

A deeper analysis of bit rates and latency for Amazon DCV connections can be found in this AWS HPC Blog post.

More information about Wacom Bridge running on Amazon DCV can be found in this AWS Wacom post and re:Invent 2024 talk.


About the authors

Andy Hayes

Andy Hayes

Andy is a Senior Visual Computing Solutions Architect at AWS. With 20 years’ experience in Visual Effects and Animation, Andy has led CG divisions and departments and added supervision to film projects, earning both Annie and Visual Effects Society Awards along the way. At AWS, he partners with studios and creative technology companies across the Media & Entertainment industry, spanning VFX and animation to rendering, virtual production, and games, helping them design and scale the cloud infrastructure and workflows behind their content pipelines. Drawing on his production background, Andy bridges the worlds of artists and engineers, guiding customers as they adopt technologies like high-performance rendering, GPU compute, and increasingly AI and machine learning to solve real creative and technical challenges. He remains passionate about the marriage of art and science that leads to the creation of compelling imagery, and about aligning technology to the evolving needs of the industry.

Mike Owen

Mike Owen

Mike is a Principal Visual Computing Solutions Architect at AWS, where he has spent many years helping studios move their creative work to the cloud. He focuses on the architecture behind high-end content creation: cloud workstations, render farm scaling and storage that spans multiple sites. Mike works with studios and creative technology companies across Media & Entertainment, from VFX and animation to finishing and colour. He has pioneered the concept of “Studio in the Cloud” and helped bring industry tools such as Autodesk Flame to AWS, and has contributed to a range of AWS Thinkbox Deadline improvements. Most recently he has led development of KNFSD File Cache, an open-source NFS caching solution from AWS. It lets studios burst large render workloads into the cloud whilst their data stays on existing on-premises storage. He cares about giving artists the same seamless, colour-accurate experience in the cloud that they expect on-premises, and about removing the cost and complexity that usually come with building a global studio.

Building a GPU workstation for visual effects on Amazon EC2: Part 2 – Linux

Post Syndicated from Andy Hayes original https://aws.amazon.com/blogs/compute/building-a-gpu-workstation-for-visual-effects-on-amazon-ec2-part-2-linux/

For many years Amazon Web Services (AWS) has been guiding customers in the deployment of powerful NVIDIA GPU-backed workstations for graphics-intensive workloads. With the recent expansion of our G-series Amazon Elastic Compute Cloud (Amazon EC2) graphics-based instances, we are excited to introduce new instance families that deliver enhanced performance and capabilities. We are also refreshing our comprehensive guidance on choosing and configuring these powerful virtual workstations.

This is the second post in a five-part series, focusing on the deployment of a Rocky Linux 9.5 visual effects (VFX) workstation, using Amazon DCV as a high-performance desktop display protocol.

“Amazon DCV gives me the ability to prioritise image quality or smooth performance and playback, depending on the tasks I am performing. It adapts quickly to any monitor used and when in full screen, I forget I am operating a remote workstation.”

— Giorgio Pitino, Head of Creative Operations, Outpost VFX

Prerequisites

This post assumes that you have already configured the following:

  1. An active AWS account with increased GPU Amazon EC2 service quota limits.
  2. A valid Amazon EC2 key pair in your AWS Region of choice.
  3. A configured virtual firewall Security Group, with open TCP/UDP ports 22, 443, 8443, and 3389 limited to only your own IP address.
  4. An AWS Identity and Access Management (IAM) profile with Amazon Simple Storage Service (Amazon S3) read access, for workstation usage.

We strongly recommend that you follow Part 1 of the series, Build a GPU workstation for visual effects on Amazon EC2: Part 1 – Setup, which goes through the preceding steps in detail.

Launching the Linux workstation

The AWS Marketplace offers an official Rocky Linux 9 x86-64 Amazon Machine Image (AMI) that can be used as a base AMI for our GPU workstation.

To use this AMI, you need to subscribe to the AMI to use the updates:

  1. Log in to the AWS Management Console and navigate to the AWS Marketplace by searching for its name in the Service search bar, or by navigating:

Services > Cloud Financial Management > AWS Marketplace.

  1. Choose Discover products in the left pane.
  2. Search for “rocky linux 9 official” in the Search field.
  3. Choose Rocky Linux 9 (Official) – x86_64 to view the marketplace offering.
  4. Choose Continue to Subscribe to subscribe to this product.
This product has no actual fee: other products, such as variations of Rocky Linux with support from third parties, might charge.
  1. To continue, choose Accept Terms (if you agree).
  2. The subscription process starts, with Effective date showing pending. But when the subscription is complete (which can take a few minutes) this changes to the current date.

When the subscription process has completed, you can launch an EC2 instance from that AMI:

  1. Navigate back to the AWS Marketplace service console page and choose Manage subscriptions.
  2. Choose Launch new instance underneath the entry for Rocky Linux 9 (Official) – x86_64.
  3. Choose Launch instances.
  4. When prompted, specify your AWS Region, and choose Continue to launch through EC2.

This now opens the Amazon EC2 launch interface.

  1. Give the instance a name, for example “my_linux_workstation”.
  2. Note that the AMI has been pre-filled for you.
  3. Under Instance type, choose g6.4xlarge.
  4. Under Key pair name, open the list and choose your key pair. In the previous post we called it “ec2_test_keypair” to specify the password security credentials.
  5. Edit the Network settings to adjust security and network placement:
    • Leave the Subnet to No preference.
    • Make sure that Auto-assign public IP is set to Enable, so that you can access the instance through the internet.
    • Under the Firewall (security groups) section of the Network Settings, choose Select existing security group, and specify your security group. In the previous post we named it “ec2_test_sg” to set up the firewall for the machine.
  6. In Configuration storage, expand the default root volume from 10 GiB to 50 GiB of gp3 storage.Important: Enable encryption in the Advanced options by checking the Encrypted option and select the default AWS managed key (aws/ebs) or a customer managed AWS KMS key for enhanced control. Amazon Elastic Block Store (Amazon EBS) encryption protects your VFX project files, renders, and application data at rest. Encryption uses industry-standard AES-256 encryption and is transparent to applications with minimal performance impact.
  7. Under Advanced details, specify the role that you created for workstation usage. In the previous post we called it “basic_workstation_role”, through the IAM instance profile list, so your workstation can access additional services.
  8. Choose Launch instance.

You can now choose the link to the EC2 instance that has been launched or view it through the Instances pane in the Amazon EC2 Service page. The following screen capture shows the newly launched workstation in the EC2 Instances list, where you can confirm it is ready to use. When the “my_linux_workstation” instance is in the Running state, and the Status check is 3/3 checks passed, you can continue.

EC2 Instances console page showing my_linux_workstation running with 3 of 3 status checks passed

Figure 1: A screen capture of the EC2 Instances page, showing my_linux_workstation in a ready state

Configuring the Linux workstation

Now you proceed to configure the instance with the necessary libraries and software needed to make it a capable workstation for making 2D and 3D content. From a Mac terminal, Linux terminal, or Windows PowerShell, you can now SSH into the rocky user over the public IP address. To find the Public IPv4 address, choose the EC2 instance in the Instances pane (as shown in the preceding figure), and you should find the address located in the top/middle part of the Details section.

ssh rocky@<<PUBLIC_IP_ADDRESS>> -i <<YOUR_EC2_TEST_KEYPAIR_PEM_FILE>>

When you are logged in to the machine, you can start with password updates and general necessary packages:

# Set a password for your rocky user.
sudo passwd rocky

# Install some useful packages for later use.
sudo yum install wget nano unzip -y

# Install AWS CLI.
# https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
cd ~
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Install pip.
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user

# Make sure all packages are up-to-date.
sudo yum update -y

# Install GNOME.
sudo yum groupinstall "Server with GUI" -y

# Install kernel-devel and reboot.
sudo yum install kernel-devel -y
sudo reboot

Reconnect to the instance to disable nouveau drivers, in preparation for installing the GPU drivers:

# We need to disable the nouveau driver which conflicts with NVIDIA graphics cards.
# Make a module configuration to disable it.
sudo nano /etc/modprobe.d/nvidia_nouveau_off.conf

# Add the following lines
blacklist nouveau
options nouveau modeset=0

# CTRL-X and "y" to exit and save changes

# Ensure this file is also in usr/lib.
sudo cp /etc/modprobe.d/nvidia_nouveau_off.conf /usr/lib/modprobe.d/

# Rebuild the initramfs / boot process.
sudo dracut --force

# Set a new boot config based on the running system, and reboot.
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Reconnect to the instance to begin installing the GPU drivers:

# Firstly, you can now double check that nouveau is not present on the system.
# You should see no output from the below:
lsmod | grep nouveau

You must use AWS approved NVIDIA GPU drivers, issued through Amazon S3, not only for optimal compatibility and performance, but also to make sure that the necessary professional GRID drivers are automatically provisioned:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html

At the time of writing, these are the current steps to install the correct NVIDIA GPU drivers.

# Install supporting libraries for NVIDIA GRID driver.
sudo dnf install -y make gcc elfutils-libelf-devel libglvnd-devel kernel-devel-$(uname -r)

mkdir ~/nvidia
cd ~/nvidia

# Copy the latest GRID driver from the public AWS S3 bucket.
aws s3 cp --recursive s3://ec2-linux-nvidia-drivers/latest/ .
chmod +x NVIDIA-Linux-x86_64*.run
sudo /bin/sh ./NVIDIA-Linux-x86_64*.run

# Follow the prompts to install:
# - you do not need to install 32 bit drivers
# - you should rebuild initramfs if requested.

# Check the driver is installed.
nvidia-smi -q | head

You should get an output as follows:

==============NVSMI LOG==============

Timestamp : Wed Jan 8 13:56:10 2025
Driver Version : 550.127.05
CUDA Version : 12.4
Attached GPUs : 1
GPU 00000000:31:00.0
Product Name : NVIDIA L4

Now set up the Amazon DCV server to stream the desktop to your local machine/client.

# Installing Amazon DCV.
# You can follow more detailed / up-to-date instructions at this link:
# https://docs.aws.amazon.com/dcv/latest/adminguide/setting-up-installing-linux-server.html

# First, disable the Wayland protocol.
sudo nano /etc/gdm/custom.conf
# Uncomment the "WaylandEnable=false" line.
# CTRL-X and "y" to exit and save changes

# Configure X Server -- check that the system is configured for graphical sessions:
sudo systemctl get-default
# If "multi.target" is returned then perform the following:
sudo systemctl set-default graphical.target

# start the X server
sudo systemctl isolate graphical.target

# Update xorg.conf with multi monitor connectivity.
sudo nvidia-xconfig --preserve-busid --enable-all-gpus --connected-monitor=DFP-0,DFP-1,DFP-2,DFP-3

# Restart X-Server.
sudo systemctl isolate multi-user.target
sudo systemctl isolate graphical.target

# Download and install the Amazon DCV server
mkdir ~/amazondcv
cd ~/amazondcv

# Install the key
sudo rpm --import https://d1uj6qtbmh3dt5.cloudfront.net/NICE-GPG-KEY

# Get the latest Rocky 9 Amazon DCV server.
# Note: The Amazon DCV team provide permanent links to the latest DCV software:
# https://www.amazondcv.com/latest.html
wget https://d1uj6qtbmh3dt5.cloudfront.net/nice-dcv-el9-x86_64.tgz
tar -xvzf ./nice-dcv-el9-x86_64.tgz

# 'cd' into the extracted folder -- note this will be version named, hence the generic line below.
cd $(ls -d */|head -n 1)

# install the server rpm -- note this will be version named, hence the generic line below.
sudo yum install $(ls nice-dcv-server*.rpm|head -n 1)

# Restart the X Server.
sudo systemctl isolate multi-user.target
sudo systemctl isolate graphical.target

The Amazon DCV server uses parameters to specify configuration options. On Linux this manifests as a file on disk.

# Adjust the Amazon DCV config files to enable
# the automatic creation of a session by the "rocky" user...
sudo nano /etc/dcv/dcv.conf

# Under [session-management] add the following line:
create-session = true

# Under [session-management/automatic-console-session] add the following line:
owner = "rocky"

# CTRL-X and "y" to exit and save changes.

# Start DCV and enable it to automatically restart on boot
sudo systemctl restart dcvserver
sudo systemctl enable dcvserver

# Finally, open up the firewall to allow Amazon DCV access.
sudo firewall-cmd --add-port=8443/tcp --permanent
sudo firewall-cmd --add-port=8443/udp --permanent
sudo firewall-cmd --reload

Note: Looking to the future of Linux based workstations, DCV now supports Wayland display drivers on RHEL 10, Rocky Linux 10, CentOS 10 and Ubuntu 26.04.

Now install the Wacom drivers:

mkdir ~/wacom
cd ~/wacom

# Download the drivers from (again replacing the link below with the latest version):
# https://github.com/linuxwacom/input-wacom/releases
wget https://github.com/linuxwacom/input-wacom/releases/download/v1.3.0/input-wacom-1.3.0.tar.bz2
tar jxf input-wacom-1.3.0.tar.bz2
cd input-wacom-1.3.0
sudo su
./configure
make && make install
modprobe wacom
dracut --force

# Setup Wacom Pressure Sensitivity
nano /etc/X11/xorg.conf.d/99-wacom-pressure2k.conf

# Add the following lines:
Section "InputClass"
    Identifier "Wacom pressure compatibility"
    MatchDriver "wacom"
    Option "Pressure2K" "true"
EndSection

# CTRL-X and "y" to exit and save changes.

# Setup Wacom with Amazon DCV and Pressure Sensitivity
# Described here: https://docs.aws.amazon.com/dcv/latest/adminguide/enable-stylus.html
sudo nano /etc/X11/xorg.conf

# Add these 2 sections to the end of the file
Section "InputDevice"
    Identifier "DCV Stylus Pen"
    Driver "dcvinput"
    Option "Pressure2K" "true"
EndSection

Section "InputDevice"
    Identifier "DCV Stylus Eraser"
    Driver "dcvinput"
    Option "Pressure2K" "true"
EndSection

# Add the following inside the ServerLayout section
InputDevice "DCV Stylus Pen"
InputDevice "DCV Stylus Eraser"

# CTRL-X and "y" to exit and save changes.

# Restart X display
sudo systemctl isolate multi-user.target
sudo systemctl isolate graphical.target

The local instance store NVMe devices need mounting. For more information, visit the Amazon EC2 User Guide.

# View the volumes -- note that only our nvme0n1 drive is available.
# Our secondary instance store is not listed.
df -h

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.4G     0  7.4G   0% /dev/shm
tmpfs           3.0G  1.3M  3.0G   1% /run
efivarfs        128K  3.8K  120K   4% /sys/firmware/efi/efivars
/dev/nvme0n1p4   49G   45G  3.9G  93% /
/dev/nvme0n1p3  936M  716M  221M  77% /boot
/dev/nvme0n1p2  100M  7.0M   93M   8% /boot/efi
tmpfs           1.5G   52K  1.5G   1% /run/user/42
tmpfs           1.5G   36K  1.5G   1% /run/user/1000

# View volumes mapped but not formatted or mounted.
# Note nvme1n1 -- our instance store - is listed but not mounted.
lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0    50G  0 disk 
├─nvme0n1p1 259:2    0     2M  0 part 
├─nvme0n1p2 259:3    0   100M  0 part /boot/efi
├─nvme0n1p3 259:4    0  1000M  0 part /boot
└─nvme0n1p4 259:5    0  48.9G  0 part /
nvme1n1     259:1    0 558.8G  0 disk

# Create a file system on that device
sudo mkfs -t xfs /dev/nvme1n1

# Create a directory to mount the new device.
sudo mkdir /data

# Mount the device to your new directory.
sudo mount /dev/nvme1n1 /data

# Confirm our instance store - nvme1n1 - is now mounted.
df -h

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs            31G     0   31G   0% /dev/shm
tmpfs            13G  1.2M   13G   1% /run
efivarfs        128K  3.8K  120K   4% /sys/firmware/efi/efivars
/dev/nvme0n1p4   49G   44G  5.1G  90% /
/dev/nvme0n1p3  936M  716M  221M  77% /boot
/dev/nvme0n1p2  100M  7.0M   93M   8% /boot/efi
tmpfs           6.1G   36K  6.1G   1% /run/user/1000
/dev/nvme1n1    559G  4.0G  555G   1% /data

Finally, we show some examples of application-specific library setup. Each application potentially has its own list of necessary libraries to install, which can be referenced online. The following are some examples:

# SideFX Houdini :
# https://www.sidefx.com/Support/system-requirements/linux-package-requirements-for-houdini-205/
sudo yum install alsa-lib compat-openssl11 dbus-libs expat fontconfig glibc libatomic libevent libglvnd-glx libglvnd-opengl libICE libSM libX11 libX11-xcb libxcb libXcomposite libXcursor libXdamage libXext libXfixes libXi libxkbcommon libxkbcommon-x11 libXrandr libXrender libXScrnSaver libXt libXtst libzstd nspr nss nss-util openldap pciutils-libs tbb xcb-util xcb-util-image xcb-util-keysyms xcb-util-renderutil xcb-util-wm zlib -y

# Autodesk Maya
# https://help.autodesk.com/view/MAYAUL/2025/ENU/?guid=GUID-D2B5433C-E0D2-421B-9BD8-24FED217FD7F
sudo yum install epel-release -y
sudo yum install libnsl.x86_64 audiofile-dev compat-openssl11 gtk2 libffi libmng libglvnd-opengl libpng15 libpq libXp mesa-libGLU xcb-util-cursor xcb-util-wm xcb-util-image xcb-util-keysyms xcb-util-renderutil xorg-x11-fonts-ISO8859-1-75dpi xorg-x11-fonts-ISO8859-1-100dpi -y

And lastly, it is time to clean up.

cd ~
rm -rf ~/nvidia
rm -rf ~/amazondcv
rm -rf ~/Wacom

# Reboot before logging in
sudo reboot

Logging in to your workstation

To log in to your machine using Amazon DCV, you must set up the client software on your local machine:

  1. On your local machine/laptop, open the Amazon DCV download webpage.
  2. Expand the Amazon DCV Client entry that matches the Server version that you installed on your workstation.
  3. Choose the relevant OS for your machine/laptop, for example Windows (x86_64).
  4. Download the installer.
  5. Run the installer.

When it is installed, we need to determine the Public IP address of the workstation to connect to:

  1. In the Amazon EC2 Console, choose Amazon EC2 Dashboard.
  2. Choose Instances (running).
  3. Choose the “my_linux_workstation” EC2 instance, in the top pane of the screen.
  4. Choose the copy icon next to the Public IPv4 address in the lower pane to copy the IP address.

Finally, use the Amazon DCV client to view the workstation:

  1. Open the Amazon DCV client on your local machine.
  2. Paste the preceding IP address and press Enter, then trust the connection.
  3. You will see a security warning about an untrusted certificate. This is expected behavior because DCV uses a self-signed certificate by default. For this initial setup, it’s safe to accept the certificate and proceed with the connection. For a detailed explanation of why this warning appears and how to implement proper certificate security for production environments, see Amazon DCV Connection Security in Building a GPU workstation for visual effects with Amazon EC2: Part 5 – Security.
  4. In the next Username / Password entry field, choose rocky, and the appropriate password to which you set that user.

After the connection is established, Amazon DCV opens and displays the remote workstation desktop. The following screen capture shows the running workstation, confirming that your remote session is active and the environment is ready to use. In this example, the Rocky Linux desktop is shown with SideFX Houdini open, displaying a rendered scene in the viewport against the Rocky Linux desktop background.

Rocky Linux workstation desktop running SideFX Houdini with a rendered scene in the viewport

Figure 2: A screen capture of a Rocky Linux-based workstation

Cleaning up

Finally, you must turn off your instances when not in use, either through the AWS Management Console, or through other programmatic methods such as the AWS Command Line Interface (AWS CLI). You incur costs associated with a running machine, and so deactivating machines when not in use reduces your expenditure.

Deactivating instances can be done by choosing Terminate instance from the Instance state options of the workstation on the Amazon EC2 console. The instance shuts down and reports a “Terminated” state on the console, but it automatically disappears after a short time.

However, if you want to learn more about the various methods to deactivate an EC2 instance, and creating a persistent machine image of your Linux instance, then refer to Part 4 of the series: Building a GPU workstation for visual effects with Amazon EC2: Part 4 – Optimizations. Additionally, Part 4 will cover essential information on security practices for Linux EC2 instances, including build-time hardening and automated patch management to protect your workstation infrastructure from evolving threats.

Conclusion

In this post you have subscribed to and deployed an AWS Marketplace AMI of Rocky Linux to serve as your base image. Then, you modified it with GPU drivers, Wacom drivers, and other modules to make it a capable Linux-backed VFX workstation, using Amazon EC2 GPU accelerated G instance types.

You may also be interested in Part 3 of our series, Building a GPU workstation for visual effects with Amazon EC2: Part 3 – Windows, to learn how to create a Windows-backed VFX workstation.

We also recommend that you continue to Part 4 of our series, Building a GPU workstation for visual effects with Amazon EC2: Part 4 – Optimizations, to learn how to implement GPU optimizations and infrastructure to streamline the provisioning of workstations.

For more information contact us at .

Further reading

If you want to augment a VFX workstation environment with a dedicated render farm, then consider the managed service AWS Deadline Cloud.

EC2 instance metrics, such as CPU and memory usage, are readily sent to Amazon CloudWatch logs, so you can access this data through the console and AWS CLI. This information can be pertinent to discern if an EC2 instance is being over-provisioned or under-provisioned for the work at hand. An over-provisioned instance has excessive CPU, memory, and networking resources for the current tasks and could be optimized for price. Alternatively, an under-provisioned instance does not have enough resources to cope. For more information visit the Amazon EC2 User Guide.

More advanced examples of using Amazon DCV can be found in the aws-samples GitHub repo. This includes methods to broker multiple users and instances. Furthermore, DCV Connection Gateway provides a single entry point so you can connect to multiple DCV sessions across different instances, streamlining access management and improving security through centralized authentication.

Commercial options also exist to perform artist/instance brokering, using Amazon DCV and Amazon EC2 Launch templates. An example of such an offering is Leostream.

For reference, examples of setting up Amazon DCV capable instances, using AWS CloudFormation templates as a method of deployment, can be found in this GitHub repo.

A greater depth of analysis of bit rates and latency surrounding Amazon DCV connections can be found in this AWS HPC post.


About the authors

Andy Hayes

Andy Hayes

Andy is a Senior Visual Computing Solutions Architect at AWS. With 20 years’ experience in Visual Effects and Animation, Andy has led CG divisions and departments and added supervision to film projects, earning both Annie and Visual Effects Society Awards along the way. At AWS, he partners with studios and creative technology companies across the Media & Entertainment industry, spanning VFX and animation to rendering, virtual production, and games, helping them design and scale the cloud infrastructure and workflows behind their content pipelines. Drawing on his production background, Andy bridges the worlds of artists and engineers, guiding customers as they adopt technologies like high-performance rendering, GPU compute, and increasingly AI and machine learning to solve real creative and technical challenges. He remains passionate about the marriage of art and science that leads to the creation of compelling imagery, and about aligning technology to the evolving needs of the industry.

Mike Owen

Mike Owen

Mike is a Principal Visual Computing Solutions Architect at AWS, where he has spent many years helping studios move their creative work to the cloud. He focuses on the architecture behind high-end content creation: cloud workstations, render farm scaling and storage that spans multiple sites. Mike works with studios and creative technology companies across Media & Entertainment, from VFX and animation to finishing and colour. He has pioneered the concept of “Studio in the Cloud” and helped bring industry tools such as Autodesk Flame to AWS, and has contributed to a range of AWS Thinkbox Deadline improvements. Most recently he has led development of KNFSD File Cache, an open-source NFS caching solution from AWS. It lets studios burst large render workloads into the cloud whilst their data stays on existing on-premises storage. He cares about giving artists the same seamless, colour-accurate experience in the cloud that they expect on-premises, and about removing the cost and complexity that usually come with building a global studio.

Build a GPU workstation for visual effects on Amazon EC2: Part 1 – Setup

Post Syndicated from Andy Hayes original https://aws.amazon.com/blogs/compute/build-a-gpu-workstation-for-visual-effects-on-amazon-ec2-part-1-setup/

For many years Amazon Web Services (AWS) has been guiding users in the deployment of powerful NVIDIA GPU-backed workstations for graphics-intensive workloads. With the recent expansion of our G-series Amazon Elastic Compute Cloud (Amazon EC2) graphics-based instances, we’re excited to introduce new instance families that deliver enhanced performance and capabilities. We’re also refreshing our comprehensive guidance on choosing and configuring these powerful virtual workstations.

This is the first in a five-part series providing detailed guidance on instance types, and deployment instructions for building Windows and Linux based NVIDIA GPU powered workstation on GPU accelerated EC2 instances. In this initial post, we explore the instance types, and outline essential AWS configurations for account preparation, security, policies, and other prerequisites. Subsequent posts cover Linux deployment (Part 2), Windows deployment (Part 3), optimizations and infrastructure for standardizing workstations deployment at scale (Part 4), and security considerations (Part 5).

G instance families

The Amazon EC2 G-series instances come in multiple families, with various types and fractionalization of GPUs, memory-to-CPU ratios, and sizes of vCPU and RAM. The g6 and gr6 instances, equipped with NVIDIA L4 GPUs, deliver up to 200 percent performance improvement in graphics-intensive workloads compared to the widely-adopted g4dn family, while offering enhanced performance and cost efficiency over their g5 predecessors. The g6f and gr6f options introduce more GPU partitioning capabilities with the NVIDIA L4, so you can provision fractional GPU allocations. This supports more cost-effective scaling for workloads that do not need a full GPU, with options for half, quarter, and one-eighth GPU. Fractional instances are ideal for remote workstation applications for staff that need only partial GPU usage, as well as other tasks such as lightweight machine learning (ML) research.

The g6e instances, featuring the NVIDIA L40S GPU, go further with double the GPU memory, 2.9x faster GPU memory bandwidth, and up to 2.5 times overall performance improvement compared to g5 instances. All three families offer flexible configurations with 1, 4, or 8 GPUs, providing specialized cores optimized for GPU processing, ray tracing, and machine learning workloads, as shown in the following figure.

The latest 7th generation instances, the g7 and g7e offer greater performance in GPU speed and memory with the latest NVIDIA RTX PRO 4500 and 6000 respectively. These instances are well suited for high-end 3D and 2D workloads and artificial intelligence (AI) workloads, delivering 4.6 times the AI inference performance of previous g6 instances. With up to 700 Gbps EFA-enabled network bandwidth, g7 instances help you run graphics and data workloads more efficiently.

NVIDIA
L4
NVIDIA
L40S
NVIDIA
RTX PRO 4500
NVIDIA
RTX PRO 6000
Architecture Ada Lovelace Ada Lovelace Blackwell Blackwell
Memory 24 GB GDDR6 48 GB GDDR6 32 GB GDDR7 96 GB GDDR7
Memory Bandwith 300 GB/s 864 GB/s 896 GB/s 1790 GB/s
Cuda Cores 7680 18176 10496 24064
Tensor Cores 240 568 328 752
Ray Tracing Cores 60 140 82 188

Figure 1: The list of GPU architecture statistics for the L4, L40S, RTX PRO 4500, and RTX PRO 6000 NVIDIA GPUs

The 6th and 7th generation of G instances are available in multiple configurations to accommodate diverse GPU, CPU, memory, and networking requirements. The g6, g6f, and g7 instances maintain the traditional 1:4 ratio of vCPUs to memory (GiB) established by previous generations such as g4dn and g5. The gr6 and gr6f families introduce an enhanced 1:8 ratio option, while the g6e and g7e instances natively possess the expanded 1:8 ratio. These higher allocations suit memory-intensive 2D and 3D workloads that frequently encounter memory constraints, providing support for demanding visual computing applications.

For the non-fractional g6 and gr6 instances, the configurations of sizes are as follows:

Instance type NVIDIA L4 GPU count vCPUs Memory (GiB) EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
Single GPU g6.xlarge 1 4 16 Up to 5 Up to 10
g6.2xlarge 1 8 32 Up to 5 Up to 10
g6.4xlarge 1 16 64 8 Up to 25
g6.8xlarge 1 32 128 16 25
g6.16xlarge 1 64 256 20 25
Single GPU / Memory Biased gr6.4xlarge 1 16 128 8 Up to 25
gr6.8xlarge 1 32 256 16 25
Multi GPU g6.12xlarge 4 48 192 20 40
g6.24xlarge 4 96 384 30 50
g6.48xlarge 8 192 768 60 100

Figure 2: The list of g6 and gr6 instance types with corresponding GPU amounts, vCPUs, network, and bandwidth data

The fractional g6f and gr6f configurations are:

Memory (GiB) to vCPU ratio Instance type NVIDIA L4 GPU count NVIDIA L4 GPU RAM (GiB) vCPUs Memory (GiB) EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
4-to-1 g6f.large 1/8 3 2 8 Up to 5 Up to 10
g6f.xlarge 1/8 3 4 16 Up to 5 Up to 10
g6f.2xlarge 1/4 6 8 32 Up to 5 Up to 10
g6f.4xlarge 1/2 12 16 64 6 Up to 25
8-to-1 gr6f.4xlarge 1/2 12 16 128 8 Up to 25

Figure 3: The list of g6f and gr6f instance types with corresponding fractional GPU amounts, vCPUs, network, and bandwidth data

For the g7 instances, the configurations of sizes are as follows:

Instance type NVIDIA RTX PRO 4500 GPU count vCPUs Memory (GiB) EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
Single GPU g7.2xlarge 1 8 32 Up to 8 Up to 60
g7.4xlarge 1 16 64 8 Up to 100
g7.8xlarge 1 32 128 16 100
Multi GPU g7.12xlarge 2 48 192 20 175
g7.24xlarge 4 96 384 40 350
g7.48xlarge 8 192 768 80 700
g7.metal* 8 192 768 80 700

* Coming soon

Figure 4: The list of g7 instance types with corresponding GPU amounts, vCPUs, network, and bandwidth data

For the enhanced g6e the following instance size configurations are:

Instance type NVIDIA L40S GPU count vCPUs Memory (GiB) EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
Single GPU g6e.xlarge 1 4 32 Up to 5 Up to 20
g6e.2xlarge 1 8 64 Up to 5 Up to 20
g6e.4xlarge 1 16 128 8 20
g6e.8xlarge 1 32 256 16 25
g6e.16xlarge 1 64 512 20 35
Multi GPU g6e.12xlarge 4 48 384 20 100
g6e.24xlarge 4 96 768 30 200
g6e.48xlarge 8 192 1536 60 400

Figure 5: The list of g6e instance types with corresponding GPU amounts, vCPUs, network, and bandwidth data

Moreover, the g7e has the following instance size configurations:

Instance type NVIDIA L40S GPU count vCPUs Memory (GiB) EBS Bandwidth (Gbps) Network Bandwidth (Gbps)
Single GPU g7e.2xlarge 1 8 64 Up to 5 50
g7e.4xlarge 1 16 128 8 50
g7e.8xlarge 1 32 256 16 100
Multi GPU g7e.12xlarge 2 48 512 25 400
g7e.24xlarge 4 96 1024 50 800
g7e.48xlarge 8 192 2048 100 1600

Figure 6: The list of g7e instance types with corresponding GPU amounts, vCPUs, network, and bandwidth data

EC2 instances mount one (or more) Amazon Elastic Block Store (Amazon EBS) volumes for durable, block-level storage. These can be detached from the instance and retained independently from the lifecycle of that instance, allowing modifications of the instance itself (such as changing the size and type) without the loss of storage.

Ephemeral NVMe storage on each instance is also provided by Instance Stores, aligned for temporary scratch data that benefits from performant local I/O access, which can be used by 2D/3D applications (for example Foundry Nuke’s cache directory). In the follow-up posts we discuss setting up and configuring these instance stores to maximize efficiency. The 6th and 7th generation EC2 instances automatically have instance stores allocated, dependent on their sizes, as shown in the following figure:

Instance Type Instance store volumes
g6f.large, g6f.xlarge 1 x 100GB
g6f.2xlarge 1 x 200 GB
g6.xlarge, g6e.xlarge 1 x 250 GB
g6.2xlarge, g6e.2xlarge, g6f.4xlarge, gr6f.4xlarge 1 x 450 GB
g6.4xlarge, g6e.4xlarge, gr6.4xlarge, g7.2xlarge, g7.4xlarge 1 x 600 GB
g6.8xlarge, g6e.8xlarge, gr6.8xlarge 2 x 450 GB
g6.12xlarge, g6e.12xlarge 4 x 940 GB
g6.16xlarge, g6e.16xlarge 2 x 940 GB
g6.24xlarge, g6e.24xlarge 4 x 940 GB
g6.48xlarge, g6e.48xlarge 8 x 940 GB
g7.8xlarge 1 x 950 GB
g7.12xlarge, g7e.2xlarge, g7e.4xlarge, g7e.8xlarge 1 x 1.9 TB
g7.24xlarge, g7e.12xlarge 1 x 3.8 TB
g7.48xlarge, g7.metal*, g7e.24xlarge 2 x 3.8 TB
g7e.48xlarge 4 x 3.8 TB

* Coming soon

Figure 7: The configuration of instance stores on the 6th and 7th generation EC2 instances

The 6th and 7th generation G instances represent our latest offering in GPU-powered workstations capable servers, but we still offer previous generations, notably the g5 and g4dn, giving many choices in machine types to match performance and cost needs.

You can find specifications of all our instances types, on our Amazon EC2 webpage (G instances are classed as Accelerated Computing types).

Desktop streaming with Amazon DCV

Typically, cost-effective Mini PCs or laptops are used to connect to the Amazon EC2 workstation through a desktop streaming protocol. AWS offers a high-end desktop experience through Amazon DCV. This is a performant remote display protocol capable of streaming 4k UHD resolution images over multiple monitors (up to 4), 60 frames per second, QUIC UDP support for high performance image streaming, 4:4:4 8-bit chroma sub-sampling, and stylus/touch device support for M&E workloads. Our posts show how to set up both the server and client for Amazon DCV on Windows and Linux.

The recent addition of Wacom Bridge with Amazon DCV has optimized stylus operations further (on Windows at the time of writing, Linux in beta currently). This is both in terms of real-time user feedback on the local device, mitigating the effect of any potential latency delay, but also dedicated channel support for hardware communications to provide even lower latency performance, as well as the synchronization of settings.

Workstation connectivity

When establishing connectivity between your on-premises workstation and your EC2 instance, you have three primary options, each offering different levels of security, performance, and complexity.

Internet connectivity (used in this post)

For this post, we use the most basic connection method: direct internet connectivity. This approach connects your local workstation to your Amazon EC2 instance over the public internet, with security groups (virtual firewalls) configured to restrict access to your specific IP address and only the necessary ports.

This method uses application-layer security to provide built-in encryption for all Amazon DCV streaming sessions, which helps keep your visual data protected during transmission. Security groups operate at the network and transport layers to control which IP addresses and ports can reach your instances. Together, these layers provide complementary security properties:

  • Confidentiality in transit: DCV session traffic is encrypted, which helps prevent an intermediary who intercepts the traffic from reading its contents.
  • Access control at the instance boundary: Security groups accept connections only from IP addresses on your approved list, reducing the instance’s exposure to unsolicited inbound connection attempts.
  • Reduced attack surface: Only the specific ports required for DCV are exposed, which limits how much of the instance is reachable from the network.

This method provides a straightforward setup suitable for individual users or small teams, combining quick configuration with secure encrypted streaming. We discuss the security group configuration in detail later in this post.

The following diagram illustrates this internet-based connectivity architecture. It shows how an artist’s local workstation connects across the public internet to a GPU-backed EC2 instance in AWS, with Amazon DCV carrying the session between them.

Architecture diagram of on-premises Windows and Linux workstations connecting over the public internet through Amazon DCV to GPU-backed Amazon EC2 instances in an Amazon VPC

Figure 8: A streamlined architectural diagram showing on-premises and home workstations integrating with Amazon EC2 instances, using Amazon Machine Images and EBS storage, connecting through the internet

VPN connectivity

For organizations requiring additional security beyond basic internet connectivity, AWS Site-to-Site VPN provides encrypted tunnels that use the internet securely. This method creates a secure connection between your on-premises location and your AWS resources, allowing you to access EC2 workstations through encrypted channels while maintaining the high-performance capabilities needed for graphics-intensive work.

AWS Site-to-Site VPN adds network-layer encryption on top of the application-layer protection provided by Amazon DCV. The VPN establishes encrypted IPsec tunnels between your on-premises network and AWS, so traffic is protected at the network layer before it reaches the application. In practice this means your DCV session traffic travels inside an additional encrypted tunnel alongside the rest of your network connection.

Choosing Site-to-Site VPN over direct internet connectivity comes down to a few practical factors:

  • Security posture: VPN extends your existing corporate network controls and policies into AWS, so the same governance you apply on-premises continues to apply to traffic reaching your instances. This is useful if your organization requires network-layer encryption or treats AWS as an extension of the internal network.
  • Operational complexity: VPN requires configuring and maintaining the tunnel, a customer gateway, and routing on both sides. This is more involved than direct connectivity, which needs only a security group.
  • Cost: Site-to-Site VPN carries an hourly connection charge plus data transfer.

VPN connectivity is a good fit when you need to extend your corporate network into AWS and maintain consistent security policies across both environments, and when the added setup and cost are justified by that requirement.

The following diagram shows the VPN-based connectivity architecture. The key difference from the direct-internet setup is that the EC2 instances now sit in a private subnet with no public internet access. The Site-to-Site VPN carries all traffic between your on-premises network and AWS instead.

Architecture diagram of on-premises workstations connecting through an IPsec Site-to-Site VPN to GPU-backed Amazon EC2 instances in a private subnet

Figure 9: A streamlined architectural diagram showing on-premises and home workstations integrating with Amazon EC2 instances, using Amazon Machine Images and EBS storage, connecting through a Site-to-Site VPN

AWS Direct Connect

For workloads that need consistent network performance and a connection that doesn’t traverse the public internet, AWS Direct Connect establishes a dedicated physical connection between your on-premises location and AWS. Through your chosen network provider, it creates a private link from your premises to the AWS Region containing your Amazon EC2 G instances, which can offer more predictable latency and bandwidth than an internet-based connection.

AWS Direct Connect provides additional isolation at the physical and data link layers by using a dedicated connection that doesn’t traverse the public internet. Your traffic travels over private network infrastructure between your location and AWS, which keeps it off shared internet paths and gives you more consistent, predictable network behavior than an internet-based connection. Because the connection is private and dedicated, it isn’t exposed to the general internet in the way a public connection is.

AWS Direct Connect is ideal for organizations requiring consistent network performance, enhanced security, and reduced data transfer costs when working with graphics workstations in the cloud. Combined with the application-layer encryption of Amazon DCV, this provides defense-in-depth across the entire network stack, from the physical cables through to the application data itself. This approach provides the most predictable latency and bandwidth for demanding creative workflows.

The following diagram depicts the Direct Connect connectivity architecture. As with the VPN option, the EC2 instances sit in private subnets with no need for public internet access, but here the connection runs over a dedicated Direct Connect link between your on-premises location and AWS.

Architecture diagram of on-premises workstations connecting over an AWS Direct Connect link to GPU-backed Amazon EC2 instances in a private subnet

Figure 10: A streamlined architectural diagram showing on-premises and home workstations integrating with Amazon EC2 instances, using Amazon Machine Images and EBS storage, connecting through a Direct Connect link

Consult the documentation for further details and options around VPN offerings, as well as Direct Connect, to inform your security decisions. For further information, and additional firewall and protection services that can enhance protection against exploits such as ransomware and DDoS, please see Security, Identity and Compliance on AWS.

Latency

A superior virtual workstation experience relies on low-latency and high-bandwidth connectivity from your local client to your EC2 instance. Bandwidth requirements vary based on several factors such as the number of monitors, screen resolution, frame rate, and visual quality settings. For creative professionals using Wacom tablets or similar input devices, we recommend a latency round trip of 50 ms or less when using Amazon DCV. To evaluate your connection’s suitability, there are a number of third-party websites that you can use to test latency between your location and the nearest Region, such as https://aws-latency-test.com/ or https://www.cloudping.info/.

“Our collaboration with the AWS Visual Computing team and the use of EC2 instances as virtual workstations, has exceeded our expectations, delivering exceptional result.”

— John Coyne, Senior Technical Manager, Sky TV

Workstation build overview

For the purposes of this post, we focus on deploying a g6 family instance.

The following overview outlines the complete build across the whole series. Part 1 covers the prerequisites and base image selection. The remaining phases such as the instance configuration, creating Amazon Machine Images, and setting up launch templates are covered in later parts.

  1. Prerequisites.
    • Account setup and GPU instance quotas.
    • Security configurations (access, firewalls, permissions).
  2. Base image selection.
    • Linux and Windows options.
    • Customization for VFX needs.
  3. Instance configuration.
    • Linux and Windows set up.
    • GPU and system tools installation.
  4. Creating Amazon Machine Images (AMIs) to capture the resultant machine image.
  5. Setting up EC2 Launch Templates to encapsulate all boot requirements.
    • Standardizing deployment parameters.
    • Streamlining future workstation launches.
  6. GPU optimization tips and resources.

Cost consideration

In these posts we use different AWS services, with each service having a cost associated with its usage. EC2 instances are charged by the second, so you must make sure that you shut down or terminate the instances when you are finished to avoid unwanted costs.

Accessing AWS services through the AWS Command Line Interface

AWS has many API/SDK entry points for creating and manipulating services. In this post we use the AWS Command Line Interface (AWS CLI), a convenient utility that you can use to query and set AWS service properties through the command line. In both the Windows and Linux deployment posts we set up the CLI for later convenience.

For more information about how to configure the credentials for usage with the CLI, as well as the range of functionality that you can use with it, go to the AWS CLI User Guide.

Workstation prerequisites Part 1: account setup and service quotas

The following steps are used to set up a secure Amazon Virtual Private Cloud (Amazon VPC) network platform for your instances, providing security and permissions for accessing and configuring your instance:

  1. Create an AWS account.
  2. Open and configure the AWS Management Console to gain access to AWS.
    • Go to the Console.

Setting the AWS Region

At the time of writing, AWS has 39 geographic Regions, each composed of multiple AWS Availability Zones (AZs) to make sure of service availability and durability, and over 35 AWS Local Zones (which are infrastructure deployments that place compute, storage, and other select AWS services closer to large population and industry centers). For more information, see the AWS Documentation on Global Infrastructure. As we previously mentioned, latency can be critical for optimal virtual workstation experiences. Therefore, we recommend building your workstations in a Region that offers the best latency experience. This is configured on the console itself:

  1. Choose your AWS Region that best suits your needs (a drop down of AWS Regions is available at the top right of the Console).

AWS services can vary across AWS Regions, so before proceeding you must confirm that g6 instances are available in your AWS Region of choice. This can be done through the Amazon EC2 service page, as shown in the following figure:

  1. Open the Amazon EC2 service page, the central service page for deploying EC2 instances.
    • Search for “EC2” in the Service search bar (top left), or choose Services -> Compute -> EC2.
  2. Choose Instance Types in the left pane.
  3. Enter “g6” into the Instance Types search bar and confirm that the various g6 instance types are listed as available in that AWS Region. The following image shows a filtered response from the Instance Types pane.
Amazon EC2 Instance Types console page filtered to show the available g6 instance types in a Region

Figure 11: A screen capture of the Instance Types page, reporting g6 Amazon EC2 availability in a Region

Service limits

Limits exist around many AWS services. They act as guard rails for new users and prevent misuse of AWS services. New AWS accounts start with low limits to prevent unintentional spend, so you need to check the account limits to make sure that there is enough GPU vCPU quota to launch your workstation. You can check and request limit changes through the Service Quotas service page:

  1. Navigate to the Service Quotas page by searching for its name in the Services search bar (top left of the Console window). Or alternatively choose Services -> Management and Governance -> Service Quotas.

You can choose the star icon next to each service to bookmark it for access later at the top of the Console.

  1. Under Manage quotas (top right), enter EC2 in the search field, choose Amazon Elastic Compute Cloud (Amazon EC2), and choose View quotas.
  2. In the Service quotas search field that appears, enter On-Demand G and the list of quotas should reduce to just Running On-Demand G and VT instances, the quota controlling your access to G instance machines, as shown in the following figure.
Service Quotas console showing the Running On-Demand G and VT instances quota with an applied value of 64

Figure 12: A screen capture of the Service Quotas page, requesting an increase to the GPU limit

You can see the current Applied account-level quota value, which is your limit on how many vCPUs you can use at any given time in this Region. This is not the number of instances (a common misconception)! We recommend making sure that you have at least a quota of 32, because this is enough to run two g6.4xlarge instances, a common size that suits basic 2D and 3D workflows. If the value in the applied account-level quota is below this, then you must submit a request to increase it:

  1. Choose Request Increase at account level.
  2. Enter “32” in the Increase quota value field.
  3. Choose Request.

When you receive an email confirmation of the increase, check the quota limit again for a value of 32 and you are safe to proceed with the rest of the post.

Workstation prerequisites Part 2: networking and security

When you have received your Service Quota increase, you can continue back in the AWS Management Console to start setting up some auxiliary components needed for workstations, for example security keys, firewall groups, and so on.

  1. Open the Amazon EC2 service page, the central service page for deploying EC2 instances.
    • Search for “EC2” in the Service search bar (top left) or choose Services -> Compute -> EC2.

Amazon EC2 key pair security

Amazon EC2 key pairs are security credentials that validate your identity when accessing EC2 instances.

To set up an Amazon EC2 key pair:

  1. Under Network and Security on the left side of the panel, choose Key Pairs.
  2. Choose Create key pair in the top right.
  3. Give your new key pair a name, for example ec2_test_keypair.
  4. Leave the key pair as an RSA type, with a .pem extension, and choose Create key pair.
  5. Store the automatically downloaded .pem file in a safe place, because you need it later.

You should restrict the permissions on the .pem file so that only you can read it.

On a Mac or Linux machine you can change the permission in a terminal:

chmod 400 <<YOUR_EC2_TEST_KEYPAIR_PEM_FILE>>

On Windows, open the context menu for the file, choose Properties, then the Security tab, choose Advanced, choose Disable inheritance, and remove access to all users except the current user. For more information surrounding the best use of key pairs, go to the relevant section in the Amazon EC2 User Guide.

Virtual firewalls

Security groups act as virtual firewalls, restricting incoming and outgoing traffic to your G instances. You should set up a security group working on the principle of least privilege, which means you must be very specific about which ports and which IP addresses can access it:

  1. Under Network and Security on the left pane, choose Security groups.
  2. Choose Create Security Group in the top left.
  3. Give your security group a name, for example ec2_test_sg.
  4. Give your security group a meaningful description, for example SG for basic EC2 interaction.
  5. Leave the VPC as-is. This automatically presents the ‘default’ VPC in your account and AWS Region.
  6. The following Inbound rules ports must be added to the security group to allow access to that machine from a specific client’s source IP address: in this case your local machine. Under Inbound rules, use the Add rule feature multiple times to add the definitions defined in the following figure, using the My IP drop-down under Source to automatically determine and use your current IP address, appending the /32 CIDR to limit the connectivity to that single address.The following table contains a list of TCP and UDP ports to add to your security group.
Type Protocol Port Range Source Description
SSH TCP 22 <YOUR “CUSTOM” CLIENT SOURCE IP> SSH for Linux platform.
Custom TCP Rule TCP 443 <YOUR “CUSTOM” CLIENT SOURCE IP> HTTPS for both platforms.
Custom TCP Rule TCP 8443 <YOUR “CUSTOM” CLIENT SOURCE IP> Amazon DCV for both platforms.
Custom UDP Rule UDP 8443 <YOUR “CUSTOM” CLIENT SOURCE IP> Amazon DCV for both platforms.
RDP TCP 3389 <YOUR “CUSTOM” CLIENT SOURCE IP> RDP for Windows platform.

Figure 13: The list of inbound ports that need to be opened to allow user access to the EC2 instance for the purpose of this post

For the purposes of this post, we’re making a single security group that encompasses both Linux and Windows. If you’re only focusing on a specific operating system, then a security consideration would be to remove the unneeded ports.
  1. For Outbound rules, the default allows all outbound traffic. This is enough for the instance to download packages and reach AWS service endpoints during setup, and because inbound access is already restricted to your IP, it doesn’t open an inbound path. Open outbound traffic is, however, the route that data exfiltration or a compromised host’s callbacks would use. Once your instance is running and you know which destinations it genuinely needs (for example, HTTPS 443 to AWS and your package repositories), restrict outbound rules to those destinations rather than leaving them fully open.
  2. Choose Create security group to finish making the security group.

Identity and Access Management

AWS advocates a policy of least permissions, establishing a strong base security standpoint, which necessitates that you add the permissions needed for access to services. To allow your Amazon EC2 workstation instance to access services such as Amazon Simple Storage Service (Amazon S3) (for graphics drivers), you must create a role to encompass these permissions for your instance. You should set up an AWS Identity and Access Management (IAM) profile to allow the instance access to the necessary services.

  1. Navigate to the IAM service page by entering “IAM” into the Service search field (top left), or choose Security, Identity & Compliance > IAM.
  2. Choose Roles (in the left side pane).
  3. Choose Create role, in the top right.
  4. Leave the Trusted entity type as AWS service.

You must use AWS approved NVIDIA GPU drivers, issued through Amazon S3, not only for optimal compatibility and performance, but also to make sure that the necessary professional GRID drivers are automatically provisioned. Therefore, you need a role that can read from Amazon S3. You should use a very generic Amazon S3 read-only policy to allow read access to public S3 buckets (such as the bucket on which AWS shares GPU drivers).

  1. Under Use case, enter “S3” and then choose Next (bottom right).
  2. Search for AmazonS3ReadOnlyAccess and choose the box to the left of the policy to add it to the role. Then, choose Next.
  3. Give your new role a name, for example basic_workstation_role.
  4. Choose Create role.
If later on you want to add more service features to your workstation role (for example to enable write permissions from the workstation to one of your own storage buckets), then you can use the same process to add pre-defined permissions or create your own in JSON.

You now have an account, and the resources needed to launch EC2 instances, which are covered in our next posts.

Conclusion

In this post we have discussed the different generation Amazon EC2 instance families: their memory, network, storage, and GPU options. We have outlined the steps needed not only to prepare an AWS account for running GPU workstations but also to build key infrastructure that is needed to deploy them.

We recommend that you continue to Part 2 of our series, Building a GPU workstation for visual effects on Amazon EC2: Part 2 – Linux, where you can learn how to create a Linux-backed workstation. You can also read Part 3 of our series, Building a GPU workstation for visual effects with Amazon EC2: Part 3 – Windows, to learn how to create a Windows-backed workstation.

For more information contact us at .

Further reading

If you want to augment a VFX workstation environment with a dedicated render farm, then consider the managed service AWS Deadline Cloud.

EC2 instance metrics, such as CPU and memory usage, are sent to Amazon CloudWatch, where you can access this data through the AWS Management Console and AWS CLI. This information can be pertinent to discern if an EC2 instance is being over or under provisioned for the work at hand. In other words, it has excessive CPU, memory, and networking resources for the current tasks, and it could be optimized for price, or the alternative where it has not been allocated enough resources to cope. For more information, go to the Amazon EC2 User Guide.

More advanced examples of using Amazon DCV can be found in this “aws-samples” GitHub repo. This includes methods to broker multiple users and instances. Furthermore, DCV Connection Gateway provides a single-entry point for users to connect to multiple DCV sessions across different instances, streamlining access management and improving security through centralized authentication.

Commercial options also exist to perform artist/instance brokering, using Amazon DCV and Amazon EC2 Launch templates. An example of such an offering is Leostream.

For reference, examples of setting up Amazon DCV capable instances, and using AWS CloudFormation templates as a method of deployment, can be found in this GitHub repo.

A greater depth of analysis of bit rates, and latency, surrounding Amazon DCV connections can be found in this AWS HPC post.


About the authors

Andy Hayes

Andy Hayes

Andy is a Senior Visual Computing Solutions Architect at AWS. With 20 years’ experience in Visual Effects and Animation, Andy has led CG divisions and departments and added supervision to film projects, earning both Annie and Visual Effects Society Awards along the way. At AWS, he partners with studios and creative technology companies across the Media & Entertainment industry, spanning VFX and animation to rendering, virtual production, and games, helping them design and scale the cloud infrastructure and workflows behind their content pipelines. Drawing on his production background, Andy bridges the worlds of artists and engineers, guiding customers as they adopt technologies like high-performance rendering, GPU compute, and increasingly AI and machine learning to solve real creative and technical challenges. He remains passionate about the marriage of art and science that leads to the creation of compelling imagery, and about aligning technology to the evolving needs of the industry.

Mike Owen

Mike Owen

Mike is a Principal Visual Computing Solutions Architect at AWS, where he has spent many years helping studios move their creative work to the cloud. He focuses on the architecture behind high-end content creation: cloud workstations, render farm scaling and storage that spans multiple sites. Mike works with studios and creative technology companies across Media & Entertainment, from VFX and animation to finishing and colour. He has pioneered the concept of “Studio in the Cloud” and helped bring industry tools such as Autodesk Flame to AWS, and has contributed to a range of AWS Thinkbox Deadline improvements. Most recently he has led development of KNFSD File Cache, an open-source NFS caching solution from AWS. It lets studios burst large render workloads into the cloud whilst their data stays on existing on-premises storage. He cares about giving artists the same seamless, colour-accurate experience in the cloud that they expect on-premises, and about removing the cost and complexity that usually come with building a global studio.

[$] Additional hardening for the glibc dynamic linker

Post Syndicated from daroc original https://lwn.net/Articles/1105134/

The

Node.js project
recently had a

security vulnerability
that
affected its

Brotli
decoder. The bug

was a buffer overflow
,
but the particular way that official Node.js binaries are built for Linux meant that one of the
main protections against turning a buffer overflow into remote code execution,

address space layout randomization
(ASLR) could not be turned on. If ASLR had been
enabled, the bug would still be exploitable in theory, but much more difficult
to exploit in practice. In response, the GNU C Library (glibc) is

considering
a new set of

tunable settings
to allow administrators to forbid programs that do not
support ASLR. The new settings, while a welcome hardening measure, may not
address the root problem.

Choosing QuePaxa, an unusual consensus algorithm

Post Syndicated from João Pedro Leite original https://blog.cloudflare.com/choosing-quepaxa/

Cloudflare recently announced Meerkat, an experimental distributed consensus service designed to manage our internal global control plane. It is powered by a new consensus algorithm called QuePaxa.

QuePaxa is an unusual choice; most consensus systems use well-established consensus algorithms like Raft, (Multi-)Paxos or Viewstamped Replication. These algorithms are well understood, have mature implementations, and usually work well.

Unfortunately, common consensus algorithms often suffer in wide-area networks with volatile network connectivity and latency, i.e., networks like the Internet, which is where Meerkat runs. These algorithms rely on a single leader process being available and having predictable latency. If the leader goes down, experiences higher latency than usual, or is disconnected, the entire consensus system slows down or becomes unavailable until the leader is replaced. This is unacceptable for Meerkat.

Meerkat’s global deployment environment led us to a newer consensus algorithm, QuePaxa, which promises better stability despite volatility, in large part by not relying on a leader to make progress. In this post we’ll explain why we chose QuePaxa and how the QuePaxa algorithm works.

What we’re building and why it’s unique

Meerkat is a distributed system in which replicas agree upon an ordered log of events, where each event is decided by a consensus algorithm.

Consensus algorithms allow a group of machines to agree on a single shared truth, even if some of those machines crash or disconnect. Consensus algorithms provide an important guarantee: machines never decide conflicting values. This ensures Meerkat replicas’ logs never diverge.

Meerkat uses its consensus-powered log to expose applications like a key-value store, where each entry in the log is a get or a put. Because the consensus algorithm guarantees replicas’ logs never diverge, they all construct the same key-value (KV) state. Comparable consensus-powered systems include Consul, etcd, and Apache ZooKeeper.

Unlike similar systems, however, Meerkat was designed for global scale. A Meerkat cluster may be distributed across continents and connected by links that often exhibit high and variable latency. This is an unusual setup for consensus systems, which are usually deployed in smaller, more local networks, and have strict latency requirements.

The challenges of consensus at a global scale

While all consensus algorithms ensure that replicas agree on the same value, many of them become disproportionately unavailable or slow when network conditions degrade (but machines do not crash). We have seen this happen firsthand: in 2020, the partial failure of a core Cloudflare network switch caused a consensus-powered system to degrade and the dashboard to operate 80x slower than usual, for six hours. Critically, the system lost no data. But it became unavailable for writes until the issue was resolved, which is unacceptable.

Ideally, a consensus algorithm should be able to make progress despite latency spikes and broken connections, not just replica failures. This is particularly a problem at global scale, where faults are more likely to originate in the network, and not our own servers.

Let’s examine two common traits of consensus algorithms that can lead to degradation under hostile network conditions: destructive interference and fault detection. Later, we’ll see how QuePaxa sidesteps both of these issues.

Destructive interference and leaders

Consider Paxos, the veritable grandfather of most consensus algorithms. A lot of consensus algorithms inherit the structure of Paxos – both the good and the not-so-good bits.

Leaving an egregious amount of detail out, here’s how Paxos works. There are two kinds of processes: proposers and acceptors. Abstractly, the goal of the system is for a majority of acceptors to agree on a value proposed by some proposer. There are two phases: the Prepare phase, and the Accept phase. In the Prepare phase, any proposer can send a ballot number out to a majority of acceptors. This ballot number serves as a promise – acceptors will only accept proposals with that ballot number or higher. Then, in the Accept phase, the proposer sends its proposal along with the ballot number to a majority of acceptors again. If a majority of acceptors still hold the ballot number of the proposal, the majority accepts it, and the proposer can safely decide the proposal.

Challenges arise when two or more proposers submit proposals at the same time. Specifically, another proposer can interleave its own higher ballot number in between the two phases, blocking any lower ballots. Two proposers can repeatedly block each other from progressing – effectively competing for acceptance.

When proposers compete and block each other in this way, we say that they destructively interfere. Destructive interference gets worse as more proposers propose their own ballots. The more concurrent proposers there are, the more likely it is that the system gets stuck repeatedly proposing. Even if none of the replicas have crashed or are unavailable, multiple proposers and misbehaving networks can block Paxos. Formally, we say that Paxos does not guarantee liveness (i.e., that proposals are eventually decided) without assuming that, at some point, all messages arrive within a bounded amount of time — an assumption known as partial synchrony.

To avoid multiple proposers constantly competing, most deployed consensus algorithms, like Raft, Multi-Paxos, and Viewstamped Replication, add a special leader proposer. The leader is the only replica that can make proposals. Centralizing the proposing power in one replica forces an order on concurrent requests, avoids competing proposals, and has the added benefit of getting decisions done in an optimal single network round-trip, instead of two. While the leader is healthy and the network is well-behaved, leader-based protocols are fast.

The trade-off is that the leader must remain operational and reachable. When it fails, becomes too slow, or is disconnected from the cluster, the system stalls. The only solution is for the remaining replicas to detect that the leader has failed, and elect a new one in its place.

When leader recovery is unstable

When a leader fails, the system must recover by electing a new one. Interestingly, leader recovery is often how destructive interference sneaks right back into a leader-based system.

Consider Raft. For safety reasons, Raft elects a new leader using a majority vote. A replica that suspects that the leader has failed announces itself as a candidate for leadership and requests a vote from other replicas. If more than half of the replicas agree to vote for the candidate, it becomes the new leader. If they don’t, then a new election is called, and the candidates try again.

Multiple replicas can become candidates — all they need to do is detect that the leader is down. The candidates may then all try to acquire a majority vote simultaneously. The problem is that the candidates will compete for the majority vote, which means Raft’s elections suffer from the destructive interference we just saw in Paxos: two replicas can mutually prevent each other from getting support from a majority and becoming the leader. The more elections fail, the more replicas will try to get themselves elected leader.

Even if the leader stops being faulty, elections continue to disrupt what could have been normal system functioning. This is known as a metastable failure: a failure pattern that perpetuates itself even when the initial cause of failure is removed. A two-second network outage — or a momentarily faulty network switch — can make the leader appear to be faulty and force a leader election, which can fail, and force further elections. The outage will be over in two seconds, but the election can continue for much longer as candidates continue to block each other.

Ideally, consensus algorithms wouldn’t need any recovery process, as they are complicated, prone to interference, and introduce another source of downtime.

When fault detection goes awry

The problem of destructive interference in leader elections is compounded by trigger-happy fault-detection, especially in wide-area networks. In order for leader-based consensus algorithms to choose a new leader, replicas must first detect that the leader has failed. However, if replicas detect leader-failures inaccurately, the system may spend an inordinate amount of time in recovery, not handling requests.

In a distributed system, where machines are separated over a network, fault detection will ultimately depend on expectations. A process cannot directly observe that a remote process has failed; it can only observe that an expected message has or hasn’t arrived. We need a predictable pattern of communication, such as heartbeats or acknowledgements, to tell us that a machine is healthy. If an expected message does not arrive within the expected time, we suspect a fault. The difficulty lies in managing those expectations.

In real networks, especially wide-area networks, messages don’t always arrive when predicted. There is no guaranteed upper bound on the time that it takes for messages to arrive. That’s not to say that the network is slow, or that it is unreliable — most messages arrive within the time we expect them to. Instead, sudden bursts of latency and losses in connectivity can, momentarily, and without warning, increase the time it takes for any message to traverse the network. We call such networks asynchronous.

The big issue with asynchronous networks is that if some server doesn’t reply in a given amount of time, we don’t really know if the server has failed or if the network is just slow — the two are indistinguishable over the network. Any practical fault detector has to make a guess based on how long it has been waiting compared to its expectation, and sometimes that guess will be wrong.

In leader-based algorithms, that guess is exactly what triggers leader elections. Sometimes the leader is down; sometimes its messages are late, or the connection is silently broken. The protocol cannot know which one happened before it reacts. If the guess is wrong, and a replica thinks the leader is faulty when it actually isn’t, then an election begins and the entire system must stop. Until the leader is replaced, no progress is made — and as we just saw, that can take a while.

Fault detection is fundamentally error-prone in asynchronous networks, and when the detector makes a mistake, we get downtime.

When fault detection is slow

Even when a fault detector guesses correctly, there are issues. Fault detectors react to failures by observing messages on the network, and so their response time has to scale with network latency — the detector’s timeout can’t be below the network latency. This is why Raft is commonly deployed with an election timeout of five to ten times the network round-trip time. It has to be high enough to give a chance for the leader to signal that it is alive, and avoid falsely triggering elections. The election timeout can be much lower than that, but since leader elections completely stop the system, timeouts are usually set conservatively high.

Roughly speaking, a round trip halfway across the world is measured in hundreds of milliseconds. If an election timeout is several round trips, detecting a failed coordinator can take around a second. Even when fault detection is completely accurate, it is slow, and it is slowest in high-latency global networks. Until the detector reacts, the system is down.

Why QuePaxa is the right fit

We realized that any algorithm relying on a strict leader or accurate timeouts was inevitably going to stall in a consensus system that runs over the Internet. We searched for consensus algorithms designed for asynchronous environments — protocols that didn't rely on a well-behaved network or precise timing to function. This led us to QuePaxa, published in 2023 by Tennage & Băsescu et al.

The most notable thing about QuePaxa is how it can efficiently reach consensus in a single round trip under favorable network conditions, but also gracefully degrades performance during latency spikes, connectivity issues, and server failures. QuePaxa achieves this by solving the exact problems we just discussed through three key design choices: having an optional leader, completely removing the reliance on fault detection, and designing for constructive interference between proposers.

In QuePaxa, much like in Paxos, there are two roles: proposers and recorders. Proposers submit proposals to be decided on, and recorders serve as intermediaries between proposers, recording every proposal they see.

Instead of having proposers compete to get their own proposals decided, in QuePaxa, proposers will cooperatively work towards a proposal that everyone supports. QuePaxa does this by assigning a priority to proposals, in the form of a random number. The optional leader gets an exclusive maximum priority. During execution, proposers submit their proposals to the recorders, who propagate the proposal to other active proposers. When proposers receive a proposal from a recorder with a higher priority than theirs, they adopt the new proposal as their own, i.e., begin broadcasting the higher priority proposal instead. In general, proposers in QuePaxa work together to decide on the highest priority proposal that they collectively know. This means proposers can’t lock each other out of deciding on a proposal, indefinitely.

The caveat is that it might not be possible to spread proposals around well enough to agree on a proposal, and so proposers have to try again, with each attempt constituting a round. However, as more proposers submit proposals, the more likely they are to converge on the highest priority proposal. This is what makes QuePaxa a randomized algorithm: the randomly generated priorities of non-leader proposers are likely — but not guaranteed — to lead to consensus on the highest known priority, in a round. Yet, given enough rounds, it is guaranteed that proposers agree on a proposal.

One proposer is designated the leader. The leader is the only proposer capable of deciding in a single round trip. The leader always proposes with the highest priority; if the leader knows it spread its proposal before any other proposer, then it can decide immediately. Clients aren’t forced to make requests to the leader — any proposer can propose. If the leader goes down, then clients can keep making requests to other proposers. The leader is completely optional. The leader is simply more likely to decide, than any other proposer, and do so faster, in a single round trip.

However, because the leader is the only one that can decide in a single round trip, we want to use it whenever possible. To exploit this fast path without creating a single point of failure, clients sending proposals to the system hedge their bets: they send their requests for proposals to every replica, and the non-leader replicas will wait for a small timeout, so the leader can propose and decide on its own. In our own implementation of QuePaxa, we actually deviate from the paper and have clients wait the hedge timeout instead of the replicas, to reduce the amount of messages we send over the network, but the overall result is the same.

This hedging timeout is very different from the timeouts used in Raft or Multi-Paxos. It is not fault detection, and so isn’t constrained by network latency; it is locally determined by replicas and can be as small as they want it to be. In Raft, having a five-millisecond timeout in a network with 100-millisecond latency would completely freeze the system. But in QuePaxa, a five-millisecond hedge timeout only increases the number of active proposers.

However, hedging isn’t free: the more proposers there are, the more proposals get broadcast, and the more likely it is that replicas fail to decide in a round. Instead, hedging provides a tradeoff: we can react faster to leader changes by lowering the hedge timeout, but we incur extra bandwidth costs and may take more time to decide. That’s a tradeoff we're more than willing to make.

If the QuePaxa leader goes down, we can recover as fast as we want — we’re not bound by network latency. This is how QuePaxa gets around fault detection: since anyone can propose, not just the leader, QuePaxa can proactively deal with leader failures by hedging, rather than reacting to failures.

How QuePaxa works under the hood

To understand how QuePaxa gives us all of these nice properties, let’s look at how QuePaxa actually works.  

QuePaxa works in rounds made up of four phases. If a proposal hasn’t been decided on in the last phase, a new round is started, and QuePaxa starts over. At the start of each phase, proposers broadcast their currently adopted proposal to all recorders. Once a proposer receives a reply from a majority of recorders, they advance to the next phase. Here’s the general outline of the phases:

  • Phase 0 (Propose and Adopt): Proposers randomly select a priority, send their proposals to recorders, and adopt the highest-priority proposal they receive from a majority of recorders. If it’s the first round and the leader knows its proposal reached a majority of recorders first, the leader decides immediately.
  • Phase 1 (Spread): Proposers spread their adopted proposal to a majority of recorders.
  • Phase 2 (Check and Decide): Proposers ask a majority of recorders for the highest-priority proposal seen in the previous phase. If it matches their own adopted proposal, they decide on it.
  • Phase 3 (Carry Forward): Any proposer that failed to decide prepares for the next round by adopting the best proposal they see from a majority of recorders.

Let’s walk through a couple of example executions and unpack how each phase contributes to the overall behavior we want out of QuePaxa. Each execution will have two proposers (one of which is the leader) and three recorders.

The Leader Fast Path (Phase 0): The first phase does three important things: it randomizes priorities (which it must do to circumvent the FLP impossibility theorem), checks if the leader reached a majority of support first, and helps proposers converge if the leader didn't. The leader’s proposal is not randomized, and will always have the highest possible priority.

In the example above, the leader's proposal reaches a majority of recorders before anyone else, and receives replies confirming it. The leader can safely decide right away. This might seem dangerous, but it is entirely safe thanks to a property called majority intersection: any two majorities of recorders always share at least one recorder. Because the leader reached a majority, any other proposer querying a majority will inevitably encounter the leader's proposal and adopt it. Since the leader’s proposal has the highest priority, any other proposer will adopt the leader’s proposal. It is impossible to decide on any other proposal.

Concurrent Slow Path (Phases 0-3): When there are concurrent proposers, however, it is not guaranteed that the leader will immediately decide. The leader might fail to reach a majority of recorders first, or it might receive a single reply supporting another proposer. Let's look at what happens when another proposer contests the leader.

Phase 0: Pictured below, a second proposer (shown in red) reaches the second and third recorders first. The fast path is blocked. Instead of competing, proposers adopt the highest-priority proposal they receive from a majority and move to the second phase. The two proposers could both converge to the leader’s proposal in this phase. For illustrative purposes, our example will prevent convergence by delaying the leader’s messages.  

Phase 1: Proposers spread the proposal they just adopted in Phase 0 to a majority of recorders. The recorders keep track of the highest-priority proposal they observe. It is a simple but critical step: it prevents the protocol from later deciding on a lower-priority proposal. This is once again thanks to majority intersection: any majority queried in the following phase must include at least one recorder that saw the adopted proposal, in this phase. For illustrative purposes, we prevent convergence in this phase by delaying the leader’s message to the third recorder, so the recorder does not adopt the leader’s proposal.

Phase 2: Now, a decision can happen. Each proposer determines if a higher-priority proposal gained enough support to overtake its own by asking a majority of recorders for the highest-priority proposal observed during Phase 1. If the highest proposal returned is still their adopted proposal, they decide. In our diagram, the leader sees its own proposal returned by a majority and safely decides. The other proposer receives conflicting summaries, so it cannot decide yet. Like in Phase 1, recorders keep track of the highest priority proposal they received.

Phase 3: If a proposer doesn’t decide in Phase 2, it moves to the final cleanup phase to prepare for the next round. Proposers spread their proposals again and adopt the best they find among a majority of recorders. Here, the remaining proposer discovers the decided proposal and adopts it, heading into the next round.

This is deliberate — if there was some decision in the previous phase, then a majority of the recorders will have that proposal, and any remaining proposers are guaranteed to adopt it. We see here another instance of cooperation between proposers — even when a round fails, proposers try to converge on the same proposal heading into the next round.

Moving to the next round:  After the fourth phase, proposers move on to the next round. At this point, there is no more leader fast path, and the only way to decide is by converging in the third phase. Importantly, due to proposal adoption, it becomes exponentially easier to decide as rounds progress. The QuePaxa paper predicts that each round has a 50% chance of deciding, but in practice, that probability is much higher, thanks to convergence.

Putting all four phases together, we can see where QuePaxa gets its resilience. The first phase gives the leader its one-round-trip fast path. If that fast path fails, random priorities and proposal adoption help proposers converge instead of compete. The second and third phases make it safe to decide, thanks to majority intersection, while the fourth carries what proposers learned into the next round, in case proposers don’t decide.

No proposer has to detect that the leader has failed before making progress, and a failed round doesn’t throw away the work already done. When the network behaves, QuePaxa is fast. When it doesn’t, QuePaxa keeps moving.

How QuePaxa makes Meerkat possible

In our own evaluations of Meerkat, where we had servers spanning nearly all continents, we’ve directly experienced the impact that QuePaxa has on the overall performance and availability of the system. Meerkat continuously processes requests, even when the leader crashes, disconnects, or just becomes slow. At no point have we ever seen it completely stall. This is exactly what we wanted.

The rate at which Meerkat handles requests has also been generally predictable. Most decisions — upwards of 95% — are made by the leader in the first phase, which keeps our throughput fairly stable. In the instances where a decision isn't reached in the first phase, the throughput drops momentarily, down to at most a third of the usual rate, signaling that three round trips — so three phases — were needed to decide instead of just one. The system slows down, but does not grind to a halt.

QuePaxa also allows us to confidently switch out which replica acts as leader while the cluster is running. Since the leader isn’t the exclusive driver behind decisions and proposers are cooperative, a temporarily non-optimal leader isn’t as limiting as in other algorithms, where a leader misconfiguration could stall the system completely. We can dynamically tune which proposer is the leader without worrying too much about impacting the stability of a Meerkat cluster.

What we’ll be showing next

Designing Meerkat for a global network forced us to re-evaluate how we approach distributed consensus. Instead of sticking to familiar defaults that didn't quite fit our constraints, we looked to the decades of research produced by academia and industry. That search led us to QuePaxa.

Choosing QuePaxa meant choosing a less-traveled path. It has not been battle-tested as extensively as Raft or Paxos, and we’re also extending the original crash-stop design to support crash recovery. This demands careful engineering and stronger evidence of correctness than usual systems.

That is why we are formally verifying our QuePaxa implementation and Meerkat as a whole, alongside extensive simulation and fault-injection testing. Future posts in this series will explain how we verify Meerkat and QuePaxa’s Rust implementation, how we test failures that are difficult to reproduce in production, benchmarks, and what we have learned while turning an academic protocol into a production system.

Stay on the lookout for further Meerkat updates on the Cloudflare Blog, and if you want to learn more about research at Cloudflare, have a look at our work at Cloudflare Research.

Security updates for Wednesday

Post Syndicated from jzb original https://lwn.net/Articles/1106876/

Security updates have been issued by AlmaLinux (edk2), Debian (openexr, rsyslog, and srt), Fedora (alertmanager, dotnet11.0, gdal, mstflint, nextcloud, python-crate-ingredients, rust-ingredients, rust-pore, rust-pulldown-cmark, rust-rbw, rust-sequoia-sq, rust-siguldry-fedora-autopen, tigervnc, unbound, and zchunk), Gentoo (Brotli, git, and libde265), Mageia (mopidy, python-dulwich, rust-similar3, rclone, and sofia-sip), Oracle (dpdk, edk2, firefox, glibc, kernel, kernel-uek, nodejs-nodemon, nodejs22, nodejs:24, patch, ruby, ruby4.0, squid:4, and thunderbird), Red Hat (.NET 10.0, .NET 11.0, .NET 8.0, .NET 9.0, bind, bind9.16, bind9.18, buildah, c-ares, containernetworking-plugins, delve, flatpak-builder, gcc, gcc-toolset-15-gcc, gcc-toolset-16-gcc, go-fdo-client, go-fdo-server, golang-github-openprinting-ipp-usb, grafana, grafana-pcp, gstreamer1-plugins-bad-free, gstreamer1-plugins-good, gvfs, gvisor-tap-vsock, nginx, nginx:1.24, nginx:1.26, oci-delta, podman, python-oauthlib, python3, python3.12, python3.9, qt5-qtbase, qt6-qtbase, rhc, runc, skopeo, squid, squid:4, tesseract, util-linux, xdg-dbus-proxy, and yggdrasil), Slackware (xorg-server), SUSE (aliyun-cli, amazon-cloudwatch-agent, ansible-core, aspell, azure-storage-azcopy, bind, cups, dlib-devel, docker, dolphin, dpdk, evolution, firefox, flannel, fuse-overlayfs, gnome-podcasts, google-cloud-sap-agent, GraphicsMagick, graphicsmagick, haproxy, istioctl, kernel, kubevirt, kured, libvips-devel, libXi, mozilla-nss, nvidia-open-driver-G06-signed-check, okteto, openssl-1_1, pcre2, poppler, poppler-qt, poppler-qt5, poppler, poppler-qt5, proftpd, psqlodbc, python-graphql-core, python-M2Crypto, python-pillow, python-soupsieve, python-urllib3_1, python313-wyoming, qt6-coap, rust-keylime, sbc, scribus, SDL2_image, tar, terraform-provider-susepubliccloud, the linux kernel (live patch 18 for suse linux, vsftpd, xen, xorg-x11-server, and zchunk), and Ubuntu (ca-certificates-java, gst-plugins-good1.0, gst-plugins-base1.0, hplip, linux-azure-fde, linux-nvidia, linux-nvidia-7.0, linux-nvidia-bos, linux-nvidia-lowlatency, python-tornado, rust-coreutils, and unbound).

Three People Arrested for AI-Enhanced Impersonation Fraud

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/11/three-people-arrested-for-ai-enhanced-impersonation-fraud.html

It’s hard to believe how lucrative this crime can be:

According to a press release from the United States attorney’s office for the southern district of New York, three men ­ Jacob Soall, Sharon Hazan, and Guy Sheetrit ­ were arrested for using AI to alter and impersonate the voices and images of executives and attorneys at various companies in order to convince employees to send over large payments. Soall and Hazan were arrested in Spain over the summer, while Sheetrit was arrested in the United Kingdom in September. The alleged fraud took place between 2024 and 2026.

With the help of AI, the men allegedly acted as a corporate executive, targeting employees of various companies and telling them they needed their help with sensitive financial transactions. They would then instruct the employees to get in touch with lawyers at a large international law firm. The men would then impersonate a lawyer at these firms and have the employees wire a large amount of money.

The men ultimately stole more than $129m from companies all over the world including in the US, according to the DoJ.

Always remember that there are three parts of AI security: security of AI, security with AI, and security from AI. This is the third.

По бухалките ще ги познаете. Как Радев копира санкциите на Орбан

Post Syndicated from Емилия Милчева original https://www.toest.bg/po-buhalkite-shte-gi-poznaete-kak-radev-kopira-sanktsiite-na-orban/

По бухалките ще ги познаете. Как Радев копира санкциите на Орбан

Когато една власт не може да се справи с криза, започва да търси виновници за нея – външни или вътрешни врагове, върху които да прехвърли общественото недоволство и собствената си отговорност. В България такава криза е стремглавото повишение на инфлацията до 6,1% на годишна база за октомври, което ни отрежда второ място в еврозоната по този показател.

След месеци на провалени опити да овладеят поскъпването – чрез „Кошница с грижа“ например, правителството на Румен Радев намери „виновниците“ – големите търговски вериги, всички те чуждестранни инвеститори от държави членки на Европейския съюз. 

Удобна мишена са – бизнесът им е на светло, оборотите им са огромни. Всяка касова бележка напомня колко са поскъпнали храните, а гневът на българите се пренася лесно от тях към виновното за всичко евро, а значи и ЕС, а значи и всичко европейско. 

(Разбира се, мамицата ѝ и на демокрацията, защото, нали, „една кифла беше шест стотинки при социализма“, когато през 70-те заплатите едва стигаха сто лева!) 

Така вместо отговор на въпроса защо държавата не се справя, чрез глобата, която Комисията за защита на конкуренцията (КЗК) наложи на „Кауфланд България“ (за общо 75 653 706 евро), на обществото беше посочен „виновникът“. 

Тази политическа технология не е българско изобретение. 

Сваленият от власт преди седем месеца Виктор Орбан я прилагаше години наред в Унгария, където чуждестранните търговски вериги се превърнаха в мишена на специални данъци, ценови ограничения и административни санкции. В името на „защитата на потребителите“ унгарската държава започна да определя не само колко да струват стоките, но и в какви количества да се продават, какви отстъпки да предлагат магазините и какви наличности да поддържат. През 2024 и 2026 г. Съдът на ЕС обяви част от тези ограничения за противоречащи на европейското право, а Европейската комисия започна отделни дела срещу Унгария за специалния данък върху оборота и за ограниченията на търговските надценки.

След призивите за бойкот на веригите от началото на годината, ударните проверки на магазинната и складовата им мрежа от КЗК и Комисията за защита на потребителите (КЗП) България поема по същия път.

Какво прави българската „бухалка“?

КЗК наложи безпрецедентната по размера си глоба за търговска верига не заради цените, които плащат потребителите, а заради отношенията на компанията с нейните доставчици. Комисията твърди, че през 2026 г. германската верига е извършила три нарушения на забраната за нелоялни търговски практики спрямо български производители на хранителни продукти. 

Според регулатора „Кауфланд“ е използвал по-силната си позиция при преговорите с производители, за да изисква намаляване на доставните цени. За това нарушение е наложена санкция от близо 21,9 млн. евро. 

Втората глоба, почти 31,9 млн. евро, е за недобросъвестно приспадане от доставните цени на възнаграждения за услуги, предоставяни от веригата. 

Третата, отново близо 21,9 млн. евро, е за създаване на необосновани пречки пред доставчиците, които са поискали увеличение на цените, включително чрез забавяне и удължаване на преговорите. Всяка от санкциите е изчислена като процент от общия оборот на веригата за 2025 г., а не от печалбата ѝ.

Как точно глоба от близо 76 млн. евро, наложена на търговска верига за отношенията с нейните доставчици, ще намали цените за потребителите? 

„Кауфланд“ отхвърли обвиненията. В публична позиция компанията обяви, че ще защити правата си пред компетентните български и европейски институции и ще преразгледа плановете си за инвестиции в страната ни за 200 млн. евро в следващите три години. Компанията определи размера на глобата като необоснован и несъразмерен спрямо европейската практика. Решението на КЗК подлежи на обжалване пред Административен съд – София-област в 14-дневен срок. 

Защитата на доставчиците от нелоялни практики е цел, залегнала и в европейското законодателство. Известно е, че големите търговски вериги действително разполагат със значителна сила при преговорите с малките производители. Но въпросът е дали конкретните нарушения са доказани, дали размерът на санкциите е адекватен и прилага ли регулаторът еднакви критерии към всички участници на пазара. 

Битката срещу „чужденците“ 

В Унгария на Виктор Орбан конфликтът между правителството и големите чуждестранни търговски вериги има дълга история. След завръщането на Орбан на власт през 2010 г. правителството постепенно въвежда различни форми на специално облагане и административни ограничения за големите търговци. Аргументите се променят според икономическата ситуация – защита на местния бизнес, справедливо разпределение на тежестите, овладяване на инфлацията, подкрепа за домакинствата. 

Особено показателен е специалният данък върху оборота на търговските вериги, въведен в сегашния си вид през 2020 г. За разлика от корпоративния данък върху печалбата, той се начислява върху приходите от продажби. Това е от съществено значение за бизнес като търговията с хранителни продукти, където оборотите са огромни, а нетните маржове на печалба – сравнително ниски. През 2024 г. най-високата ставка достига 4,5% от оборота. Австрийската SPAR твърди, че подобна тежест може да принуди търговците да работят на загуба дори когато продават успешно.

На пръв поглед унгарският данък е неутрален – облага оборота, а не националността на собственика. Но в действителност не е така и обяснението се крие в организацията на търговията в страната. 

Чуждестранните вериги, като SPAR, Lidl, Tesco, Aldi и Penny, обикновено оперират чрез интегрирани дружества, чиито обороти се сумират – магазините са обединени в едно дружество. За разлика от тях, много от големите унгарски търговски вериги работят на франчайзингов принцип, при който отделните магазини се управляват от самостоятелни дружества и се облагат поотделно. 

Така дори при сходен общ оборот данъчната тежест може да бъде значително по-висока за чуждестранните вериги. Според Европейската комисия това поставя въпросните компании в неблагоприятно положение и ограничава свободата им на установяване в ЕС.

На 29 април 2026 г. Европейската комисия реши да предяви иск срещу Унгария пред Съда на ЕС заради този данъчен режим. Процедурата е започнала с официално уведомително писмо през октомври 2024 г., последвано от мотивирано становище през юни 2025 г. Будапеща не приема констатациите на Брюксел. Затова спорът преминава в съдебна фаза. Съдебно решение все още няма. 

Цената на държавната намеса

Австрийската SPAR се превръща в един от най-откритите противници на унгарската политика по отношение на чуждестранните вериги. През март 2024 г. компанията подава жалба до ЕК, а австрийското правителство също поставя въпроса пред Брюксел. От компанията посочват, че унгарските мерки не само нарушават правилата на вътрешния пазар, но и поставят под съмнение възможността чуждестранните инвеститори да работят при равни условия с местните компании.

Числата, представени от компанията, са показателни – специалните данъци и задължителните ценови отстъпки са увеличили разходите ѝ с приблизително 90 млн. евро, а през 2023 г. тя е отчела загуба от близо 50 млн. евро за дейността си в Унгария. 

Правителството на Орбан защитава ограниченията като мерки срещу високите цени на храните, за чието повишение правителството обвинява големите търговски компании. Унгария действително преживява тежък инфлационен шок, като през 2023 г. поскъпването на храните става сериозен обществен проблем. Но според критиците на мерките административните ограничения не премахват причините за инфлацията, а прехвърлят част от икономическата тежест върху определени предприятия.

Унгарското правителство не спира до данъците. През 2022 г. въвежда таван на цените на основни храни, а през 2023 г. преминава към система на задължителни промоции. Големите търговци биват задължени да предлагат продукти от определени категории с намаления, които впоследствие е заложено да са поне 15% спрямо най-ниската цена за конкретната стока през предходните 30 дни. Към това се добавя и задължението за поддържане на определени наличности. 

Точно този режим доведе до делото на Penny Market, собственост на германската REWE Group. През март 2024 г. унгарските власти глобяват компанията, след като при проверка установяват, че в конкретен магазин липсват два продукта от задължителните промоционални категории – ябълки и минерални води или безалкохолни напитки. 

Penny оспорва санкцията, като посочва забавена доставка и наличието на заместващ продукт. Унгарският съд отправя преюдициално запитване¹ до Съда на ЕС за съвместимостта на режима с европейското право.

На 18 юни 2026 г. Съдът на ЕС постановява, че унгарските правила за задължителните намаления противоречат на европейското право. Според Съда те ограничават свободата на търговците да определят цените и количествата въз основа на икономически съображения, без да е представена достатъчна обосновка за това ограничение. 

Решението е важно, защото показва докъде е допустима държавната намеса.

В съдебното производство се появява и друг съществен детайл. Според изложените пред Съда на ЕС данни всички търговци с чуждестранна собственост са попадали в обхвата на оспорваните изисквания, докато унгарските компании с оборот под определения праг са били изключени. 

И Lidl печели дела срещу унгарската КЗК

В България Kaufland тепърва започва съдебната битка. Lidl, която принадлежи към същата група Schwarz като Kaufland, води съдебни спорове с унгарския орган за защита на конкуренцията Gazdasági Versenyhivatal (GVH).

За разлика от делата на SPAR и Penny, които са свързани с правителственото регулиране на цените, тук става дума за решения на самия антимонополен регулатор. През 2026 г. унгарските съдилища постановяват две решения, благоприятни за Lidl, макар и с различен обхват.

На 17 февруари 2026 г. Върховният съд на Унгария (Kúria) отменя решение на GVH по дело VJ-19/2023 и разпорежда на регулатора да проведе ново производство. Съдът променя предходното решение на първоинстанционния съд и задължава антимонополния орган да плати на Lidl 1,5 млн. форинта съдебни разноски (близо 4132 евро).

Решението не подлежи на ново обжалване. То показва, че съдебният контрол върху действията на регулатора може да доведе не само до намаляване на санкция, но и до цялостна отмяна на административния акт. Веднага след това обаче унгарският регулатор започва ново производство срещу Lidl…

В друго дело – VJ-1/2024, Столичният съд на Будапеща отменя частта от решение на GVH, в която Lidl е санкционирана със 186 млн. форинта (512 346,30 евро). Съдът връща тази част от преписката за ново разглеждане, но отхвърля останалите искания на компанията. Следователно не става дума за пълно оправдаване на търговската верига, а за съдебна отмяна на конкретната имуществена санкция. Решението е публикувано на сайта на унгарския регулатор на 4 ноември 2026 г., само дни преди българската КЗК да обяви глобата срещу Kaufland.

Сравнението е показателно и заради разликата в мащаба. 

В унгарския случай оспорената санкция е половин милион евро, докато в България КЗК налага на Kaufland общо близо 76 млн. евро. Макар размерите да не могат да се сравняват механично, тъй като нарушенията, законовите основания и методиките за определяне на санкциите са различни, въпросът е един и същ: какви са гаранциите, че регулаторът не е бухалка на властта?

Съдебните спорове на Lidl в Унгария обаче не се изчерпват с решенията на антимонополния регулатор. На 14 януари 2026 г. до Съда на ЕС е отправено преюдициално запитване по дело Lidl Magyarország, C-10/26, свързано с наложена на компанията санкция. Унгарските власти са задължили търговците да продават определени хранителни продукти при регулирани цени и да поддържат количества, съответстващи на среднодневните продажби през референтната година. 

Мерките са мотивирани с необходимостта да бъдат защитени унгарските семейства от поскъпването заради войната в Украйна. Съдът в Сегед пита дали подобна намеса е съвместима с европейското право и дали при незаконосъобразно наложена глоба държавата дължи и лихви върху възстановената сума. Делото все още е висящо.

Брюксел срещу унгарските ограничения

През 2026 г. Европейската комисия предприе още една стъпка срещу унгарската политика в търговията. На 8 юли тя реши да сезира Съда на ЕС по две отделни процедури заради ограничения върху надценките: едната – за хранителните магазини, другата – за търговците на нехранителни стоки, включително дрогериите. Според Комисията ограниченията засягат предимно чуждестранни компании, а допустимите надценки са толкова ниски, че не позволяват на търговците да покриват разходите си.

Брюксел оспорва и начина, по който унгарското правителство представя разликата между доставната и продажната цена. Според Комисията тази разлика не е равнозначна на печалба, тъй като от нея трябва да се покриват заплати, наеми, логистика, данъци и други разходи. Възможно е търговецът да има положителна надценка върху отделен продукт и въпреки това да продава на загуба, след като се отчетат всички разходи. Комисията смята, че унгарските ограничения са непропорционални и могат да нарушават свободата на установяване и правилата на Директивата за услугите.

В доклада си за Унгария от 2026 г. Европейската комисия отбелязва, че чуждестранните компании продължават да понасят непропорционална данъчна тежест въпреки известното облекчаване на специалния данък през ноември 2025 г. В доклада се разглеждат и ограниченията за откриване и преустройство на големи търговски обекти. 

Според Комисията подобни мерки ограничават конкуренцията, възпират инвестициите и водят до по-високи цени в засегнатите сектори. Така политиката, която официално се представя като защита на потребителите, може да има обратен ефект в дългосрочен план.

Прилики и разлики

Независимият по закон регулатор се оказва част от обявена от правителството кампания. Девет дни преди решението на КЗК за глобата на Kaufland вицепремиерът и министър на икономиката Александър Пулев обяви по БНТ предстоящи многомилионни санкции срещу търговските вериги: 

… и доколкото разбирам, в медийното пространство ще има много сериозен анонс в близката седмица от страна на КЗК, която е образувала редица производства. 

Пулев уточни, че става дума за глоби, достигащи почти 10% от оборота, които ще имат „много сериозен структурен ефект и дисциплиниращ ефект върху целия пазар и веригата на стойността“.

В същото интервю Пулев коментира и предизборното обещание на „Прогресивна България“ за намаляване с 15% на цените на хранителните продукти.

На 9 ноември КЗК обяви новината за санкцията си. Ден по-късно Пулев вече говори за действията на регулаторите като за част от правителствената политика. 

Ние никога не сме били в някаква битка с търговските вериги. Ние сме в битка с нелоялните търговски практики и вече виждате видими резултати от цялата тази кампания на правителството.

Пулев е категоричен и по отношение на чуждестранните инвеститори: 

Няма да се огънем под силата на лобизма, работим само в интерес на българските граждани.

Председателят на КЗК Росен Карадимов твърди, че те просто спазват закона. Според него глобите са в долната част на законовия диапазон: за две от нарушенията са определени на 1,58% от оборота, а за третото – на 2,2%, при законов максимум от 10% за всяко нарушение. Той посочва като пример Полша, където санкция от 170 млн. евро срещу търговска верига впоследствие е намалена от съда на 110 млн. евро.

Карадимов отказва да коментира предупреждението на Kaufland, че може да преразгледа планираните си инвестиции в България. Според него КЗК не може да прави компромиси със закона заради нечии инвестиционни планове.

Поръчка „от най-високо място“

Докато КЗК налага многомилионни санкции, други контролни органи провеждат масирани проверки в големите търговски вериги. На 21 октомври министърът на земеделието Пламен Абровски лично присъства на проверка на Българската агенция по безопасност на храните (БАБХ), КЗП и НАП в логистичната база на германската Lidl край Равно поле. Тогава той обявява началото на проверки във всички централни складове на веригите, последвани от инспекции в магазините и на борсите. Официалната цел е да се изясни ценообразуването по цялата верига на доставки.

На 10 ноември Lidl България излезе с безпрецедентно остра позиция, в която обвини държавните институции в административен тормоз. 

Според компанията само за последните две седмици на октомври в магазините ѝ са извършени 171 проверки при средно 86 месечно от началото на годината. 

Към 10 ноември остават 158 отворени проверки от различни държавни органи. Особено показателен е случаят във Видин, където един магазин е проверен осем пъти за пет седмици.

На 9 ноември инспектори на БАБХ затварят временно магазина във Видин заради нарушения при съхранението на хранителни продукти и хигиената в пекарната. Според Агенцията са установени 343 кг храни, съхранявани при неправилна температура, и 270 кг мухлясало зеле. Всички 613 кг са насочени за унищожаване. 

Lidl оспорва констатациите, като твърди, че компрометираните продукти са били отделени от продажба, а техническият проблем с хладилното оборудване е отстранен още по време на проверката. Магазинът възобновява работа на следващия ден в 12 часа, като пекарната остава временно затворена до излизане на лабораторните резултати.

Компанията определя затварянето на целия магазин като несъразмерна мярка и обявява, че ще потърси съдебна защита. 

Още по-сериозно е твърдението ѝ, че инспектори неформално са споделяли за натиск да търсят основания за затваряне на обекти по разпореждане от по-високо ниво. 

Има и още един въпрос, на който КЗК дължи отговор. На 21 май 2026 г. Комисията образува едновременно производства срещу Kaufland и „Максима България“, която управлява веригата Т-Маркет. И двете са за предполагаеми нелоялни търговски практики спрямо доставчици. На 9 ноември регулаторът обявява глоба от близо 76 млн. евро срещу Kaufland, но не съобщава за резултат по производството срещу Т-Маркет. 

Медийни публикации свързват веригата с най-богатия в Северна Македония Орце Камчев, макар според фирмените данни собственик на капитала да е нидерландското дружество PARETAS B.V., а като действителен собственик да е вписан литовецът Нериус Нума. А производството срещу нея няма как да бъде обявено нито за приключило, нито за прекратено без съответен акт.

В Унгария войната срещу чуждестранните вериги не свали цените, но доведе до съдебни дела и конфликти с Брюксел. В България властта тепърва я започва, но също ще разбере, че може да глобява търговците, но не и да санкционира инфлацията. 

1 Процедура, при която национален съд се обръща към Съда на ЕС, за да получи разяснение как трябва да се тълкува европейското право или дали даден акт на ЕС е валиден.

Език свещен, почти узаконен

Post Syndicated from Павлина Върбанова original https://www.toest.bg/ezik-sveshten-pochti-uzakonen/

Език свещен, почти узаконен

Един законопроект за българския език, внесен от депутати от „Възраждане“, разбуни духовете миналата седмица и предизвика повече емоционални, отколкото рационални реакции. Когато става въпрос за насъщния ни език, е естествено да се чувстваме силно засегнати и да изригнем с подигравки и саркастични коментари. След като тази вълна поотмина, нека да погледнем малко по-хладнокръвно на законопроекта.

Законопроект или декларация? 

Вероятно първо трябва да си зададем въпроса дали се нуждаем от закон за българския език. Как се изготвя нормативен акт за една толкова жива, многообразна и трудно контролируема материя, каквато е изобщо езикът? Вероятно точно хора, които не се занимават професионално с него, дръзват да го правят, без да си дават сметка, колко е сложно да вкараш употребата му в членове и алинеи.

Докато четях разпоредбите, не ме напускаше усещането, че авторите са сбъркали жанра и вместо закон са писали програмна декларация. Държавата „предприема всички необходими действия“, „насърчава доставчиците на медийни услуги“, „провежда целенасочена езикова политика“, „приема национални програми“, „създава необходимите условия“. Какво по-конкретно ще върши държавата обаче, така и не разбрах. Най-вече ще насърчава разни неща, включително и самия български книжовен език – поне на пет места срещнах тази дума в законопроекта и придружаващите го документи.

Всъщност има нещо по-конкретно и то е в глава четвърта, в която се регламентира създаването на „държавен стандарт за българска форма на кирилица“ и използването на български кирилски шрифтове в електронна среда. Според мен това е единственият светъл лъч в законопроекта. С разпоредбите се подкрепят дългогодишните усилия на една общност, която си поставя за цел да популяризира именно българския вариант на кирилицата. Текстовете обаче са общи и не става ясно поне кой орган ще изработи и въведе държавния стандарт.

Разбират ли вносителите езика?

Когато се съставя закон в дадена област, е нужно много добре да се познават основните понятия в нея и да се използват внимателно. Законопроектът е за българския език, но в самия текст паралелно се употребява български книжовен език и съвременен български книжовен език. Още в чл. 2 пише: „Официалният език в Република България е българският книжовен език“, което най-малкото се различава от текста на Конституцията – според нея официален е „българският език“. Какво ще стане, ако един човек, който не е достатъчно образован, напише писмо до някоя институция и в него има правописни, пунктуационни и граматични грешки, тоест не е спазил книжовноезиковите норми, не е използвал книжовния език? Той ще бъде ли санкциониран?

Неразбирането на материята, върху която се упражняват вносителите, е отбелязано и в становището на Института за български език при БАН:

Чл. 1, ал. 1, т. 2 и 3 не съответстват на същността на книжовния език като обществено явление. Езикът е естествено възникнала знакова система, която функционира в обществото. Той се развива и променя по свои закони, които могат да се анализират от специалистите, но не и да се регулират с административни мерки.

Проблемът с транслитерацията, или голямото объркване

Обществената енергия и креативност в критиката към законопроекта се изляха в постове, изображения и коментари на чл. 4, ал. 3, според който се допуска транслитерация на латиница под текст на кирилица. За да се придържаме максимално близо до законопроекта, следва да прочетем по-внимателно текста. 

Транслитерация се допуска съгласно едноименния закон, а в него е указано, че правилата за транслитерация важат за „български географски имена, имена на исторически личности, културни реалии, както и термини с български произход в съответната научна област“. Тоест няма как да бъдат транслитерирани имена на институции от типа на Народно събрание и Министерство на културата, както разясни председателят на „Възраждане“ Костадин Костадинов, и те си остават в този вид – „само на български и само на кирилица“, без превод, без транслитерация. Не сме били разбрали правилно текста на разпоредбите, затова трябваше Костадинов да обяснява какво е искал да каже авторът. 

А защо всички вкупом не сме разбрали? Може би е трябвало някой да се потруди върху формулировките на текстовете. 

Вероятно тези алинеи ще претърпят някакви промени, защото в момента са доста неясни, а и Костадинов каза, че няма да налагат диктат, и заговори за по-дребен шрифт на текста, „когато говорим за надписи на други езици и други графични системи“.

Във вида, в който са в момента, разпоредбите нищо не уреждат, а напротив – объркват. Изглежда, сякаш са включени повече с пропагандна цел, отколкото с намерението реално да издигнат авторитета на българския език и на кирилицата. Да вземем за пример Софийски университет „Св. Климент Охридски“. Според Закона за транслитерация името на патрона може да се транслитерира, но първата част – не и следователно цялото име си остава на кирилица, така ли? Обаче пък според проекта за новия закон на табела в метрото съответната станция може да бъде обозначена и в превод на английски, защото градският транспорт се ползва от туристите, посещаващи София.

Защо изобщо са нужни подобни рестрикции в XXI век и как е възможно да се мисли по този начин – че като забраняваш превод на говорения по цялото земно кълбо английски език, създаваш благоприятни условия за утвърждаването на своя език и своята азбука? Защо се затваряме за света, който идва при нас, застава пред сградата с надпис Народно събрание и се чуди: какво ли е пък това голямо здание? Защо пречим на другите да ни разберат – кои сме, какви сме, кои са институциите в центъра на столицата и къде се помещават? Забраната на паралелни преводни надписи на английски е опит отново да се затворим в черупката си, за да се предпазим от чуждо „вредно“, разбирай западно влияние. 

Още една безполезна комисия в държавата

В законопроекта се предвижда да се създаде 7-членен Консултативен съвет за българския език към Министерския съвет. Основната му дейност ще се състои в получаване, обобщаване и изготвяне на доклади, предлагане на „препоръки относно правилата за ползване на книжовния български език“, даване на становища и др. Кому е нужен подобен съвет, след като вече има Институт за български език при БАН и той е в състояние да извършва почти същата работа? Но най-вече

какъв практически резултат ще има от тези доклади, становища и препоръки, които никой не е задължен да взема предвид? 

А, видяхме го този доклад, добре, има някои хубави изводи в него. Благодарим за становището, прочетохме го, но ще си караме, както си знаем.

Основателни по-подробни критики към замислената нова комисия в държавата са отправени и от ИБЕ при БАН, и от Министерството на образованието и науката. В становището по законопроекта, подписано от министъра на образованието проф. Георги Вълчев, се посочва, че уредбата на Консултативния съвет влиза в противоречие с няколко разпоредби от действащия в момента Закон за администрацията.

Направили ли са си труда авторите да съобразят законопроекта си със съществуващото законодателство? Явно не – по-важно е било да се „защити“ нашият език.

Не ме гледай какво правя, слушай ме какво ти говоря 

Предполагам, че това искат да ни внушат вносителите – колко много са загрижени за българския език, за неговото опазване, обогатяване, както и за „правилата за официална употреба […] от публичните институции“, фигуриращи в чл. 1, ал. 1, т. 3 от потенциалния закон.

А дали самите вносители са спазили правилата на българския книжовен език? Не са –

откриваме 57 грешки в текста на законопроекта и на придружаващите го документи. 

Език свещен, почти узаконен
Кликнете върху изображението, за да прегледате целия текст

Поне основното правило за употреба на пълния член да беше уважено, но не би! Няма научаване ни от ученици, ни от депутати. И така, един урок по граматика, който е лесен и върши работа вероятно в 99% от случаите: ако членуваната дума или словосъчетанието, в което е включена, може да се замести с той, тази дума се членува с пълен член; ако не може – с кратък член. Консултативният съвет [той] избира от състава си заместник-председател. Специално в този случай употребата е типична и няма нищо сложно – подлогът (предшестван от съгласувано определение) е в началото на изречението, където обичайно си стои.

Друг тип граматична грешка е допусната при съгласуването на определения, които са определения към едно съществително, но се отнасят за различни обекти или групи от обекти: устни и писмени изрази вместо правилното устните и писмените изрази; научните и образователни среди вместо научните и образователните среди.

На някои места има излишни запетаи. В придружаващото писмо например виждаме една често срещана грешка в езиковата практика – запетая след моля, когато следва да-изречение. В случаите, в които молим някого да направи нещо или някакво действие да се осъществи, не поставяме запетая: Молим законопроектът да бъде поставен за разглеждане и гласуване… 

Забелязват се и правописни грешки. Научната област кирилометодиевистика се пише с малка буква, но пък когато названието на Института за български език се свежда до една дума, главната буква се запазва – Института. Съюзът и не бива да се бърка с притежателното местоимение ѝ – употребата и в администрацията е грешно, когато става дума за употребата на кирилицата.

Няма как да подмина една лексикална грешка в самото название на документа – Законопроект на Закон за българския език. Ето ви пример за повторение. Редно е да бъде само Проект на Закон за българския език. Възможно е авторите да са се повлияли от много по-често внасяните законопроекти за изменение и допълнение на различни закони, но там употребата на думата е оправдана, защото, след като се одобри например Законопроектът [тоест проектът на закон] за изменение и допълнение на Закона за здравното осигуряване, ще бъде приет Закон за изменение и допълнение на Закона за здравното осигуряване. 

Целта ми, разбира се, не е да правя пълен разбор на забелязаните грешки, а и няма особен смисъл от такава изчерпателност. По-важно е, че те показват немарливото отношение на вносителите към българския книжовен език и незачитането на неговите норми. И не, не ги обвинявам толкова в неграмотност, колкото в двойствено отношение към езика и неговите правила: загриженост на думи – пренебрежение на практика.

Както заяви проф. Владислав Миланов,

добре е хората, които пишат законите за езика, да познават първо законите на езика.

Подсказвам на вносителите какво бих направила аз на тяхно място. Първо щях да си кажа, че този законопроект, щом е за българския език, трябва да бъде изпипан до последната запетая. И ако не съм нито професионален редактор, нито коректор, щях да потърся услугите на специалисти, за да редактират и коригират текста.

Да осъзнаваш доколко си компетентен в дадена област, е важно и бих казала, задължително умение за един съвременен човек. Ако пък един депутат не го притежава, липсата може да стане опасна. Законопроектът на Закона за българския език е нагледен пример за това. 


Езикът може да е вкусен и извън блюдото – онзи, българският език, на който говорим от малки и на който около 24 май се кълнем в обич. А той в същността си е средство за общуване и за да ни служи добре, непрекъснато се променя. Да го погледнем в неговата динамика и да се опитаме да разберем какво става и защо, кои са движещите механизми и как те са свързани с обществените процеси. И тъй като задачата не е лека, ще го правим постепенно – на порции.

What’s New in Zabbix 8.0 LTS

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/whats-new-in-zabbix-8-0-lts/33755/

Zabbix 8.0 LTS might just be the largest Zabbix update yet. With a focus on application performance monitoring, new event processing workflows, scalability improvements, and MCP server for connecting Zabbix to your AI agents, Zabbix 8.0 stands out as a feature-rich and scalable solution for organizations of all types and sizes.

Application performance monitoring

Application performance monitoring (APM) has been one of our most requested features, and with the release of Zabbix 8.0 we’re ready to introduce it to our users.

Zabbix 8.0 LTS APM features enable Zabbix to collect and analyze metrics, logs, and traces collected using the OpenTelemetry protocol (OTLP). In our current implementation, OTLP data must be stored in a ClickHouse history backend, which can be configured as a Zabbix APM data source.

OpenTelemetry protocol data can be written to the ClickHouse source database either by a Zabbix proxy (which listens and collects incoming OTLP data and writes it to the ClickHouse database) or it can be sent directly to the ClickHouse database by the application itself using third-party plugins and solutions.

Traces, metrics, and logs can be viewed in the new APM section

Telemetry query item

Existing item types have also been expanded to support OpenTelemetry data collection. The new Telemetry query item type enables the collection of specific OpenTelemetry traces, metrics, and logs. The item supports filtering based on OpenTelemetry data fields – meaning that only the matching data is picked up by the item.

In addition, the item can be configured to collect and aggregate only specific OpenTelemetry data columns, returning a JSON with the required data. The data collection buckets can also be configured with parameters such as Time shift, Lookback limit, and Granularity.

We also offer a Generic OpenTelemetry by OTLP template which can serve as a starting point for Telemetry Query item creation and problem detection.

More detailed information regarding Telemetry query item configuration can be found in Zabbix documentation.

Telemetry query items can be used to build Zabbix dashboards

ClickHouse history backend

The ClickHouse history backend was specifically selected as the new history backend option because of the scalability and performance supported by ClickHouse.

ClickHouse can be configured as an additional history backend, meaning a regular MySQL or PostgreSQL database is still required to store Zabbix configuration. ClickHouse then can be configured to store specific types of data. There are a variety of benefits to implementing a ClickHouse history backend in your environment:

  • Column-based storage enables faster queries
  • Horizontal scaling and parallel processing by using sharding
  • In synthetic tests proven to support millions of values per second
  • ClickHouse is used as OpenTelemetry data storage

Complex event processing

Over time we’ve introduced a variety of features and made design decisions to support the eventual introduction of our new complex event processing engine. The goal of the newly introduced complex event processing features is to support a variety of event processing scenarios, such as:

  • Automated cause and symptom event ranking
  • Event deduplication
  • Event enrichment with contextual information
  • Automatic event closure and discarding

Under the hood, the complex event processing rules are based on conditions (which events to match), a time window (the window during which event processing logic will be performed) and operations (what is going to be done with matched events).

For most complex use cases where the available event matching methods might not suffice, Zabbix administrators have the option to use JavaScript-based event pattern matching.

Complex event processing offers granular event matching and correlation options

Zabbix MCP server

With the release of Zabbix 8.0 LTS, we’re introducing our official Zabbix MCP server. The MCP server can be installed from the official Zabbix repositories via the new zabbix-mcp-server package.

The MCP server contains definitions of multiple tools used to communicate with the Zabbix API:

  • Read Host, template, item and related configuration
  • Retrieve problems, maintenance and stored item history
  • Perform problem updates and manage maintenance periods

The full list of tools supported by the MCP server is available in our MCP server documentation.

MCP server can be used to enable AI data and problem analysis workflows

Ad-hoc maintenance

The existing maintenance functionality has been expanded from both the UX and the feature perspective.

Need to suppress a noisy trigger or a set of problems? Maintenance windows can now be created directly from the problem context menu. Here you can suppress not just the whole host, but also choose to suppress an individual trigger or a set of events based on their event names.

To achieve this, maintenance period configuration now has 2 new options:

  • Suppress problems generated by an individual trigger
  • Suppress problem events based on their name

Suppressing the trigger or event from the problem context menu will automatically fill in the corresponding configuration field. The fields can also be freely customized to fine-tune the maintenance window to fit your requirements.

Configurable table fields

One of the largest user experience improvements coming in Zabbix 8.0 LTS is the ability to fully customize table views in the Monitoring and Data collection sections of the Zabbix user interface:

  • Table columns can be resized and reordered
  • Existing columns can be hidden
  • Columns for individual tag values can be added
  • Tables can be extended with custom data columns

Custom text columns can be populated with built-in and user macro values in addition to static text.

With the ability to fully customize table fields, Zabbix administrators can create views tailored for individual use cases and scenarios. For example, tables can be enriched with columns containing data such as trigger threshold values, inventory information, host details, or event IDs. This way, instead of navigating across a variety of Zabbix sections, a lot of additional context information can be provided in a single table view.

Configuring the MCP server requires a few simple steps:

  • Install the zabbix-mcp-server package
  • Provide the frontend URL in the MCP server configuration file
  • Configure additional security settings in the MCP server configuration file
  • Configure Zabbix user roles and permissions and assign an API token to the user
  • Connect your AI agent to the MCP server using the configured Zabbix user

From a security standpoint, the MCP server utilizes the Zabbix API, which means the usual user roles and permissions are fully considered when working with Zabbix via the MCP server. In addition to the usual set of Zabbix user permissions, the MCP server supports defining tool Allow and Deny lists, adding another layer of optional permission logic.

Custom fields with user and built-in macro values can be added to table views

New dashboard theme

We have expanded the list of available frontend themes with a new addition – the dark blue frontend theme. The new theme can be accessed in user profile settings.

New Dark blue frontend theme is available in Zabbix 8.0 LTS

Visualization and UX/UI improvements

As always, our latest LTS release comes with a variety of improvements to UX/UI and Zabbix visualization features.

New and improved widgets:

  • A new scatter plot widget can be used to display relations between 2 metrics
  • The graph widget now supports inverting value display on the Y axis
  • Hovering over an item in the tooltip highlights its graph while dimming others
  • A geomap widget clustering parameter has been added
  • The Top items widget can now aggregate item values into a single column or row
Inverted graph data sets enable quick metric comparisons in graphs

In addition to new widget features, Zabbix 8.0 introduces multiple improvements to the Zabbix user interface:

  • We have continued with expanding inline validation to various Zabbix configuration fields
  • Inherited tags are now displayed and returned consistently across templates, hosts, items, web scenarios, and triggers
  • Suppression icons have been expanded to display whether the problem suppression was performed manually or by a maintenance period
  • General improvements to Zabbix GUI look and feel
The new Scatter plot widget can be used to display relationship between two metrics

Dashboard import/export

In addition to widget improvements, we’ve also introduced the ability to import and export dashboards. This way, anyone can share their dashboards with other Zabbix users or use this feature to migrate a dashboard between multiple Zabbix instances.

When importing a dashboard, users will also see a full list of imported dashboards, dashboard pages, and widgets. If the imported dashboard contains references to entities that are missing in the current environment, a warning will be displayed. In case of missing entities, the dashboard structure is still preserved so users can link the widgets to existing entities after the import is done.

During dashboard import all of added dashboards, pages and widgets are displayed

We’re also introducing our own public dashboard repository, where anyone can download and import official Zabbix dashboards into their own Zabbix environment.

New and updated templates

Zabbix 8.0 LTS comes packaged with a variety of new templates. The full list of templates can be found in our What’s new and upgrade notes documentation page.

In addition to new templates, we have continued expanding and improving upon our existing templates. Kubernetes administrators in particular might be interested in the reworked Kubernetes template – Kubernetes Cluster by HTTP. The old templates are retained for backward compatibility with existing configurations.

Final notes

But that’s not all – here we’ve listed only some of the more prominent features. In addition to what has been covered in this article, we have also introduced features such as:

  • New items
  • Improved performance related to various Zabbix components and data collection methods
  • Support for PostgreSQL cluster DB backends
  • Optimized Elasticsearch history backend
  • Extended native security features
  • Added the ability to import SAML certificates directly in Zabbix frontend
  • New low-level discovery features
  • And much more!

Check out the What’s new and upgrade notes documentation page to see the full list of features and improvements, and watch our Zabbix 8.0 webinar recording on YouTube to see the new updates in action.

The post What’s New in Zabbix 8.0 LTS appeared first on Zabbix Blog.

Светият синод на БТА

Post Syndicated from Дарина Сарелска original https://www.toest.bg/svetiyat-sinod-na-bta/

Светият синод на БТА

Добре че бяха президентските избори, за да си спомним, че има нещо като медия, нещо като държавно предприятие, нещо като министерство на официалната информация, нещо като пропаганден амвон, нещо, което в 2026 година прилича повече на ТАСС¹, отколкото на Reuters или AFP. Българската телеграфна агенция –

невзривеният мавзолей на медийното ни минало. 

Реформирана отвън, сложила костюма на съвременната агенционна журналистика, кръстила се за пред чужденците Bulgarian News Agency (между другото, този текст в Закона за БТА сега сигурно ще трябва да се оправя, след като управляващите се заеха да пазят кирилицата от преводачите), все пак БТА си остава вярна на своя член 1, който я дефинира като информационен институт на бюджетна издръжка. 

За разлика от БНТ и БНР, които се преоблякоха в обществени медии и чиито генерални директори от кумова срама се избират от поне-на-теория-независим-орган (СЕМ), тук генералният директор се назначава за срок от 5 години пряко от парламента по предложение на политическите партии. Което го прави нещо като министър на държавната информация с удължен мандат, ако се съди само по процедурата. 

Това даже стига, за да направи връзката медии–власт проблематична, защото е пряка, близка и причинно-следствена. Но не е само то. 

Кирил Вълчев изкара цяла предизборна кампания и беше дори избран за вицепрезидент, бидейки все още в неплатен отпуск като генерален директор на БТА. Междувременно Агенцията продължи да отразява шефа си в неговата двойна роля, произвеждайки дописки като тази, достойни за антипримери в часовете по журналистическа етика. 

Светият синод на БТА
Заглавие на статия в БТА от 4 ноември 2026 г.
Светият синод на БТА
Началото на статия в БТА от 4 ноември 2026 г.

В предизборната кампания Кирил Вълчев се запомни основно с това, че се превърна в меме с ярката си защита на неясно от кого нападнатата кирилица и с прословутия призив „на следващите президентски избори да не си говорим с арабски цифри, а да си говорим с българските букви“. В изборната нощ бетеанци се шегуваха, че те именно са големият победител от изборите, защото след шест години най-накрая ще сменят Вълчев, избиран единодушно от всяка власт. 

Аз вече не съм сигурна, че това е необходимо. Щом един човек може цяла предизборна кампания да е едновременно директор на държавната информационна агенция и кандидат за „Дондуков“ 2, а накрая и избран за вицепрезидент, без никой да се смути, защо изобщо да търсим наследник? Нека го узаконим: една длъжност на „Дондуков“ 2, една на „Цариградско шосе“. 

Може даже да му добавим и светски патриарх по съвместителство, след като Вълчев се записа и с приумицата да вкара Евангелието в новините и да задължи кореспондентите да отразяват проповедите на БПЦ като факт. Истински официален пазител на българщината и буквите. С две думи, 

Кирил на републиката (без Методий)

Кирил Вълчев не е първият преминал през въртящата се врата между медии и политика. Но определено е от ярките примери, редом с настоящата директорка на БНТ Милена Милотинова, която също направи едно кръгче по върховете между медиите и властта (беше депутатка от НДСВ, 2001–2005). И понеже всичко от един масраф се плаща – нашия, а и ретрото очевидно отново е на мода, си струва да припомним, че и двете организации имат дългогодишно минало на пропагандни апарати на социалистическата държава. 

Признанието като самопризнание

Защо прескачането на журналисти в политиката е саморазобличително и за политиката, и за журналистиката? Как журналистическото доверие се превръща в политически капитал и какво остава след това от самата журналистика – от Дарина Сарелска.

БТА като централен информационен и пропаганден институт на комунистическия Министерски съвет, а Българската телевизия като държавен монопол, чието ръководство и съдържание са под контрола на БКП. И двете – с дълга история на високоговорители на властта вместо на нейни корективи. На ретранслатори на прессъобщения и официални държавни комюникета. 

Минали са много години оттогава, но изглежда, разбирането за ролята на журналистите като прессекретарки на властта си е останало там. Поне за „прогресивното“ мнозинство на Румен Радев, което показа, че очакванията му от медиите са в най-добрите авторитарни традиции: да слушат и да препредават. 

Достоверно и в пълнота

В края на октомври правителствената информационна служба разпространи официална позиция, в която се казва, че професионалните стандарти изисквали думите на премиера да се отразяват „достоверно и в пълнота“ дори когато журналистите не ги харесват. А медиите, които съкращават и подбират информацията си, бяха обвинени в създаването на „политически наративи“. БТА, разбира се, публикува това. Без контекст що е то новина и как именно професионалният подбор на база обществен интерес и новинарска ценност отделя новините от суровата информация, която се лее в партийните Facebook страници. 

Асоциацията на европейските журналисти – България (АЕЖ – България) реагира веднага, че това издава дълбоко непознаване на журналистическата работа. Медиите не се свеждат до платформи за публикуване на прессъобщения на властта, а журналистите не са просто „стойки за микрофони“, четем в официалната позиция на професионалната организация. БТА не публикува това. 

Прочетена през Закона за БТА обаче, грешката е вярна! 

Защото онова, което АЕЖ – България с основание определя като нежурналистика, е всъщност законовото основание за съществуването на националната информационна агенция на държавна издръжка. 

Член 5 задължава БТА да разпространява безвъзмездно официалните актове и съобщения на президента, премиера, парламента, главния прокурор и практически на целия държавен апарат. Нещо повече, член 9 освобождава Агенцията от всякаква редакционна отговорност, когато прави това, стига да цитира точно и в пълнота. 

Вътрешните правила, утвърдени от Кирил Вълчев, отиват и по-далеч: съобщенията се публикуват „в пълен обем, във вида, в който са получени“. 

Ето ви го златния стандарт. Описан и прошнурован. Не трябва да се учудваме, значи, че сега премиерът Радев викна „равнис“ за всички медии по този явно доказано работещ (за него) модел. 

Good night, and good luck, motherf*ckers

Историята на американското предаване „60 минути“ е разказ за механизмите, чрез които се опитомяват медиите. И тези механизми са удивително сходни, независимо от пазара или знамето пред сградата на телевизията. Един текст в стил „Думам ти, дъще, сещай се, журналистическа снахо“ от Дарина Сарелска.

Държавен високоговорител с журналистическа карта

То така си беше, ще кажете вие. И ще сте прави – и при Борисов беше същото. Просто този е военен и командите се викат ясно и на плаца. Онзи беше пожарникар и гасеше пожарите в медиите с други достъпни му средства (опразване на столчета, държавна реклама и „размазване“ през регулаторите).

Но да се върнем на журналистиката. За разлика от устава на БТА, в нормалната агенционна журналистика прессъобщението е само източник: проверява се, поставя се в контекст и най-често не става новина само̀ по себе си. Ако не е вярно, даже става обект на разследване.

Професионалните агенции нямат нито задължението, нито лиценза да съобщават безнаказано всичко – дори неистини, само защото ги е изрекъл някой важен. Напротив, много от тях разполагат с цели екипи, чиято работа е да опровергават лъжите на важните (справка: Reuters Fact Check, AFP Proveri). 

В БТА не просто няма такива екипи. Няма и стимул за подобно разбиране и практикуване на журналистиката. Законът на практика окуражава липсата на въпроси и безкритичното цитиране. Това е удобна защита за държавния куриер, но е и вграден рефлекс за автоцензура. 

Когато не питаш, не рискуваш нищо.

Когато препечатваш, не създаваш „наративи“. Защото когато питаш, сравняваш и поставяш в контекст, може да ядосаш точно този, който ти плаща. Законът е превърнал липсата на въпрос и папагалската нежурналистика в начин на оцеляване и в бизнес модел. А за някои, като бонус – очевидно и във възможност за кариерно издигане. 

„Няма места.“ Журналистиката след журналистите

Ако се чудите какво стана с медиите в България, този текст ви е напълно достатъчен, за да си дадете отговори на много въпроси. А иначе, вие въпроси може и да си задавате, но в много медии у нас вече няма кой да ги задава – гледайте какво нещо… Защо стана така – от Дарина Сарелска.

Агенция, която никой не купува, но всички плащаме

И тук няма как в наратива да не включим малко арабски цифри (с извинение към вицепрезидента – генерален директор). 

Защото именно числата рисуват БТА като любопитен икономически феномен: куха държавна фирма, от която никой не иска да купува, но явно ни е скъпа. Толкова скъпа, че са налага данъкоплатците да плащат двойно – и на входа, и на изхода. 

Веднъж осигуряват бюджетната издръжка на радиоточката, тоест явяват се неин собственик по неволя, и втори път – клиент по неволя, защото пак с публични средства се плаща сметката на държавните институции, които са нейни законово задължени абонати. Ето това е бизнес модел! 

Явно за това говори Кирил Вълчев, когато казва, че вижда БТА като институция на съгласие и национално приобщаване и обединение. Със сигурност сме обединени в плащането на сметката. 

От началото на управлението на Кирил Вълчев държавната субсидия на Агенцията се е увеличила повече от два пъти, от около 6 млн. лв. през 2020 г. до на над 15 млн. лв. за 2025 г., докато собствените приходи между 2020 и 2025 г. са намалели от 1,258 млн. до 981 000 лв., тоест с около 22%. И дори тези скромни постъпления не са от новините, които БТА произвежда, а основно от пресконференции, архивни материали, снимки, реклама и наеми. 

За сравнение, AFP също получава от френската държава 120 млн. евро субсидия, но това е около една трета от общите ѝ приходи. Останалото идва от реални клиенти. При БТА собствените приходи са едва 5%, всичко друго е държавна помощ. 

Качествената агенционна журналистика по правило не е евтина, камо ли безплатна. Бизнес моделът ѝ е споделяне на разходите: никоя редакция не може сама да си позволи кореспондент във всяка столица и екип за всяко събитие, затова десетки медии плащат по малко за общ агенционен екип, който работи за всички. Редакцията възлага тази част от работата си на Агенцията срещу абонамент. 

Финансовата криза на традиционните медии постави на изпитание модела и на много места се наложи предоговаряне на условията. Абонамент за AFP или Associated Press можеше да достигне до 500 000 евро през 2009-та, след оптимизации сега стига до 80–100 000 евро на година. На абонат. Но медиите, включително българските, намират пари и плащат, когато виждат стойност. 

Като обществени медии БНТ и БНР са обявили публично тези абонаменти, големите частни телевизии също ги ползват. Учудващо, но за периода ноември 2024 г. – април 2025 г. излиза, че общественото радио плаща повече за български агенции, отколкото за международни, въпреки разликите в качеството и обема информация. 

Но тъй като радиото остава на практика сред малкото реални медийни клиенти на БТА, се наложи проблемът да бъде решен генерално: вместо да се конкурира за клиентите, през 2022-ра БТА стана безплатна по закон и получи допълнителна държавна гаранция за издръжката си. Това идва след дълги години мижав интерес (поради липса на качество) към бюлетините на БТА, предназначени за другите български и чуждестранни медии. 

Краят на журналистиката

Журналистиката губи не само пари и трафик, а и необходимостта да я има. Алгоритми, нюзинфлуенсъри и политици, които вече говорят директно на публиката, променят правилата на играта. Въпросът вече e не дали медиите са в криза, а дали обществото изобщо още иска журналистика. От Дарина Сарелска.

Докато лобира за реформата, през 2021 г. самият Кирил Вълчев признава пред парламентарната Комисия по културата и медиите, че дори БНТ вече се е отказала от абонамента си. От 145 акредитирани посланици в България към онзи момент едва 37 плащат за англоезичния бюлетин Daily News, а другата английска емисия има само 7 абонати, повечето на бартер. Вълчев обяснява отлива на клиенти с финансовите затруднения на медиите. Но като видите водещата страница на БТА и я сравните с тези на Reuters или Associated Press, става ясно, че причините са други. 

Ето ви малко БТА заглавия с дата 7 ноември 2026 г. 

ОБНОВЕНА Бюджетът задава визията за развитие на държавата, а нашата цел е българите да живеят по-добре и по-сигурно, заяви премиерът Радев

Вицепремиерът и министър на финансите Гълъб Донев заяви, че за 2027 г. представя „бюджет на вразумяването“ и на реалните цифри

От 2017 г. не е имало увеличение на ученическите стипендии и се предвижда да бъдат повишени с 20 процента през 2027 г., каза министър Георги Вълчев

Ученически театър и срещи на кварталната общност ще се състоят в 104-о ОУ „Захари Стоянов“

Националният етнографски музей представя изложба за азербайджанската килимарска традиция

Пожарната служба в Петрич предупреждава за повишен риск от пожари през отоплителния сезон

Творчеството на Вежди Рашидов остава значима част от българската култура, написаха от НХА

В Самоков бе отслужена панихида в памет на героите, загинали във войнитe

Има и 1550 новини годишно само за български фолклорни състави в чужбина според отчетите на самия Вълчев. Стилът „Работническо дело“ не може да се сбърка – същият безпочвен оптимизъм, същият клиширан език, същото усещане, че няма значение какво се е случило, важното е как от държавния амвон се разказва за него. Само дето годината е 2026-та, има интернет и социални мрежи и това вече не работи. 

Но продължава да ни струва пари. 

За 2026 г. за издръжката на БТА-буквите и за публикуването на прессъобщения в пълнота бюджетът ще плати близо 10 млн. евро. Отделно са платените партньорства, които държавни институции подписват с БТА – нещо като задължителен втори стълб на издръжка. Общото е, че винаги е на принципа „от единия джоб в другия“. А платецът винаги сме всички ние, данъкоплатците. 

Кой има нужда от мавзолея на „Цариградско шосе“?

Големият победител наистина е БТА и нейните държавни ръководители. Предстои да видим как ще бъде избран наследникът на Вълчев. Ще го има ли традиционния консенсус на партийни предложения като през 2021 и 2025 г., когато Вълчев беше избран без нито един против? Предложен първоначално от ГЕРБ, преизбран с гласовете на ГЕРБ, „Има такъв народ“ и БСП, а сега вицепрезидент, издигнат от днешното мнозинство. За това „единство на нацията“ говорим. Или просто за доказателството, че партиите не са имали интерес да сменят един функционер, който не им създава проблеми. 

Затова не питайте колко струва БТА. Питайте какво ни казва фактът, че никой в парламента не иска да се замисли защо я издържаме на загуба. 

1 Руската държавна информационна агенция, създадена през 1925 г. като Телеграфна агенция на Съветския съюз. След разпадането на СССР през 1992 г. е преименувана на ИТАР-ТАСС (Информационна телеграфна агенция на Русия – ТАСС), като запазва и познатата съветска абревиатура. През 2014 г. връща старото си име ТАСС.

Reduce alert fatigue by customizing threat detection features in Amazon GuardDuty

Post Syndicated from Allan Holmes original https://aws.amazon.com/blogs/security/reduce-alert-fatigue-by-customizing-threat-detection-features-in-amazon-guardduty/

The teams responsible for security today don’t have a problem with the number of alerts they receive; they have a problem turning those alerts into something truly actionable. When an analyst concludes an investigation with “this was caused by our vulnerability scanner” or “this account is configured to share Amazon Machine Images (AMIs) publicly on purpose,” the detection worked as designed, it just wasn’t applicable to the environment. When this happens continuously at scale, alert fatigue sets in. As teams sort through these events, incident response slows down or, worse, teams stop paying attention to the alerts at all.

Amazon GuardDuty is a threat detection service that continuously monitors, analyzes, and processes Amazon Web Services (AWS) data sources and logs in your AWS environment for suspicious or potentially malicious activity. GuardDuty then delivers detailed security findings to you for visibility and remediation. GuardDuty accomplishes this by analyzing independent streams of available sources. This includes logging of management events in AWS CloudTrail, information about IP traffic through Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, and Amazon Route 53 DNS query logs to name a few. These sources don’t need to be enabled or configured within your accounts for GuardDuty to use them. When GuardDuty observes activity that looks suspicious or anomalous, it produces a finding that’s a detailed security alert naming the behavior, the resources involved, and any supporting evidence. A finding is a signal to investigate, not a final verdict.

While the managed detections in GuardDuty work consistently across every AWS customer, personalizing detections for your specific architectures and environments is equally important. Over the past year, AWS has released a set of capabilities within GuardDuty that you can use to supply that missing context, most recently with the launch of Custom Detection Rules.

In this post, we walk through those capabilities in three distinct stages. These stages explore how to customize what GuardDuty detects, triage the findings that remain, and accelerate investigations. Throughout this blog post you’ll learn how to:

  • Efficiently detect anomalous behavior that shouldn’t occur in a specific account, without building detection logic yourself
  • Measure if a detection will be noisy before it generates a single finding, flooding security teams
  • Feed your own custom threat intelligence, including file hashes, into GuardDuty
  • Scope a suppression rule narrowly enough to archive expected activity while surfacing the same behavior elsewhere
  • Get an automated analysis of a finding, with a risk level and recommended actions, before the finding is opened manually

Prerequisites

For the walkthrough in this post, you must have the following prerequisites in place.

  • GuardDuty must be enabled in at least one AWS account and AWS Region. Note that these customization features aren’t currently available in all Regions, for more information, see Region-specific feature availability.
  • Your user or role must have AWS Identity and Access Management (IAM) permissions to configure these features in GuardDuty. The AmazonGuardDutyFullAccess_v2 managed policy covers the required actions.
  • If you manage multiple accounts through AWS Organizations, AWS recommends administering GuardDuty using the delegated administrator account. By using this method, you can manage lists, suppression rules, and conduct investigations from a centralized location.
  • GuardDuty Runtime Monitoring must be enabled to conduct file hash matching and to use Custom Detection Rules for Runtime Monitoring. This protection plan isn’t enabled automatically in GuardDuty.
  • An Amazon Simple Storage Service (Amazon S3) bucket that you control for storage of custom threat entity lists.
  • The AWS Command Line Interface (AWS CLI) installed.

Stage 1: How to customize what GuardDuty detects

An AMI is a pre-configured template that contains the software configurations needed to launch an Amazon Elastic Compute Cloud (Amazon EC2) instance on AWS. Sharing an AMI publicly can be a routine operation in an account whose purpose is distributing images, and a potential security event in any other account. The same might be said for disabling VPC Flow Logs, or signing in without multi-factor authentication for a legacy service account. Until the release of Custom Detection Rules, there was no way to account for these differences in GuardDuty.

Custom Detection Rules

Custom Detection Rules is a library of prebuilt, opt-in rules that AWS builds and maintains. At launch, it holds 90 rules across AWS CloudTrail management events and GuardDuty Runtime Monitoring events, producing 38 finding types mapped to 13 MITRE ATT&CK® tactics, described using vocabulary from ATT&CK and the Threat Technique Catalog for AWS. The catalog is maintained by the AWS Customer Incident Response Team (CIRT) and is based on the MITRE ATT&CK Cloud Matrix, documenting threat actor tactics, techniques, and procedures specific to AWS as observed by the AWS CIRT during customer engagements. Each entry includes detection guidance and mitigations specific to AWS environments, so a technique ID on a finding points to a public entry that your team can read.

Previously, detecting environment-specific behavior meant ingesting your own CloudTrail logs or host telemetry, normalizing them, building the logic, and maintaining that logic as APIs changed. Custom Detection Rules remove that work. You select the rules for GuardDuty to evaluate against the CloudTrail management events it already analyzes and, where Runtime Monitoring is enabled, the runtime events from the GuardDuty security agent, without the undifferentiated heavy lifting of log ingestion, normalization, and storage. Coverage is extended without adding unnecessary alerts that don’t add value to your teams.

Each rule runs in one of two modes. In dry run mode, GuardDuty evaluates the rule and publishes Amazon CloudWatch metrics showing how often an event matched the rule, but generates no findings. Dry run mode expires automatically after 14 days. In live mode, events that match generate findings, and live mode has no expiration.

Custom Detection Rules are currently available in all AWS commercial Regions and the AWS GovCloud (US) Regions. Custom Detection Rules for Runtime Monitoring are available in every region where GuardDuty Runtime Monitoring is available.

Walkthrough: Measure a rule, then turn it on

Start in dry run so you can measure the noise level before receiving a finding. This walkthrough uses the AMIExternalAccess rule, which detects when an AMI is shared with an external AWS account or made public, and applies it from the GuardDuty delegated administrator account. In an organization, the delegated administrator manages Custom Detection Rules for all member accounts through an organization configuration, and accounts that join the organization later inherit it.

Using the GuardDuty console

  1. Open the AWS Management Console for GuardDuty and choose Custom Detection Rules.

    Figure 1 – The Custom Detection Rules catalog with the detection logic for the AMIExternalAccess rule

    Figure 1. The Custom Detection Rules catalog with the detection logic for the AMIExternalAccess rule

  2. Select a rule name to see its detection logic on the Details tab. Rules are SQL conditions over CloudTrail fields, so you can review exactly what a rule matches before enabling it.
  3. Choose the rule’s Accounts tab, then under Organization configuration, choose Configure.

    Figure 2. Configuring the AMIExternalAccess rule in dry run mode

    Figure 2. Configuring the AMIExternalAccess rule in dry run mode

  4. Under Configuration strategy, select Whole organization. Under Status, select Dry run. Under Accounts, select Apply rule status to all accounts, then choose Save.
  5. Watch the CustomRuleDryRunSignalsMatched metric in CloudWatch. GuardDuty publishes the metric only when the rule matches an event, so a rule with no matching activity shows no data. Because dry run mode expires automatically after 14 days, a full dry-run period is the longest continuous sample GuardDuty collects without you re-enabling the rule. The metric data persists in CloudWatch under its standard retention after the dry run ends, so you can review it shortly before or after expiry and still base your decision on the most complete data available. Volume alone doesn’t tell you whether the activity is benign, so account for what you see before deciding. If a rule matches constantly and someone can point to the process that causes it, that account is a poor candidate for live mode. If nobody can explain the matches, investigate them as they stand, because a dry run measuring real attacker activity looks the same as one measuring routine operations.

    Figure 3. Dry-run match volume for a rule over one week, shown as hourly totals

    Figure 3. Dry-run match volume for a rule over one week, shown as hourly totals

  6. When you’re satisfied the rule is worth acting on, return to the rule’s organization configuration and change Status to Live to start generating findings. To keep the rule off in accounts where the activity is expected, such as an image-build account, choose Exclude accounts manually under Accounts and select those accounts.

Using the AWS CLI

  1. From the delegated administrator account, apply the rule in dry run to the whole organization. Omitting account lists targets every member account, including accounts that join later.
    aws guardduty create-custom-detection-rule-org-configuration \
    --rule-id ami-external-access \
    --mode DRY_RUN

  2. Review the CustomRuleDryRunSignalsMatched metric over the dry-run period, applying the same judgment described in the preceding console procedure.
  3. Move the rule to live mode by updating the existing configuration. Creating a second whole-organization configuration for the same rule fails with a conflict error. To exclude accounts where the activity is expected, add --exclude-account-ids.
    aws guardduty update-custom-detection-rule-org-configuration \
    --rule-id ami-external-access \
    --mode LIVE \
    --exclude-account-ids <image-build-account-id>

Note: In a standalone account that isn’t part of an organization, you enable a rule for that account alone. With the AWS CLI, use create-custom-detection-rule-association with the same --rule-id and --mode parameters. To change the mode later, use update-custom-detection-rule-association with the association ID. Member accounts in an organization can’t enable or change a rule themselves; the delegated administrator manages rules on their behalf. For details, see Managing Custom Detection Rules in multiple-account environments.

Threat entity lists

If your threat intelligence team already tracks known-bad infrastructure and files, you can pass those indicators to GuardDuty and generate findings when they appear in your AWS account.

GuardDuty has two list formats. An IP address list holds IP addresses only. An entity list, the newer format, can hold IP addresses, CIDR ranges, and domain names together in a single file. The difference that matters is coverage: entity list entries apply to CloudTrail, VPC Flow Logs, and Amazon Route 53 Resolver DNS query log findings, while IP address lists cover CloudTrail and VPC Flow Log findings only. If you still use IP address lists, the additional DNS coverage is a reason to move.

In addition, entity lists now also accept SHA-256 file hashes. Hashes sit alongside network indicators in the same file, and GuardDuty checks them against Runtime Monitoring. When the runtime agent sees an executable whose hash matches an active list, it generates a high severity Execution:Runtime/MaliciousFileExecuted.Custom finding and names the matching list, so your responder knows which intelligence source was used.

Note: If the entity list is changed, the list must be reactivated to include the updates. Uploading a new version of the file isn’t sufficient on its own. If this step is automated, build that reactivation step into the same process that refreshes the file.

Entity lists are currently available in all AWS commercial Regions, but not in the AWS GovCloud (US) Regions.

Stage 2: How to triage the findings that remain

Customizing detection reduces findings at the source, but the work is rarely finished there. New workloads appear, teams change how they operate, and some findings still describe activity that’s expected in your environment. You also need a precise way to route those events away from the queue your analysts work from.

A filter selects findings. A suppression rule is a filter that automatically archives a GuardDuty finding it matches. Archiving isn’t deletion: the finding is retained for 90 days and stays viewable, but the finding isn’t forwarded to AWS Security Hub, Amazon S3, Amazon Detective, or Amazon EventBridge, so it stays out of your security information and event management (SIEM) system and ticket queue.

Two changes have made that routing more customizable.

Wildcards

Filter criteria now support the Matches and NotMatches operators, where * matches any number of characters and ? matches one. Criteria used to be exact-match only, so suppressing findings from every AWS Lambda function in a service meant listing each name and updating it on every deployment. One pattern now covers the naming convention, including functions that don’t exist yet, so the rule stops going stale.

Additional property filter fields

You can now use over 1,000 supported fields from a finding as suppression criteria, including runtime process details, Kubernetes workload metadata, network connection information, and attack sequence signals. Take a vulnerability scanner in one Kubernetes namespace generating expected findings. With no namespace criterion available, you previously either suppressed the finding type more broadly than you wanted or wrote downstream logic to strip them. Now you scope the rule to that specific namespace and leave detection intact everywhere else in the cluster. This expanded field set is available in the aws partition; other partitions continue to use the console-supported fields only. See Filtering findings in GuardDuty for the full list of supported fields.

The recommended method is to assemble your criteria on the Findings page first, confirm the filtered list contains only findings you’re willing to archive, then choose Create suppression rule. The console and CLI procedures are detailed in the user guide.

Figure 4. Findings filtered by finding type and Kubernetes namespace, before creating a suppression rule

Figure 4. Findings filtered by finding type and Kubernetes namespace, before creating a suppression rule

Before you write wildcard criteria, know that they can’t be validated with ListFindings or GetFindingsStatistics, the testing workflow AWS recommends for ordinary filters. Instead, start from a filter that takes no action and watch the suppression rule metrics to see how much it would match before you rely on it. Nothing is lost if the pattern is too broad, matched findings are archived rather than deleted, so you can lift the suppression rule and return them to your queue.

Stage 3: How to accelerate investigations

What remains consumes analyst hours. To decide whether a finding is a real event, an analyst pivots into CloudTrail for the operations that identity performed, lines up timestamps, checks IPs and domains against intelligence, and looks for a change ticket that explains the work. That can take hours, and most such investigations end in expected. When one doesn’t, all that time was spent before any actual containment could begin.

GuardDuty Investigation, also called AI-powered investigations performs that correlation automatically. GuardDuty Investigation analyzes the finding alongside related activity from the past 90 days, affected resources, threat intelligence, and threat indicators, and returns a structured summary in minutes: a risk level, a confidence rating, a written summary, supporting detail, MITRE ATT&CK classification, and recommended actions including the CLI commands to carry them out.

Note: Read the confidence rating as carefully as the risk level. High risk with low confidence is a very different indicator to an analyst than high risk with high confidence.

You can investigate a single finding, an account, or a whole organization, from the console, the AWS CLI, the API, or an AWS MCP Server. To turn on AI-powered investigations, go to Settings, find AI powered investigations – Preview, and choose Enable.

Figure 5. A completed account investigation with risk level, confidence, analysis summary, and recommended actions

Figure 5. A completed account investigation with risk level, confidence, analysis summary, and recommended actions

The preview runs in 10 AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and Asia Pacific (Tokyo). It allows 10 investigations per account per day and 100 per account in total, and can only be started from an administrator account, member accounts can’t create investigations from their own accounts. The feature uses Amazon Bedrock models with Cross-Region Inference (CRIS), which automatically selects a Region within your geography to process the analysis and generate the report. Your data remains stored only in the Region where the investigation request originates, but investigation data and summary results might be processed outside that Region. Review that routing if you have data residency commitments. For GuardDuty Region-specific differences more broadly, see Region-specific feature availability in the user guide.

Putting it all together

Consider a team running containerized workloads on Amazon Elastic Kubernetes Service (Amazon EKS) across a handful of accounts in one organization.

The threat intelligence team uploads SHA-256 hashes from a recent brief as a threat entity list. Security engineering enables Custom Detection Rules in dry run mode and finds after 2 weeks that AMI sharing fires constantly in the image-build account and nowhere else, so they promote the rule to live across the organization and exclude the build account.

The team’s vulnerability scanner runs in a dedicated Kubernetes namespace, so they suppress that namespace and finding type while leaving the same behavior detected elsewhere. A second rule uses a wildcard on their Lambda naming convention, replacing a hand-maintained list.

Suppressed findings are archived instead of being sent to Security Hub, Amazon S3, EventBridge, or the SIEM, so they never reach the analyst queue. When a finding does warrant attention, an analyst starts an investigation and gets a disposition before manual work begins.

Each stage reduces the work of the next. Customizing detection means less to triage, effective triage means less to investigate, and automated investigation means the findings that reach a human get a decision faster.

Next steps

If you haven’t enabled GuardDuty, start with the 30-day free trial. Most protection plans enable automatically and are included, so you can evaluate them. Turn on Runtime Monitoring separately if you want to test hash matching or Custom Detection Rules for Runtime Monitoring..

If you’re already using GuardDuty:

  • Enable two or three Custom Detection Rules in dry run mode. Choose behavior that shouldn’t occur in specific accounts, measure across the dry-run period, then promote selectively.
  • Revisit existing suppression rules. Any rule written before the April 2026 field expansion might be broader than you intended, because the field you wanted wasn’t available then; narrow it to that field now.
  • Audit filters for wildcard opportunities. Any filter with a growing list of exact-match names is maintenance a single pattern can replace.
  • Migrate IP address lists to entity lists for the DNS query log coverage.

Conclusion

GuardDuty’s managed detections keep expanding, driven by AWS threat intelligence, security research, and customer feedback. These capabilities enable you to decide which activity is worth alerting on in which accounts, and which findings never need to be delivered to a human at all.

Learn more


Allan Holmes

Allan Holmes

Allan brings over 20 years of experience spanning security & compliance, networking, and DevOps to his current role as a Security Specialist. Giving him a uniquely holistic view of cloud security challenges. Allan holds multiple technical certifications from AWS, ISC2, CompTIA, and an MBA, enabling him to bridge deep technical expertise with business strategy. Outside of work, Allan is an avid gardener and electronics enthusiast who enjoys exploring innovative technologies hands-on.

Austin Buettner

Austin Buettner

Austin is a Senior Technical Account Manager (TAM) at AWS, where he helps enterprise companies build and operate secure, resilient workloads at scale. A former architect with over 15 years of experience in IT, he focuses on security, networking, and cloud operations, turning AWS capabilities into practical guidance customers can put to work. Outside of work, Austin enjoys golfing, hitting the gym, and spending time with his family.

Zhuo Zhang

Zhuo Zhang

Zhuo is a Technical Lead at AWS GuardDuty, where he builds the software and systems behind the service’s threat detection mission. Since 2016, he has been part of GuardDuty from concept through many expansions. Zhuo uses his background in software engineering and cybersecurity to develop the capabilities of GuardDuty. Outside of work, Zhuo enjoys trying new technologies and spending time with his family.

AWS refreshes ProcessUnity Global Risk Exchange assessment to enhance customer’s third-party due diligence

Post Syndicated from Naranjan Goklani original https://aws.amazon.com/blogs/security/aws-refreshes-processunity-global-risk-exchange-assessment-to-enhance-customers-third-party-due-diligence/

Amazon Web Services (AWS) is pleased to announce renewal of the AWS ProcessUnity Global Risk Exchange assessment. This validated assessment report helps customers perform effective cloud supplier due diligence on AWS and enhances their third-party risk management process.

What is the AWS ProcessUnity Global Risk Exchange assessment

Many customers use third-party cyber risk management (TPCRM) services such as ProcessUnity to better manage risks from their evolving third-party environments and to drive operational efficiencies. To help with such efforts, AWS has completed the ProcessUnity Global Risk Exchange assessment of its security posture. ProcessUnity’s assessors perform the review of Global Risk Exchange assessment and validate the results annually.

This assessment approach replaces static spreadsheets with a dynamic system that combines advanced analytics, threat intelligence, and risk models with vendor input. Vendor profiles continuously update as cloud provider risk levels and AWS security posture evolve, keeping risk matrices current in near real time. Additionally, customers can also use Framework View which maps AWS assessment controls to threat profiles, MITRE ATT&CK scenarios, and global compliance standards and frameworks including National Institute of Standards and Technology (NIST) 800-53, International Organization for Standardization (ISO) 27001, Payment Card Industry Data Security Standard (PCI DSS), and Health Insurance Portability and Accountability Act (HIPAA), significantly reducing customers’ third-party due diligence burden.

How can you access the assessment?

Customers can request access to the report by completing an access request form, available on the ProcessUnity Global Risk Exchange page. If you are an existing ProcessUnity Global Risk Exchange member, you can request access from within ProcessUnity Global Risk Exchange portal.

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.


Naranjan Goklani

Naranjan is an WS Compliance, Security Assurance and Privacy (CAPS) Lead for Canada. He has experience leading audits, attestations, certifications, and assessments across the Americas. Naranjan has more than 16 years of experience in technology risk management, customer security assurance, and technology audits. He previously worked in the Big 4 accounting firms and supported clients from the financial services, technology, retail, and utilities industries.

Nitro Sandboxes: Hardware isolation for every EKS pod, ECS task, and EC2 workload

Post Syndicated from Alex Graf original https://aws.amazon.com/blogs/compute/nitro-sandboxes-hardware-isolation-for-every-eks-pod-ecs-task-and-ec2-workload/

Security has been the foundation of AWS cloud computing since day one. When we built Amazon Elastic Compute Cloud (Amazon EC2), we designed it with strong isolation that keeps customer workloads and data separated from every other tenant on the same hardware. The AWS Nitro System took that further by moving virtualization, storage, and networking onto dedicated hardware, making zero operator access to customer data the default. Customers then asked us for more. They wanted tightly scoped, attestable environments for their most sensitive processing: the code and data that required protection even from their own administrators. That’s why we built Nitro Enclaves. Today, more of those sensitive workloads run in containers on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and self-managed Kubernetes and container runtimes on Amazon EC2. Customers are asking for the same level of isolation and confidentiality for container environments.

Container environments today host many independent workloads on a single EC2 instance by using Linux container technology. For a lot of customers, Linux containers have become the prevalent workload delivery vehicle because of their ease of deployment and the ability to have them share resources in one host through a shared kernel. This sharing has some downsides. Operators of a shared cluster have access to all the tasks running in it. A weakness in the shared kernel reachable by one container also exposes every other container on the host. With the current rate of Linux Common Vulnerabilities and Exposures (CVEs), containerized workloads need an architecture that isolates each workload from other workloads and from cluster operators.

Introducing AWS Nitro Sandboxes

This week we introduced AWS Nitro Sandboxes, a new Amazon EC2 building block that gives each container the boundary of a hardware-isolated virtual machine (VM). Each sandbox gets the security properties of the Nitro System (strong isolation, zero operator access to customer data, and attestation) while keeping the familiar container experience. And isolation is designed so that no one outside the Nitro Sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory.

A Nitro Sandbox is a lightweight virtual machine that you launch and control from your EC2 instance. To your instance, it behaves like a child VM. Under the hood, it runs as a separate VM alongside your instance. The Nitro System enforces separation between the sandbox and its parent instance, between sandboxes, and between the sandbox and AWS operators. The parent instance contributes vCPU, memory, and I/O. Nitro keeps the workload isolated and, when you ask, proves what code it is running.

Because a Nitro Sandbox is a first-level guest with no nested hypervisor, and the Nitro Hypervisor enforces its CPU and memory boundary in hardware, compute and memory run at near-native EC2 performance. Instead of needing a separate instance for every workload or customer that needs a hard boundary, you can consolidate many isolated workloads onto instances you already run. You can then improve utilization and lower cost without giving up the boundary that made separate instances attractive in the first place.

Nitro Sandboxes share the same Nitro System isolation technology that we have operated for years as Nitro Enclaves. With Nitro Enclaves, we designed everything around the smallest possible surface: You dedicate full CPU cores and memory to a separate VM that you can only reach over a vsock channel. With Nitro Sandboxes, we built on the same foundation, but adjusted the experience to the needs of container workloads in three ways. First, a sandbox’s vCPUs can share cores with the parent instance through time slicing. Second, the parent can attach standard virtio network and block devices, not only a vsock channel, so the sandbox gets real networking and storage. Third, you can run the sandbox through a standard virtual machine monitor (VMM) such as Firecracker. The result is a general-purpose, hardware-isolated environment that behaves like a normal VM and runs almost anywhere EC2 runs.

We then took Nitro Sandboxes and natively integrated them into Amazon EKS and Amazon ECS, providing a turnkey solution to Nitro-backed, confidential virtual-machine-based container isolation at scale. Nitro Sandboxes are broadly available with the following features:

  • Instances: Every EC2 instance type that supports Nitro Enclaves, across general purpose, compute optimized, memory optimized, storage optimized, and accelerated families, on AWS Graviton, Intel, and AMD processors, in commercial Regions worldwide, at no additional charge.
  • Storage: Sandboxes reach storage the same way an EC2 instance does: local instance store for scratch space, Amazon Elastic Block Store (Amazon EBS) for persistent volumes, and Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS) over the network. To keep data confidential from the parent, you can use attestation-gated AWS Key Management Service (AWS KMS) keys, the same model as Nitro Enclaves, now integrated into Nitro Sandboxes, EKS, and ECS.
  • Networking: Each sandbox runs its own kernel, and therefore its own network stack. The parent connects that stack to your Amazon Virtual Private Cloud (Amazon VPC) the same way it does for containers today. On EKS, pods keep their own VPC IP address through the Amazon VPC CNI. On ECS, each task gets its own elastic network interface in awsvpc mode. Network policies and security groups are enforced on the parent instance, outside the sandbox, so a compromised workload cannot turn them off.

With Nitro Sandboxes, you can give every workload that needs a hard boundary its own VM, without giving it its own instance. AI agents that write and run their own code can run each session in a separate sandbox. Code that a prompt injection gets the agent to run stays inside one VM, which is deleted when the session ends. Software as a service (SaaS) providers can run many tenants on shared EKS nodes or ECS Managed Instances, each in its own sandbox, instead of giving each tenant a dedicated instance. Continuous integration and continuous delivery (CI/CD) pipelines can run privileged builds and untrusted pull request code inside a per-job VM rather than on dedicated build machines. And with confidential sandboxes, model providers can deploy closed-weight models in a customer’s account, knowing the weights are decrypted only inside an attested sandbox. Regulated industries can give auditors cryptographic proof of exactly what ran.

Isolation by design

Containers on a shared node all trust one Linux kernel. A single kernel issue can lead to data exposure across pods or tasks. With Nitro Sandboxes, each pod, task, or agent runs in its own VM with its own kernel. A kernel issue in one sandbox doesn’t affect its neighbors.

The Nitro Hypervisor establishes strong, virtual-machine-based security isolation through its leading security posture and built-in confidential computing which we continuously extend. The hypervisor removes the parent’s access to the sandbox’s memory and CPU register state. Only I/O buffers that the sandbox explicitly shares stay visible to the parent.

Nitro Sandboxes can therefore treat the parent instance as untrusted and attest to what runs inside. They use the Nitro-based Nitro Secure Module (NSM) to gain an attestation document in the same style as Nitro Enclaves. A workload can be configured to get its secrets only after it proves it is running the code you approved. The system is designed so that no one outside the sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory. You write AWS KMS key policies that require specific attestation measurements before AWS KMS releases a decryption key. This creates a verifiable link between who can decrypt and what code is actually running.

Isolation modes in EKS and ECS

On top of Nitro Sandboxes, EKS and ECS provide two isolation modes. With these modes, you can choose between faster launch times and cryptographic attestation:

  • Confidential sandbox: Sandboxes are fully attestable. Image loading and other runtime operations are gated by a Cedar policy, image contents are verified and extracted inside the sandbox, and the ephemeral layer is encrypted. AWS KMS releases your secrets only to workloads that prove they run the code you approved. Use this mode for anything that handles secrets, including when you want protection from your own operators. SaaS providers can show their customers that tenant data is protected through attestation. Regulated industries can give auditors cryptographic evidence in addition to documentation.
  • Isolated sandbox: Each pod (EKS) or task (ECS) runs in its own hardware-isolated VM with its own kernel, protecting the node and neighboring workloads from anything that happens inside the sandbox. The node and its operators are trusted, just as they are for pods and tasks today, and image validation works the same as it does without sandboxes. For workloads that also need protection from the node operator, use confidential sandbox.

Getting started on Amazon EKS

To get started on EKS, use EKS Auto Mode. Add runtimeClassName to your pod spec, and EKS handles instance selection, Nitro component setup, and scheduling. Use confidential-sandbox for a confidential sandbox or isolated-sandbox if you do not need protection from node operators. For self-managed EKS, you will need a compatible Amazon Machine Image (AMI) to support Nitro Sandboxes.

To learn more about the EKS experience with Nitro Sandboxes, see the Amazon EKS blog post.

Getting started on Amazon ECS

You can use Nitro Sandboxes on ECS with ECS Managed Instances. ECS launches compatible EC2 instances in your account, sets up a fully configured Bottlerocket image, and runs each task in its own Nitro Sandbox. ECS supports the two isolation modes, CONFIDENTIAL_SANDBOX and ISOLATED_SANDBOX.

Each task reports its isolation mode in a read-only trustedExecutionSummary field when you describe it. Per-task security groups, VPC Flow Logs, Application Load Balancer, Network Load Balancer, and Amazon VPC Lattice work as before. To learn more about the ECS experience with Nitro Sandboxes, see the Amazon ECS Developer Guide.

Getting started on Amazon EC2

If you manage your own instances, or build your own platform, you can launch Nitro Sandboxes directly on nearly any instance built on the AWS Nitro System. Amazon Linux and Firecracker support Nitro Sandboxes today. We have started to upstream support for Nitro Sandboxes in Linux and other virtual machine monitors. We also worked with operating system vendor partners to add support early, so you can keep the stack your workload already depends on.

In addition to containers, Nitro Sandboxes serve as a building block for more in-instance virtualization use cases. Nitro Sandboxes support a special “Direct Mode” switch which disables attestation, but allows the parent to retain access to sandbox memory. We have seen early adopters use this mode to execute non-enlightened operating systems, including Windows or Android, in sandboxes. Other customers have used Nitro Sandboxes to create fully attested package build environments, where each built package receives a tamper-proof signature of its build inputs. See the Amazon EC2 User Guide for details.

Pricing and availability

Nitro Sandboxes, along with the EKS and ECS integrations, are available at no additional charge. You pay for the EC2 instances, plus the standard EKS Auto Mode or ECS Managed Instances management fee if you use those capabilities. Nitro Sandboxes run on nearly every EC2 instance, EKS cluster, and ECS cluster, so strong per-container, per-task, and in-instance isolation is available to everyone.

For more information, see the Nitro Sandboxes documentation in the Amazon EKS User Guide, the Amazon ECS Developer Guide, or the Amazon EC2 User Guide for direct use on EC2 instances.


About the authors

Alex Graf

Alex is a Principal Engineer on the EC2 team at AWS.

46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service

Post Syndicated from Sukanth Rajan original https://aws.amazon.com/blogs/big-data/46-fewer-nodes-2x-faster-indexing-confluent-on-amazon-opensearch-service/

Operating one of the world’s largest data-streaming platforms, Confluent, the company founded by the original creators of Apache Kafka and now part of IBM, continuously modernizes its infrastructure to stay ahead of rapid data growth. Confluent’s data-streaming products, including its Apache Kafka, Connect, and Flink fleets, are available on the AWS Cloud, and its internal logging system runs on Amazon OpenSearch Service, ingesting hundreds of terabytes of logs per day to support operations across that fleet.

At that ingest volume, indexing work dominates the hot tier. The question Confluent’s engineering team set out to answer was not “can we run on cheaper hardware,” but “how much more ingest can we absorb per node, and how much headroom does that free up?” This post shows how a measure-first migration of the OpenSearch hot tier to the purpose-built OpenSearch optimized instance type, oi2, improved indexing performance and ingestion headroom first. oi2 costs more per node, but we needed far fewer of them. The lower cost came from that efficiency, not from a lower instance price.

Why instance choice matters for ingestion

Confluent’s hot tier ran on r6gd, a previous-generation, general-purpose Graviton2 instance with local NVMe added as a bolt-on. The team migrated to the current-generation OpenSearch Optimized Instance family (oi2), purpose-built for OpenSearch. On oi2, local NVMe is the primary storage tier with no separate Amazon Elastic Block Store (Amazon EBS) volume. Data is copied synchronously to Amazon Simple Storage Service (Amazon S3) for durability, and the storage and network path is integrated specifically for indexing and merge I/O. For an ingest-bound logging workload, that storage path is the lever. It’s what lets each node sustain a higher indexing rate at lower latency.

A domain with OpenSearch Optimized Instances uses local storage as the primary tier, with data copied synchronously to Amazon S3 as it arrives. This provides increased indexing throughput with high durability (11 nines). Within the family, OR1, OR2, and OM2 use Amazon EBS volumes (gp3 or io1) locally, whereas oi2 instances use local NVMe disks. These instances benefit indexing-heavy operational analytics workloads such as log analytics, observability, and security analytics.

OpenSearch supports two replication strategies: logical (document) and physical (segment) replication. With logical replication, data is indexed on all the copies independently, leading to redundant computation on the cluster. OpenSearch Optimized Instances use the physical replication model, where data is indexed only on the primary copy. Additional copies are created by copying the segments from the primary (durably persisted to Amazon S3). Because this frees up compute resources on replicas that would otherwise redundantly index data, and network overhead on the primary, the cluster can churn more throughput. When Optimized Instances are configured with replicas, the indexing rate might appear lower than it actually is (for example, showing a rate of 1,000 when the actual indexing rate is 2,000). This is because indexing is only performed on primary shards.

Reference architecture

The following diagram shows the OpenSearch multi-domain topology, split by priority (P0/P1) and volume, behind Cross-Cluster Search.

Kubernetes clusters feed the Confluent ingestion pipeline that writes to four OpenSearch domains read through cross-cluster search

Figure 1: OpenSearch multi-domain topology, with writes going directly to the P0 and P1 domains and reads served through cross-cluster search

A measure-first approach

Each domain was modernized in two logical phases, so the efficiency of the new instance could be measured in production before capacity was removed:

  1. Phase 1, a type swap at constant node count: an in-place Amazon OpenSearch Service blue/green re-provision at the same node count, to measure real production efficiency at fixed ingest. At equal nodes, oi2 ran at roughly one-third the CPU of r6gd for the same ingest, about 2.7x the headroom per node.
  2. Phase 2, iterative rightsizing: nodes were removed in steps, holding at each step to confirm cluster health, until a guardrail became binding. That guardrail turned out to be JVM heap, not CPU, because heap pressure is driven by query patterns, and removing nodes concentrates the same shards onto fewer of them.

Spending the Phase 1 headroom this way let the fleet consolidate to roughly half the nodes. Each oi2 node carried about 1.9x the load of an r6gd node (168 ÷ 90), while indexing got faster, not slower.

Production results: Ingestion efficiency first

The following figures are production averages at the same ingest (7-day baseline on r6gd compared to a 24-hour window on oi2 at the final node counts). Because the log traffic is identical, the comparison reflects a real change per unit of work. We didn’t benchmark R8gd for this migration. Our evaluation focused on the OpenSearch optimized (oi2) family, which is built specifically for log-analytics ingestion and storage economics.

Fleet node count dropping from 168 on r6gd to 90 on oi2, a 46% reduction

Figure 2: Total hot-tier fleet size, 168 r6gd nodes compared to 90 oi2 nodes, a 46% reduction

Node count per domain falling from r6gd to oi2: Domain A 30 to 12, B 36 to 18, C 39 to 27, D 63 to 33

Figure 3: Node count per domain on r6gd compared to oi2, with reductions of 31 to 60 percent

Indexing latency per domain roughly halving from r6gd at 0.18 to 0.19 ms to oi2 at 0.08 to 0.10 ms

Figure 4: Average indexing latency per domain on r6gd compared to oi2, roughly halved across all four domains

Write queue depth per domain dropping from double digits on r6gd to near zero on oi2

Figure 5: Write queue depth in queued operations per domain on r6gd compared to oi2, cleared to near zero

Domain (anonymized) Nodes Avg CPU Indexing latency Write queue (ops)
Domain A (high-volume) 30 → 12 (-60%) 49% → 43% 0.19 → 0.08 ms 17.6 → 0.2
Domain B (streaming logs) 36 → 18 (-50%) 60% → 40% 0.18 → 0.08 ms 20.6 → 0.3
Domain C (high-volume) 39 → 27 (-31%) 58% → 39% 0.19 → 0.09 ms 37.9 → 0.2
Domain D (low-volume) 63 → 33 (-48%) 43% → 56% 0.18 → 0.10 ms 8.4 → 1.0
Fleet 168 → 90 (-46%) — ~halved <1 everywhere
  • Indexing latency roughly halved everywhere: approximately 0.18–0.19 ms on r6gd to approximately 0.08–0.10 ms on oi2, reflecting oi2’s faster storage path.
  • Write backpressure cleared. Every r6gd domain carried a double-digit write queue (8–38 queued ops, the indexing-backpressure signal). On oi2, all four dropped below approximately 1. Faster storage keeps indexing current.
  • CPU fell even on far fewer nodes for three of four domains, the clearest sign of per-node efficiency (for example, Domain B: 60% → 40% on half the nodes).
  • Heap, not CPU, is the ceiling. oi2 has the same heap per node as r6gd, so concentrating shards onto fewer nodes raised average heap on every domain. This is what set the floor on rightsizing.
  • The honest outlier. Domain D was cut the hardest relative to its load. Its CPU rose (43% → 56%) and its write queue, although far better, is the only one not near zero. It’s the domain to watch through peak traffic before calling it settled.
  • Stability: zero write/primary-write rejections in the final state. All clusters healthy throughout.

In AWS internal benchmarks, oi2 instances deliver up to 9 percent higher indexing throughput than OR2 instances and up to 33 percent more than I8g instances. Each OpenSearch Optimized instance is provisioned with compute, third-generation AWS Nitro SSDs for caching, and remote managed storage on Amazon S3. The oi2 family is available in sizes from large through 24xlarge with up to 22.5 TB of storage.

To maximize indexing throughput from OpenSearch Optimized Instances, AWS recommends:

  • Use large bulk sizes to improve buffer utilization. The recommended size is 10 MB.
  • Use multiple clients to improve parallel processing performance.
  • Consider more primary shards to raise indexing throughput, but note that over-sharding every index drives up per-node shard density and lowers it.
  • Keep the index refresh interval at 10 seconds or higher (the default for Optimized Instances is 10 seconds).

Fewer nodes, lower bill — even at a higher per-node price

oi2 carries a higher per-node price than r6gd, so a like-for-like swap alone would have raised spend. The saving came entirely from needing fewer nodes: consolidating the hot tier by approximately 46 percent more than offset the premium, for an estimated approximately 18 percent lower hot-tier cost at on-demand pricing. Committing the final footprint to 1-year reserved instances is estimated to take the hot tier to roughly 47–51 percent below the original r6gd on-demand baseline.

Cost figures are approximate, list-derived percentages at on-demand pricing (reserved-instance figures are estimates pending a quoted rate). Storage (UltraWarm and snapshots) and dedicated-master tiers were unchanged and are excluded. The takeaway for teams modernizing OpenSearch: optimize efficiency per unit of work, not instance unit price.

Conclusion

With the measure-first approach in two phases, Confluent raised ingestion efficiency and halved indexing latency on its Amazon OpenSearch Service cluster. Before removing any capacity, let the real sizing signal (heap in this case) guide the optimization work. This approach is what improved the efficiency of the system and lowered the overall cost. The same measure-first approach applies to ingest-bound OpenSearch workloads evaluating purpose-built storage-optimized instances. To learn more, see the Amazon OpenSearch Service documentation on instance types and sizing domains.

 


About the authors

Sukanth Rajan

Sukanth Rajan

Sukanth is a Technical Account Manager in Strategic Accounts at Amazon Web Services. He helps AWS’s largest customers architect, migrate, and optimize their mission-critical workloads on the cloud. He leverages his deep technical expertise, especially in databases, to help customers solve critical issues and build highly reliable, secure, and cost-effective solutions using AWS services.

Aravind Karthik

Aravind Karthik

Aravind is a software engineer on the Cloud Infrastructure & Platform team at Confluent. He helps build highly scalable observability systems that process terabytes of data every day. He is currently working on regionalizing Confluent’s metrics pipelines and optimizing the logging pipeline to be lean and efficient for the next wave of telemetry volume growth.

The collective thoughts of the interwebz