Post Syndicated from xkcd.com original https://xkcd.com/3230/
Introducing the UX Research Working Group
Post Syndicated from Amy Super original https://prometheus.io/blog/2026/04/08/introducing-ux-research-working-group/
Prometheus has always prioritized solving complex technical challenges to deliver a reliable, performant open-source monitoring system. Over time, however, users have expressed a variety of experience-related pain points. Those pain points range from onboarding and configuration to documentation, mental models, and interoperability across the ecosystem.
At PromCon 2025, a user research study was presented that highlighted several of these issues. Although the central area of investigation involved Prometheus and OpenTelemetry workflows, the broader takeaway was clear: Prometheus would benefit from a dedicated, ongoing effort to understand user needs and improve the overall user experience.
Recognizing this, the Prometheus team established a Working Group focused on improving user experience through design and user research. This group is meant to support all areas of Prometheus by bringing structured research, user insights, and usability perspectives into the community’s development and decision-making processes.
How we can help Prometheus maintainers
Building something where the user needs are unclear? Maybe you’re looking at two competing solutions and you’d like to understand the user tradeoffs alongside the technical ones.
That’s where we can be of help.
The UX Working Group will partner with you to conduct user research or provide feedback on your plans for user outreach. That could include:
- User research reports and summaries
- User journeys, personas, wireframes, prototypes, and other UX artifacts
- Recommendations for improving usability, onboarding, interoperability, and documentation
- Prioritized lists of user pain points
- Suggestions for community discussions or decision-making topics
To get started, tell us what you’re trying to do, and we’ll work with you to determine what type and scope of research is most appropriate.
How we can help Prometheus end users
We want to hear from you! Let us know if you’re interested in participating in a research study and we’ll contact you when we’re working on one that’s a good fit. Having an issue with the Prometheus user experience? We can help you open an issue and direct it to the appropriate community members.
Interested in helping?
New contributors to the working group are always welcome! Get in touch and let us know what you’d like to work on.
Where to find us
Drop us a message in Slack, join a meeting, or raise an issue in GitHub.
- Slack: #prometheus-ux-wg
- Meetings: We meet biweekly, currently on Wednesdays at 14:00 UTC (subject to change depending on contributor availability).
- GitHub: prometheus-community/ux-research
Cloudflare targets 2029 for full post-quantum security
Post Syndicated from Bas Westerbaan original https://blog.cloudflare.com/post-quantum-roadmap/
Cloudflare is accelerating its post-quantum roadmap. We now target 2029 to be fully post-quantum (PQ) secure including, crucially, post-quantum authentication.
At Cloudflare, we believe in making the Internet private and secure by default. We started by offering free universal SSL certificates in 2014, began preparing our post-quantum migration in 2019, and enabled post-quantum encryption for all websites and APIs in 2022, mitigating harvest-now/decrypt-later attacks. While we’re excited by the fact that over 65% of human traffic to Cloudflare is post-quantum encrypted, our work is not done until authentication is also upgraded. Credible new research and rapid industry developments suggest that the deadline to migrate is much sooner than expected. This is a challenge that any organization must treat with urgency, which is why we’re expediting our own internal Q-Day readiness timeline.
What happened? Last week, Google announced they had drastically improved upon the quantum algorithm to break elliptic curve cryptography, which is widely used to secure the Internet. They did not reveal the algorithm, but instead provided a zero-knowledge proof that they have one.
This is not even the biggest breakthrough. That same day, Oratomic published a resource estimate for breaking RSA-2048 and P-256 on a neutral atom computer. For P-256, it only requires a shockingly low 10,000 qubits. Google’s motivation behind their recent announcement to also pursue neutral atoms alongside superconducting quantum computers becomes clear now. Although Oratomic explains their basic approach, they still leave out crucial details on purpose.
These independent advances prompted Google to accelerate their post-quantum migration timeline to 2029. What’s more, in their announcement and other talks, Google has placed a priority on quantum-secure authentication over mitigating harvest-now/decrypt-later attacks. As we discuss next, this priority indicates that Google is concerned about Q-Day coming as soon as 2030. Following the announcements, IBM Quantum Safe’s CTO is more pessimistic and can’t rule out quantum “moonshot attacks” on high value targets as early as 2029.
The quantum threat is well known: Q-Day is the day that sufficiently capable quantum computers can break essential cryptography used to protect data and access across systems today. Cryptographically relevant quantum computers (CRQCs) don’t exist yet, but many labs across the world are pursuing different approaches to building one. Until recently, progress on CRQCs has been mostly public, but there is no reason to expect that will continue. Indeed, there is ample reason to expect that progress will leave the public eye. As quantum computer scientist Scott Aaronson warned at the end of 2025:
[A]t some point, the people doing detailed estimates of how many physical qubits and gates it’ll take to break actually deployed cryptosystems using Shor’s algorithm are going to stop publishing those estimates, if for no other reason than the risk of giving too much information to adversaries. Indeed, for all we know, that point may have been passed already.
That point has now passed indeed.
We’d like to spend some words on why it’s difficult to predict progress on quantum computing. Sudden “quantum” leaps in understanding, like the one we witnessed last week, can occur even if everything happens in the public eye. Simply put, breaking cryptography with a quantum computer requires engineering on three independent fronts: quantum hardware, error correction, and quantum software. Progress on each front compounds progress on the others.
Hardware. There are many different competing approaches. We mentioned neutral atoms and superconducting qubits, but there are also ion-trap, photonics, and moonshots like topological qubits. Complementary approaches can even be combined. Most of these approaches are pursued by several labs around the world. They all have their distinct engineering challenges and problems to solve before they can scale up. A few years ago, all of them had a long list of open challenges, and it was unclear if any of them would scale. Today most of them have made good progress. None have been demonstrated to scale yet: if they had, we wouldn’t have a couple of years left. But these approaches are much closer now, especially neutral atoms. To ignore this progress, you’d have to believe that every single approach will hit a wall.
Error correction. All quantum computers are noisy and require error-correcting codes to perform meaningful computation. This adds quite a bit of overhead, though how much depends on the architecture. More noise requires more error correction, but more interestingly, improved qubit connectivity allows for much more efficient codes. For a sense of scale: typically around a thousand physical qubits are required for one logical qubit for the superconducting quantum computers that are noisy and only have neighbor qubit connectivity. We knew “reconfigurable qubits” such as those of neutral-atom machines allow for an order of magnitude better error-correcting codes. Surprisingly, Oratomic showed the advantage is even larger: only about 3-4 physical neutral atom qubits are required per logical qubit.
Software. Lastly, the quantum algorithms to crack cryptography can be improved. This is Google’s breakthrough: they massively sped up the algorithm to crack P-256. On top of that, Oratomic showed further architecture specific optimizations for reconfigurable qubits.
The picture comes together: in 2025 neutral atoms turned out to be more scalable than expected, and now Oratomic figured out how to do much better error-correcting codes with such highly connected qubits. On top of that, breaking P-256 requires much less work. The result is that Q-Day has been pulled forward significantly from typical 2035+ timelines, with neutral atoms in the lead, and other approaches not far behind.
In previous blog posts we’ve discussed how different quantum computers compare on physical qubit count and fidelity, compared to the conservative goalpost of cracking RSA-2048 on a superconducting qubit architecture. This analysis gives us a rough idea of how much time we have, and it’s certainly better than tracking quantum factoring records, but it misses architecture-specific optimization and software improvements. What to watch for now is when the final missing capabilities for each architecture are achieved.
Historically, the industry’s focus on post-quantum cryptography (PQC) has been based largely on PQ encryption, which stops harvest-now/decrypt-later (HNDL) attacks. In an HNDL attack, an adversary harvests sensitive encrypted network traffic today and stores it until a future date when it can use a powerful quantum computer to decrypt the data. HNDL attacks are the primary threat when Q-Day is far away. That’s why our focus, thus far, has been on mitigating this risk, by adopting post-quantum encryption by default in our products since 2022. Today, as we mentioned above, most Cloudflare products are secure against HNDL attacks, and we’re working to upgrade the rest as we speak.
The other category of attacks is against authentication: adversaries armed with functioning quantum computers impersonate servers or forge access credentials. If Q-Day is far off, authentication is not urgent: deploying PQ certificates and signatures does not add any value, only effort.
An imminent Q-Day flips the script: data leaks are severe, but broken authentication is catastrophic. Any overlooked quantum-vulnerable remote-login key is an access point for an attacker to do as they wish, whether that’s to extort, take down, or snoop on your system. Any automatic software-update mechanism becomes a remote code execution vector. An active quantum attacker has it easy — they only need to find one trusted quantum-vulnerable key to get in.
When experts in the field of building quantum computers start patching authentication systems, we should all listen. The question is no longer “when will our encrypted data be at risk?” but “how long before an attacker walks in the front door with a quantum-forged key?”
If quantum computers arrive in the next few years, they will be scarce and expensive. Attackers will prioritize high-value targets, like long-lived keys that unlock substantial assets or persistent access such as root certificates, API auth keys and code-signing certs. If an attacker is able to compromise one such key, they retain indefinite access until they are discovered or that key is revoked.
This suggests long-lived keys should be prioritized. That is certainly true if the quantum attack of a single key is expensive and slow, which is to be expected for the first generation of neutral atom quantum computers. That’s not the case for scalable superconducting quantum computers and later generations of neutral atom quantum computers, which could well crack keys much faster. Such fast CRQCs flip the script again, and an adversary with one might focus purely on HNDL attacks so that their attacks remain undetected. Google’s Sophie Schmieg compares this scenario to Enigma’s cryptanalysis that changed the direction of World War II.
Adding support for PQ cryptography is not enough. Systems must disable support for quantum-vulnerable cryptography to be secure against downgrade attacks. In larger, especially federated systems such as the web, this is not feasible because not every client (browser) will support post-quantum certificates, and servers need to keep supporting these legacy clients. However, downgrade protection for HTTPS is still achievable using “PQ HSTS” and/or certificate transparency.
Disabling quantum-vulnerable cryptography is not the last step: once done, all secrets such as passwords and access tokens previously exposed in the quantum-vulnerable system need to be rotated. Unlike post-quantum encryption, which takes one big push, migrating to post-quantum authentication has a long dependency chain — not to mention third-party validation and fraud monitoring. This will take years, not months.
It’s natural for organizations reading this to rush out and think about which internal systems they need to upgrade. But that’s not the end of the story. Q-day threatens all systems. As such, it’s important to understand the impact of a potential Q-day on third-party dependencies, both direct and indirect. Not just the third-parties you speak cryptography to, but also any third parties that are critical business dependencies like financial services and utilities.
With Q-day approaching on a shorter timeline, post-quantum authentication is top priority. Long-term keys should be upgraded first. Deep dependency chains and the fact that everyone has third-party vendors means this effort will take on the order of years, not months. Upgrading to post-quantum cryptography is not enough: to prevent downgrades, quantum-vulnerable cryptography must also be turned off.
Today, Cloudflare provides post-quantum encryption for the majority of our products mitigating harvest-now/decrypt-later. This is the product of work we started over a decade ago to protect our customers and the Internet at large.
We are targeting full post-quantum security including authentication for our entire product suite by 2029. Here we’re sharing some intermediate milestones we’ve set, subject to change as our understanding of the risk and deployment challenges evolve.
For businesses, we recommend making post-quantum support a requirement for any procurement. Common best practices, like keeping software updated and automating certificate issuance, are meaningful and will get you pretty far. We recommend assessing critical vendors early for what their failure to take action would mean for your business.
For regulatory agencies and governments: leading by setting early timelines has been crucial for industry-wide progress so far. We are now in a pivotal position where fragmentation in standards and effort between and within jurisdictions could put progress at risk. We recommend that governments assign and empower a lead agency to coordinate the migration on a clear timeline, stay security-focused, and promote the use of existing international standards. Governments need not panic, but can lead migration with confidence.
For Cloudflare customers, with respect to our services, you do not need to take any mitigating action. We are following the latest advancements in quantum computing closely and taking proactive steps to protect your data. As we have done in the past, we will turn on post-quantum security by default, with no switches to flip. What we don’t control is the other side: browsers, applications, and origins need to upgrade. Corporate network traffic on Cloudflare need not worry: Cloudflare One offers end-to-end protection when tunnelling traffic through our post-quantum encrypted infrastructure.
Privacy and security are table stakes for the Internet. That’s why every post-quantum upgrade we build will continue to be available to all customers, on every plan, at no additional cost. Making post-quantum security the default is the only way to protect the Internet at scale.
Free TLS helped encrypt the web. Free post-quantum cryptography will help secure it for what comes next.
Fareed Zakaria: The U.S. is behaving like a rogue superpower
Post Syndicated from The Atlantic original https://www.youtube.com/shorts/MtoScIOT16w
Launching S3 Files, making S3 buckets accessible as file systems
Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/
I’m excited to announce Amazon S3 Files, a new file system that seamlessly connects any AWS compute resource with Amazon Simple Storage Service (Amazon S3).
More than a decade ago, as an AWS trainer, I spent countless hours explaining the fundamental differences between object storage and file systems. My favorite analogy was comparing S3 objects to books in a library (you can’t edit a page, you need to replace the whole book) versus files on your computer that you can modify page by page. I drew diagrams, created metaphors, and helped customers understand why they needed different storage types for different workloads. Well, today that distinction becomes a bit more flexible.
With S3 Files, Amazon S3 is the first and only cloud object store that offers fully-featured, high-performance file system access to your data. It makes your buckets accessible as file systems. This means changes to data on the file system are automatically reflected in the S3 bucket and you have fine-grained control over synchronization. S3 Files can be attached to multiple compute resources enabling data sharing across clusters without duplication.
Until now, you had to choose between Amazon S3 cost, durability, and the services that can natively consume data from it or a file system’s interactive capabilities. S3 Files eliminates that tradeoff. S3 becomes the central hub for all your organization’s data. It’s accessible directly from any AWS compute instance, container, or function, whether you’re running production applications, training ML models, or building agentic AI systems.
You can access any general purpose bucket as a native file system on your Amazon Elastic Compute Cloud (Amazon EC2) instances, containers running on Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS), or AWS Lambda functions. The file system presents S3 objects as files and directories, supporting all Network File System (NFS) v4.1+ operations like creating, reading, updating, and deleting files.
As you work with specific files and directories through the file system, associated file metadata and contents are placed onto the file system’s high-performance storage. By default, files that benefit from low-latency access are stored and served from the high performance storage. For files not stored on high performance storage such as those needing large sequential reads, S3 Files automatically serves those files directly from Amazon S3 to maximize throughput. For byte-range reads, only the requested bytes are transferred, minimizing data movement and costs.
The system also supports intelligent pre-fetching to anticipate your data access needs. You also have fine-grained control over what gets stored on the file system’s high performance storage. You can decide whether to load full file data or metadata only, which means you can optimize for your specific access patterns.
Under the hood, S3 Files uses Amazon Elastic File System (Amazon EFS) and delivers ~1ms latencies for active data. The file system supports concurrent access from multiple compute resources with NFS close-to-open consistency, making it ideal for interactive, shared workloads that mutate data, from agentic AI agents collaborating through file-based tools to ML training pipelines processing datasets.
Let me show you how to get started.
Creating my first Amazon S3 file system, mounting, and using it from an EC2 instance is straightforward.
I have an EC2 instance and a general purpose bucket. In this demo, I configure an S3 file system and access the bucket from an EC2 instance, using regular file system commands.
For this demo, I use the AWS Management Console. You can also use the AWS Command Line Interface (AWS CLI) or infrastructure as code (IaC).
Here is the architecture diagram for this demo.
Step 1: Create an S3 file system.
On the Amazon S3 section of the console, I choose File systems and then Create file system.
I enter the name of the bucket I want to expose as a file system and choose Create file system.
Step 2: Discover the mount target.
A mount target is a network endpoint that will live in my virtual private cloud (VPC). It allows my EC2 instance to access the S3 file system.
The console creates the mount targets automatically. I take notes of the Mount target IDs on the Mount targets tab.
When using the CLI, two separate commands are necessary to create the file system and its mount targets. First, I create the S3 file system with create-file-system. Then, I create the mount target with create-mount target.
Step 3: Mount the file system on my EC2 instance.
After it’s connected to an EC2 instance, I type:
sudo mkdir /home/ec2-user/s3files sudo mount -t s3files fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files
I can now work with my S3 data directly through the mounted file system in ~/s3files, using standard file operations.
When I make updates to my files in the file system, S3 automatically manages and exports all updates as a new object or a new version on an existing object back in my S3 bucket within minutes.
Changes made to objects on the S3 bucket are visible in the file system within a few seconds but can sometimes take a minute or longer.
# Create a file on the EC2 file system
echo "Hello S3 Files" > s3files/hello.txt
# and verify it's here
ls -al s3files/hello.txt
-rw-r--r--. 1 ec2-user ec2-user 15 Oct 22 13:03 s3files/hello.txt
# See? the file is also on S3
aws s3 ls s3://s3files-aws-news-blog/hello.txt
2025-10-22 13:04:04 15 hello.txt
# And the content is identical!
aws s3 cp s3://s3files-aws-news-blog/hello.txt . && cat hello.txt
Hello S3 FilesThings to know
Let me share some important technical details that I think you’ll find useful.
- S3 Files integrates with AWS Identity and Access Management (IAM) for access control and encryption. You can use identity and resource policies to manage permissions at both the file system and object level.
- Data is always encrypted in transit using TLS 1.3 and at rest using Amazon S3 managed keys (SSE-S3) or customer-managed keys with AWS Key Management Service (AWS KMS).
- S3 Files uses POSIX permissions for files and directories, checking user ID (UID) and group ID (GID) against file permissions stored as object metadata in the S3 bucket.
- Monitor S3 Files using Amazon CloudWatch metrics for drive performance and updates and AWS CloudTrail for logging management events.
- Verify that the latest version of the EFS driver (amazon-efs-utils package) is installed on your EC2 instances. This package is preinstalled on the Amazon Machine Image (AMI) provided by AWS. At the time of writing, you can update it to the latest version.
- In this post, I showed you how to use S3 Files from an EC2 instance. You can also mount your S3 bucket as a file system from your ECS or EKS containers, on AWS Fargate or not, and from your Lambda functions.
Another question I frequently hear in customer conversations is about choosing the right file service for your workloads. Yes, I know what you’re thinking: AWS and its seemingly overlapping services, keeping cloud architects entertained during their architecture review meetings. Let me help demystify this one.
S3 Files works best when you need interactive, shared access to data that lives in Amazon S3 through a high performance file system interface. It’s ideal for workloads where multiple compute resources—whether production applications, agentic AI agents using Python libraries and CLI tools, or machine learning (ML) training pipelines—need to read, write, and mutate data collaboratively. You get shared access across compute clusters without data duplication, sub-millisecond latency, and automatic synchronization with your S3 bucket.
For workloads migrating from on-premises NAS environments, Amazon FSx provides the familiar features and compatibility you need. Amazon FSx is also ideal for high-performance computing (HPC) and GPU cluster storage with Amazon FSx for Lustre. It’s particularly valuable when your applications require specific file system capabilities from Amazon FSx for NetApp ONTAP, Amazon FSx for OpenZFS, or Amazon FSx for Windows File Server.
Pricing and availability
S3 Files is available today in all commercial AWS Regions.
You pay for the portion of data stored in your S3 file system, for small file read and all write operations to the file system, and for S3 requests during data synchronization between the file system and the S3 bucket. The Amazon S3 pricing page has all the details.
From discussions with customers, I believe S3 Files helps simplify cloud architectures by eliminating data silos, synchronization complexity, and manual data movement between objects and files. Whether you’re running production tools that already work with file systems, building agentic AI systems that rely on file-based Python libraries and shell scripts, or preparing datasets for ML training, S3 Files lets these interactive, shared, hierarchical workloads access S3 data directly without choosing between the durability of Amazon S3 and cost benefits and a file system’s interactive capabilities. You can now use Amazon S3 as the place for all your organizations’ data, knowing the data is accessible directly from any AWS compute instance, container, and function.
To learn more and get started, visit the S3 Files documentation.
I’d love to hear how you use this new capability. Feel free to share your feedback in the comments below.
Building AI defenses at scale: Before the threats emerge
Post Syndicated from Amy Herzog original https://aws.amazon.com/blogs/security/building-ai-defenses-at-scale-before-the-threats-emerge/
At AWS, we’ve spent decades developing processes and tools that enable us to defend millions of customers simultaneously, wherever they operate around the world. Every day, our security and threat intelligence teams are doing work with AI and automation that most people never see. Our AI-powered log analysis system has reduced the time SecOps engineers spend analyzing security logs from an average of six hours to just seven minutes, a 50x productivity increase that lets us detect and respond to threats faster than ever. Across AWS, we analyze over 400 trillion network flows per day to detect patterns that signal emerging threats. In 2025 alone, we blocked over 300 million attempts to maliciously encrypt customer files hosted on Amazon S3.
What we learn protecting one customer helps protect all customers. At this scale, every threat we see makes our defenses stronger for everyone, and AI is already central to how we do it.
A new class of AI for cybersecurity
Today, Anthropic announced Project Glasswing, a cybersecurity initiative designed to secure the world’s most critical software and advance the cybersecurity practices the industry will need as AI grows more capable. Organizations that build or maintain critical digital infrastructure are getting early access to Claude Mythos Preview, a new class of AI model, to find and patch vulnerabilities in the systems the world depends on. Given our role in securing some of the world’s most essential infrastructure, AWS is playing an integral part in advancing this work.
Powering the project is Claude Mythos Preview, Anthropic’s most advanced AI model to date and a step-change in reasoning and AI capabilities for cybersecurity. Claude Mythos Preview represents a fundamentally new model class: more intelligent and capable than Anthropic’s previous frontier models, with higher performance on cybersecurity, software coding, and complex reasoning tasks.
As part of Project Glasswing, we’ve already applied Claude Mythos Preview to critical AWS codebases that undergo continuous AI-powered security reviews, and even in those well-tested environments, it’s helped us identify additional opportunities to strengthen our code. In our internal testing, Claude Mythos Preview has proven more productive than previous models at surfacing security findings, requiring less manual guidance from our engineers to deliver actionable results. We’ve also given early access to a select group of AWS customers, who are deploying Claude Mythos Preview in their own security workflows and helping shape how the model evolves. For us, Claude Mythos Preview is a natural extension of the AI tools we already use, and as the technology grows more powerful, so must our defenses.
This is exactly the kind of innovation that drives our work, and we’ve been working closely with Anthropic to help ensure Claude Mythos Preview is ready for enterprise use. AWS is Anthropic’s primary cloud provider for mission-critical workloads, safety research, and foundation model development. More broadly, AWS provides the foundational infrastructure that the world’s leading AI companies rely on to build, train, and deploy their most advanced models. We’re bringing decades of security experience to this partnership, helping to ensure Claude Mythos Preview is ready for even more organizations to build upon and operate securely at scale.
Claude Mythos Preview signals an upcoming wave of models that can find vulnerabilities and build working exploits at a scale and speed we haven’t seen before. Anthropic and AWS are taking a deliberately cautious approach to release. Access begins with a small number of organizations, prioritizing internet-critical companies and open-source maintainers whose software and digital services impact hundreds of millions of users. The goal: find and fix vulnerabilities in the world’s most critical software. Claude Mythos Preview is available in gated research preview through Amazon Bedrock with enterprise-grade security controls, including customer-managed encryption, VPC isolation, and detailed logging, so your team can explore Claude Mythos Preview’s capabilities without exposing production assets to unnecessary risk.
AWS architects services with security at the core
Our work with Project Glasswing is grounded in a philosophy we’ve developed over two decades of securing mission-critical workloads: you can’t wait for threats to materialize before building your defenses. You have to look around corners, adopt new technologies, build protections first, deploy them in your own operations at scale, and refine them based on what you learn.
That’s exactly what we’ve done at AWS with AI and security. Our approach spans the full spectrum: proactive defense through threat hunting and vulnerability research, dynamic response to active campaigns, and third-party certifications that verify our security practices meet the highest industry standards. This operational experience has taught us where AI accelerates security work and where human judgment remains essential. And it’s reinforced that security innovation must be pragmatic: proven in production before we ask you to rely on it.
That’s also why we help define what secure AI looks like. We became the first major cloud provider to achieve ISO 42001 certification for AI services. We’re active participants in OWASP, the Coalition for Secure AI, and the Frontier Model Safety Framework. And we co-founded the Open Cybersecurity Schema Framework (OCSF) to enable better threat intelligence sharing across the ecosystem. The AWS Nitro System provides mathematical isolation for workloads. Our zero-operator access architecture means AWS personnel can’t access your data. These aren’t aspirational goals. They’re how we operate today, at scale, every day.
Amazon Bedrock is where these principles come to life for AI. It provides policy-enforced access controls, built-in evaluation tools to measure how effectively models identify and validate vulnerabilities, and the ability to run workloads inside your own virtual private cloud. AWS is also the first cloud provider to achieve FedRAMP High and Department of Defense Security Requirements Guide Impact Level 4 and 5 authorizations for generally available Claude foundation models, reinforcing that Amazon Bedrock is where the most security-sensitive organizations already trust Anthropic’s technology.
How to get started today
The same principles that guide our work at AWS scale apply regardless of which AI tools you’re using: comprehensive observability, defense in depth, automation where it adds value, and human judgment where it’s essential. Here’s how to put them into practice.
Prepare for the next generation of AI security. Claude Mythos Preview signals an upcoming wave of AI models that will transform cybersecurity. Start strengthening your security posture now so your organization is ready as these capabilities become more broadly available. Claude Mythos Preview is available in gated preview through Amazon Bedrock, and access is limited to an initial allow-list of organizations. If your organization has been allow-listed, your AWS account team will reach out directly.
Run on-demand penetration testing with AWS Security Agent. Now generally available, AWS Security Agent delivers autonomous penetration testing that operates 24/7 at a fraction of the cost of manual penetration tests. It transforms penetration testing from a periodic bottleneck into an on-demand capability that scales with your development velocity across AWS, Azure, GCP, other cloud providers, and on-premises. AWS Security Agent represents a new class of frontier agents: autonomous systems that work independently to achieve goals, scale to tackle concurrent tasks, and run persistently without constant human oversight. It deploys specialized AI agents to discover, validate, and report security vulnerabilities through sophisticated multi-step attack scenarios. Unlike traditional scanners that generate findings without validation, AWS Security Agent identifies potential vulnerabilities, then attempts to exploit them with targeted payloads and attack chains to confirm they are legitimate security risks. Each finding includes CVSS risk scores, application-specific severity ratings, detailed reproduction steps, and remediation suggestions. The result: penetration testing that once took weeks now completes in hours, and security coverage that scales across your entire application portfolio, not just your most critical systems. New customers can explore AWS Security Agent with a 2-month free trial.
Build AI applications you can trust with Amazon Bedrock. For teams building with generative AI, the challenge isn’t just making AI work, it’s making AI work safely. Amazon Bedrock provides the security and safety controls you need to deploy AI responsibly. Its Automated Reasoning capability is the first and only AI safeguard to use formal logic to help prevent factual errors from hallucinations, providing verifiable explanations with 99% accuracy, a capability we’ve refined over more than a decade of applying formal methods across AWS storage, identity, and networking. Amazon Bedrock also provides customizable guardrails that block harmful content and enforce your content policies, along with comprehensive observability to track AI behavior and detect anomalies across your workloads.
The threat landscape isn’t waiting
The threat landscape isn’t waiting for us to catch up. Nation-state actors, ransomware operators, and supply chain attackers are already using AI to scale their operations. Our job is to stay ahead by building defenses first, deploying them at scale, and sharing what we learn so the entire community benefits.
That’s what we do every day at AWS. We prove technology works in our own operations before we ask customers to rely on it. We set standards rather than follow them. And we look around corners to address tomorrow’s challenges today.
As AI capabilities continue to evolve, this approach won’t change. We’ll keep building defenses first, refining them at scale, and working with partners like Anthropic to ensure the next generation of AI security tools meets the real-world needs of enterprises defending at this scale.
Learn More
- Get started with AWS Security Agent
- Explore Amazon Bedrock Guardrails for AI content safety
- See how we’re Securing AI at AWS
- Learn about AWS Responsible AI
- Read about AWS AI Compliance
- Review our AWS Security Bulletins on emerging threats
If you have feedback about this post, submit comments in the Comments section below.
Cybersecurity in the Age of Instant Software
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/04/cybersecurity-in-the-age-of-instant-software.html
AI is rapidly changing how software is written, deployed, and used. Trends point to a future where AIs can write custom software quickly and easily: “instant software.” Taken to an extreme, it might become easier for a user to have an AI write an application on demand—a spreadsheet, for example—and delete it when you’re done using it than to buy one commercially. Future systems could include a mix: both traditional long-term software and ephemeral instant software that is constantly being written, deployed, modified, and deleted.
AI is changing cybersecurity as well. In particular, AI systems are getting better at finding and patching vulnerabilities in code. This has implications for both attackers and defenders, depending on the ways this and related technologies improve.
In this essay, I want to take an optimistic view of AI’s progress, and to speculate what AI-dominated cybersecurity in an age of instant software might look like. There are a number of unknowns that will factor into how the arms race between attacker and defender might play out.
How flaw discovery might work
On the attacker side, the ability of AIs to automatically find and exploit vulnerabilities has increased dramatically over the past few months. We are already seeing both government and criminal hackers using AI to attack systems. The exploitation part is critical here, because it gives an unsophisticated attacker capabilities far beyond their understanding. As AIs get better, expect more attackers to automate their attacks using AI. And as individuals and organizations can increasingly run powerful AI models locally, AI companies monitoring and disrupting malicious AI use will become increasingly irrelevant.
Expect open-source software, including open-source libraries incorporated in proprietary software, to be the most targeted, because vulnerabilities are easier to find in source code. Unknown No. 1 is how well AI vulnerability discovery tools will work against closed-source commercial software packages. I believe they will soon be good enough to find vulnerabilities just by analyzing a copy of a shipped product, without access to the source code. If that’s true, commercial software will be vulnerable as well.
Particularly vulnerable will be software in IoT devices: things like internet-connected cars, refrigerators, and security cameras. Also industrial IoT software in our internet-connected power grid, oil refineries and pipelines, chemical plants, and so on. IoT software tends to be of much lower quality, and industrial IoT software tends to be legacy.
Instant software is differently vulnerable. It’s not mass market. It’s created for a particular person, organization, or network. The attacker generally won’t have access to any code to analyze, which makes it less likely to be exploited by external attackers. If it’s ephemeral, any vulnerabilities will have a short lifetime. But lots of instant software will live on networks for a long time. And if it gets uploaded to shared tool libraries, attackers will be able to download and analyze that code.
All of this points to a future where AIs will become powerful tools of cyberattack, able to automatically find and exploit vulnerabilities in systems worldwide.
Automating patch creation
But that’s just half of the arms race. Defenders get to use AI, too. These same AI vulnerability-finding technologies are even more valuable for defense. When the defensive side finds an exploitable vulnerability, it can patch the code and deny it to attackers forever.
How this works in practice depends on another related capability: the ability of AIs to patch vulnerable software, which is closely related to their ability to write secure code in the first place.
AIs are not very good at this today; the instant software that AIs create is generally filled with vulnerabilities, both because AIs write insecure code and because the people vibe coding don’t understand security. OpenClaw is a good example of this.
Unknown No. 2 is how much better AIs will get at writing secure code. The fact that they’re trained on massive corpuses of poorly written and insecure code is a handicap, but they are getting better. If they can reliably write vulnerability-free code, it would be an enormous advantage for the defender. And AI-based vulnerability-finding makes it easier for an AI to train on writing secure code.
We can envision a future where AI tools that find and patch vulnerabilities are part of the typical software development process. We can’t say that the code would be vulnerability-free—that’s an impossible goal—but it could be without any easily findable vulnerabilities. If the technology got really good, the code could become essentially vulnerability-free.
Patching lags and legacy software
For new software—both commercial and instant—this future favors the defender. For commercial and conventional open-source software, it’s not that simple. Right now, the world is filled with legacy software. Much of it—like IoT device software—has no dedicated security team to update it. Sometimes it is incapable of being patched. Just as it’s harder for AIs to find vulnerabilities when they don’t have access to the source code, it’s harder for AIs to patch software when they are not embedded in the development process.
I’m not as confident that AI systems will be able to patch vulnerabilities as easily as they can find them, because patching often requires more holistic testing and understanding. That’s Unknown No. 3: how quickly AIs will be able to create reliable software updates for the vulnerabilities they find, and how quickly customers can update their systems.
Today, there is a time lag between when a vendor issues a patch and customers install that update. That time lag is even longer for large organizational software; the risk of an update breaking the underlying software system is just too great for organizations to roll out updates without testing them first. But if AI can help speed up that process, by writing patches faster and more reliably, and by testing them in some AI-generated twin environment, the advantage goes to the defender. If not, the attacker will still have a window to attack systems until a vulnerability is patched.
Toward self-healing
In a truly optimistic future, we can imagine a self-healing network. AI agents continuously scan the ever-evolving corpus of commercial and custom AI-generated software for vulnerabilities, and automatically patch them on discovery.
For that to work, software license agreements will need to change. Right now, software vendors control the cadence of security patches. Giving software purchasers this ability has implications about compatibility, the right to repair, and liability. Any solutions here are the realm of policy, not tech.
If the defense can find, but can’t reliably patch, flaws in legacy software, that’s where attackers will focus their efforts. If that’s the case, we can imagine a continuously evolving AI-powered intrusion detection, continuously scanning inputs and blocking malicious attacks before they get to vulnerable software. Not as transformative as automatically patching vulnerabilities in running code, but nevertheless valuable.
The power of these defensive AI systems increases if they are able to coordinate with each other, and share vulnerabilities and updates. A discovery by one AI can quickly spread to everyone using the affected software. Again: Advantage defender.
There are other variables to consider. The relative success of attackers and defenders also depends on how plentiful vulnerabilities are, how easy they are to find, whether AIs will be able to find the more subtle and obscure vulnerabilities, and how much coordination there is among different attackers. All this comprises Unknown No. 4.
Vulnerability economics
Presumably, AIs will clean up the obvious stuff first, which means that any remaining vulnerabilities will be subtle. Finding them will take AI computing resources. In the optimistic scenario, defenders pool resources through information sharing, effectively amortizing the cost of defense. If information sharing doesn’t work for some reason, defense becomes much more expensive, as individual defenders will need to do their own research. But instant software means much more diversity in code: an advantage to the defender.
This needs to be balanced with the relative cost of attackers finding vulnerabilities. Attackers already have an inherent way to amortize the costs of finding a new vulnerability and create a new exploit. They can vulnerability hunt cross-platform, cross-vendor, and cross-system, and can use what they find to attack multiple targets simultaneously. Fixing a common vulnerability often requires cooperation among all the relevant platforms, vendors, and systems. Again, instant software is an advantage to the defender.
But those hard-to-find vulnerabilities become more valuable. Attackers will attempt to do what the major intelligence agencies do today: find “nobody but us” zero-day exploits. They will either use them slowly and sparingly to minimize detection or quickly and broadly to maximize profit before they’re patched. Meanwhile, defenders will be both vulnerability hunting and intrusion detecting, with the goal of patching vulnerabilities before the attackers find them.
We can even imagine a market for vulnerability sharing, where the defender who finds a vulnerability and creates a patch is compensated by everyone else in the information-sharing/repair network. This might be a stretch, but maybe.
Up the stack
Even in the most optimistic future, attackers aren’t going to just give up. They will attack the non-software parts of the system, such as the users. Or they’re going to look for loopholes in the system: things that the system technically allows but were unintended and unanticipated by the designers—whether human or AI—and can be used by attackers to their advantage.
What’s left in this world are attacks that don’t depend on finding and exploiting software vulnerabilities, like social engineering and credential stealing attacks. And we have already seen how AI-generated deepfakes make social engineering easier. But here, too, we can imagine defensive AI agents that monitor users’ behaviors, watching for signs of attack. This is another AI use case, and one that I’m not even sure how to think about in terms of the attacker/defender arms race. But at least we’re pushing attacks up the stack.
Also, attackers will attempt to infiltrate and influence defensive AIs and the networks they use to communicate, poisoning their output and degrading their capabilities. AI systems are vulnerable to all sorts of manipulations, such as prompt injection, and it’s unclear whether we will ever be able to solve that. This is Unknown No. 5, and it’s a biggie. There might always be a “trusting trust problem.”
No future is guaranteed. We truly don’t know whether these technologies will continue to improve and when they will plateau. But given the pace at which AI software development has improved in just the past few months, we need to start thinking about how cybersecurity works in this instant software world.
This essay originally appeared in CSO.
EDITED TO ADD: Two essays published after I wrote this. Both are good illustrations of where we are re AI vulnerability discovery. Things are changing very fast.
Introducing workload simulation workbench for Amazon MSK Express broker
Post Syndicated from Manu Mishra original https://aws.amazon.com/blogs/big-data/introducing-workload-simulation-workbench-for-amazon-msk-express-broker/
Validating Kafka configurations before production deployment can be challenging. In this post, we introduce the workload simulation workbench for Amazon Managed Streaming for Apache Kafka (Amazon MSK) Express Broker. The simulation workbench is a tool that you can use to safely validate your streaming configurations through realistic testing scenarios.
Solution overview
Varying message sizes, partition strategies, throughput requirements, and scaling patterns make it challenging for you to predict how your Apache Kafka configurations will perform in production. The traditional approaches to test these variables create significant barriers: ad-hoc testing lacks consistency, manual set up of temporary clusters is time-consuming and error-prone, production-like environments require dedicated infrastructure teams, and team training often happens in isolation without realistic scenarios. You need a structured way to test and validate these configurations safely before deployment. The workload simulation workbench for MSK Express Broker addresses these challenges by providing a configurable, infrastructure as code (IaC) solution using AWS Cloud Development Kit (AWS CDK) deployments for realistic Apache Kafka testing. The workbench supports configurable workload scenarios, and real-time performance insights.
Express brokers for MSK Provisioned make managing Apache Kafka more streamlined, more cost-effective to run at scale, and more elastic with the low latency that you expect. Each broker node can provide up to 3x more throughput per broker, scale up to 20x faster, and recover 90% quicker compared to standard Apache Kafka brokers. The workload simulation workbench for Amazon MSK Express broker facilitates systematic experimentation with consistent, repeatable results. You can use the workbench for multiple use cases like production capacity planning, progressive training to prepare developers for Apache Kafka operations with increasing complexity, and architecture validation to prove streaming designs and compare different approaches before making production commitments.
Architecture overview
The workbench creates an isolated Apache Kafka testing environment in your AWS account. It deploys a private subnet where consumer and producer applications run as containers, connects to a private MSK Express broker and monitors for performance metrics and visibility. This architecture mirrors the production deployment pattern for experimentation. The following image describes this architecture using AWS services.
This architecture is deployed using the following AWS services:
Amazon Elastic Container Service (Amazon ECS) generate configurable workloads with Java-based producers and consumers, simulating various real-world scenarios through different message sizes and throughput patterns.
Amazon MSK Express Cluster runs Apache Kafka 3.9.0 on Graviton-based instances with hands-free storage management and enhanced performance characteristics.
Dynamic Amazon CloudWatch Dashboards automatically adapt to your configuration, displaying real-time throughput, latency, and resource utilization across different test scenarios.
Secure Amazon Virtual Private Cloud (Amazon VPC) Infrastructure provides private subnets across three Availability Zones with VPC endpoints for secure service communication.
Configuration-driven testing
The workbench provides different configuration options for your Apache Kafka testing environment, so you can customize instance types, broker count, topic distribution, message characteristics, and ingress rate. You can adjust the number of topics, partitions per topic, sender and receiver service instances, and message sizes to match your testing needs. These flexible configurations support two distinct testing approaches to validate different aspects of your Kafka deployment:
Approach 1: Workload validation (single deployment)
Test different workload patterns against the same MSK Express cluster configuration. This is useful for comparing partition strategies, message sizes, and load patterns.
Approach 2: Infrastructure rightsizing (redeploy and compare)
Test different MSK Express cluster configurations by redeploying the workbench with different broker settings while keeping the same workload. This is recommended for rightsizing experiments and understanding the impact of vertical compared to horizontal scaling.
Each redeployment uses the same workload configuration, so you can isolate the impact of infrastructure changes on performance.
Workload testing scenarios (single deployment)
These scenarios test different workload patterns against the same MSK Express cluster:
Partition strategy impact testing
Scenario: You are debating the usage of fewer topics with many partitions compared to many topics with fewer partitions for your microservices architecture. You want to understand how partition count affects throughput and consumer group coordination before making this architectural decision.
Message size performance analysis
Scenario: Your application handles different types of events – small IoT sensor readings (256 bytes), medium user activity events (1 KB), and large document processing events (8KB). You must understand how message size impacts your overall system performance and if you should separate these into different topics or handle them together.
Load testing and scaling validation
Scenario: You expect traffic to vary significantly throughout the day, with peak loads requiring 10× more processing capacity than off-peak hours. You want to validate how your Apache Kafka topics and partitions handle different load levels and understand the performance characteristics before production deployment.
Infrastructure rightsizing experiments (redeploy and compare)
These scenarios help you understand the impact of different MSK Express cluster configurations by redeploying the workbench with different broker settings:
MSK broker rightsizing analysis
Scenario: You deploy a cluster with basic configuration and put load on it to establish baseline performance. Then you want to experiment with different broker configurations to see the effect of vertical scaling (larger instances) and horizontal scaling (more brokers) to find the right cost-performance balance for your production deployment.
Step 1: Deploy with baseline configuration
Step 2: Redeploy with vertical scaling
Step 3: Redeploy with horizontal scaling
This rightsizing approach helps you understand how broker configuration changes affect the same workload, so you can improve both performance and cost for your specific requirements.
Performance insights
The workbench provides detailed insights into your Apache Kafka configurations through monitoring and analytics, creating a CloudWatch dashboard that adapts to your configuration. The dashboard starts with a configuration summary showing your MSK Express cluster details and workbench service configurations, helping you to understand what you’re testing. The following image shows the dashboard configuration summary:
The second section of dashboard shows real-time MSK Express cluster metrics including:
- Broker performance: CPU utilization and memory usage across brokers in your cluster
- Network activity: Monitor bytes in/out and packet counts per broker to understand network utilization patterns
- Connection monitoring: Displays active connections and connection patterns to help identify potential bottlenecks
- Resource utilization: Broker-level resource tracking provides insights into overall cluster health
The following image shows the MSK cluster monitoring dashboard:
The third section of the dashboard shows the Intelligent Rebalancing and Cluster Capacity insights showing:
- Intelligent rebalancing: in progress: Shows whether a rebalancing operation is currently in progress or has occurred in the past. A value of 1 indicates that rebalancing is actively running, while 0 means that the cluster is in a steady state.
- Cluster under-provisioned: Indicates whether the cluster has insufficient broker capacity to perform partition rebalancing. A value of 1 means that the cluster is under-provisioned and Intelligent Rebalancing can’t redistribute partitions until more brokers are added or the instance type is upgraded.
- Global partition count: Displays the total number of unique partitions across all topics in the cluster, excluding replicas. Use this to track partition growth over time and validate your deployment configuration.
- Leader count per broker: Shows the number of leader partitions assigned to each broker. An uneven distribution indicates partition leadership skew, which can lead to hotspots where certain brokers handle disproportionate read/write traffic.
- Partition count per broker: Shows the total number of partition replicas hosted on each broker. This metric includes both leader and follower replicas and is key to identifying replica distribution imbalances across the cluster.
The following image shows the Intelligent Rebalancing and Cluster Capacity section of the dashboard:
The fourth section of the dashboard shows the application-level insights showing:
- System throughput: Displays the total number of messages per second across services, giving you a complete view of system performance
- Service comparisons: Performs side-by-side performance analysis of different configurations to understand which approaches fit
- Individual service performance: Each configured service has dedicated throughput tracking widgets for detailed analysis
- Latency analysis: The end-to-end message delivery times and latency comparisons across different service configurations
- Message size impact: Performance analysis across different payload sizes helps you understand how message size affects overall system behavior
The following image shows the application performance metrics section of the dashboard:
Getting started
This section walks you through setting up and deploying the workbench in your AWS environment. You will configure the necessary prerequisites, deploy the infrastructure using AWS CDK, and customize your first test.
Prerequisites
You can deploy the solution from the GitHub Repo. You can clone it and run it on your AWS environment. To deploy the artifacts, you will require:
- AWS account with administrative credentials configured for creating AWS resources.
- AWS Command Line Interface (AWS CLI) must be configured with appropriate permissions for AWS resource management.
- AWS Cloud Development Kit (AWS CDK) should be installed globally using npm install -g aws-cdk for infrastructure deployment.
- Node.js version 20.9 or higher is required, with version 22+ recommended.
- Docker engine must be installed and running locally as the CDK builds container images during deployment. Docker daemon should be running and accessible to CDK for building the workbench application containers.
Deployment
After deployment is completed, you will receive a CloudWatch dashboard URL to monitor the workbench performance in real-time.You can also deploy multiple isolated instances of the workbench in the same AWS account for different teams, environments, or testing scenarios. Each instance operates independently with its own MSK cluster, ECS services, and CloudWatch dashboards.To deploy additional instances, modify the Environment Configuration in cdk/lib/config.ts:
Each combination of AppPrefix and EnvPrefix creates completely isolated AWS resources so that multiple teams or environments can use the workbench simultaneously without conflicts.
Customizing your first test
You can edit the configuration file located at folder “cdk/lib/config-types.ts” to define your testing scenarios and run the deployment. It is preconfigured with the following configuration:
Best practices
Following a structured approach to benchmarking ensures that your results are reliable and actionable. These best practices will help you isolate performance variables and build a clear understanding of how each configuration change affects your system’s behavior. Begin with single-service configurations to establish baseline performance:
After you understand the baseline, add comparison scenarios.
Change one variable at a time
For clear insights, modify only one parameter between services:
This approach helps you understand the impact of specific configuration changes.
Important considerations and limitations
Before relying on workbench results for production decisions, it is important to understand the tool’s intended scope and boundaries. The following considerations will help you set appropriate expectations and make the most effective use of the workbench in your planning process.
Performance testing disclaimer
The workbench is designed as an educational and sizing estimation tool to help teams prepare for MSK Express production deployments. While it provides valuable insights into performance characteristics:
- Results can vary based on your specific use cases, network conditions, and configurations
- Use workbench results as guidance for initial sizing and planning
- Conduct comprehensive performance validation with your actual workloads in production-like environments before final deployment
Recommended usage approach
Production readiness training – Use the workbench to prepare teams for MSK Express capabilities and operations.
Architecture validation – Test streaming architectures and performance expectations using MSK Express enhanced performance characteristics.
Capacity planning – Use MSK Express streamlined sizing approach (throughput-based rather than storage-based) for initial estimates.
Team preparation – Build confidence and expertise with production Apache Kafka implementations using MSK Express.
Conclusion
In this post, we showed how the workload simulation workbench for Amazon MSK Express Broker supports learning and preparation for production deployments through configurable, hands-on testing and experiments. You can use the workbench to validate configurations, build expertise, and improve performance before production deployment. If you’re preparing for your first Apache Kafka deployment, training a team, or improving existing architectures, the workbench provides practical experience and insights needed for success. Refer to Amazon MSK documentation – Complete MSK Express documentation, best practices, and sizing guidance for more information.
About the authors
Proactive monitoring for Amazon Redshift Serverless using AWS Lambda and Slack alerts
Post Syndicated from Cristian Restrepo Lopez original https://aws.amazon.com/blogs/big-data/proactive-monitoring-for-amazon-redshift-serverless-using-aws-lambda-and-slack-alerts/
Performance issues in analytics environments often remain invisible until they disrupt dashboards, delay ETL jobs, or impact business decisions. For teams running Amazon Redshift Serverless, unmonitored query queues, long-running queries, or unexpected spikes in compute capacity can degrade performance and increase costs if left undetected.
Amazon Redshift Serverless streamlines running analytics at scale by removing the need to provision or manage infrastructure. However, even in a serverless environment, maintaining visibility into performance and usage is essential for efficient operation and predictable costs. While Amazon Redshift Serverless provides advanced built-in dashboards for monitoring performance metrics, delivering notifications directly to platforms like Slack, brings another level of agility. Real-time alerts in the team’s workflow enable faster response times and more informed decision-making without requiring constant dashboard monitoring.
In this post, we show you how to build a serverless, low-cost monitoring solution for Amazon Redshift Serverless that proactively detects performance anomalies and sends actionable alerts directly to your selected Slack channels. This approach helps your analytics team identify and address issues early, often before your users notice a problem.
Solution overview
The solution presented in this post uses AWS services to collect key performance metrics from Amazon Redshift Serverless, evaluate them against thresholds that you can flexibly configure, and notify you when anomalies are detected.
The workflow operates as follows:
- Scheduled execution – An Amazon EventBridge rule triggers an AWS Lambda function on a configurable schedule (by default, every 15 minutes during business hours).
- Metric collection – The AWS Lambda function gathers metrics including queued queries, running queries, compute capacity (RPUs), data storage usage, table count, database connections, and slow-running queries using Amazon CloudWatch and the Amazon Redshift Data API.
- Threshold evaluation – Collected metrics are compared against your predefined thresholds that reflect acceptable performance and usage limits.
- Alerting – When a threshold is exceeded, the Lambda function publishes a notification to an Amazon SNS topic.
- Slack notification – Amazon Q Developer in Chat applications (formerly AWS Chatbot) delivers the alert to your designated Slack channel.
- Observability – Lambda execution logs are stored in Amazon CloudWatch Logs for troubleshooting and auditing.
This architecture is fully serverless and requires no changes to your existing Amazon Redshift Serverless workloads. To simplify deployment, we provide an AWS CloudFormation template that provisions all required resources.
Prerequisites
Before deploying this solution, you must collect information about your existing Amazon Redshift Serverless workgroup and namespace that you want to monitor. To identify your Amazon Redshift Serverless resources:
- Open the Amazon Redshift console.
- In the navigation pane, choose Serverless dashboard.
- Note down your workgroup and namespace names. You will use these values when launching this blog’s AWS CloudFormation template.
Deploy the solution
You can launch the CloudFormation stack and deploy the solution via the provided link.
When launching the CloudFormation stack, complete the following steps in the AWS CloudFormation Console:
- For Stack name, enter a descriptive name such as redshift-serverless-monitoring.
- Review and modify the parameters as needed for your environment.
- Acknowledge that AWS CloudFormation may create IAM resources with custom names.
- Choose Submit.
CloudFormation parameters
Amazon Redshift Serverless Workgroup configuration
Provide details for your existing Amazon Redshift Serverless environment. These values connect the monitoring solution to your Redshift environment. Some parameters come with the default values that you can replace with your actual configuration.
| Parameter | Default value | Description |
| Amazon Redshift Workgroup Name | Your Amazon Redshift Serverless workgroup name. | |
| Amazon Redshift Namespace Name | Your Amazon Redshift Serverless namespace name. | |
| Amazon Redshift Workgroup ID | Workgroup ID (UUID) of the Amazon Redshift Serverless workgroup to monitor. Must follow the UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (lowercase hexadecimal with hyphens). |
|
Namespace ID (UUID) of the Amazon Redshift Serverless namespace. Must follow the UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (lowercase hexadecimal with hyphens). |
||
| Database Name | dev |
Target Amazon Redshift database for SQL-based diagnostic and monitoring queries. |
Monitoring schedule
The default schedule runs diagnostic SQL queries every 15 minutes during business hours, balancing responsiveness and cost efficiency. Running more frequently might increase costs, while less frequent monitoring could delay detection of performance issues. You can adjust this schedule to your actual need.
| Parameter | Default value | Description |
| Schedule Expression | cron(0/15 8-17 ? * MON-FRI *) | EventBridge schedule expression for Lambda function execution. Default runs every 15 minutes, Monday through Friday, 8 AM to 5 PM UTC. |
Threshold configuration
Thresholds should be tuned based on your workload characteristics.
| Parameter | Default value | Description |
| Queries Queued Threshold | 20 | Alerts threshold for queued queries. |
| Queries Running Threshold | 20 | Alerts threshold for running queries. |
| Compute Capacity Threshold (RPUs) | 64 | Alert threshold for compute capacity (RPUs). |
| Data Storage Threshold (MB) | 5242880 | Threshold for data storage in MB (default 5 TB). |
| Table Count Threshold (MB) | 1000 | Alerts threshold for total table count. |
| Database Connections Threshold | 50 | Alert threshold for database connections. |
| Slow Query Threshold (seconds) | 10 | Thresholds in seconds for slow query detection. |
| Query Timeout (Seconds) | 30 | Timeout for SQL diagnostics queries. |
Tip: Start with conservative thresholds and refine them after observing baseline behavior for one to two weeks.
Lambda configuration
Configure the AWS Lambda function settings. The selected default values are appropriate for most monitoring scenarios. You may want to change them only in case of troubleshooting.
| Parameter | Default value | Description |
| Lambda Memory Size (MB) | 256 | Lambda function memory size in MB. |
| Lambda Time Out (Seconds) | 240 | Lambda function timeout in seconds. |
Security Configuration – Amazon Virtual Private Cloud (VPC)
If your organization has network isolation requirements, you can optionally enable VPC deployment for the Lambda function. When enabled, the Lambda function runs within your specified VPC subnets, providing network isolation and allowing access to VPC-only resources.
| Parameter | Default value | Description |
| VPC ID | VPC ID for Lambda deployment (required if EnableVPC is true). The Lambda function will be deployed in this VPC. Ensure that the VPC has appropriate routing (NAT Gateway or VPC Endpoints) to allow Lambda to access AWS services like CloudWatch, Amazon Redshift, and Amazon SNS. |
|
| VPC Subnet IDs | Comma-separated list of subnet IDs for Lambda deployment (required if EnableVPC is true). |
|
| Security Group IDs | Comma-separated list of security group IDs for Lambda (optional). If not provided and EnableVPC is true, a default security group will be created with outbound HTTPS access. Custom security groups must allow outbound HTTPS (port 443) to AWS service endpoints. |
Note that VPC deployment might increase cold start times and requires an NAT Gateway or VPC endpoints for AWS service access. We recommend provisioning interface VPC endpoints (through AWS PrivateLink) for the five services the Lambda function calls which keeps all traffic private without the recurring cost of a NAT Gateway.
Security configuration – Encryption
If your organization requires encryption of data at rest, you can optionally enable AWS Key Management Service (AWS KMS) encryption for the Lambda function’s environment variables, CloudWatch Logs, and SNS topic. When enabled, the template encrypts each resource using the AWS KMS keys that you provide, either a single shared key for all three services, or individual keys for granular key management and audit separation.
| Parameter | Default value | Description |
| Shared KMS Key ARN | AWS KMS key ARN to use for all encryption (Lambda, Logs, and SNS) unless service-specific keys are provided. This streamlines key management by using a single key for all services. The key policy must grant encrypt/decrypt permissions to Lambda, CloudWatch Logs, and SNS. | |
| Lambda KMS Key ARN | AWS KMS key ARN for Lambda environment variable encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant decrypt permissions to the Lambda execution role. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true. |
|
| CloudWatch Logs KMS Key ARN | AWS KMS key ARN for CloudWatch Logs encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to the CloudWatch Logs service. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true. |
|
| SNS Topic KMS Key ARN | AWS KMS key ARN for SNS topic encryption (optional, overrides SharedKMSKeyArn). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to SNS service and the Lambda execution role. If not provided, SharedKMSKeyArn will be used when EnableKMSEncryption is true. |
|
| Enable Dead Letter Queue | False | Optionally enable Dead Letter Queue (DLQ) for failed Lambda invocations to improve reliability and security monitoring. When enabled, events that fail after all retry attempts will be sent to an SQS queue for investigation and potential replay. This helps prevent data loss, provides visibility into failures, and enables security audit trails for monitoring anomalies. The DLQ retains messages for 14 days. |
Note that AWS KMS encryption requires the key policy to grant appropriate permissions to each consuming service (Lambda, CloudWatch Logs, and SNS).
- On the review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
- Choose Submit.
Resources created
The CloudFormation stack creates the following resources:
- EventBridge rule for scheduled execution
- AWS Lambda function (Python 3.12 runtime)
- Amazon SNS topic for alerts
- IAM role with permissions for CloudWatch, Amazon Redshift Data API, and SNS
- CloudWatch Log Group for Lambda logs
Note: CloudFormation deployment typically takes 10–15 minutes to complete. You can monitor progress in real time under the Events tab of your CloudFormation stack.
Post-deployment configuration
After the CloudFormation stack has been successfully created, complete the following steps.
Step 1: Record CloudFormation outputs
- Navigate to the AWS CloudFormation console.
- Select your stack and choose the Outputs tab.
- Note the values for LambdaRoleArn and SNSTopicArn. You will need these in subsequent steps.
Step 2: Grant Amazon Redshift permissions
Grant permissions to the Lambda function to query Amazon Redshift system tables for monitoring data. Complete the following steps to grant the necessary access:
- Navigate to the Amazon Redshift console.
- In the left navigation pane, choose Query Editor V2.
- Connect to your Amazon Redshift Serverless workgroup.
- Execute the following SQL commands, replacing <IAM Role ARN> with the LambdaRoleArn value from your CloudFormation outputs:
These commands create an AmazonRedshift user associated with the Lambda IAM role and grant it the sys:monitor Amazon Redshift role. This role provides read-only access to catalog and system tables without granting permissions to user data tables.
Step 3: Configure Slack notifications
Amazon Q Developer in chat applications provides native AWS integration and managed authentication, removing custom webhook code and reducing setup complexity. To receive alerts in Slack, configure Amazon Q Developer in Chat Applications to connect your SNS topic to your preferred Slack channel:
- Navigate to Amazon Q Developer in chat applications (formerly AWS Chatbot) in the AWS console.
- Follow the instructions in the Slack integration documentation to authorize AWS access to your Slack workspace.
- When configuring the Slack channel, ensure that you select the correct AWS Region where you deployed the CloudFormation stack.
- In the Notifications section, select the SNS topic created by your CloudFormation stack (refer to the SNSTopicArn output value).
- Keep the default IAM read-only permissions for the channel configuration.
After configured, alerts automatically appear in Slack whenever thresholds are exceeded.
Cost considerations
With the default configuration, this solution incurs minimal ongoing costs. The Lambda function executes approximately 693 times per month (every 15 minutes during an 8-hour business day, Monday through Friday), resulting in a monthly cost of approximately $0.33 USD. This includes Lambda compute costs ($0.26) and CloudWatch GetMetricData API calls ($0.07). All other services (EventBridge, SNS, CloudWatch Logs, and Amazon Redshift Data API). The Amazon Redshift Data API has no additional charges beyond the minimal Amazon Redshift Serverless RPU consumption for the Amazon Redshift Serverless system table query execution. You can reduce costs by decreasing the monitoring frequency (such as, every 30 minutes) or increase responsiveness by running more frequently (such as, every 5 minutes) with a proportional cost increase.
All costs are estimates and may vary based on your environment. Variations often occur because queries scanning system tables may take longer or require additional resources depending on the system complexity
Security best practices
This solution implements the following security controls:
- IAM policies scoped to specific resource ARNs for the Amazon Redshift workgroup, namespace, SNS topic, and log group.
- Data API statement access restricted to the Lambda function’s own IAM user ID.
- Read-only
sys:monitordatabase role for operational metadata access. Limit to the role created by the CloudFormation template. - Reserved concurrent executions capped at five.
To further strengthen your security posture, consider the following enhancements:
- Enable
EnableKMSEncryptionto encrypt environment variables, logs, and SNS messages at rest. - Enable
EnableVPCto deploy the function within a VPC for network isolation. - Audit access through AWS CloudTrail.
Important: This is sample code for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. This solution demonstrates monitoring capabilities but requires additional security hardening for production environments, including encryption configuration, IAM policy scoping, VPC deployment, and comprehensive testing.
Clean up
To remove all resources and avoid ongoing charges if you don’t want to use the solution anymore:
- Delete the CloudFormation stack.
- Remove the Slack integration from Amazon Q Developer in chat applications.
Troubleshooting
- If no metrics or incomplete SQL diagnostics are returned, verify that the Amazon Redshift Serverless workgroup is active with recent query activity, and ensure the database user has the
sys:monitorrole (GRANT ROLE sys:monitor TO <user>) in the query editor. Without this role, queries execute successfully but only return data visible to that user’s permissions rather than the full cluster activity. - For VPC-deployed functions that fail to reach AWS services, confirm that VPC endpoints or a NAT Gateway are configured for CloudWatch, Amazon Redshift Data API, Amazon Redshift Serverless, SNS, and CloudWatch Logs.
- If the Lambda function times out, increase the
LambdaTimeoutandQueryTimeoutSecondsparameters. The default timeout of 240 seconds accommodates most workloads, but clusters with many active queries may require additional time for SQL diagnostics to complete.
Conclusion
In this post, we showed how you can build a proactive monitoring solution for Amazon Redshift Serverless using AWS Lambda, Amazon CloudWatch, and Amazon SNS with Slack integration. By automatically collecting metrics, evaluating thresholds, and delivering alerts in near real time to Slack or your preferred collaborative platform, this solution helps detect performance and cost issues early. Because the solution itself is serverless, it aligns with the operational simplicity goals of Amazon Redshift Serverless—scaling automatically, requiring minimal maintenance, and delivering high value at low cost. You can extend this foundation with additional metrics, diagnostic logic, or alternative notification channels to meet your organization’s needs.
To learn more, see the Amazon Redshift documentation on monitoring and performance optimization.
About the authors
Distributed by Design: Building a Truly Remote Backblaze
Post Syndicated from Elisa Ramos Miller original https://www.backblaze.com/blog/distributed-by-design-building-a-truly-remote-backblaze/
In 2007, Backblaze started in a one-bedroom Palo Alto apartment. Since those days of hand-assembling Storage Pods, we’ve grown to manage over 5 exabytes of data for 500,000 customers across 175 countries. As our mission to make customers unstoppable grew, our team naturally grew with it, moving far beyond the walls of any single office.
The pandemic gave us the final piece of data we needed to evolve. When the world reopened, our San Mateo headquarters never quite looked the same. The office became a bit of a paradox: a large, quiet space where a handful of people moved through a space designed for hundreds. Meanwhile, the true heart and soul of the company was already thriving elsewhere, solving the world’s toughest storage challenges from data centers in Phoenix, spare bedrooms in Austin, and kitchen tables in Cheltenham.
We didn’t rush this. We experimented, explored, and really listened to our team. When we looked at the data, we found only around 5% of our teams worked out of the San Mateo office regularly. By testing new ways of collaborating, we confirmed what we already suspected: our culture isn’t tied to a physical floor plan, it’s tied to our shared commitment to our customers and each other. As of February 1, 2026, we made it official: Backblaze has moved beyond the hybrid model to become a fully remote, distributed company.
Listening, learning, and being practical
This transition isn’t about a grand corporate strategy, but about meeting our employees where they’re at—everywhere. While we appreciate the flexibility, we noticed that hybrid models can unintentionally create two different cultures, one for those who can make it into the office and another for everyone else. Whether an employee is in San Jose or the Philippines, we want them to have the same access to leadership, and each other. We’re now a team that finally matches the diversity and reach of the 175 countries we serve.
Beyond the culture, this is also about being good stewards of our financial resources. Holding onto an expensive, mostly empty office in one of the world’s costliest real estate markets was not a good investment. By letting go of the San Mateo office, we can more responsibly direct those investments toward our people—funding intentional collaboration tools, resources, and smaller local gatherings that strengthen connection and culture.
Let’s talk about the bedrock : Our data centers
While the “cloud” often feels like a metaphor, our Data Center Technicians know it’s built on hardware, precision, and physical presence. When it comes to data centers, there is no shifting the in-person work. Someone needs to be there to swap out a drive, and when it comes to making sure we’re always-on, that is a 24/7/365 commitment.
They are the heartbeat of Backblaze. Every day, they show up in person to the facilities where data actually lives, keeping the drives spinning to ensure that your data is always available and accessible.
Remote flexibility is a privilege afforded to the rest of the company by the physical excellence of our Data Center teams. They are the essential anchor that allows the rest of us to be weightless. We celebrate their work not as an exception to our model, but as the very foundation that makes our distributed future possible.
We’re still figuring it out
We are still finding new ways to connect. This isn’t a static policy, but a continuous effort to make sure people can be successful wherever they work. We’re replacing accidental hallway chats with deliberate rituals, from cross-functional coffee chats and regional off-sites with local talent and execs, to providing on-demand access to professional workspaces for solo work or we want to assemble together.
Closing the San Mateo office isn’t a retreat from our history; it’s an embrace of the company we’ve already become. As we grow across new markets and time zones, we’re committed to raising the bar on what distributed work can look like so our teams can collaborate in ways that drive real impact around the world. We are building a smarter, more intentional Backblaze, one that is as distributed as the data we protect and as resilient as the customers we serve.
The post Distributed by Design: Building a Truly Remote Backblaze appeared first on Backblaze Blog | Cloud Storage & Cloud Backup
The Dangers of Outsourcing Critical Tasks to AI
Post Syndicated from The Atlantic original https://www.youtube.com/shorts/KkwEh5Bxdio
All of Wikipedia + World Maps + AI Models = How Much Space?
Post Syndicated from Crosstalk Solutions original https://www.youtube.com/shorts/BNXpXPYV-5A
[$] Sharing stories on Scuttlebutt
Post Syndicated from daroc original https://lwn.net/Articles/1065471/
Not many people live on sailboats. Things may be better these days, but
back in 2014 sailboat dwellers had
to contend with lag-prone,
intermittent, low-bandwidth internet connections. Dominic Tarr
decided
to fix the problem of keeping up with his friends by developing a delay-tolerant,
fully distributed social-media protocol called
Scuttlebutt. Nearly twelve
years later, the protocol has gained a number of users who have their own,
non-sailboat-related reasons to prefer a censorship-resistant,
offline-first social-media system.
Security updates for Tuesday
Post Syndicated from jzb original https://lwn.net/Articles/1066665/
Security updates have been issued by AlmaLinux (crun, kernel, and kernel-rt), Debian (dovecot), Fedora (calibre and nextcloud), Mageia (freerdp, polkit-122, python-nltk, python-pyasn1, vim, and xz), Red Hat (edk2 and openssl), SUSE (avahi, cockpit, python-pyOpenSSL, python311, and tar), and Ubuntu (lambdaisland-uri-clojure, linux-gcp, linux-gcp-4.15, linux-gcp-fips, linux-oem-6.17, and linux-realtime-6.17).
A First Look at Our Speaker Lineup and Agenda for the Rapid7 2026 Global Cybersecurity Summit
Post Syndicated from Emma Burdett original https://www.rapid7.com/blog/post/it-speaker-lineup-first-look-agenda-rapid7-2026-global-cybersecurity-summit
The agenda for the Rapid7 2026 Global Cybersecurity Summit is starting to take shape, and with it, a clearer picture of the conversations security teams need to be having right now.
Taking place May 12–13, this year’s summit brings together a mix of security leaders, practitioners, analysts, and industry voices to explore how organizations are moving from reactive defense to preemptive security operations. The focus is practical. What is changing, what is not working, and what teams need to do differently.
Voices from across the industry
This year’s lineup reflects that shift. Alongside Rapid7 experts and customer speakers, the summit will feature well-known voices from across the security community.
Rachel Tobac, CEO of SocialProof Security, joins the keynote panel The Reality of Running a SOC in 2026, bringing a perspective grounded in how modern attacks actually begin and how attackers adapt in real time. She is joined by cybersecurity speaker and “Smashing Security” podcast host Graham Cluley, whose work has long focused on translating complex threats into practical understanding for security teams.
From an analyst perspective, Craig Robinson of IDC and Dave Gruber of Omdia add an external view on how the market is evolving, where organizations are investing, and how security programs are being measured. Their contributions help ground the discussion in broader industry trends, not just individual experiences.
Customer voices also play a central role. Leaders from organizations such as Netscout Systems, Target RWE, and Miltenyi Biotecwill share how they are navigating complexity, validating decisions around MDR and platform consolidation, and focusing on outcomes rather than activity.
What to expect during the show
Across two days, the summit is structured to reflect how security teams actually operate.
Day one focuses on shared context with sessions like Defense Starts Earlier Than You Think and The Reality of Running a SOC in 2026 examining how the threat landscape has shifted and why traditional approaches are struggling to keep pace. From there, sessions such as Inside the Modern SOC and Using Red Teaming to Power Preemptive MDR move into how detection, response, and validation work in practice.
The goal is to connect the full picture: how attacks begin, how they progress, and how teams respond when it matters.
Day two is more focused on the unique needs of particular security roles. The two dedicated tracks allow attendees to go deeper into the implications of modern security evolution based on their daily realities.
For security leaders, sessions such as The CISO’s Role in Enterprise Transformation and A CISO’s Guide to MDR Accountability and Outcomes explore governance, accountability, and ways to measure effectiveness that reflect real business risk.
For practitioners, sessions like Hunt or Be Hunted and IR in Practice focus on the mechanics of investigation, detection and response. These sessions look closely at how analysts triage signals, how decisions are made under pressure, and how response workflows hold up in real environments.
Across both days, the agenda is designed to move beyond theory with each session connecting back to the same core concept. Security teams need to act earlier, reduce uncertainty, and make decisions with greater confidence.
Secure your spot
If you are looking to understand how security operations are evolving, and how teams are putting that into practice, this is where those conversations come together.
Join us May 12–13 and see how organizations are building more resilient, preemptive security operations.
THG Podcast – Counterfactuals: The Gregorian Calendar
Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=mTaUUMvlyLo
Hong Kong Police Can Force You to Reveal Your Encryption Keys
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/04/hong-kong-police-can-force-you-to-reveal-your-encryption-keys.html
According to a new law, the Hong Kong police can demand that you reveal the encryption keys protecting your computer, phone, hard drives, etc.—even if you are just transiting the airport.
In a security alert dated March 26, the U.S. Consulate General said that, on March 23, 2026, Hong Kong authorities changed the rules governing enforcement of the National Security Law. Under the revised framework, police can require individuals to provide passwords or other assistance to access personal electronic devices, including cellphones and laptops.
The consulate warned that refusal to comply is now a criminal offense. It also said authorities have expanded powers to take and keep personal electronic devices as evidence if they claim the devices are linked to national security offenses.
Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale
Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e
By Ben Sykes
In a previous post, we described how Netflix uses Apache Druid to ingest millions of events per second and query trillions of rows, providing the real-time insights needed to ensure a high-quality experience for our members. Since that post, our scale has grown considerably.
With our database holding over 10 trillion rows and regularly ingesting up to 15 million events per second, the value of our real-time data is undeniable. But this massive scale introduced a new challenge: queries. The live show monitoring, dashboards, automated alerting, canary analysis, and A/B test monitoring that are built on top of Druid became so heavily relied upon that the repetitive query load started to become a scaling concern in itself.
This post describes an experimental caching layer we built to address this problem, and the trade-offs we chose to accept.
The Problem
Our internal dashboards are heavily used for real-time monitoring, especially during high-profile live shows or global launches. A typical dashboard has 10+ charts, each triggering one or more Druid queries; one popular dashboard with 26 charts and stats generates 64 queries per load. When dozens of engineers view the same dashboards and metrics for the same event, the query volume quickly becomes unmanageable.
Take the popular dashboard above: 64 queries per load, refreshing every 10 seconds, viewed by 30 people. That’s 192 queries per second from one dashboard, mostly for nearly identical data. We still need Druid capacity for automated alerting, canary analysis, and ad-hoc queries. And because these dashboards request a rolling last-few-hours window, each refresh changes slightly as the time range advances.
Druid’s built-in caches are effective. Both the full-result cache and the per-segment cache. But neither is designed to handle the continuous, overlapping time-window shifts inherent to rolling-window dashboards. The full-result cache misses for two reasons.
- If the time window shifts even slightly, the query is different, so it’s a cache miss.
- Druid deliberately refuses to cache results that involve realtime segments (those still being indexed), because it values deterministic, stable cache results and query correctness over a higher cache hit rate.
The per-segment cache does help avoid redundant scans on historical nodes, but we still need to collect those cached segment results from each data node and merge them in the brokers with data from the realtime nodes for every query.
During major shows, rolling-window dashboards can generate a flood of near-duplicate queries that Druid’s caches mostly miss, creating heavy redundant load. At our scale, solving this by simply adding more hardware is prohibitively expensive.
We needed a smarter approach.
The Insight
When a dashboard requests the last 3 hours of data, the vast majority of that data, everything except the most recent few minutes, is already settled. The data from 2 hours ago won’t change.
What if we could remember the older portions of the result and only ask Druid for the part that’s actually new?
This is the core idea behind a new caching service that understands the structure of Druid queries and serves previously-seen results from cache while fetching only the freshest portion from Druid.
A Deliberate Trade-Off
Before diving into the implementation, it’s worth being explicit about the trade-off we’re making. Caching query results introduces some staleness, specifically, up to 5 seconds for the newest data. This is acceptable for most of our operational dashboards, which refresh every 10 to 30 seconds. In practice, many of our queries already set an end time of now-1m or now-5s to avoid the “flappy tail” that can occur with currently-arriving data.
Since our end-to-end data pipeline latency is typically under 5 seconds at P90, a 5-second cache TTL on the freshest data introduces negligible additional staleness on top of what’s already inherent in the system. We decided it was better to accept this small amount of staleness in exchange for significantly lower query load on Druid. But a 5s cache on its own is not very useful.
Exponential TTLs
Not all data points are equally trustworthy. In real-time analytics, there’s a well-known late-arriving data problem. Events can arrive out of order or be delayed in the ingestion pipeline. A data point from 30 seconds ago might still change as late-arriving events trickle in. A data point from 30 minutes ago is almost certainly final.
We use this observation to set cache TTLs that increase exponentially with the age of the data. Data less than 2 minutes old gets a minimum TTL of 5 seconds. After that, the TTL doubles for each additional minute of age: 10 seconds at 2 minutes old, 20 seconds at 3 minutes, 40 seconds at 4 minutes, and so on, up to a maximum TTL of 1 hour.
The effect is that fresh data cycles through the cache rapidly, so any corrections from late-arriving events in the most recent couple of minutes are picked up quickly. Older data lingers much longer, because our confidence in its accuracy grows with time.
For a 3-hour rolling window, the exponential TTL ensures the vast majority of the query is served from the cache, leaving Druid to only scan the most recent, unsettled data.
Bucketing
If we were to use a single-level cache key for the query and interval, similar to Druid’s existing result-level cache, we wouldn’t be able to extract only the relevant time range from cached results. A shifted window means a different key, which means a cache miss.
Instead, we use a map-of-maps. The top-level key is the query hash without the time interval; the inner keys are timestamps bucketed to the query granularity (or 1 minute, whichever is larger) and encoded as big-endian bytes so lexicographic order matches time. This enables efficient range scans; fetching all cached buckets between times A and B for a query hash. A 3-hour query at 1-minute granularity becomes 180 independent cached buckets, each with its own TTL; when the window shifts (e.g., 30 seconds later), we reuse most buckets from cache and only query Druid for the new data.
How It Works
Today, the cache runs as an external service integrated transparently by intercepting requests at the Druid Router and redirecting them to the cache. If the cache fully satisfies a request, it returns the result; otherwise it shrinks the time interval to the uncached portion and calls back into the Router, bypassing the redirect to query Druid normally. Non-cached requests (e.g., metadata queries or queries without time group-bys) pass straight through to Druid unchanged.
This intercepting proxy design allows us to enable or disable caching without any client changes and is a key to its adoption. We see this setup as temporary while we work out a way to better integrate this capability into Druid more natively.
When a cacheable query arrives, those that are grouping-by time (timeseries, groupBy), the cache performs the following steps.
Parsing and Hashing. We parse each incoming query to extract the time interval, granularity, and structure, then compute a SHA-256 hash of the query with the time interval and parts of the context removed. That hash is the cache key: it encodes what is being asked (datasource, filters, aggregations, granularity) but not when, so the same logical query over different overlapping time windows maps to the same cache entry. There are some context properties that can alter the response structure or contents, so these are included in the cache-key.
Cache Lookup. Using the cache key, we fetch cached points within the requested range, but only if they’re contiguous from the start. Because bucket TTLs can expire unevenly, gaps can appear; when we hit a gap, we stop and fetch all newer data from Druid. This guarantees a complete, unbroken result set while sending at most one Druid query, rather than “filling gaps” with multiple small, fragmented queries that would increase Druid load.
Fetching the Missing Tail. On a partial cache hit (e.g., 2h 50m of a 3h window), we rebuild the query with a narrowed interval for the missing 10 minutes and send only that to Druid. Since Druid then scans just the recent segments for a small time range, the query is usually faster and cheaper than the original.
Combining. The cached data and fresh data are concatenated, sorted by timestamp, and returned to the client. From the client’s perspective, the response looks identical to what Druid would have returned, same JSON format, same fields.
Asynchronous Caching. The fresh data from Druid is parsed into individual time-granularity buckets and written back to the cache asynchronously, so we don’t add latency to the response path.
Negative Caching
Some metrics are sparse. Certain time buckets may genuinely have no data. Without special handling, the cache would treat these empty buckets as gaps and re-query Druid for them every time.
We handle this by caching empty sentinel values for time buckets where Druid returned no data. Our gap-detection logic recognizes these empty entries as valid cached data rather than missing data, preventing needless re-queries for naturally sparse metrics.
However, we’re careful not to negative-cache trailing empty buckets. If a query returns data up to minute 45 and nothing after, we only cache empty entries for gaps between data points, not after the last one. This avoids incorrectly caching “no data” for time periods where events simply haven’t arrived yet, which would exacerbate the chart delays of late arriving data.
The Storage Layer
For the backing store, we use Netflix’s Key-Value Data Abstraction Layer (KVDAL), backed by Cassandra. KVDAL provides a two-level map abstraction, a natural fit for our needs. The outer key is the query hash, and the inner keys are timestamps. Crucially, KVDAL supports independent TTLs on each inner key-value pair, eliminating the need for us to manage cache eviction manually.
This two-level structure gives us efficient range queries over the inner keys, which is exactly what we need for partial cache lookups: “give me all cached buckets between time A and time B for query hash X.”
Results
The biggest win is during high-volume events (e.g., live shows): when many users view the same dashboards, the cache serves most identical queries as full hits, so the query rate reaching Druid is essentially the same with 1 viewer or 100. The scaling bottleneck moves from Druid’s query capacity to the much cheaper-to-scale cache, and with ~5.5 ms P90 cache responses, dashboards load faster for everyone.
On a typical day, 82% of real user queries get at least a partial cache hit, and 84% of result data is served from cache. As a result, the queries that reach Druid scan much narrower time ranges, touching fewer segments and processing less data, freeing Druid to focus on aggregating the newest data instead of repeatedly re-querying historical segments.
An experiment validated this, showing about a 33% drop in queries to Druid and a 66% improvement in overall P90 query times. It also cut result bytes and segments queried, and in some cases, enabling the cache reduced result bytes by more than 14x. Caveat: the size of these gains depends heavily on how similar and repetitive the query workload is.
Looking Ahead
This caching layer is still experimental, but results are promising and we’re exploring next steps. We’ve added partial support for templated SQL so dashboard tools can benefit without writing native Druid queries.
Longer term, we’d like interval-aware caching to be built into Druid: an external proxy adds infrastructure to manage, extra network hops, and workarounds (like SQL templating) to extract intervals. Implemented inside Druid, it could be more efficient, with direct access to the query planner and segment metadata, and benefit the broader community without custom infrastructure. We’d likely ship it as an opt-in, configurable, result-level cache in the Brokers, with metrics to tune TTLs and measure effectiveness. Please leave a comment if you have a use-case that could benefit from this feature.
More broadly, this strategy, splitting time-series results into independently cached, granularity-aligned buckets with age-based exponential TTLs, isn’t Druid-specific and could apply to any time-series database with frequent overlapping-window queries.
Summary
As more Netflix teams rely on real-time analytics, query volume grows too. Dashboards are essential at our scale, but their popularity can become a scaling bottleneck. By inserting an intelligent cache between dashboards and Druid, one that understands query structure, breaks results into granularity-aligned buckets, and trades a small amount of staleness for much lower Druid load, we’ve increased query capacity without scaling infrastructure proportionally, and hope to deliver these benefits to the Druid community soon as a built-in Druid feature.
Sometimes the best way to handle a flood of queries is to stop answering the same question twice.
Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Unlock efficient model deployment: Simplified Inference Operator setup on Amazon SageMaker HyperPod
Post Syndicated from Shreya Gangishetty original https://aws.amazon.com/blogs/architecture/unlock-efficient-model-deployment-simplified-inference-operator-setup-on-amazon-sagemaker-hyperpod/
Amazon SageMaker HyperPod offers an end-to-end experience supporting the full lifecycle of AI development—from interactive experimentation and training to inference and post-training workflows. The SageMaker HyperPod Inference Operator is a Kubernetes controller that manages the deployment and lifecycle of models on HyperPod clusters, offering flexible deployment interfaces (kubectl, Python SDK, SageMaker Studio UI, or HyperPod CLI), advanced autoscaling with dynamic resource allocation, and comprehensive observability that tracks critical metrics like time-to-first-token, latency, and GPU utilization.
Deploying inference workloads on Kubernetes-native infrastructure has traditionally required AI teams to navigate a maze of Helm charts, IAM role configurations, dependency management, and manual upgrades — often taking hours before a single model can serve predictions. Today, we’re announcing the Amazon SageMaker HyperPod Inference Operator as a native EKS add-on, enabling one-click installation and managed upgrades directly from the SageMaker console. This eliminates the need for manual Helm charts, complex IAM configuration tweaks, and downtime during upgrades.
In this post, we walk through the new installation experience, demonstrate three deployment methods (console, CLI, and Terraform), and show how features like multi-instance-type deployment and native node affinity give you fine-grained control over inference scheduling
Simplified installation experience
The new installation experience addresses three key customer scenarios with streamlined workflows:
New HyperPod clusters: Automatic installation
When creating new HyperPod clusters through the SageMaker console’s Quick Setup or Custom Setup workflows, the Inference Operator along with necessary dependencies is now installed through EKS add-on automatically as part of the cluster creation process. This eliminates the need for post-deployment configuration and ensures your cluster is ready for model deployments immediately upon creation along with one click upgrades.
Existing clusters: One-click installation
For existing HyperPod clusters, customers can install the Inference Operator with a single click through the SageMaker console. The installation automatically:
- Creates required IAM roles with appropriate trust relationships and permissions
- Sets up S3 buckets for TLS certificate storage
- Configures VPC endpoints for secure S3 access
- Installs dependency add-ons (cert-manager, S3 CSI driver, FSx CSI driver, metrics-server)
- Deploys the Inference Operator as an EKS add-on
Managed upgrades and lifecycle
The EKS add-on integration provides standardized version management with one-click upgrades through the AWS console or CLI. This ensures customers can easily adopt new features and security updates without complex manual procedures.
The below prerequisite resources are needed to be setup before installing the Inference operator add-on. These prerequisites will be setup if SageMaker AI console is used to setup Inference operator. However, if EKS cli or console is used, these prerequisites will need to be created manually and passed to the add-on through configuration parameters. We discuss these approaches in Installation Methods.
List of prerequisites
- EKS add-ons (S3 Mountpoint csi driver add-on, FsX add-on, Cert Manager add-on, Metrics server add-on)
- IAM roles (Inference operator execution role, ALB role, KEDA role, Optional JumpStart Gated models role)
- Infrastructure (S3 bucket to manage TLS certificates, OIDC association on the cluster,
For more information refer to this trouble shooting guide.
Installation methods
Method 1: Install SageMaker HyperPod Inference Add-on through SageMaker UI (Recommended)
The SageMaker console provides the most streamlined experience with two installation options:
Quick install: Automatically creates all required resources with optimized defaults, including IAM roles, S3 buckets, and dependency add-ons. This option is ideal for getting started quickly with minimal configuration decisions.
Custom install: Provides flexibility to specify existing resources or customize configurations while maintaining the one-click experience. Customers can choose to reuse existing IAM roles, S3 buckets, or dependency add-ons based on their organizational requirements.
Prerequisites
- An existing Amazon SageMaker HyperPod cluster with EKS orchestration
- IAM permissions for EKS cluster administration
- kubectl configured for cluster access
Installation steps
- Navigate to the SageMaker Console: Go to HyperPod Clusters → Cluster Management
- Select Your Cluster: Choose the cluster where you want to install the Inference Operator
- Choose Installation Type: Navigate to Inference tab. Select Quick Install for automated setup or Custom Install for configuration flexibility
- Configure Options: If choosing Custom Install, specify existing resources or customize settings as needed
- Install: Choose Install to begin the automated installation process
- Verify: Check the installation status through the console, or by running
kubectl get pods -n hyperpod-inference-system, or by checking the add-on status withaws eks describe-addon --cluster-name CLUSTER-NAME --addon-name amazon-sagemaker-hyperpod-inference --region REGION
After the add-on is successfully installed, you can deploy models using the Model deployments document or navigate to Deploying Your First Model section below.
Method 2: Install SageMaker HyperPod Inference add-on through EKS APIs
For customers preferring command-line workflows, the Inference Operator can be installed directly using the EKS CLI. Note that all prerequisite resources (IAM roles, S3 buckets, VPC endpoints) and dependency add-ons must be created manually before installing the Inference Operator add-on. For detailed setup instructions, see the installation guide.
aws eks create-addon \
--cluster-name my-hyperpod-cluster \
--addon-name amazon-sagemaker-hyperpod-inference \
--addon-version v1.0.0-eksbuild.1 \
--configuration-values '{
"executionRoleArn": "arn:aws:iam::ACCOUNT-ID:role/SageMakerHyperPodInference-inference-role",
"tlsCertificateS3Bucket": "hyperpod-tls-certificate-bucket",
"hyperpodClusterArn": "arn:aws:sagemaker:REGION:ACCOUNT-ID:cluster/CLUSTER-ID",
"alb": {
"serviceAccount": {
"create": true,
"roleArn": "arn:aws:iam::ACCOUNT-ID:role/alb-controller-role"
}
},
"keda": {
"auth": {
"aws": {
"irsa": {
"roleArn": "arn:aws:iam::ACCOUNT-ID:role/keda-operator-role"
}
}
}
}
}' \
--region us-west-2
Method 3: Install SageMaker HyperPod Inference add-on through Terraform deployment
Organizations utilizing Terraform for Infrastructure as Code (IaC) can deploy HyperPod clusters using the provided modules in the awesome-distributed-training GitHub repository.
To enable the HyperPod inference operator, set the create_hyperpod_inference_operator_module variable to true within your custom.tfvars file, as shown below:
kubernetes_version = "1.33"
eks_cluster_name = "tf-eks-cluster"
hyperpod_cluster_name = "tf-hp-cluster"
resource_name_prefix = "tf-eks-test"
aws_region = "us-east-1"
instance_groups = [
{
name = "accelerated-instance-group-1"
instance_type = "ml.g5.8xlarge",
instance_count = 2,
availability_zone_id = "use1-az2",
ebs_volume_size_in_gb = 100,
threads_per_core = 1,
enable_stress_check = false,
enable_connectivity_check = false,
lifecycle_script = "on_create.sh"
}
]
create_hyperpod_inference_operator_module = true
In addition to the HyperPod inference operator add-on, the Terraform modules also support the task governance, training operator, and observability add-ons as well. Check out the documentation for enabling optional add-ons for more details.
Dependency management
The HyperPod inference operator includes several additional dependencies, which are enabled by default but can be toggled off if they already exist on your EKS cluster:
| Dependency | Module/Variable | Toggle to Disable |
| cert-manager | Installed via the HyperPod module | enable_cert_manager = false |
| Amazon FSx for Lustre CSI | Installed via FSx module | create_fsx_module = false |
| Mountpoint for Amazon S3 CSI | Bundled with Inference Operator Module | enable_s3_csi_driver = false |
| AWS Load Balancer Controller | Bundled with Inference Operator EKS add-on | enable_alb_controller = false |
| KEDA Operator | Bundled with Inference Operator EKS add-on | enable_keda = false |
Key benefits
Faster time to value
Teams can now deploy their first inference endpoint within minutes of cluster creation, compared to the previous multi-hour setup process. This acceleration enables faster experimentation and reduces the barrier to adoption for new teams.
Reduced complexity
The new installation experience eliminates the need to manually create and configure multiple AWS resources. Previously, customers needed to create IAM roles, policies, S3 buckets, VPC endpoints, and install multiple Kubernetes operators. Now, a single action handles all these requirements automatically.
Consistent configuration
Automated resource creation ensures consistent, secure configurations across environments. The installation process follows AWS best practices for IAM permissions, network security, and resource naming conventions.
Simplified upgrades
EKS Add-on integration provides standardized upgrade paths with rollback capabilities. Customers can confidently adopt new features and security updates through the familiar AWS console or CLI interfaces.
Advanced features integration
The simplified installation experience seamlessly integrates with advanced HyperPod inference capabilities:
Managed tiered KV cache
During installation, customers can optionally enable managed tiered KV cache with intelligent memory allocation based on instance types. This feature can reduce inference latency by up to 40% for long-context workloads while optimizing memory utilization across the cluster.
Intelligent routing
The installation automatically configures intelligent routing capabilities with multiple strategies (prefix-aware, KV-aware, round-robin) to maximize cache efficiency and minimize inference latency based on workload characteristics.
Observability integration
Built-in integration with HyperPod Observability provides immediate visibility into inference metrics, cache performance, and routing efficiency through Amazon Managed Grafana dashboards.
Deploying your first model
Once the add-on is installed, you can deploy models using the InferenceEndpointConfig or JumpStart models custom resources. Here’s an example configuration for deploying a Llama model:
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: JumpStartModel
metadata:
name: deepseek-test-endpoint
spec:
model:
modelId: "deepseek-llm-r1-distill-qwen-1-5b"
sageMakerEndpoint:
name: deepseek-test-endpoint
server:
instanceType: "ml.g5.8xlarge"
New features
Multi-Instance Type Deployment HyperPod Inference supports multi-instance type deployment, enhancing deployment reliability and resource utilization. You can specify a prioritized list of instance types in your deployment configuration, and the system automatically selects from available alternatives when your preferred instance type lacks capacity. The Kubernetes scheduler evaluates instance types in priority order using node affinity rules based scheduling, seamlessly placing workloads on the highest-priority available instance type. In the example below, when deploying a model from S3, ml.p4d.24xlarge has the highest priority and will be selected first if memory capacity is available. If ml.p4d.24xlarge is unavailable, the scheduler automatically falls back to ml.g5.24xlarge, and finally to ml.g5.8xlarge as the last resort.
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
name: lmcache-test-1
namespace: default
spec:
replicas: 13
modelName: Llama-3.1-8B-Instruct
instanceTypes: ["ml.p4d.24xlarge","ml.g5.24xlarge","ml.g5.8xlarge"]
This is implemented using Kubernetes node affinity rules with requiredDuringSchedulingIgnoredDuringExecution to restrict scheduling to the specified instance types, and preferredDuringSchedulingIgnoredDuringExecution with descending weights to enforce priority ordering.
Node affinity
For scenarios requiring more granular scheduling control — such as excluding spot instances, preferring specific availability zones, or targeting nodes with custom labels — HyperPod Inference exposes Kubernetes’ native nodeAffinity directly in the InferenceEndpointConfig spec. This gives you the full expressiveness of Kubernetes scheduling primitives.
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
name: lmcache-test-1
namespace: default
spec:
replicas: 15
modelName: Llama-3.1-8B-Instruct
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node.kubernetes.io/instanceType
operator: In
values: ["ml.g5.4xlarge"]
worker:
resources:
limits:
nvidia.com/gpu: "1"
requests:
cpu: "6"
memory: 30Gi
nvidia.com/gpu: "1"
Clean up
To clean up your environment after completing this walkthrough, follow these steps to remove the deployed models and uninstall the Inference Operator add-on from your HyperPod cluster.
Removing Inference Operator add-on
Through the SageMaker console:
- Navigate to SageMaker Console → HyperPod Clusters → Cluster Management
- Select your cluster and go to the Inference tab
- Choose Remove to uninstall the Inference Operator add-on and associated resources
Alternatively, using the AWS CLI:
aws eks delete-addon \
--cluster-name <my-hyperpod-cluster> \
--addon-name amazon-sagemaker-hyperpod-inference \
--region <region>Delete the deployed models
# Delete JumpStartModel deployment
kubectl delete jumpstartmodel <model-name> -n <namespace>
# Or for InferenceEndpointConfig deployment
kubectl delete inferenceendpointconfig <endpoint-name> -n <namespace>Migration path for existing users
Automated migration script is hosted in public GitHub that transitions the HyperPod Inference Operator from Helm to EKS add-on with built-in rollback capabilities if add-on installation fails. Backup files are stored in /tmp/hyperpod-migration-backup-<timestamp>/ for manual rollback if needed.
Key features
- Auto-Discovery: Derives configuration from existing Helm deployment (roles, buckets, dependencies)
- Safe Migration: Scales down Helm deployments before add-on installation, validates prerequisites
- Dependency Handling: Migrates S3/FSx CSI drivers, cert-manager, and metrics-server to add-ons
- Rollback Support: Preserves original resources and restores on failure
IAM Roles created
- Execution Role (Inference Operator + S3 TLS access)
- JumpStart Gated Model Role
- ALB Controller Role
- KEDA Operator Role
Examples for running the script
# to follow step by step guide
./helm_to_addon.sh --cluster-name <my-cluster> --region us-east-1
# no prompts needed except for initiating rollback in case of failure
./helm_to_addon.sh --cluster-name <my-cluster> --region us-east-1 --auto-approve
# To skip the dependencies FSX, S3, Metricsserver, cert manager migration from Inference operator helm to respective add-ons
./helm_to_addon.sh --cluster-name my-cluster —region us-east-1 --skip-dependencies-migration
Migration flow
- Validate existing Helm installation
- Auto-derive configuration and create new IAM roles
- Tag resources (ALBs, ACM certs, S3 objects) with
CreatedBy: HyperPodInference - Install dependency add-ons (S3, FSx, cert-manager) if dependent CRDs don’t exist
- Scale down Helm deployments for ALB, KEDA and Inference operator
- Install Inference Operator add-on with
OVERWRITEflag - Clean up old Helm resources
- Migrate Helm-installed dependencies that are installed through Inference operator main chart to add-ons. To skip this step provide
--skip-dependenciesflag.
Benefits
- Simplified management through EKS console/APIs
- Automated updates via EKS add-on mechanisms
- Native EKS integration
- Zero downtime migration with rollback safety
Conclusion
The streamlined Inference Operator installation experience for Amazon SageMaker HyperPod eliminates infrastructure complexity and accelerates time to value for machine learning teams. With one-click installation, automated resource management, and seamless upgrade capabilities, teams can focus on deploying and optimizing their inference workloads rather than managing underlying infrastructure.
The EKS Add-on integration provides enterprise-grade lifecycle management while maintaining the flexibility to customize configurations for specific organizational requirements. Combined with advanced features like managed tiered KV cache and intelligent routing, this simplified installation experience makes high-performance inference deployment accessible to teams of all sizes.
Get started today by creating a new HyperPod cluster with the Inference Operator pre-installed, or add it to your existing clusters with a single click through the SageMaker console. For detailed add-on installation instructions and configuration options see this guide and for troubleshooting see this guide.
Appendix
About the authors
How we built Organizations to help enterprises manage Cloudflare at scale
Post Syndicated from Justin Hutchings original https://blog.cloudflare.com/organizations-beta/
Cloudflare was designed to be simple to use for even the smallest customers, but it’s also critical that it scales to meet the needs of the largest enterprises. While smaller customers might work solo or in a small team, enterprises often have thousands of users making use of Cloudflare’s developer, security, and networking capabilities. This scale can add complexity, as these users represent multiple teams and job functions.
Enterprise customers often use multiple Cloudflare Accounts to segment their teams (allowing more autonomy and separation of roles), but this can cause a new set of problems for the administrators by fragmenting their controls.
That’s why today, we’re launching our new Organizations feature in beta — to provide a cohesive place for administrators to manage users, configurations, and view analytics across many Cloudflare Accounts.
The principle of least privilege is one of the driving factors behind enterprises using multiple accounts. While Cloudflare’s role-based access control (RBAC) system now offers fine-grained permissions for many resources, it can be cumbersome to enumerate all the resources one by one. Instead, we see enterprises use multiple accounts, so each team’s resources are managed by that team alone. This allows organic growth within the account: they can add new resources as needed, without giving Administrative control too widely.
While multiple accounts are great at limiting permissions for most of the users within an organization, they complicate things for the administrators, as the administrators need to be added to every account and given the appropriate permissions to handle tasks like reporting or setting policies. This situation is fragile, as other administrators could remove them.
We designed Cloudflare Organizations with these scenarios in mind. Organizations adds a new layer to the hierarchy so that administrators can manage a collection of accounts together. Organizations is built on top of the Tenant system, which we created to support the needs of Cloudflare’s partner ecosystem. This provides a strong foundation for the many new features we’ve built with enterprises in mind.
The account list is at the core of the organization. This is a flat list of all the accounts that have been onboarded to the organization. “Org Super Administrator” is a new user role that is managed at the organization level; users with this role can add more accounts to the list as long as they are a Super Administrator of the account as well.
Org Super Administrators have Super Administrator permissions to every account in the organization. They do not require a membership in any of the child accounts and will not be listed in the account level UI. Org Super Administrator is the first of many roles we anticipate adding at the organization layer over the course of the year.
This feature was the culmination of a major innersource development project that we ran within the organization to remove legacy codepaths and consolidate every authorization check on our domain-scoped roles system. We added almost 133,000 lines of new code and removed about 32,000 lines of old code in support of this, making it one of the largest changes to our permissions system ever. This foundational improvement will make it easier to deliver additional roles in the future, both at the organization and account levels. We also made a 27% performance improvement in how we check permissions on enumeration calls like /accounts or /zones, which previously struggled with users that have access to thousands of accounts.
Org super administrators can view a roll-up dashboard complete with analytics about their HTTP traffic from across all accounts and zones. HTTP traffic analytics is the first of many analytics dashboards that we expect to deliver over the course of the year as we add this feature for more products.
Managing shared policies across your organization allows one team to centrally manage features like WAF (Web Application Firewall) or Gateway policies. Org Super Administrators will have the ability to share a policy set from one account to the rest of the accounts within the organization. That means any users in the source account with permission to manage those configurations can update the policy sets. So security analysts can update WAF rules for an entire enterprise centrally, without needing to be org administrators or administrators of other accounts in the organization.
We’ve limited the initial launch of Organizations only to enterprise customers, but will be expanding it to all customers in the coming months starting with pay-as-you-go customers. We’ll be working to extend this to our partner ecosystem too, but have a number of special scenarios we need to address for them before we do.
There’s a lot more on the roadmap in this space. Keep an eye on the changelog for capabilities coming soon:
- Organization-level audit logs
- Organization-level billing reports
- More organization-level analytics reports
- Additional organization user roles
- Self-serve account creation
Organizations is rolling out in public beta over the next several days to enterprise customers. In introducing Organizations, our own key requirements are that we do not elevate privilege for any users, and that customers create just one organization each. To deliver on those requirements, we elected not to do a backfill and create organizations on your behalf, and are instead using a self-serve invitation process.
If you are a Super Administrator of an enterprise account, and nobody else has created an organization for your company, then you will see an invitation to create an organization in your Cloudflare dashboard. Once you have created an organization, you can add accounts to the organization if you are a super administrator of that account as well.
If another user in your company has already claimed the organization, then they can either invite you as an Org Super Administrator so that you can add your accounts to the organization, or you can invite them as a Super Administrator of your account, so they can add your account to the organization. This process ensures that no user ever gets permission to a Cloudflare account where a Super Administrator was not involved in approving it. Cloudflare support will not be making configuration changes on behalf of customers, so plan to work with other administrators to complete your internal rollout of Organizations.
If you’re a Super Administrator of an enterprise account, claim your company’s organization now. There is no additional fee for using Organizations. You can find more details on how to get started in the Dashboard under the new Organizations tab, or at our developer docs.
If you’re not an enterprise customer, keep an eye on our changelog for more information about when Organizations will be available for your plan. And to learn more about our enterprise offerings, our enterprise sales team can get you started today.