Detect Issues in Your Zabbix Instance Before It’s Too Late

Post Syndicated from Janis Eidaks original https://blog.zabbix.com/detect-issues-in-your-zabbix-instance-before-its-too-late/32741/

In this blog post, I will show you how to detect performance issues in your Zabbix instance – in advance!

You might be using Zabbix to monitor your infrastructure, devices, and applications, but are you also monitoring your own instance? It might seem unnecessary – after all, what’s there to monitor, right? Your instance just works, so everything is good. What else is needed?

Remember though, if your Zabbix database system runs out of disk space, data collection will come to a halt. If the data collectors are insufficient, the collected data will be inconsistent, and this will also affect the problem detection.

If you run out of cache space on your Zabbix server, depending on which cache is affected, your Zabbix server might crash immediately or experience degraded performance. A lot of things can go wrong, and you need to stay ahead of them! Here’s how.

Tune your database

If you are using the default settings for your database, you are missing out on significant performance improvements that are just unused! Your actual instance performance is tied to the database performance. If the database performance is low, you will have a degraded Zabbix monitoring experience as well.

Do at least minimal fine-tuning, only change the settings you understand: read the documentation, check the official Zabbix blogposts, Zabbix community forum, and perform testing. Of course, you can use every tool at your disposal to make it work, such as AI, but always test the settings in the test environment.

The database tuning is a complex task. Initial parameters that you could tune for the MySQL DB are these:

innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
optimizer_switch=index_condition_pushdown=off
innodb_buffer_pool_size= ~75% of RAM if only DB engine running or less if shared with other applications

For a PostgreSQL database, you can use online tuner PGTUNE for initial configuration:

https://pgtune.leopard.in.ua/

Monitor the Zabbix database

It is important to monitor your database. Zabbix offers several out-of-the-box options to monitor the most popular databases through different methods: either by Zabbix agent or Zabbix agent2, by ODBC checks, using Zabbix Java gateway or by HTTP checks. If an issue is detected, you will get a corresponding problem event. Don’t forget to manually update the old Zabbix templates to the current version after the Zabbix server upgrades.

Fig 1. Some of the available out-of-the-box templates for database monitoring

Of course, depending on the approach you have selected to monitor the database, you will need to do some additional steps for that to work. More information on how to configure it is available on the Zabbix integration page.

Fig 2. Example of the configuration required to monitor the MySQL database with Zabbix agent2

Monitor the Zabbix server

The next thing you should check is the Zabbix server host dashboards. In new instances, the Zabbix server host has already been included out of the box with two templates: Zabbix server health and Linux by Zabbix agent. If such a host has not been retained for some reason, now it’s time to create it and start monitoring your Zabbix server.

The Zabbix server health template uses Zabbix internal items that do not require any interface. The Linux by Zabbix agent template does require a running Zabbix agent on the Zabbix server system in order to gather the OS related metrics.

Fig 3. Zabbix server host with linked templates

Check the current state of your Zabbix server

Once you have such a host, go to the menu Monitoring > Hosts and use the main filter to find the Zabbix server host and select its Host Dashboards.

Fig 4. Host dashboards

Select the Zabbix server health dashboard. Below, you will see the following pages under it – Performance, Processes, and Statuses.

Fig 5. Zabbix server health dashboard page: Performance

Check the cache utilization

In the Performance page, you can see the usage of Zabbix server caches. You should make sure that all caches except the history cache are at least ~ 50 % free. Technically, you can make the caches as large as possible; at worst, they will just be under-utilised. So, adjust the cache sizes accordingly.

Consequences of running out of configuration cache

If you add a lot of hosts in an automated way and have a relatively small or default configuration cache size [configcache], you could fill this cache quickly. The consequences of it are:

  • The Zabbix server will crash
  • The Zabbix server will be unable to start
  • The Zabbix server will not collect any data

You will also see a warning message in the Zabbix frontend:

Fig 6. Zabbix server health dashboard page when running out of config cache

If the config cache does not fill up instantly, the problem event will be generated shortly after, and matching action operations will be executed while the Zabbix server is still running, for example, notifying admins about the issue. In the screenshot below, you can see that one action operation step was executed before the server crashed.

Fig 7. Generated problem event

When a Zabbix component is not working as expected, your best source of information is the log file, as it informs you about the issues. Here is the error message in the Zabbix Server log file below.

Fig 8. Zabbix server log file error: out of memory for config cache

The solution is very simple: just increase the configuration cache size (two times or more) and restart the Zabbix server. If you expect a significant increase in hosts in the near future, you can be more generous and allocate more memory. My current Zabbix server is monitoring approximately 400 hosts.

Fig 9. The system information of my Zabbix server

Consequences of running out of value and history cache

So, what happens if you run out of value cache? Zabbix server performance will degrade, and the frontend will become noticeably less responsive. Why is that? Value cache stores item values used for calculated items and evaluating triggers. Now, for each trigger calculation that does not contain an item metric in the history cache will be retrieved directly from the database.

Fig 10. Zabbix server health dashboard performance page for cache usage

The history cache stores historical data that will be written to the database. If it’s mostly full, it means you might have issues with your database performance – the Zabbix server is unable to write data fast enough to the database. This can trigger a cascading performance degradation with a negative feedback loop. In my case:

  • A full value cache leads to additional DB read queries
  • DB performance drops, which leads to slow historical data writes to the database
  • The history cache also starts to fill up
  • The data collection is delayed due to the full history cache
  • As more data is collected, database read queries retrieve more data, progressively worsening the cycle

Technically, it does not require your value cache to be 100% full to have this issue – if a lot of triggers use a long-time interval for evaluation, you could have a situation where your value cache is only 85% or 90% full, but the Zabbix server is unable to fit the required item history records in available memory.

The issue with running out of value cache will also be logged in the Zabbix server’s log file.

Fig 11. The Zabbix server log file with value cache error

The solution to this issue is simple: increase the value cache size and restart the Zabbix server.
If you monitor your Zabbix server with the health template, problem events will be automatically generated when:

  • Value cache works in a low memory mode
  • History cache utilization exceeds 75 %
Fig 12. The generated problem events about the value cache issue

The issue with the Value cache working in low memory mode can also be seen in the graph below. Here you can see how many historical item values were present in cache, and how many had to be retrieved from the database directly.

Fig 13. Value cache effectiveness graph

Due to the terrible performance of the untuned database, when my history write cache fills up the data collectors are throttled, causing a pileup of delayed item collection.

Fig 14. Zabbix server performance graph

Slow database queries will appear in the Zabbix server log file.

Fig 15. The Zabbix server log file with slow query errors

The result of cache tuning and database tuning

Increasing the value cache only partly solved one issue. After database tuning, database performance has improved significantly:

  • The history cache is now empty
  • No more value cache misses
  • No more delayed items
Fig 16. The Cache usage, server performance, and value cache effectiveness graphs

After the Database tuning, the agent poller process and history syncer utilization also decreased to a low level.

Fig 17. Data collector and internal process utilization graphs

Tune the Zabbix server configuration parameters

Check the Processes page in the Zabbix Health dashboard and adjust the parameters accordingly. Only adjust the parameters that you understand. Changing the parameters arbitrarily can lead to the following:

  • Wasted resources without effective performance improvement
  • Reduced Zabbix server performance
  • Zabbix server crashes

For the data collectors, generally you require only a relatively small number of asynchronous data collectors, as they are very efficient, relatively larger number of synchronous data collectors. The graphs showing the utilisation of the gathering processes are extremely useful for determining which ones need to be increased – if they are close to 100% utilized, it’s now time for you to take action and add more.

Pitfalls of misconfiguration

Now, regarding the pitfalls of misconfiguration or lack of tuning. Here is a scenario: installed the Zabbix components, MySQL database without any configuration tuning, except the configuration cache to avoid the Zabbix server crashing immediately. The Zabbix server is monitoring around ~400 hosts. The Zabbix agent poller process and history syncers are utilized close to 100%, like in the Fig.17 before the tuning.

You might think that increasing both of these processes would improve the situation, for example, by doubling the count of them: more parallel agent processes should collect more data, and the more history syncers should write more data to the database.

After restarting the Zabbix server and checking the graph, both processes are close to 100% busy and the metric collection is significantly delayed. This is much worse.

Fig 18. Async data collector and internal process utilization graphs

By quadrupling both processes, the result is even worse, with significantly delayed item value collection.

Fig 19. Async data collector and internal process utilization graphs

So, what is happening behind the frontend? Just increasing the number of agent poller collectors and history syncers results in even worse performance. Seems counterintuitive, right: more data collectors should mean more data will be collected, and more history syncers – should allow more data to be written in parallel to the database.

However, increasing the data collector count in this specific situation will just make things much worse: you can collect more data at the same time, but will still face the same bottleneck: the database. Increasing the history syncers in this case makes the situation much worse, as more simultaneous queries to the database force it to slow down even further. So once again, tune your database engine and get more performance out of it.

Summary

You should monitor all of your Zabbix components and react when issues occur. Also make sure that you receive the notifications in your preferred media type, so you can act immediately. The complete list of what to monitor is more extensive, but this blog post should provide you with some examples and inspiration. It is always a good idea to react proactively rather than deal with the issues after they occur.

 

The post Detect Issues in Your Zabbix Instance Before It’s Too Late appeared first on Zabbix Blog.

Automating data classification in Amazon SageMaker Catalog using an AI agent

Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/automating-data-classification-in-amazon-sagemaker-catalog-using-an-ai-agent/

If you’re struggling with manual data classification in your organization, the new Amazon SageMaker Catalog AI agent can automate this process for you. Most large organizations face challenges with the manual tagging of data assets, which doesn’t scale and is unreliable. In some cases, business terms aren’t applied consistently across teams. Different groups name and tag data assets based on local conventions. This creates a fragmented catalog where discovery becomes unreliable and governance teams spend more time normalizing metadata than governing.

In this post, we show you how to implement this automated classification to help reduce the manual tagging effort and improve metadata consistency across your organization.

Amazon SageMaker Catalog provides automated data classification that suggests business glossary terms during data publishing. This helps to reduce the manual tagging effort and improve metadata consistency across organizations. This capability analyzes table metadata and schema information using Amazon Bedrock language models to recommend relevant terms from organizational business glossaries. Data producers receive AI-generated suggestions for business terms defined within their glossaries. These suggestions include both functional terms and sensitive data classifications such as PII and PHI, making it straightforward to tag their datasets with standardized vocabulary. Producers can accept or modify these suggestions before publishing, facilitating consistent terminology across data assets and improving data discoverability for business users.

The problem with manual classification

Manual tagging doesn’t scale effectively. Data producers interpret business terms differently, especially across domains. Critical labels like PII and PHI get missed because the publishing workflow is already complex. After assets enter the catalog with inconsistent terminology, search functionality and access controls quickly degrade.The solution isn’t only better training—it’s making the classification process predictable and consistent.

How automated classification works

The capability runs directly inside the publish workflow:

  1. The catalog looks at the table’s structure—column names, types, whatever metadata exists.
  2. That structure is sent to an Amazon Bedrock model that matches patterns against the organization’s glossary.
  3. Producers receive a set of suggestions from the defined business glossary terms for classification that might include both functional and sensitive-data glossary terms.
  4. They accept or adjust the suggestions before publishing.
  5. The final list is written into the asset’s metadata using the controlled vocabulary.

The model evaluates column names, data types, schema patterns, and existing metadata. It maps those signals to the terms defined in the organization’s glossary. The suggestions are generated inline during publishing, with no separate Extract, Transform and Load (ETL) or batch processes to maintain. The accepted terms become part of the asset’s metadata and flow into downstream catalog operations immediately.

Under the hood: intelligent agent-based classification

Automated business glossary assignment goes beyond simple metadata lookups using a reasoning-driven approach. The AI agent functions like a virtual data steward, following human-like reasoning patterns such as:

  • Reviews asset details and context
  • Searches the catalog for relevant terms
  • Evaluates whether results make sense
  • Refines strategy if initial searches don’t surface appropriate terms
  • Learns from each step to improve recommendations

Key approaches:

Reasoning over static queries – The agent interprets asset attributes and context rather than treating metadata as a fixed index, generating dynamic search intents instead of relying on predefined queries.
Iterative adaptive search – When initial results are weak, the agent automatically adjusts queries—broadening, narrowing, or shifting terms through a feedback loop that helps improve discovery quality.
Structured semantic search – The agent performs semantic querying across entity types, applies filtering and relevance scoring, and conducts multi-directional exploration until strong matches are found.

This allows the agent to explore multiple directions until strong matches are found, improving recall and precision over static methods like direct vector search when asset metadata is incomplete or ambiguous.

Things to keep in mind

This feature is only as strong as the glossary it sits on top of. If the glossary is incomplete or inconsistent, the suggestions reflect that. Producers should still review each recommendation, especially for regulatory labels. Governance teams should monitor how often suggestions are accepted or overridden to understand model accuracy and glossary gaps.

Prerequisites

To follow along, you must have an Amazon SageMaker Unified Studio domain set up with a domain owner or domain unit owner permissions. You must have a project that you can use to publish assets. For instructions on setting up a new domain, refer to the SageMaker Unified Studio Getting started guide. We will also use Amazon Redshift to catalog data. If you are not familiar, read Learn Amazon Redshift concepts to learn more.

Step 1: Define business glossary and terms

AI recommendations suggest terms only from glossaries and definitions already present in the system. As a first step we create high-quality, well-described glossary entries so the AI can return accurate and meaningful suggestions.

We create the following business glossaries in our domain. For information about how to create a business glossary, see Create a business glossary in Amazon SageMaker Unified Studio.

Domain: Terms – Customer Profile, Policy, Order, Invoice.

The following is the view of ‘Domain’ business glossary with all terms added.

Data sensitivity: Terms – PII, PHI, Confidential, Internal.

The following is the view of ‘Data sensitivity’ business glossary with all terms added.

Business Unit: Terms – KYC, Credit Risk, Marketing Analytics

The following is the view of ‘Business Unit’ business glossary with all terms added.

We recommend that you use glossary descriptions to make terms unambiguous. Ambiguous or overlapping definitions confuse AI models and humans equally.

Step 2: Create data assets

Create the following table in Amazon Redshift. For information about how to bring Amazon Redshift data to Amazon SageMaker Catalog, see Amazon Redshift compute connections in Amazon SageMaker Unified Studio.

CREATE TABLE  dev.public.customer_analytics_data (
    customer_id VARCHAR(50) NOT NULL,
    customer_full_name VARCHAR(200),
    customer_email VARCHAR(255),
    customer_phone VARCHAR(20),
    customer_dob DATE,
    customer_tax_id VARCHAR(256),
    policy_id VARCHAR(50),
    policy_type VARCHAR(100),
    policy_start_date DATE,
    policy_end_date DATE,
    policy_coverage_amount DECIMAL(18,2),
    order_id VARCHAR(50),
    order_date TIMESTAMP,
    order_status VARCHAR(50),
    order_total DECIMAL(18,2),
    invoice_id VARCHAR(50),
    invoice_date DATE,
    invoice_amount DECIMAL(18,2),
    invoice_payment_status VARCHAR(50),
    customer_profile_created_timestamp TIMESTAMP DEFAULT GETDATE(),
    customer_profile_updated_timestamp TIMESTAMP DEFAULT GETDATE(),

    PRIMARY KEY (customer_id, order_id)
)
DISTSTYLE KEY
DISTKEY (customer_id)
SORTKEY (customer_id, order_date);

Once the Redshift is onboarded with above steps, navigate to Project catalog from left navigation menu and choose Data sources. Run the Data Source to add the table to Project inventory assets.

‘customer_analytics_data’ should be Project Assets inventory.

Verify navigating to ‘Project catalog’ menu on the left and choose ‘Assets’.

Step 3: Generate classification recommendations

To automatically generate terms, select GENERATE TERMS in ‘GLOSSARY TERMS’ section of the asset.

AI recommendations for glossary terms automatically analyze asset metadata and context to determine the most relevant business glossary terms for each asset and its columns. Instead of relying on manual tagging or static rules, it reasons about the data and performs iterative searches across what already exists in the environment to identify the most relevant glossary term concepts.

After recommendations are generated, review the terms both at table and column level. Table level suggested terms can be viewed as shown in the following image:

Select the SCHEMA tab to review column level tags as shown in the following image:

Review and accept individually by selecting the AI icon shown in below image.

In this case, we select ACCEPT ALL and then select PUBLISH ASSET as shown below.

The tags are now added to the asset and columns without manual search and addition. Select PUBLISH ASSET.

The asset is now published to the catalog as shown in the following image in the upper left corner.

Step 4: Improve data discovery

Users can now experience enhanced search results and find assets in the catalog based on the associated terms.

Browse by TermsUsers can now explore the catalog and filter by terms as shown in left navigation “APPLY FILTER” section

Search and FilterUsers can also search assets by glossary terms as shown below:

Cleanup

Conclusion

By standardizing terminology at publication, organizations can reduce metadata drift and improve discovery reliability. The feature integrates with existing workflows, requiring minimal process changes while helping deliver immediate catalog consistency improvements.

By tagging data at publication rather than correcting it later, data teams can spend less time fixing metadata and more time using it. For more information on SageMaker capabilities, see the Amazon SageMaker Catalog User Guide.

About the authors

Ramesh Singh

Ramesh Singh

Ramesh is a Senior Product Manager Technical (External Services) at AWS in Seattle, Washington, currently with the Amazon SageMaker team. He is passionate about building high-performance ML/AI and analytics products that help enterprise customers achieve their critical goals using cutting-edge technology.

Pradeep Misra

Pradeep Misra

Pradeep is a Principal Analytics and Applied AI leader at AWS. He is passionate about solving customer challenges using data, analytics, and AI/ML. Outside of work, he likes exploring new places, trying new cuisines, and playing badminton with his family. He also likes doing science experiments, building LEGOs, and watching movies with his daughters.

Mohit Dawar

Mohit Dawar

Mohit is a Senior Software Engineer at Amazon Web Services (AWS) working on Amazon DataZone. Over the past 3 years, he has led efforts around the core metadata catalog, generative AI–powered metadata curation, and lineage visualization. He enjoys working on large-scale distributed systems, experimenting with AI to improve user experience, and building tools that make data governance feel effortless.

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 2

Post Syndicated from Ankush Goyal original https://aws.amazon.com/blogs/big-data/designing-centralized-and-distributed-network-connectivity-patterns-for-amazon-opensearch-serverless-part-2/

This post is Part 2 of our two-part series on hybrid multi-account access patterns for Amazon OpenSearch Serverless. In Part 1, we explored a centralized architecture where a single account hosts multiple OpenSearch Serverless collections and a shared VPC endpoint. This approach works well when a single business unit or team manages collections on behalf of the organization.

However, many enterprises have multiple business units that need independent ownership of their OpenSearch Serverless infrastructure. When each business unit wants to manage their own collections, security policies, and VPC endpoints within their own AWS accounts, the centralized model from Part 1 no longer fits.

In this post, we address this multi-business unit scenario by introducing a pattern where the central networking account manages a custom private hosted zone (PHZ) with CNAME records pointing to each business unit’s VPC endpoint. This approach maintains centralized DNS management and connectivity while giving each business unit full autonomy over their collections and infrastructure.

The challenge with multiple business units

When multiple business units independently manage their own OpenSearch Serverless collections in separate AWS accounts, each account has its own VPC endpoint with its own private hosted zones. These private hosted zones only work within their respective VPCs, creating DNS fragmentation across the organization. Consumers in spoke accounts and on-premises environments can’t resolve collection endpoints in other accounts without additional DNS configuration.

Managing individual PHZ associations for each consumer VPC doesn’t scale, and asking each business unit to coordinate DNS with every consumer creates operational overhead. You need a network architecture that gives each team autonomy while keeping DNS management and connectivity centralized.

Solution overview

This architecture solves the problem by centralizing DNS management in the networking account while leaving collection and VPC endpoint ownership with each business unit. The networking account maintains a custom PHZ with CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. This custom PHZ is associated with a Route 53 Profile and shared through AWS Resource Access Manager (AWS RAM) to spoke accounts. On-premises DNS resolution flows through the Route 53 Resolver inbound endpoint in the central networking VPC, which uses the same custom PHZ.

We cover two complementary patterns: Pattern 1 for on-premises access to collections across multiple business unit accounts, and Pattern 2 for spoke account access to those same collections. Both patterns rely on the centralized custom PHZ managed by your networking team.

Pattern 1: On-premises access to OpenSearch Serverless collections across multiple business unit accounts

With this pattern, your on-premises clients can privately access OpenSearch Serverless collections hosted across multiple business unit accounts, each with its own VPC endpoint. The following diagram illustrates this multi-business-unit architecture. It shows how on-premises DNS queries are resolved through the custom PHZ in the central networking account and routed to the correct business unit’s VPC endpoint through AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

(E) This automatically creates the following PHZs during VPC endpoint creation and associates them with the business unit’s VPC, so DNS resolution works locally within that VPC without depending on the Route 53 Profiles.

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

  1. Your on-premises client initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com.
  2. The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
  3. The inbound Resolver endpoint passes the query to the Route 53 VPC Resolver in the central networking VPC.
  4. The VPC Resolver finds the custom PHZ (bu-1-collection-id-1.us-east-1.aoss.amazonaws.com) associated with the central networking VPC. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint (for example, vpce-1234567890abcdefghi.a2oselk.vpce-svc-0c3ebf9a1a3ad247b.us-east-1.vpce.amazonaws.com).
  5. The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
  6. The traffic reaches BU1’s VPC endpoint elastic network interfaces (ENIs) through private network connectivity because the on-premises client connects over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.

Data flow

  1. Your on-premises client sends an HTTPS request to the resolved IP address with the TLS Server Name Indication (SNI) header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
  2. Traffic reaches the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
  3. The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, your client follows the same flow using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ contains a separate CNAME record pointing to BU2’s VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Spoke account access to OpenSearch Serverless collections across multiple business unit accounts

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications in spoke accounts to OpenSearch Serverless collections across multiple business unit accounts. With this pattern, compute resources in spoke account VPCs can privately access OpenSearch Serverless collections across multiple business unit accounts through the centralized private hosted zone (PHZ) in the networking account.

In the following example, we use an Amazon Elastic Compute Cloud (Amazon EC2) instance as a compute resource to illustrate the pattern. However, the same approach applies to any compute resource within the spoke VPC. The following diagram illustrates this multi-business-unit, multi-spoke architecture, showing how spoke VPCs resolve DNS through the shared Route 53 Profile and custom PHZ, then route traffic to the correct business unit’s OpenSearch Serverless collections through AWS PrivateLink.

(A) The Route 53 Profiles, created in the central networking account, are shared through AWS RAM with all spoke accounts and with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

(E) This automatically creates the following PHZs during VPC endpoint creation and associates them with the business unit’s VPC, so DNS resolution works locally within that VPC without depending on the Route 53 Profiles.

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

  1. An Amazon EC2 instance in BU1 Spoke VPC 1 initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
  2. The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC.
  3. The Profiles reference the custom PHZ (us-east-1.aoss.amazonaws.com) managed in the central networking account. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint.
  4. The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
  5. Traffic reaches the VPC endpoint ENIs through private network connectivity because the spoke VPC connects to BU1’s VPC through AWS Transit Gateway or AWS Cloud WAN.

Data flow

  1. Your Amazon EC2 instance sends an HTTPS request to the resolved IP address with the TLS SNI header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, routed through AWS Transit Gateway or AWS Cloud WAN to BU1’s OpenSearch Serverless VPC.
  2. The request arrives at the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
  3. The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, the same flow applies using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ resolves to BU2’s VPC endpoint, and routes traffic through the transit gateway to BU2’s VPC. The same applies to resources in other spoke accounts with the Route 53 Profiles associated.

Custom PHZ record structure

The custom PHZ in the central networking account uses the domain us-east-1.aoss.amazonaws.com and contains CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. Note that collections within the same business unit account share the same VPC endpoint, so their CNAME records point to the same regional DNS name. Collections in different business unit accounts point to different VPC endpoints.

Custom PHZ management

Unlike Part 1, where the auto-created PHZs from the VPC endpoint handle DNS resolution, this pattern requires your networking team to manually maintain the custom PHZ. When a business unit adds a new collection, the networking team must add a corresponding CNAME record to the custom PHZ.

Cost considerations

The architecture patterns described in this post use several AWS services that can contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how you can give on-premises clients and spoke account resources private access to OpenSearch Serverless collections distributed across multiple business unit accounts. By centralizing DNS management through a custom PHZ in the networking account and sharing it through Route 53 Profiles, you avoid coordinating PHZ associations across accounts while giving each business unit full ownership of their collections and VPC endpoints.

Combined with Part 1, you now have two architectural approaches for hybrid multi-account access to OpenSearch Serverless: a centralized model where one account owns all collections and a shared VPC endpoint, and a distributed model where multiple business units each manage their own collections and VPC endpoints. Choose the centralized model when a single team manages collections on behalf of the organization. Choose the distributed model when business units need independent ownership of their OpenSearch Serverless infrastructure.

For additional details, refer to the Amazon OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.

About the authors

Ankush Goyal

Ankush Goyal

Ankush is a Senior Technical Account Manager at AWS Enterprise Support, specializing in helping customers in the travel and hospitality industries optimize their cloud infrastructure. With over 20 years of IT experience, he focuses on leveraging AWS networking services to drive operational efficiency and cloud adoption. Ankush is passionate about delivering impactful solutions and enabling clients to streamline their cloud operations.

author name

Salman Ahmed

Salman is a Senior Technical Account Manager at AWS. He specializes in guiding customers through the design, implementation, and support of AWS solutions. Combining his networking expertise with a drive to explore new technologies, he helps organizations successfully navigate their cloud journey. Outside of work, he enjoys photography, traveling, and watching his favorite sports teams.

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 1

Post Syndicated from Ankush Goyal original https://aws.amazon.com/blogs/big-data/designing-centralized-and-distributed-network-connectivity-patterns-for-amazon-opensearch-serverless-part-1/

Amazon OpenSearch Serverless is a fully managed, serverless option for Amazon OpenSearch Service that removes the operational complexity of provisioning, configuring, and tuning OpenSearch clusters. When you run OpenSearch Serverless collections in a central account and need secure, private access from both on-premises environments and multiple AWS accounts, network architecture becomes critical. In this post, we explore two patterns to help you achieve this connectivity securely.

Solution overview

Working with customers implementing OpenSearch Serverless, we published blog posts addressing various network connectivity patterns to meet their evolving requirements:

In this post, we build on those patterns to address an additional enterprise requirement. When you manage many OpenSearch Serverless collections centrally but need access from multiple accounts and on-premises, you face several key challenges:

  • Coordinating VPC endpoints across accounts: managing endpoint provisioning and lifecycle across many consumer accounts adds operational overhead
  • Managing DNS configurations for each consumer: each new account or on-premises environment requires its own DNS setup, increasing complexity
  • Separating networking responsibilities from application ownership: without clear boundaries, networking and application teams become tightly coupled, slowing down both

This architecture solves these challenges with a clear separation of responsibilities. The central networking account shares Route 53 Profiles to manage DNS propagation across spoke accounts. The OpenSearch Serverless account owner maintains full control over their VPC endpoint and the associated private hosted zones (PHZs). Application owners retain autonomy over DNS configuration and collection management.

A single VPC endpoint handles multiple OpenSearch Serverless collections in an AWS Region, which reduces complexity and cost. Your networking team manages connectivity infrastructure while your application teams independently manage their OpenSearch Serverless collections, data access policies, and collection-specific DNS configurations. This gives you connectivity from on-premises networks (through AWS Direct Connect or AWS Site-to-Site VPN) and from compute resources across multiple AWS accounts through a unified network path.

This separation means that your network administrators and application teams can work independently. The result is a governance model that scales with your organization. We cover two complementary patterns that together give you complete hybrid access coverage, Pattern 1 for on-premises access and Pattern 2 for multi-account access, both using centralized interface VPC endpoints and Route 53 Profiles.

Before proceeding, you should be familiar with OpenSearch Serverless interface VPC endpoint DNS resolution. When creating an OpenSearch Serverless interface VPC endpoint, AWS automatically provisions four private hosted zones. The zones are three visible private hosted zones (for collections, dashboards, and FIPS endpoints) and one hidden internal private hosted zone that work together to resolve collection endpoints to private IP addresses. For more details on this DNS resolution mechanism, customers can review our previous blog post.

Pattern 1: Accessing multiple OpenSearch Serverless collections from on-premises through a centralized VPC endpoint and Route 53 Profiles in a multi-account architecture

The architecture spans three components:

  • A central OpenSearch Serverless account that hosts the collections and interface VPC endpoint.
  • A central networking account that owns the Route 53 Profiles and Inbound Resolver.
  • An on-premises environment connected using AWS Direct Connect or AWS Site-to-Site VPN.

The following diagram illustrates the architecture across these three components. It shows how DNS queries from on-premises clients are resolved through the Route 53 Profile and Inbound Resolver, and how data traffic reaches the OpenSearch Serverless collections using AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account. The central OpenSearch Serverless account associates the PHZs and interface VPC endpoint association with the shared Route 53 Profiles because that’s needed for end-to-end private DNS resolution.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

(E) On the on-premises side, the DNS resolver is configured with conditional forwarding for us-east-1.aoss.amazonaws.com, directing queries to the Route 53 Resolver Inbound Endpoint in the central networking account.

DNS resolution flow

  1. Your on-premises client initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com.
  2. The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
  3. The inbound resolver receives the query and passes it to the Route 53 VPC Resolver.
  4. The VPC Resolver checks the Route 53 Profiles associated with the central networking VPC. The Profiles provide access to the visible and hidden PHZs from the central OpenSearch Serverless VPC.
  5. The VPC Resolver uses the visible PHZ to match the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the interface VPC endpoint (VPCE) DNS name. It then uses the hidden PHZ to resolve the VPCE DNS name to the private elastic network interface (ENI) IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
  6. The private ENI IP addresses are returned through the inbound resolver endpoint to the on-premises DNS resolver and back to the on-premises client.

Data flow

  1. The on-premises client sends an HTTPS request to the resolved private ENI IP address with the TLS Server Name Indication (SNI) header set to collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
  2. Traffic reaches the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
  3. The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the client follows the same flow using collection-id-2.us-east-1.aoss.amazonaws.com. The wildcard DNS resolves to the same interface VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Accessing multiple OpenSearch Serverless collections from spoke accounts using a centralized VPC endpoint and Route 53 Profiles

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications across multiple AWS accounts. With this pattern, any compute resource running within spoke account VPCs can privately access multiple OpenSearch Serverless collections hosted in a central OpenSearch Serverless VPC through a single shared interface VPC endpoint.

In the following example, we use an Amazon Elastic Compute Cloud (Amazon EC2) instance as a compute resource to illustrate the pattern. However, the same approach applies to any compute resource within the spoke VPC.

The following diagram illustrates this multi-account architecture, showing how spoke account VPCs resolve DNS and route data traffic to the central OpenSearch Serverless collections through the shared Route 53 Profile and AWS PrivateLink, alongside the on-premises access path from Pattern 1.

(A) The Route 53 Profiles, created in the central networking account, are shared via AWS RAM with both the OpenSearch Serverless account and the spoke accounts. The OpenSearch Serverless account associates its interface VPC endpoint and private hosted zones (PHZs) with the Profiles, while each spoke account associates the Profiles with its VPC. Spoke VPCs get full DNS resolution for OpenSearch Serverless collection endpoints without requiring their own interface VPC endpoints, PHZs, or manual DNS configuration.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

DNS resolution flow

  1. An Amazon EC2 instance in Spoke VPC 1 initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
  2. The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC, which carries the PHZs from the central OpenSearch Serverless account.
  3. The visible PHZ matches the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the VPCE DNS name. The hidden PHZ resolves the VPCE DNS name to the private ENI IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
  4. Route 53 returns the private ENI IP addresses to the Amazon EC2 instance.

Data flow

  1. Your Amazon EC2 instance sends an HTTPS request to the resolved private ENI IP address with the TLS SNI header set to collection-id-1.us-east-1.aoss.amazonaws.com. This is routed through AWS Transit Gateway or AWS Cloud WAN to the central OpenSearch Serverless VPC.
  2. The request arrives at the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
  3. The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the same flow applies using collection-id-2.us-east-1.aoss.amazonaws.com. The same applies to resources in Spoke Account 2 or other spoke accounts with the Route 53 Profiles associated.

AWS RAM Permission Configuration for Resource Association

When the central networking account shares the Route 53 Profiles through AWS RAM, the default AWS managed permissions policy (AWSRAMPermissionRoute53ProfileAllowAssociationActions) only grants actions for associating and disassociating the Profile with VPCs, viewing Profiles details, and listing associations. It does not include the route53profiles:AssociateResourceToProfile or route53profiles:DisassociateResourceFromProfile actions required for the OpenSearch Serverless account to associate its interface VPC endpoint and PHZs with the shared Profiles.

To enable this, the central networking account must create a custom managed permission in AWS RAM with the following actions:

  • route53profiles:AssociateProfile
  • route53profiles:AssociateResourceToProfile
  • route53profiles:DisassociateProfile
  • route53profiles:DisassociateResourceFromProfile
  • route53profiles:GetProfile
  • route53profiles:GetProfileResourceAssociation
  • route53profiles:ListProfileAssociations
  • route53profiles:ListProfileResourceAssociations
  • route53profiles:ListProfiles

This custom permission must be attached to the RAM resource share before the central OpenSearch Serverless account can associate its PHZs and interface VPC endpoint with the Profiles.

Cost considerations

The architecture patterns described in this post use several AWS services that may contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how organizations can provide secure, private access to multiple OpenSearch Serverless collections from both on-premises environments and distributed AWS accounts using a single centralized interface VPC endpoint and Route 53 Profiles. This architecture centralizes OpenSearch Serverless collections and network infrastructure in a dedicated account, using Route 53 Profiles to propagate DNS across accounts. This eliminates per-account VPC endpoints, manual PHZ associations, and custom DNS configuration in spoke accounts.

This pattern is a good fit when a single team or business unit manages OpenSearch Serverless collections centrally. However, many enterprises have business units that need to independently manage their own OpenSearch Serverless collections in separate AWS accounts, each with their own interface VPC endpoints, security policies, and collection lifecycle. In Part 2, we explore how the architecture changes to support this distributed ownership model, where each business unit runs OpenSearch Serverless in their own account while still relying on centralized network connectivity and DNS management through Route 53 Profiles. We will cover how DNS resolution and the Route 53 Profiles configuration adapt when interface VPC endpoints and collections are spread across multiple accounts.

For additional details, refer to the OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.

About the authors

Ankush Goyal

Ankush Goyal

Ankush is a Senior Technical Account Manager at AWS Enterprise Support, specializing in helping customers in the travel and hospitality industries optimize their cloud infrastructure. With over 20 years of IT experience, he focuses on leveraging AWS networking services to drive operational efficiency and cloud adoption. Ankush is passionate about delivering impactful solutions and enabling clients to streamline their cloud operations.

author name

Salman Ahmed

Salman is a Senior Technical Account Manager at AWS. He specializes in guiding customers through the design, implementation, and support of AWS solutions. Combining his networking expertise with a drive to explore new technologies, he helps organizations successfully navigate their cloud journey. Outside of work, he enjoys photography, traveling, and watching his favorite sports teams.

New Whitepaper: Exploiting Cellular-based IoT Devices

Post Syndicated from Deral Heiland original https://www.rapid7.com/blog/post/tr-new-whitepaper-exploiting-cellular-based-iot-devices

Rapid7 has released a whitepaper titled “The Weaponization of Cellular Based IoT Technology,” by Deral Heiland, principal security researcher, IoT, at Rapid7, and Carlota Bindner, lead product security researcher at Thermo Fisher Scientific. The paper examines how attackers with physical access can exploit cellular modules in Internet of Things (IoT) devices to move into cloud and backend environments, exfiltrate data, and conceal command channels within expected device traffic. Heiland presented their findings at the RSAC 2026 conference in San Francisco.

The research focuses on how these attacks work in practice. It details how interchip communications such as USB and universal asynchronous receiver-transmitter (UART) can be observed and manipulated. It also shows how hardware modifications can replace a device host, allowing an external system to assume control of the cellular module. The authors developed proof-of-concept tools, including a TCP port scanner using AT commands, an S3 bucket enumerator, a SOCKS5 proxy that routes traffic through the cellular module, and a Metasploit proxy module. These examples demonstrate how attackers can take advantage of trusted relationships between devices and connected services.

The findings highlight consistent risks across tested devices. Cellular modules often expose multiple interfaces, and unused UART or USB paths can provide direct access. With targeted printed circuit board modifications, an attacker can reroute traffic through the cellular interface. Many modules accept AT commands that support raw sockets, HTTP requests, and TCP tunnels, which can enable reconnaissance and lateral movement. All cellular devices the researchers examined lacked tamper protections and most did not encrypt sensitive data before transmission, increasing exposure in environments that use private access point names (APNs).

Organizations should treat cellular-enabled devices as privileged entry points into their networks as well as their critical data storage and management environments. This includes disabling or removing unused interchip interfaces, enforcing end-to-end encryption before data is transmitted through the cellular modules, and applying monitoring and outbound controls within APN architectures. Hardware-level security testing should be part of standard product security practices.To read the whitepaper, click here.

Building a Scalable Messaging API with AWS End User Messaging and SES

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/building-a-scalable-messaging-api-with-aws-end-user-messaging-and-ses/

Modern applications often need to send notifications across multiple channels either through email and/or SMS. However, building a reliable messaging system that manages templates, handles failures gracefully, scales, and maintains security can be challenging. Following this guide, you’ll learn how to build a template manager and messaging API using API Gateway with JWT authentication for secure access, Amazon SQS for reliable message queuing, AWS Lambda for serverless processing, AWS End User Messaging for SMS, and Amazon Simple Email Service (SES) for email.

Architecture overview

You deploy a decoupled architecture that separates message ingestion from processing, providing resilience and scalability.

Fig. 1 Message Template Manager Architecture

Fig. 1 Message Template Manager Architecture

Architecture flow

  1. Client Application sends authenticated requests with JWT tokens
  2. API Gateway validates requests using a Lambda Authorizer
  3. Lambda Authorizer retrieves the JWT secret from AWS Secrets Manager and validates the token
  4. API Gateway sends validated messages to the SQS Queue
  5. Lambda Processor polls messages from SQS in batches
  6. Lambda Processor retrieves message templates from DynamoDB (if needed)
  7. Lambda Processor sends emails through Amazon SES and SMS through AWS End User Messaging
  8. Failed messages (after 3 retries) move to the Dead Letter Queue
  9. CloudWatch Alarm triggers when messages arrive in the DLQ

If a message fails processing after three attempts, it moves to a Dead Letter Queue (DLQ) where it’s preserved for 14 days, and a CloudWatch alarm notifies you of the failure.

Key features

With this architecture, you get several important capabilities:

  • JWT Authentication: Secure API access using JSON Web Tokens stored in AWS Secrets Manager
  • Automatic Retries: Failed messages retry up to three times before moving to the DLQ
  • Partial Batch Failures: Only failed messages retry, not the entire batch
  • Template Management: Store reusable message templates in Amazon DynamoDB
  • Multi-Channel Support: Send email through Amazon SES and SMS through AWS End User Messaging
  • Configuration Set Support: Track delivery metrics and route events with per-message or deployment-level configuration sets
  • Monitoring: CloudWatch alarms alert you when messages fail

Implementation details

1. API Gateway with JWT authorization

The API Gateway uses a Lambda authorizer to validate JWT tokens before allowing requests through:

def lambda_handler(event, context):
    token = event.get('authorizationToken', '').replace('Bearer ', '')
    try:
        # Retrieve secret from AWS Secrets Manager
        jwt_secret = get_jwt_secret()
        
        # Validate JWT token
        payload = jwt.decode(
            token,
            jwt_secret,
            algorithms=['HS256'],
            issuer='messaging-api'
        )
        
        # Generate IAM policy to allow request
        return generate_policy(payload.get('sub'), 'Allow', event['methodArn'])
    except jwt.ExpiredSignatureError:
        raise Exception('Unauthorized: Token expired')
    except jwt.InvalidTokenError:
        raise Exception('Unauthorized: Invalid token')

The JWT secret is stored securely in AWS Secrets Manager and cached in the Lambda execution environment for performance.

2. SQS queue configuration

The SAM template defines two queues with appropriate settings:

MessagesQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: MessagesQueue
    VisibilityTimeout: 300  # 5 minutes
    MessageRetentionPeriod: 345600  # 4 days
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt MessagesDeadLetterQueue.Arn
      maxReceiveCount: 3

MessagesDeadLetterQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: MessagesDeadLetterQueue
    MessageRetentionPeriod: 1209600  # 14 days

The visibility timeout of 5 minutes prevents duplicate processing while giving the Lambda function enough time to complete. Messages that fail three times automatically move to the DLQ.

3. Lambda message processor

The Lambda function processes messages from SQS and sends them through the appropriate channel:

def lambda_handler(event, context):
    failed_messages = []
    
    for record in event['Records']:
        message_id = record['messageId']
        try:
            message = json.loads(record['body'])
            
            # Process email if configured
            if 'EmailMessage' in message:
                send_emails(message)
            
            # Process SMS if configured
            if 'SMSMessage' in message:
                send_sms_messages(message)
                
        except Exception as e:
            print(f"Error processing message {message_id}: {str(e)}")
            failed_messages.append({"itemIdentifier": message_id})
    
    # Return failed messages for automatic retry
    return {"batchItemFailures": failed_messages}

Configuration Sets for tracking and analytics:

Configuration Sets enable you to track delivery metrics, monitor costs, and route events to analytics pipelines. You can set defaults at deployment time and override them per-message:

  • Deployment-level defaults: Set SMSConfigurationSet parameter during deployment to apply to all messages
  • Per-message override: Include ConfigurationSetName in the SMSMessage payload to use different tracking for specific messages

This flexibility lets you separate analytics by message type, campaign, or priority without requiring redeployment.

4. Template management with DynamoDB

Message templates are stored in DynamoDB for reusability:

def get_email_template_from_dynamodb(template_name, substitutions):
    response = templates_table.get_item(Key={'TemplateName': template_name})
    
    if 'Item' not in response:
        return build_default_email(), "Account Alert"
    
    template_body = response['Item']['MessageBody']
    subject = response['Item'].get('Subject', 'Notification')
    
    # Replace {variable} placeholders with actual values
    rendered_body = replace_variables(template_body, substitutions)
    rendered_subject = replace_variables(subject, substitutions)
    
    return rendered_body, rendered_subject

You can update message content without redeploying code.

Template size considerations:

DynamoDB has a 400 KB limit per item, which includes all attribute names and values. For message templates, this means:

  • Typical email templates (5-20 KB) fit comfortably
  • SMS templates (< 1 KB) have no practical constraints

If you need to store templates larger than 400 KB, consider storing them in Amazon S3 and referencing the S3 object key in DynamoDB. This hybrid approach provides unlimited template size while maintaining fast lookups.

Prerequisites

Before deploying this solution, ensure you have the following:

  • AWS End User Messaging SMS configured with a phone pool or origination identity for SMS sending
  • Amazon SES configured with verified email identities (sender and recipient for sandbox mode)
  • IAM permissions to create Lambda functions, API Gateway, SQS queues, DynamoDB tables, and Secrets Manager secrets
  • Python 3.9 or later installed locally
  • AWS SAM CLI installed (version 1.0 or later)
  • AWS CLI installed and configured with your credentials
  • An active AWS account with appropriate permissions

Deployment

You use AWS SAM for infrastructure as code. Deploy with these commands:

sam build
sam deploy --guided

During deployment, you’ll set a JWT secret that’s stored in AWS Secrets Manager. Use a strong, random secret for production:

python -c "import secrets; print(secrets.token_urlsafe(32))"

Usage example

Once deployed, send messages by making authenticated API requests:

curl -X POST "https://your-api-endpoint/dev/" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{
    "TraceId": "12345",
    "EmailMessage": {
      "FromAddress": "[email protected]",
      "Subject": "Low Balance Alert",
      "ConfigurationSetName": "email-analytics",
      "Substitutions": {
        "productName": "CHEQUING",
        "membershipNumber": "****5493",
        "accountBalance": "100.00"
      }
    },
    "SMSMessage": {
      "MessageType": "TRANSACTIONAL",
      "OriginationNumber": "your-pool-id",
      "TemplateName": "alert-template",
      "ConfigurationSetName": "sms-analytics"
    },
    "Addresses": {
      "[email protected]": {
        "ChannelType": "EMAIL"
      },
      "+16048621234": {
        "ChannelType": "SMS",
        "Substitutions": {
          "productName": "CHEQUING",
          "membershipNumber": "****7303",
          "accountBalance": "100.00"
        }
      }
    }
  }'

Using Configuration Sets:

The example above shows optional ConfigurationSetName parameters for both email and SMS. These enable:

  • Delivery tracking: Monitor delivery rates, failures, and bounce metrics
  • Cost monitoring: Track spending per campaign or message type
  • Event routing: Send delivery events to CloudWatch, Kinesis, or SNS for analytics
  • Segmented metrics: Separate analytics by use case, priority, or customer segment

If you don’t specify a ConfigurationSetName in the request, the system uses the deployment-level default (if configured).

Monitoring and operations

CloudWatch alarms

The solution includes a CloudWatch alarm that triggers when messages arrive in the DLQ:

DLQAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub ${AWS::StackName}-DLQ-Messages
    MetricName: ApproximateNumberOfMessagesVisible
    Namespace: AWS/SQS
    Statistic: Sum
    Period: 300
    EvaluationPeriods: 1
    Threshold: 1
    ComparisonOperator: GreaterThanOrEqualToThreshold

Handling failed messages

When messages fail, you can inspect them in the DLQ and redrive them back to the main queue after fixing the issue:

# Check DLQ depth
aws sqs get-queue-attributes \
  --queue-url YOUR_DLQ_URL \
  --attribute-names ApproximateNumberOfMessages

# Redrive messages from DLQ to main queue
aws sqs start-message-move-task \
  --source-arn YOUR_DLQ_ARN \
  --destination-arn YOUR_MAIN_QUEUE_ARN

Viewing logs

Lambda automatically logs to CloudWatch Logs:

aws logs tail /aws/lambda/MessageProcessor --follow

Cost considerations

For 1 million messages per month estimated costs are:

Service Usage Cost
API Gateway 1M requests $3.50
Amazon SQS 1M messages $0.40
AWS Lambda 1M invocations (128MB, 1s avg) $2.50
Amazon SES 1M emails $100.00
AWS End User Messaging SMS 1M SMS $ Varies based on destination

Note: The serverless architecture means you only pay for what you use, with no minimum fees or upfront costs.

Security best practices

This solution follows several security best practices:

  1. JWT Authentication: All API requests require valid JWT tokens
  2. Secrets Manager: JWT secrets are stored encrypted in AWS Secrets Manager
  3. IAM Least Privilege: Each Lambda function has only the permissions it needs
  4. HTTPS Only: API Gateway enforces HTTPS for all requests
  5. Token Caching: Authorization decisions are cached for 5 minutes to reduce latency

Clean up

To avoid incurring ongoing charges, delete the resources created by this solution when you no longer need them:

  1. If you configured AWS End User Messaging phone pools or origination identities for this solution, remove them from the End User Messaging console
  2. If you created Amazon SES email identities specifically for this solution, remove them from the SES console
  3. Verify that all resources (Lambda functions, API Gateway, SQS queues, DynamoDB table, Secrets Manager secret) have been removed in the AWS Management Console
  4. Delete the CloudFormation stack by running: sam delete --stack-name <your-stack-name>

Conclusion

With this architecture, you can build a production-ready messaging API using AWS serverless services. The decoupled design gives you resilience through automatic retries and dead letter queues, while the serverless approach eliminates infrastructure management and scales automatically.

The complete solution is deployable through AWS SAM and includes:

  • JWT authentication with AWS Secrets Manager
  • Multi-channel messaging (email and SMS)
  • Template management with DynamoDB
  • Configuration Set support for tracking and analytics
  • Comprehensive monitoring and alerting
  • Automatic retry and failure handling

You can extend this architecture by adding more channels (push notifications, webhooks), implementing message scheduling, or integrating with Amazon EventBridge for event-driven workflows.

Additional resources

The complete source code for this solution is available in the accompanying GitHub repository, including SAM templates, Lambda functions, and deployment scripts.

About the authors

Enhancing auto scaling resilience by tracking worker utilization metrics

Post Syndicated from Brian Moore original https://aws.amazon.com/blogs/compute/enhancing-auto-scaling-resilience-by-tracking-worker-utilization-metrics/

A resilient auto scaling policy requires metrics that correlate with application utilization, which may not be tied to system resources. Traditionally, auto scaling policies track system resource such as CPU utilization. These metrics are easily available, but they only work when resource consumption correlates with worker capacity. Factors such as high variance in request processing time, mixed instance types, or natural changes in application behavior over time can break this assumption.

Worker utilization tracking offers an alternative approach. Using a combination of total worker slots, work in flight, and work waiting in the backlog, a utilization value can be calculated for use in an auto scaling policy. This approach remains accurate across fleets with mixed instance types, applications with variable latencies, and requires no changes as your application evolves.

The limitations of resource-based scaling

Traditional auto scaling policies track system resource metrics like CPU utilization, assuming a direct correlation between resource consumption and available application capacity. Consider an application that reads messages from Amazon Simple Queue Service (SQS), processes them, and writes results to Amazon DynamoDB. If this application uses a fixed-size thread pool to process messages, such as 10 worker threads, the application reaches maximum capacity when all threads are busy, regardless of CPU utilization.

In our example, each worker spends most of its time waiting for DynamoDB responses rather than consuming CPU. All 10 threads become occupied handling requests, but CPU utilization stays low. From the perspective of the auto scaling policy, the fleet looks like it has enough capacity because plenty of CPU headroom remains. Meanwhile, new messages accumulate in the SQS queue because no workers are available to process them.

For queue-based workloads, AWS provides guidance to scale based on an acceptable backlog per worker. This is a calculated target based on your application’s average processing latency (queue delay). This works well when processing times are consistent, but breaks down if an application has variable latency characteristics.

Consider an image processing application that initially handles thumbnails taking 500 ms each. Using the traditional guidance with a target latency of 5 seconds you calculate an acceptable backlog of 10 messages per worker and deploy your scaling policy. Over time, the application evolves to also process 4K photos which take 2 seconds each. Eventually 4K photos are 50% of your traffic and total latency for queued messages has increased to 12.5 seconds, 2.5x more than your initial target.

The scaling policy is no longer fit for its intended purpose because your original latency assumptions no longer reflect reality. To keep this type of scaling effective you must also remember to update your scaling policies as your application behavior evolves.

A shift to using mixed instance types in your application can lead to additional complexity when using traditional resource-based scaling policies. Different instance types may handle the same workload at different CPU levels leading to an unbalanced average that misrepresents your actual application health. By changing your mental model to consider how much work your application can accept instead of how much of a system resource is available you can improve your scaling rules and better model your application’s capacity.

Understanding worker utilization

Worker utilization measures the ratio of active work to available processing capacity. To calculate it, divide total work by total workers.

We use an SQS-based processing application as an example to demonstrate how worker utilization operates, but this approach can also be applied to other applications where work units and worker capacity are measurable. In our example application total work consists of messages waiting to be processed plus messages currently being processed. Amazon CloudWatch provides these values through the ApproximateNumberOfMessagesVisible metric (messages waiting in the queue) and the ApproximateNumberOfMessagesNotVisible metric (messages currently being processed or in flight). Each host in your application should publish the number of available workers as a custom CloudWatch metric with at least a 1-minute period. For Java thread pools or Python multiprocessing pools, this represents the pool or process count. The formula works regardless of the metric period. Using the shortest period possible allows more responsive target tracking and enables Fast Target Tracking if your application has sub-minute data points.

To derive the formula, we can use the following CloudWatch Metric Math expressions:

  • totalWork = FILL(backlog, REPEAT) + FILL(inFlight, REPEAT)
  • utilizationRatio = totalWork / workers

Where:

  • backlog = ApproximateNumberOfMessagesVisible with the Maximum statistic.
  • inFlight = ApproximateNumberOfMessagesNotVisible with the Maximum statistic.
  • workers = Your custom TotalWorkers metric with the Sum statistic.

Putting the components together the final expression for your target tracking scaling policy uses the following formula:

IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0))

The FILL function uses last known values if SQS metrics are delayed, and the IF statement handles the case where you have no traffic and your fleet scales to zero instances. When there are no available workers, the formula metric reports 1 to indicate that the workers are fully saturated. This prevents the application from getting stuck at zero capacity and not being able to respond to any requests.

In this formula, a value of 1 or higher represents full or over saturation, where all workers are busy with no spare capacity, like running at 100% CPU. Values below 1 indicate available capacity for your application to process more work.

For applications without a measurable backlog metric, you can track worker utilization using only the in-flight work. This approach works for APIs or other synchronous workloads where work arrives and is immediately assigned to workers rather than queuing. In these cases, the formula becomes:

IF(FILL (workers, 0) > 0, utilizationRatio, IF(FILL(inFlight, 0) > 0, 1, 0))

In this scenario the utilization ratio is calculated as follows:

  • utilizationRatio = FILL(inFlight, REPEAT) / workers

The definitions of workers and inFlight remain the same for this formula. The primary difference is that the ratio directly tracks workers available and does not consider the backlog as an option.

How worker utilization prevents outages

Worker utilization-based scaling works for any application that can define available workers and total work. When the ratio of total work to available workers exceeds your threshold, the system scales out. This approach measures whether workers are available to handle the workload and treats application bottlenecks consistently. Whether workers are waiting on network I/O, performing CPU-intensive calculations, or experiencing another bottleneck doesn’t matter; the only question is whether total work exceeds available worker capacity. Any situation causing messages to accumulate on the queue increases the utilization ratio and triggers scale-out.

Implementing worker utilization scaling

To set up worker utilization-based auto scaling, identify metrics to use in the formula discussed earlier. First, identify a metric to track the amount of work being worked on. For SQS-based processing, AWS provides this metric. Second, implement a custom metric from your application representing the total workers. Optionally you can also identify a metric to track the available backlog of work.

Using CloudWatch metric math, you calculate the utilization metric and use it in a target tracking scaling policy. Here is an example AWS CloudFormation snippet showing the metric math configuration for a Amazon EC2 Auto Scaling group. This snippet shows only the scaling policy configuration and is only an example, before using in production fully test with your application. Your complete template also needs IAM roles with appropriate permissions for SQS, DynamoDB, and CloudWatch access.

ScalingPolicy: 
  Type: AWS::AutoScaling::ScalingPolicy 
  Properties: 
    AutoScalingGroupName: !Ref AutoScalingGroup 
    PolicyType: TargetTrackingScaling 
    TargetTrackingConfiguration: 
      TargetValue: 0.7 
      CustomizedMetricSpecification: 
        Metrics: 
          - Id: backlog 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: inFlight 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesNotVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: workers 
            MetricStat: 
            Metric: 
              Namespace: YourApp 
              MetricName: TotalWorkers 
            Stat: Sum 
          - Id: totalWork 
            Expression: FILL(backlog, REPEAT) + FILL(inFlight, REPEAT) 
          - Id: utilizationRatio 
            Expression: totalWork / workers 
          - Id: utilization 
            Expression: IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0)) 
            ReturnData: true

This approach also works for Amazon ECS services using AWS Application Auto Scaling. The metric math configuration remains the same, but you create an AWS::ApplicationAutoScaling::ScalingPolicy resource instead, adapting the parameters accordingly.

Choosing a target utilization

Since the worker utilization metric directly tracks the available capacity of your application, the target utilization value you choose reflects your organization’s balance between cost efficiency and availability. Lower target values provide more headroom for traffic spikes and faster response to load changes but result in higher infrastructure costs due to lower utilization. Higher target values maximize cost efficiency by keeping workers busy but leave less headroom for sudden traffic increases.

When choosing a target consider traffic patterns, acceptable latency during scale-out events, and cost sensitivity. Applications with unpredictable traffic spikes may benefit from lower targets, while an application with predictable load can safely use higher targets. Start with a moderate value like 0.7 and adjust based on observed behavior and your business requirements. If you previously tracked a resource utilization metric such as CPU, consider starting with the same target.

Monitoring resource utilization for cost optimization

While worker utilization drives scaling decisions, CPU and latency should be regularly evaluated to ensure cost-effective operations. Resource-based metrics can identify host resizing opportunities to better match your application requirements. If no scale-in happens when CPU utilization is consistently low, you are likely running instances that are too large for your workload. By using worker utilization in an auto scaling policy, you can switch to a different instance type without adjusting the auto scaling policy. The formula automatically adapts as you add different instance types or update the capacity per worker.

Conversely, if CPU utilization is consistently high while worker utilization remains at your target, your instances might be undersized. Upgrading to larger instance types can improve per-worker throughput, allowing each worker to process tasks faster. Changes to your auto scaling policy are not needed in this situation either. As messages are processed faster, they spend less time in the in-flight state, and the utilization ratio naturally adjusts.

This approach manages application availability independent of instance size, while resource utilization guides cost optimization. Each can be optimized independently without complex coordination.

Conclusion

Worker utilization-based auto scaling reduces the operational burden of continuously validating your scaling rules as application requirements and infrastructure change. By tracking the ratio of work to workers, your auto scaling policies automatically respond to capacity constraints based on available work. The approach works across workloads with discrete processing units and remains effective when you modify instance configurations or application worker pool sizes.

Implementation requires identifying a metric for available work, publishing a custom metric representing total workers, and using CloudWatch metric math in a target tracking scaling policy. This setup provides resilience that scaling based solely on resource metrics cannot achieve, while maintaining the flexibility to optimize costs and change your instance size without impacting system availability.

To get started:

  1. Identify an application in your environment that uses a worker pool.
  2. Instrument the application to publish worker count metrics.
  3. Configure a scaling policy tracking worker utilization.
  4. Monitor how the system responds to traffic changes and capacity events.

Learn more

To learn more about auto scaling and monitoring, see the following resources:

Rapid7 Completes BSI C5 Type 2 Examination: Stronger Cloud Security for DACH Organizations

Post Syndicated from Georgeta Toth original https://www.rapid7.com/blog/post/cds-rapid7-completes-bsi-c5-type-2-examination-stronger-cloud-security-dach-organizations

If you’re a security leader operating in Germany, Austria, or Switzerland, you already know that compliance isn’t a checkbox. It’s a competitive differentiator. Rapid7 has completed BSI C5 Type 2 attestation for the Rapid7 Command Platform, including Threat Command, and it’s a milestone worth unpacking.

This isn’t just a badge on a webpage. It’s proof that our security controls work, not just on paper, but in practice, over time.

What is BSI C5 and why does it matter?

The Cloud Computing Compliance Criteria Catalogue (C5) was developed by Germany’s Federal Office for Information Security (BSI). It sets some of the most rigorous cloud security standards in the world, covering everything from data protection to operational transparency.

A Type 2 attestation is the gold standard within that framework. Unlike a point-in-time audit, Type 2 validates that security controls aren’t just well-designed, but that they’re actively working consistently over a sustained period. It’s the difference between a security promise and a security proof.

For organizations in the DACH region, C5 is more than a nice-to-have. It’s a procurement requirement for German federal agencies, critical infrastructure operators, healthcare institutions, and financial services firms. If you’re operating in any of these sectors, your cloud providers need to meet this bar. Rapid7 now does.

BSI C5 Type 2 and your cloud security strategy

Whether you’re evaluating security vendors, managing compliance obligations, or looking to strengthen your organization’s risk posture, the question is the same: How do you know your cloud security provider actually does what it says?

BSI C5 Type 2 attestation answers that question. It’s independent, rigorous, and sustained over time. While rooted in German regulatory requirements, C5 is increasingly recognized as a benchmark for secure cloud operations across Europe. It’s one of the clearest signals that a cloud provider has the operational maturity to handle sensitive environments.

The Rapid7 Command Platform unifies exposure management with detection and response, giving security teams clear visibility across their attack surface. Threat Command extends that protection further, identifying and helping remediate threats across the clear, deep, and dark web. Both are now independently validated against one of the world’s toughest cloud security frameworks.

Why independent validation of security controls matters

Trusting a security vendor shouldn’t require a leap of faith. Independent validation exists so you have the evidence to make that call with confidence. This attestation reflects our continued investment in meeting the highest security standards for customers across Germany and the wider European market. Rapid7 has achieved a milestone that speaks directly to the conversations had every day with public sector and enterprise organizations who need more than a promise. 

They need proof that a security provider’s controls have been tested, verified, and proven to hold up over time. That’s the kind of assurance that matters when the stakes are high.

Ready to see the Command Platform in action? Visit Rapid7.com for a free trial.

[$] A PHP license change is imminent

Post Syndicated from jzb original https://lwn.net/Articles/1063993/

PHP’s licensing has been a source of confusion for some time. The project is,
currently, using two licenses that cover different parts of the code base: PHP v3.01 for the
bulk of the code and Zend v2.0 for code
in the Zend directory. Much has changed
since the project settled on those licenses in 2006, and the need for custom
licensing seems to have passed. An effort to simplify PHP’s licensing, led by
Ben Ramsey, is underway; if successful, the existing licenses will be deprecated
and replaced by the BSD
three-clause license. The PHP community is now voting on the license
update RFC through April 4, 2026.

Down: Debunking zswap and zram myths

Post Syndicated from corbet original https://lwn.net/Articles/1064478/

Chris Down has posted a
detailed look at how the kernel’s zswap and zram subsystems work — and
how they differ.

Most people think of zswap and zram simply as two different
flavours of the same thing: compressed swap. At a surface level,
that’s correct – both compress pages that would otherwise end up on
disk – but they make fundamentally different bets about how the
kernel should handle memory pressure, and picking the wrong one for
your situation can actively make things worse than having no swap
at all

Krita 5.3.0 and 6.0.0 released

Post Syndicated from jzb original https://lwn.net/Articles/1064477/

The Krita project has announced
the release of Krita 5.3.0 and 6.0.0:

Krita 5.3/6.0 is the result of many years of work by the Krita
developers. Some features have been rewritten from the ground up,
others make their first appearance.

Enjoy the completely new text feature: on canvas editing, full
opentype support, text flowing into shapes. It is now easier than ever
to create vector-based panels for comic pages. Tools got extended: for
instance, the fill tool now can close gaps. The liquify mode of the
transform tool is much faster. There are new filters: a propagate
colors filter and a reset transparent filter. Support for HDR painting
has been improved. The recorder docker can now work in real
time. There is improved support for file formats, like support for
text objects in PSD files. And much, much, much more!

According to the announcement, the versions are almost functionally
identical. However, the 6.0.0 release is the first based on Qt 6;
it has more Wayland functionality but is considered experimental. It
cautions that users should stick to 5.3.0 for real work. See
the release
notes for a full list of changes.

The collective thoughts of the interwebz