Tag Archives: Amazon MQ*

How Picnic configured multiple OAuth providers for Amazon MQ

Post Syndicated from Oscar Mapfumo Sibanda original https://aws.amazon.com/blogs/big-data/how-picnic-configured-multiple-oauth-providers-for-amazon-mq/

This post is co-written with Oscar Mapfumo Sibanda from Picnic.

Picnic is an Amsterdam-based tech scale-up that reinvents how people buy food. It isn’t a supermarket with a digital layer but a tech company that happens to deliver groceries. Picnic is engineered in-house: the customer app, the fulfillment platform, the supply chain, and the routing technology that guides a fleet of thousands of electric vehicles through the Netherlands, Germany, and France. Software doesn’t merely support business. Software is a business.

At the center of this system, RabbitMQ is the core component. It’s the communication backbone connecting hundreds of microservices across the entire business lifecycle, from ordering and logistics to delivery and finance. At peak, Picnic’s platform processes close to one million messages per second. At this scale, messaging is no longer only background infrastructure. It becomes part of the company’s operational nervous system. To keep that system highly available, scalable, and resilient as Picnic grows, the company decided to use Amazon MQ as a managed service.

The next challenge was identity. Picnic’s authentication strategy clearly distinguishes between people and services. Operators sign in through Keycloak, the company’s single sign-on provider, while Picnic’s Amazon Elastic Kubernetes Service (Amazon EKS) workloads are adopting AWS Identity and Access Management (IAM) authentication to eliminate static credentials. A single broker therefore must trust two identity providers at once. The Amazon MQ documentation covers configuring OAuth 2.0 with a single provider. This post extends that guidance to a multi-provider setup on the same broker.

In this post, we show how Picnic solved that problem. You will learn how to configure an Amazon MQ for RabbitMQ broker to accept tokens from multiple OAuth 2.0 identity providers, using Keycloak and IAM as the working example. You will also see how to map each provider’s scopes to RabbitMQ permissions and how to roll the change out on a running broker without disrupting connected users.

Background and prerequisites

Amazon MQ for RabbitMQ supports OAuth 2.0 authentication and authorization, where broker users and their permissions are managed by an external identity provider. User authentication and resource permissions for vhosts, exchanges, queues, and topics are centralized through the OAuth 2.0 provider’s scope system.

RabbitMQ’s OAuth 2.0 plugin supports multiple resource servers and audiences, allowing different OAuth 2.0 providers to issue tokens that a single broker can validate. This capability is essential if you operate in multiple environments or have teams registered with separate identity providers.

Prerequisites

To follow along with this post, you need:

  1. An active AWS account.
  2. An Amazon MQ for RabbitMQ broker with OAuth 2.0 configured for at least one identity provider (see Using OAuth 2.0 authentication and authorization for Amazon MQ for RabbitMQ).
  3. A second OAuth 2.0 identity provider configured and operational.
  4. Outbound web identity federation enabled for your AWS account (if using IAM as a provider).
  5. Basic familiarity with RabbitMQ configuration and OAuth 2.0 concepts.
  6. AWS Command Line Interface (AWS CLI) version 2.27 or later (required for the get-web-identity-token command used in the testing section).

Note: The information in this post reflects Amazon MQ for RabbitMQ features and behavior at the time of publication. We recommend checking the Amazon MQ documentation, release notes and best practices before implementation.

Solution architecture

The design rests on a single idea: a RabbitMQ broker can trust more than one identity provider at the same time, and it decides which one to apply per token rather than per broker. RabbitMQ does this by reading the aud (audience) claim of each incoming token and matching it against a configured resource server. Each resource server is bound to one OAuth 2.0 provider, so the audience determines both which signing keys validate the token and which permission rules apply.

Architecture diagram

The following two diagrams show how services and operators authenticate the broker through their respective identity providers.

Services authentication flow from an Amazon EKS workload through AWS STS to the Amazon MQ for RabbitMQ broker over AMQPS

Figure 1: Services (IAM) flow

Services authenticate through IAM. An Amazon EKS workload assumes an IAM role and generates a web identity token (1), which AWS Security Token Service (AWS STS) issues with an audience of rabbitmq-iam (2). The workload presents that token as its password when it connects to the broker over Advanced Message Queuing Protocol (AMQPS) on port 5671 (3). The broker selects the matching resource server, verifies the token’s signature against the AWS STS signing keys (4), and maps the role’s Amazon Resource Name (ARN) to the permissions the workload needs (5).

Operators authentication flow from the RabbitMQ management console through Keycloak single sign-on to the broker

Figure 2: Operators (Keycloak) flow

Operators authenticate through Keycloak. An operator opens the RabbitMQ management console and initiates login (1), and the console redirects the browser to Keycloak (2). Keycloak authenticates the operator and issues a token whose audience targets the console resource server, rabbitmq-keycloak (3). The browser presents that token to the broker (4), which verifies the signature against Keycloak’s signing keys (5). The broker then reads the operator’s group membership and grants access (6): the Operator group receives read-only permissions, while the Administrator group receives full control.

Two constraints follow this design. First, audience verification is a single broker-wide setting that applies to every provider at once, so each provider must issue tokens carrying the exact audience its resource server expects. Second, the broker is private, deployed inside an Amazon Virtual Private Cloud (Amazon VPC) with no public exposure. Both providers’ endpoints must be resolvable, either through publicly addressable JSON Web Key Set (JWKS) endpoints or through private networking, because the broker fetches signing keys from those endpoints.

Implementation walkthrough

This walkthrough configures one Amazon MQ for RabbitMQ broker to trust two identity providers: Keycloak for operators and AWS IAM for services. The steps assume you already have a running broker, a Keycloak realm, and outbound web identity federation enabled for your AWS account. All configurations are applied through a RabbitMQ configuration revision using the AWS Command Line Interface (AWS CLI).

Enable OAuth 2.0 on the broker

The first block activates the OAuth 2.0 backend and keeps the internal backend in place. Internal authentication remains active deliberately: Amazon MQ creates an administrator user when the broker is provisioned, and that user is needed for break-glass access.

auth_backends.1 = oauth2
auth_backends.2 = internal
auth_oauth2.verify_aud = true

Setting verify_aud = true tells RabbitMQ to reject any token whose aud claim does not match a configured resource server. This single broker-wide setting governs every provider you add.

Add the first identity provider (Keycloak)

A resource server binds an audience value to a provider and a set of permission rules. The Keycloak resource server uses the id rabbitmq-keycloak, which is the audience the realm must place in its tokens. RabbitMQ reads the operator’s group membership from the group_membership claim and resolves it through scope aliases.

auth_oauth2.resource_servers.1.id = rabbitmq-keycloak
auth_oauth2.resource_servers.1.oauth_provider_id = keycloak
auth_oauth2.resource_servers.1.scope_prefix = rabbitmq.
auth_oauth2.resource_servers.1.additional_scopes_key = group_membership
auth_oauth2.resource_servers.1.preferred_username_claims.1 = email
auth_oauth2.resource_servers.1.scope_aliases.1.alias = Operator
auth_oauth2.resource_servers.1.scope_aliases.1.scope = rabbitmq.read:*/* rabbitmq.write:^$ rabbitmq.configure:^$ rabbitmq.tag:monitoring
auth_oauth2.resource_servers.1.scope_aliases.2.alias = Administrator
auth_oauth2.resource_servers.1.scope_aliases.2.scope = rabbitmq.read:*/* rabbitmq.write:*/* rabbitmq.configure:*/* rabbitmq.tag:administrator

The Operator group is read-only: it can read any resource and view the management UI through the monitoring tag. The Administrator group receives full permissions plus the administrator tag. This role is least privilege by design.

The provider is configured with its issuer and JWKS endpoint:

auth_oauth2.oauth_providers.keycloak.https.hostname_verification = wildcard
auth_oauth2.oauth_providers.keycloak.issuer = https://keycloak.example.com/auth/realms/test
auth_oauth2.oauth_providers.keycloak.jwks_uri = https://keycloak.example.com/auth/realms/test/protocol/openid-connect/certs

To let operators sign in from the management console, expose Keycloak as a management resource:

management.oauth_enabled = true
management.oauth_disable_basic_auth = false
management.oauth_scopes = openid email profile
management.oauth_resource_servers.1.id = rabbitmq-keycloak
management.oauth_resource_servers.1.oauth_client_id = rabbitmq-keycloak
management.oauth_resource_servers.1.label = Keycloak SSO

Add the second identity provider (AWS IAM)

Adding a second provider means adding a second resource server and a second entry under oauth_providers. The IAM resource server uses the id rabbitmq-iam because the audience is set when minting the token.

auth_oauth2.resource_servers.2.id = rabbitmq-iam
auth_oauth2.resource_servers.2.oauth_provider_id = aws_iam
auth_oauth2.resource_servers.2.scope_prefix = rabbitmq/
auth_oauth2.resource_servers.2.additional_scopes_key = sub
auth_oauth2.resource_servers.2.scope_aliases.1.alias = arn:aws:iam::123456789012:role/EKSWorkloadRole
auth_oauth2.resource_servers.2.scope_aliases.1.scope = rabbitmq/read:*/* rabbitmq/write:*/* rabbitmq/configure:*/* rabbitmq/tag:policymaker
auth_oauth2.oauth_providers.aws_iam.https.hostname_verification = wildcard
auth_oauth2.oauth_providers.aws_iam.issuer =
auth_oauth2.oauth_providers.aws_iam.jwks_uri =

The IAM workload receives the policymaker tag. It can publish, consume, and manage policies but does not receive the administrator tag.

Apply the configuration and restart the broker:

CONFIG_ID=$(aws mq describe-broker --broker-id $BROKER_ID \
  --query 'Configurations.Current.Id' --output text)
REVISION=$(aws mq update-configuration --configuration-id $CONFIG_ID \
  --data "$(cat rabbitmq.conf | base64 | tr -d '\n')" \
  --query 'LatestRevision.Revision' --output text)
aws mq update-broker --broker-id $BROKER_ID \
  --configuration Id=$CONFIG_ID,Revision=$REVISION
aws mq reboot-broker --broker-id $BROKER_ID

Note: The base64 command syntax differs between Linux and macOS. The preceding command (cat file | base64 | tr -d '\n') is portable on both operating systems. If running exclusively on Linux, you can also use base64 --wrap=0 rabbitmq.conf. On macOS, the equivalent command is base64 -i rabbitmq.conf.

Testing and validation

Validate each provider independently. For IAM, assume the role and request a token from AWS STS, then present it as the AMQP password:

TOKEN=$(aws sts get-web-identity-token \
  --audience "rabbitmq-iam" \
  --signing-algorithm ES384 \
  --duration-seconds 300 \
  --query 'WebIdentityToken' --output text)
# Username is empty (ignored by the OAuth plugin); the token is passed as the password
curl -u ":$TOKEN" https://<broker-id>.mq.<region>.on.aws/api/overview

Note: The get-web-identity-token API requires outbound web identity federation to be enabled on your AWS account and AWS CLI version 2.27 or later.

A successful response confirms the IAM resource server accepted the token. For Keycloak, open the management console, choose Keycloak SSO, and sign in as an operator.

When a login fails, decode the JSON Web Token (JWT) and check two claims. The aud claim must exactly match a resource server id. With verify_aud = true, a missing or mismatched audience is the most common cause of rejection. If the audience is correct but permissions are missing, verify the scope_prefix is set correctly.

Operational considerations

A few points deserve attention before you run this pattern in production.

  1. Key rotation: When rotating signing keys at a provider, publish the new key in the JWKS endpoint before revoking the old one. The broker caches keys, so overlapping both during the transition window prevents authentication failures while the cache refreshes.
  2. Audience validation: Audience remains the linchpin. With verify_aud enabled, every provider must issue tokens carrying the audience its resource server expects, so confirm this whenever you onboard a new one. Don’t disable audience validation in production. The RabbitMQ OAuth 2.0 plugin does not perform token revocation checks, which makes audience binding a critical control that prevents tokens issued for other services from granting access.
  3. Scope prefix: scope_prefix values are optional. They’re needed only if the tokens don’t follow the default format. RabbitMQ only reads scopes carrying the expected prefix, so a token can authenticate yet grant nothing if the prefix is missing. Map each provider to the least privilege its principals need. For example, prefer narrow scopes like read:orders over blanket read:all to limit the scope of impact if a single provider’s credentials are compromised.
  4. Token lifetime: Because the plugin does not support token revocation, token lifetime is your primary control over leaked credentials. Issue short-lived access tokens and have your client applications refresh them proactively at approximately 75 percent of the token’s lifetime to avoid connection disruptions when a token expires mid-session.
  5. Monitoring: Authentication failures and refused tokens are recorded in the broker’s connection log group in Amazon CloudWatch, which can be reached through the Amazon CloudWatch Logs link on the broker’s page in the Amazon MQ console. Beyond logs, set up CloudWatch alarms on RabbitMQMemUsed, RabbitMQDiskFree, and ConnectionCount. An unexpected spike in failed connections is often the first sign of a token or audience misconfiguration. For unaggregated, per-node visibility, consider enabling the Prometheus metrics endpoint: metrics such as rabbitmq_auth_attempts_failed_total surface OAuth rejections faster than the CloudWatch one-minute polling interval.
  6. Network controls: Enforce defense in depth by restricting broker access using security groups so that only authorized VPCs and IP ranges can reach the AMQPS and management endpoints. This matters especially in an OAuth setup because, once a token has been issued, the broker cannot revoke it before it expires.

Cleanup

To avoid incurring future costs, delete the resources created during this walkthrough if you no longer need them:

  1. Delete Amazon MQ broker and configurations.
  2. Remove test OAuth application registrations from your identity providers.
  3. Delete any IAM roles created for testing.

Conclusion

In this post, we demonstrated how Picnic configured an Amazon MQ for RabbitMQ broker to authenticate tokens from two OAuth 2.0 identity providers: Keycloak for human operators and AWS IAM for machine-to-machine services on a single broker instance. The key mechanism is RabbitMQ’s support for multiple resource servers, where the audience claim in each token determines which provider’s signing keys and permission rules apply.

With this approach, the Picnic team was able to cleanly separate human and machine authentication without the operational overhead of running separate brokers, while retaining fine-grained access control for both token issuers.

This pattern works with any combination of OAuth 2.0 providers and is particularly valuable for organizations looking to consolidate messaging infrastructure while maintaining distinct identity boundaries.

To learn more about Amazon MQ for RabbitMQ and OAuth 2.0 authentication, see Authentication and authorization for Amazon MQ. For a hands-on walkthrough of configuring OAuth 2.0 with Amazon MQ for RabbitMQ, see Using OAuth 2.0 authentication and authorization for Amazon MQ for RabbitMQ. The configuration examples in this post are broker-level settings applied through the Amazon MQ API. No standalone code repository is required.


About the authors

Oscar Mapfumo Sibanda

Oscar Mapfumo Sibanda

Oscar is a Senior Site Reliability Engineer at Picnic Technologies in the Netherlands. He builds infrastructure that supports rapid scaling, empowers engineering teams to move independently, and strengthens the security posture across the organization. Outside of work he paints and takes photographs; he is a technology enthusiast in the pursuit of happiness.

Ayush Kumar

Ayush Kumar

Ayush is a Technical Account Manager at Amazon Web Services based in the Netherlands. He works with enterprise customers to optimize their cloud architectures and accelerate innovation on AWS. You’ll find him experimenting in the kitchen in his spare time.

Amit Singh

Amit Singh

Amit is a Senior Solutions Architect at AWS, working with enterprise retail customers in the Benelux region. He helps customers design cloud-native architectures, navigate complex modernization journeys, and adopt AI/ML capabilities at scale. Outside of work, he enjoys exploring new places and chasing the perfect shot, whether through a camera lens or on a running trail.

IAM authentication with OAuth 2.0 for Amazon MQ for RabbitMQ

Post Syndicated from Vinodh Kannan Sadayamuthu original https://aws.amazon.com/blogs/big-data/iam-authentication-with-oauth-2-0-for-amazon-mq-for-rabbitmq/

This is Part 3 of a three-part series on authentication and authorization for Amazon MQ for RabbitMQ. For an overview of all available methods, see Authentication and Authorization Options for Amazon MQ for RabbitMQ. For certificate-based mTLS and SSL authentication, see Part 1. For OAuth 2.0, LDAP, Entra ID, and HTTP authentication, see Part 2.

When you run Amazon MQ for RabbitMQ at scale without AWS Identity and Access Management (IAM) authentication, you face a common challenge: managing static credentials across multiple services, each requiring its own username and password. This approach creates operational overhead through password rotation, credential distribution, and the risk of inadvertent secret disclosure. IAM authentication with OAuth 2.0 removes these static credentials. Clients authenticate with their existing IAM identity instead.

This post covers the key configuration options for using IAM as an OAuth 2.0 provider and demonstrates a multi-tenant use case with vhost-level isolation enforced by IAM roles and broker-level scope aliases.

Amazon MQ for RabbitMQ supports IAM-based authentication through OAuth 2.0, so you have centralized access control without managing broker-local credentials. The clients authenticate using their existing IAM identity. Tokens expire automatically, and access control lives entirely in IAM roles and broker configuration.

Note: IAM authentication for Amazon MQ for RabbitMQ requires RabbitMQ versions 3.13 and 4.2 or later. Amazon MQ for ActiveMQ brokers doesn’t support this feature.

Important: IAM outbound federation must be configured and available in your AWS account before you enable IAM authentication on your broker.

Overview

This post covers two aspects of IAM-based authentication for Amazon MQ for RabbitMQ:

  1. IAM as an OAuth 2.0 identity provider: How Amazon MQ uses IAM outbound federation and the RabbitMQ OAuth 2.0 plugin to authenticate clients using short-lived JSON Web Tokens (JWTs) issued by AWS Security Token Service (AWS STS), eliminating broker-local credentials.
  2. Multi-tenant isolation with IAM roles and scope aliases: How per-tenant IAM roles combined with RabbitMQ scope aliases restrict access to specific virtual hosts (vhosts), enforcing tenant isolation at both the authentication and broker layers.

Both capabilities work together to provide credential-free authentication, centralized access control, and a comprehensive audit trail through AWS CloudTrail.

How IAM authentication works

IAM authentication for Amazon MQ for RabbitMQ uses the RabbitMQ OAuth 2.0 plugin with IAM serving as the identity provider through IAM outbound federation. Instead of managing usernames and passwords in the broker, clients authenticate using short-lived JWTs issued by AWS STS.

When a client connects to a broker configured with IAM authentication:

  1. The client application uses its IAM credentials from an IAM role attached to its AWS Lambda function, Amazon Elastic Container Service (Amazon ECS) task, Amazon Elastic Kubernetes Service (Amazon EKS) pod, or Amazon Elastic Compute Cloud (Amazon EC2) instance to call AWS STS.
  2. AWS STS evaluates the caller’s IAM policies for sts:GetWebIdentityToken.
  3. If the policy allows the request, AWS STS issues a signed JWT that encodes the caller’s identity and the permitted RabbitMQ scopes.
  4. The client connects to the Amazon MQ broker and presents the JWT as an OAuth 2.0 bearer token (passed as the password).
  5. The broker retrieves the AWS STS public keys through the JSON Web Key Set (JWKS) endpoint and validates the token signature, expiration, and audience claim.
  6. The broker extracts the caller’s IAM role ARN from the token’s sub claim, matches it against configured scope aliases, and grants the corresponding RabbitMQ permissions.

The following diagram shows the IAM authentication flow.

IAM authentication flow from a client IAM role through AWS STS token issuance to broker validation through the JWKS endpoint

Benefits over traditional username/password authentication

The following table compares traditional username/password authentication with IAM-based OAuth 2.0 authentication across the operational dimensions that matter most at scale.

Aspect Traditional (username/password) IAM-based (OAuth 2.0 JWT)
Credential management Manual creation, distribution, and rotation Automatic through IAM roles. No broker-local credentials
Credential lifetime Static until manually rotated Short-lived (5 minutes–1 hour). Automatic expiration
Access control Broker-local permissions per user Centralized through IAM roles mapped to broker scope aliases
Audit trail Broker logs only AWS CloudTrail logs every token issuance and policy evaluation
Tenant isolation Manual permission configuration per user Per-role scope aliases enforce vhost restrictions at the broker
Onboarding/offboarding Create/delete RabbitMQ users and distribute credentials Create/delete IAM roles. No credential distribution needed

Key configuration

The following rabbitmq.conf snippet shows the essential settings for IAM-based OAuth 2.0 authentication:

# Enable OAuth 2.0 authentication with IAM, with internal as fallback
auth_backends.1 = oauth2
auth_backends.2 = internal

# Token validation - account-specific JWKS endpoint
auth_oauth2.jwks_uri = https://<issuer-id>.tokens.sts.global.api.aws/.well-known/jwks.json
auth_oauth2.https.hostname_verification = wildcard

# Resource server configuration
auth_oauth2.resource_server_id = rabbitmq
auth_oauth2.scope_prefix = rabbitmq/

# Required: extract identity from the 'sub' claim in STS JWTs
auth_oauth2.additional_scopes_key = sub

# Scope alias maps IAM role ARN to RabbitMQ permissions
auth_oauth2.scope_aliases.1.alias = arn:aws:iam::<account-id>:role/RabbitMqAdminRole
auth_oauth2.scope_aliases.1.scope = rabbitmq/tag:administrator rabbitmq/read:*/* rabbitmq/write:*/* rabbitmq/configure:*/*

# Enable OAuth for the Management UI
management.oauth_enabled = true

Note: The auth_oauth2.jwks_uri value is account-specific. Obtain it by running aws iam enable-outbound-web-identity-federation, which returns an issuer identifier URL. Append /.well-known/jwks.json to form the full JWKS URI.

The following table describes each configuration setting shown in the preceding snippet.

Setting Purpose
auth_backends.1 = oauth2 Enables the OAuth 2.0 authentication backend
auth_backends.2 = internal Fallback to internal auth for the system monitoring user
auth_oauth2.jwks_uri Account-specific JWKS endpoint (from IAM outbound federation) for validating token signatures
auth_oauth2.resource_server_id Identifies this broker as a resource server. Must match the --audience value used when requesting tokens
auth_oauth2.scope_prefix Prefix applied to scope values (for example, rabbitmq/)
auth_oauth2.additional_scopes_key JWT claim key where RabbitMQ looks for the identity used in scope alias matching (must be sub for STS JWTs)
auth_oauth2.scope_aliases..alias The IAM role ARN that maps to a set of RabbitMQ permissions
auth_oauth2.scope_aliases..scope The RabbitMQ permissions granted when the alias matches
auth_oauth2.https.hostname_verification Set to wildcard for AWS STS endpoint certificate validation
management.oauth_enabled Enables OAuth token authentication for the Management API/UI

IAM policy with vhost restriction

The IAM policy condition is what enforces tenant isolation at the authentication layer. The following policy restricts a role to requesting tokens scoped to a specific vhost:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetWebIdentityToken",
                "sts:TagGetWebIdentityToken"
            ],
            "Resource": "*"
        }
    ]
}
Policy element Purpose
sts:GetWebIdentityToken Authorizes JWT token issuance through STS
sts:TagGetWebIdentityToken Allows attaching request tags (such as scope) to the token request

Vhost-level isolation is enforced at the broker layer through scope aliases (see the following Multi-tenant isolation with IAM section), not through IAM policy conditions. Each IAM role maps to a specific set of RabbitMQ permissions through the broker configuration, and the broker denies any access not granted by the matching scope alias.

Important considerations

  • IAM authentication is supported on Amazon MQ for RabbitMQ versions 3.13 and 4.2 or later. It isn’t supported on Amazon MQ for ActiveMQ brokers.
  • IAM authentication requires IAM outbound federation to be configured and available in your AWS account. Make sure that the outbound federation is enabled before configuring IAM-based authentication on your broker.
  • With AWS STS, you can request web identity tokens with a duration between 300 seconds (5 minutes) and 3600 seconds (1 hour) with the --duration-seconds parameter. Implement token caching and refresh logic in your client applications to avoid requesting a new token on every connection.
  • Don’t embed IAM user credentials in application code or environment variables. Attach IAM roles to AWS Lambda functions, Amazon ECS tasks, Amazon EKS pods, or Amazon EC2 instances so that credentials are issued and rotated automatically by the AWS runtime.
  • The IAM policy evaluation happens before any broker interaction. If the policy denies the sts:GetWebIdentityToken request, AWS STS returns AccessDenied and no connection is attempted.
  • Amazon MQ automatically creates a system user named monitoring-AWS-OWNED-DO-NOT-DELETE with monitoring-only permissions. This user uses RabbitMQ’s internal authentication system even on IAM-enabled brokers, and Amazon MQ restricts it to loopback interface access only.

Limitations

  • Scope claim configuration: You can’t use a scope claim directly because the JWT token from AWS STS places the caller’s identity (IAM role ARN) in the sub claim rather than a standard scope claim. This requires setting auth_oauth2.additional_scopes_key = sub and using scope aliases in the RabbitMQ configuration to map IAM role ARNs to RabbitMQ permissions. This limitation also prevents using IAM policies for authorization fully, requiring RabbitMQ configuration for authorization instead.

For information about how to configure IAM authentication and authorization for your Amazon MQ for RabbitMQ brokers, see the following Implementation guide section.

Multi-tenant isolation with IAM

IAM-based authentication is particularly effective for multi-tenant architectures where you need to enforce data isolation across a shared RabbitMQ infrastructure. By combining per-tenant IAM roles with RabbitMQ scope aliases, you enforce isolation at three layers:

  • IAM layer: Trust policies restrict which principals (Lambda functions, ECS tasks, EKS pods) can assume each tenant’s IAM role. A service belonging to Tenant A cannot assume Tenant B’s role.
  • Broker layer: Scope aliases make sure that each role ARN only receives permissions for its own vhost. Even if a client attempts to connect to a different vhost, the broker denies access because the token’s sub claim maps to permissions for a different vhost only.
  • Audit layer: CloudTrail logs every role assumption and AWS STS token request, including the IAM principal and whether the request was granted or denied.

The following diagram shows the multi-tenant architecture.

Multi-tenant architecture where per-tenant IAM roles map through AWS STS and broker scope aliases to isolated tenant-a and tenant-b vhosts

Broker configuration for multi-tenant isolation

AMQP-only access (default): For tenants that connect through AMQP to produce and consume messages:

# Tenant A - AMQP access to tenant-a vhost only
auth_oauth2.scope_aliases.2.alias = arn:aws:iam::<account-id>:role/TenantARole
auth_oauth2.scope_aliases.2.scope = rabbitmq/configure:tenant-a/* rabbitmq/write:tenant-a/* rabbitmq/read:tenant-a/*

# Tenant B - AMQP access to tenant-b vhost only
auth_oauth2.scope_aliases.3.alias = arn:aws:iam::<account-id>:role/TenantBRole
auth_oauth2.scope_aliases.3.scope = rabbitmq/configure:tenant-b/* rabbitmq/write:tenant-b/* rabbitmq/read:tenant-b/*

With Management API access (optional): For tenants that also need HTTP API access for monitoring or management:

# Tenant A - AMQP + Management API access to tenant-a vhost
auth_oauth2.scope_aliases.2.alias = arn:aws:iam::<account-id>:role/TenantARole
auth_oauth2.scope_aliases.2.scope = rabbitmq/tag:management rabbitmq/configure:tenant-a/* rabbitmq/write:tenant-a/* rabbitmq/read:tenant-a/*

# Tenant B - AMQP + Management API access to tenant-b vhost
auth_oauth2.scope_aliases.3.alias = arn:aws:iam::<account-id>:role/TenantBRole
auth_oauth2.scope_aliases.3.scope = rabbitmq/tag:management rabbitmq/configure:tenant-b/* rabbitmq/write:tenant-b/* rabbitmq/read:tenant-b/*

The tag:management scope grants access to the RabbitMQ Management HTTP API, limited to resources the tenant already has permissions for. Most producer/consumer workloads (Lambda, ECS tasks) connect through AMQP and do not need this tag. Add it only for tenants that require monitoring or management capabilities through the HTTP API.

How isolation is enforced

When Tenant A’s service connects to the broker:

  1. The service assumes TenantARole using its attached IAM role credentials.
  2. AWS STS issues a JWT with sub = arn:aws:iam::<account-id>:role/TenantARole.
  3. The service connects to the broker with the JWT as the password.
  4. The broker matches the sub claim against scope aliases and grants configure:tenant-a/*, write:tenant-a/*, and read:tenant-a/*.
  5. If the service attempts to connect to vhost tenant-b, the broker returns NOT_ALLOWED - access to vhost 'tenant-b' refused for user 'arn:aws:iam::<account-id>:role/TenantARole'.

Trust policy for tenant isolation

Each tenant role uses a trust policy that restricts which principals can assume it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>:role/TenantAServiceRole"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

This ensures that only Tenant A’s services can obtain tokens that map to Tenant A’s vhost permissions.

Client authentication pattern

Each client application uses its IAM role credentials to obtain a short-lived token from AWS STS, then presents that token as the password when connecting to the broker:

import boto3
import pika
import ssl

class TokenManager:
    def __init__(self):
        self.sts_client = boto3.client("sts")

    def get_token(self, role_arn: str) -> str:
        # Assume the tenant's IAM role
        assumed = self.sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName="rabbitmq-session"
        )
        # Create STS client with assumed role credentials
        sts = boto3.client(
            "sts",
            aws_access_key_id=assumed["Credentials"]["AccessKeyId"],
            aws_secret_access_key=assumed["Credentials"]["SecretAccessKey"],
            aws_session_token=assumed["Credentials"]["SessionToken"],
        )
        # Get web identity token
        response = sts.get_web_identity_token(
            Audience=["rabbitmq"],
            SigningAlgorithm="ES384",
            DurationSeconds=300,
        )
        return response["WebIdentityToken"]

class RabbitMQClient:
    def __init__(self, broker_host: str, vhost: str, role_arn: str):
        self.broker_host = broker_host
        self.vhost = vhost
        self.role_arn = role_arn
        self.token_manager = TokenManager()

    def connect(self) -> pika.channel.Channel:
        token = self.token_manager.get_token(self.role_arn)
        credentials = pika.PlainCredentials(
            username="", password=token
        )
        parameters = pika.ConnectionParameters(
            host=self.broker_host,
            port=5671,
            virtual_host=self.vhost,
            credentials=credentials,
            ssl_options=pika.SSLOptions(ssl.create_default_context()),
        )
        return pika.BlockingConnection(parameters).channel()

The token manager caches tokens and refreshes them before expiration, so your application does not request a new token on every connection. For long-running connections outside Lambda (such as Amazon ECS tasks or EC2-hosted services), add connection recovery logic to handle token expiry gracefully and reconnect with a fresh token when needed.

Comparing IAM authentication with other approaches

The following table compares IAM authentication with the other authentication methods available for Amazon MQ for RabbitMQ, so you can choose the approach that best fits your security and operational requirements.

Aspect IAM (OAuth 2.0 through STS) OAuth 2.0 (external IdP) Username/Password
Identity provider IAM / STS External OAuth 2.0 IdP Broker-local
Credential type Short-lived JWT Short-lived JWT Static password
Credential management Automatic through IAM roles Managed by external IdP Manual creation and rotation
Tenant isolation Per-role scope aliases restrict vhost access at the broker layer Token scopes Manual per-user permissions
Audit trail AWS CloudTrail IdP-specific logs Broker logs only
AWS integration Native (IAM roles, STS, CloudTrail) Requires external IdP configuration None

Implementation guide

Cleaning up

To avoid ongoing charges, delete the resources you created during this walkthrough:

  1. Delete the test IAM roles (TenantARole, TenantBRole) and their associated trust policies.
  2. If you created a dedicated Amazon MQ broker for testing, delete the broker from the Amazon MQ console.
  3. Remove any test virtual hosts and their queues from your broker configuration.

For production deployments, retain your IAM roles and broker configuration but review your scope aliases periodically to remove unused tenant mappings.

Conclusion

This post demonstrated how IAM-based OAuth 2.0 authentication works for Amazon MQ for RabbitMQ, and how per-tenant IAM roles combined with broker scope aliases enforce multi-tenant isolation. Clients authenticate using their existing IAM roles, AWS STS issues short-lived JWTs, and the broker validates tokens using the AWS STS JWKS endpoint. Scope aliases map each role ARN to vhost-specific permissions, ensuring tenants can only access their own resources.

Combined with the certificate-based authentication covered in Part 1 and the OAuth 2.0, LDAP, Entra ID, and HTTP integrations covered in Part 2, you now have a detailed picture of the authentication and authorization options available for Amazon MQ for RabbitMQ. Choose the approach that fits your identity infrastructure or combine multiple methods for defense-in-depth security.

If you have questions or feedback about this post, leave a comment in the Comments section. For troubleshooting help, visit the AWS re:Post community for Amazon MQ.

For more information about Amazon MQ security, see the following resources:


About the authors

Vinodh Kannan Sadayamuthu

Vinodh Kannan Sadayamuthu

Vinodh is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.

Paras Jain

Paras Jain

Paras is a Senior Solutions Architect at AWS. He works with Security Independent Software Vendors (ISVs) to build and deploy scalable, secure, and resilient applications. He lives in Ashburn, VA and enjoys spending time with his wife, two kids, and a dog.

OAuth 2.0, LDAP, and HTTP auth for Amazon MQ for RabbitMQ

Post Syndicated from Vinodh Kannan Sadayamuthu original https://aws.amazon.com/blogs/big-data/oauth-2-0-ldap-and-http-auth-for-amazon-mq-for-rabbitmq/

This is Part 2 of a three-part series on authentication and authorization for Amazon MQ for RabbitMQ. For an overview of all available methods, see Authentication and Authorization Options for Amazon MQ for RabbitMQ. For certificate-based mTLS and SSL authentication, see Part 1. For AWS Identity and Access Management (IAM) authentication, see Part 3.

When you deploy Amazon MQ for RabbitMQ in an enterprise environment, authentication quickly becomes more complex than a single broker configuration. Your organization might already have an Active Directory managing thousands of users, or a cloud identity provider handling application access, or workloads that require short-lived, token-based credentials. Maintaining a separate set of static RabbitMQ credentials alongside these systems creates operational overhead and introduces security gaps. This is especially true when users change roles, leave the organization, or when credentials need to be rotated across multiple brokers.

Amazon MQ for RabbitMQ supports OAuth 2.0, LDAP, and HTTP-based authentication backends, so you can connect your broker directly to the identity infrastructure you already use. This post explains how each approach works, highlights the key configurations, and helps you decide which one fits your use case.

Overview

This post covers three authentication and authorization integrations for Amazon MQ for RabbitMQ:

  1. OAuth 2.0: Token-based authentication where clients obtain short-lived tokens from an identity provider and present them to the broker as bearer credentials. The broker validates tokens using JSON Web Key Sets (JWKS) and derives permissions from token scopes.
  2. LDAP: Directory-based authentication where the broker delegates credential verification to an LDAP directory such as Active Directory. Users authenticate with their directory credentials, and RabbitMQ permissions map to LDAP group memberships.
  3. HTTP authentication backend: A flexible approach where the broker delegates authentication and authorization decisions to an external HTTP service, so you can implement custom logic or integrate with identity systems that don’t support OAuth 2.0 or LDAP natively.

All three approaches eliminate the need to manage broker-local credentials. They provide centralized user management, fine-grained access control, and audit capabilities through your existing identity infrastructure.

How OAuth 2.0 authentication works

OAuth 2.0 authentication eliminates static broker credentials by using short-lived tokens issued by an external identity provider. Instead of storing usernames and passwords in the broker, clients obtain access tokens and present them as credentials when connecting.

When a client connects to a broker configured with OAuth 2.0 authentication:

  1. The client requests an access token from the OAuth 2.0 identity provider, specifying the required scopes.
  2. The identity provider validates the client credentials and issues a signed JWT (JSON Web Token) containing the granted scopes.
  3. The client connects to the Amazon MQ broker and presents the JWT as the password.
  4. The broker retrieves the identity provider’s public keys through the JWKS endpoint.
  5. The broker validates the token signature, expiration, and audience claim.
  6. The broker extracts RabbitMQ permissions from the token scopes and grants access accordingly.

The following diagram shows the OAuth 2.0 authentication flow.

OAuth 2.0 authentication flow between a client, an identity provider, and the Amazon MQ for RabbitMQ broker

Figure 1: OAuth 2.0 authentication flow for Amazon MQ for RabbitMQ

Scope-to-permission mapping

The broker maps OAuth 2.0 scopes to RabbitMQ permissions using a configurable prefix. For example, with the resource server ID rabbitmq, the following scopes grant specific access:

OAuth 2.0 scope RabbitMQ permission
rabbitmq.read:*/* Read access to all resources in all vhosts
rabbitmq.write:*/* Write access to all resources in all vhosts
rabbitmq.configure:*/* Configure access to all resources in all vhosts
rabbitmq.read:orders/* Read access to all resources in the orders vhost
rabbitmq.tag:management Management UI access
rabbitmq.tag:administrator Administrator access

Key configuration

The following rabbitmq.conf snippet shows the essential settings for OAuth 2.0 authentication:

# Enable OAuth 2.0 authentication (with internal fallback for the monitoring user)
auth_backends.1 = oauth2
auth_backends.2 = internal

# OAuth 2.0 resource server configuration
auth_oauth2.resource_server_id = rabbitmq
auth_oauth2.preferred_username_claims.1 = sub

# JWKS endpoint for token validation
auth_oauth2.jwks_uri = https://your-idp.example.com/.well-known/jwks.json

# Additional token validation
auth_oauth2.issuer = https://your-idp.example.com
auth_oauth2.scope_prefix = rabbitmq.

# Skip audience validation for IdPs that do not emit an aud claim matching resource_server_id
auth_oauth2.verify_aud = false

The following table describes each configuration setting.

Setting Purpose
auth_backends.1 = oauth2 Enables the OAuth 2.0 authentication backend (use auth_backends.2 = internal for the monitoring user fallback)
auth_oauth2.resource_server_id Identifies this broker as a resource server. Used as the scope prefix
auth_oauth2.preferred_username_claims.1 JWT claim used to extract the username for display and logging
auth_oauth2.jwks_uri URL of the identity provider’s JWKS endpoint for token signature validation (named jwks_url on RabbitMQ 3.x, jwks_uri on 4.x)
auth_oauth2.issuer Expected token issuer. Tokens from other issuers are rejected
auth_oauth2.verify_aud Whether the broker validates the token’s aud claim against resource_server_id. Set to false for IdPs that do not emit a matching aud
auth_oauth2.scope_prefix Prefix applied to scopes when mapping to RabbitMQ permissions

Important considerations

  1. By default the broker validates the token’s aud (audience) claim against the resource_server_id and rejects tokens without a match. Some identity providers (for example, Amazon Cognito) don’t emit an aud claim matching the resource server. For those, set auth_oauth2.verify_aud = false.
  2. If your identity provider cannot issue scopes in the native RabbitMQ form (for example, it disallows the * wildcard), use auth_oauth2.scope_aliases entries to translate the provider’s scope names to RabbitMQ scopes such as rabbitmq.read:*/*.
  3. Configure short-lived tokens (one hour or less) and implement token refresh logic in your client applications.
  4. The JWKS endpoint must be reachable from the broker’s network. For private identity providers, verify network connectivity and DNS resolution.
  5. On RabbitMQ 3.x the JWKS endpoint setting is auth_oauth2.jwks_url. On RabbitMQ 4.x it is auth_oauth2.jwks_uri. Use the setting name that matches your broker engine version.
  6. Amazon MQ automatically creates a system user named monitoring-AWS-OWNED-DO-NOT-DELETE with monitoring-only permissions. This user uses the internal RabbitMQ authentication system even on OAuth 2.0-enabled brokers.

How LDAP authentication works

LDAP authentication connects your RabbitMQ broker to an existing directory service such as Active Directory. Instead of managing users locally in the broker, the broker delegates authentication to the LDAP server and derives permissions from directory group memberships. This centralizes user management and lets you apply your existing password policies, account lockout rules, and audit trails to broker access.

When a client connects to a broker configured with LDAP authentication:

  1. The client connects to the Amazon MQ broker with a username and password.
  2. The broker constructs a Distinguished Name (DN) from the username using the configured user_dn_pattern.
  3. The broker performs an LDAP bind operation against the directory server using the constructed DN and the client’s password.
  4. If the bind succeeds, the broker queries the directory for the user’s group memberships.
  5. The broker maps group memberships to RabbitMQ permissions (vhost access, resource permissions, and management tags).
  6. The client is authenticated and authorized based on the LDAP query results.

The following diagram shows the LDAP authentication flow.

LDAP authentication flow showing the Amazon MQ broker binding to a directory server and mapping group memberships to permissions

Figure 2: LDAP authentication flow for Amazon MQ for RabbitMQ

LDAP directory structure

This implementation uses a group-centric LDAP model where RabbitMQ concepts (vhosts, exchanges, queues, and tags) are represented as sub-OUs under a single groups hierarchy:

OU=rabbitmq
├── OU=users
│   ├── CN=app-orders-producer
│   └── CN=app-orders-consumer
│
└── OU=groups
    ├── OU=vhosts
    │   ├── CN=vhost-orders
    │   └── CN=vhost-payments
    │
    ├── OU=exchanges
    │   ├── CN=orders-publisher
    │   └── CN=payments-publisher
    │
    ├── OU=queues
    │   ├── CN=orders-consumer
    │   └── CN=payments-consumer
    │
    └── OU=tags
        ├── CN=rmq-admin
        └── CN=rmq-monitor

Users are assigned to groups based on their required access. For example, app-orders-producer would be a member of vhost-orders and orders-publisher, granting it access to the orders vhost and write permissions on the orders exchange.

Key configuration

The following rabbitmq.conf snippet shows the essential settings for LDAP authentication:

# Enable LDAP as primary backend with internal as fallback
auth_backends.1 = ldap
auth_backends.2 = internal

# LDAP server connection (LDAPS on port 636)
auth_ldap.servers.1 = your-active-directory-server.example.com
auth_ldap.port = 636
auth_ldap.user_dn_pattern = CN=${username},OU=users,OU=rabbitmq,DC=example,DC=com
auth_ldap.use_ssl = true
auth_ldap.ssl_options.verify = verify_peer
auth_ldap.log = true

# AWS integration: assume an IAM role to retrieve the CA certificate for LDAPS
aws.arns.assume_role_arn = arn:aws:iam::111122223333:role/AmazonMqLdapRole
aws.arns.auth_ldap.ssl_options.cacertfile = arn:aws:s3:::your-ca-cert-bucket/ca-cert.pem

# Management console tags
auth_ldap.queries.tags = '''
[{administrator, {in_group, "CN=rmq-admin,OU=tags,OU=groups,OU=rabbitmq,DC=example,DC=com"}},
{management, {in_group, "CN=rmq-monitor,OU=tags,OU=groups,OU=rabbitmq,DC=example,DC=com"}}]
'''

# Vhost access control
auth_ldap.queries.vhost_access = '''
{in_group, "CN=vhost-${vhost},OU=vhosts,OU=groups,OU=rabbitmq,DC=example,DC=com"}
'''

# Resource access control
auth_ldap.queries.resource_access = '''
{for, [{permission, configure,
{in_group, "CN=rmq-admin,OU=tags,OU=groups,OU=rabbitmq,DC=example,DC=com"}},
{permission, write,
{for, [{resource, exchange,
{in_group, "CN=orders-publisher,OU=exchanges,OU=groups,OU=rabbitmq,DC=example,DC=com"}}]}},
{permission, read,
{for, [{resource, queue,
{in_group, "CN=orders-consumer,OU=queues,OU=groups,OU=rabbitmq,DC=example,DC=com"}}]}}]}
'''

The following table describes each configuration setting.

Setting Purpose
auth_backends.1 = ldap Sets LDAP as the primary authentication backend
auth_backends.2 = internal Falls back to internal authentication if LDAP is unavailable
auth_ldap.servers.1 LDAP server hostname or IP address
auth_ldap.user_dn_pattern Template for constructing the user DN from the provided username
auth_ldap.port LDAP server port; 636 for LDAPS
auth_ldap.use_ssl Enables an encrypted LDAPS connection to the directory server. Amazon MQ requires that you explicitly set either auth_ldap.use_ssl = true or auth_ldap.use_starttls = true. The broker fails configuration validation if neither is set.
auth_ldap.ssl_options.verify Certificate verification mode for the LDAPS connection. Verify_peer validates the server certificate
aws.arns.assume_role_arn ARN of the IAM role the broker assumes to retrieve the CA certificate
aws.arns.auth_ldap.ssl_options.cacertfile ARN of the CA certificate (in S3) used to validate the LDAP server’s TLS certificate
auth_ldap.queries.tags Maps directory group membership to the administrator and management console tags
auth_ldap.queries.vhost_access LDAP query that determines which vhosts a user can access based on group membership
auth_ldap.queries.resource_access LDAP query that determines resource-level permissions (configure, write, read) based on group membership

Important considerations

  1. Amazon MQ requires an encrypted LDAP connection: you must explicitly set either auth_ldap.use_ssl = true (LDAPS on port 636) or auth_ldap.use_starttls = true (StartTLS on port 389). The broker rejects the configuration if neither is set. Unencrypted LDAP transmits credentials in plaintext, so always use one of these options to protect credentials in transit between the broker and your directory server.
  2. The user_dn_pattern must match your directory’s organizational structure exactly. Verify the pattern with an LDAP browser before applying it to the broker.
  3. With Active Directory, user DNs are usually based on the display name rather than the sign-in name, so a fixed user_dn_pattern often will not match. In that case, configure DN lookup (auth_ldap.dn_lookup_bind, auth_ldap.dn_lookup_base, and auth_ldap.dn_lookup_attribute = sAMAccountName) so the broker resolves each username to its full DN before binding.
  4. LDAP configuration changes require a broker reboot to take effect. However, user permission changes in the directory (group membership additions or removals) take effect immediately for new connections.
  5. Configure the internal backend as a fallback to maintain access if the LDAP server becomes temporarily unavailable.

How HTTP authentication works

The HTTP authentication backend delegates all authentication and authorization decisions to an external HTTP service. When a client connects, the broker sends requests over HTTPS to your service, which responds with allow or deny decisions. Amazon MQ requires encrypted connections and rejects any configuration that uses a plain http endpoint. This approach provides maximum flexibility for integrating with identity systems that don’t support OAuth 2.0 or LDAP natively, or when you need custom authentication logic. The HTTP authentication backend is available on Amazon MQ for RabbitMQ version 4 and above.

When a client connects to a broker configured with HTTP authentication:

  1. The client connects to the Amazon MQ broker with a username and password.
  2. The broker sends an HTTPS POST request to the configured authentication endpoint with the username and password.
  3. The external authentication service validates the credentials against its identity store and responds with allow or deny.
  4. For each authorization check (vhost access, resource permissions, topic permissions), the broker sends additional HTTPS requests to the corresponding endpoints.
  5. The authentication service evaluates the authorization request and responds with allow, deny, or allow with tags.
  6. The client is authenticated and authorized based on the authentication service responses.

The following diagram shows the HTTP authentication flow.

HTTP authentication flow showing the Amazon MQ broker sending credential and authorization checks to an external HTTP service

Figure 3: HTTP authentication flow for Amazon MQ for RabbitMQ

The broker sends HTTPS POST requests to four endpoints. Each endpoint must return a plain-text response:

Endpoint Request parameters Expected response
/auth/user username, password allow [tag1, tag2] or deny
/auth/vhost username, vhost, ip allow or deny
/auth/resource username, vhost, resource, name, permission allow or deny
/auth/topic username, vhost, resource, name, permission, routing_key allow or deny

Key configuration

The following rabbitmq.conf snippet shows the essential settings for HTTP authentication:

# Enable the HTTP backend with caching to reduce load on the auth service
auth_backends.1 = cache
auth_backends.2 = http
auth_cache.cached_backend = http

# HTTP authentication endpoints (HTTPS required)
auth_http.http_method = post
auth_http.user_path = https://your-auth-service.example.com/auth/user
auth_http.vhost_path = https://your-auth-service.example.com/auth/vhost
auth_http.resource_path = https://your-auth-service.example.com/auth/resource
auth_http.topic_path = https://your-auth-service.example.com/auth/topic

# TLS configuration for the HTTPS connection to the auth service
auth_http.ssl_options.verify = verify_peer
auth_http.ssl_options.sni = your-auth-service.example.com

# AWS integration: IAM role and CA certificate for secure credential retrieval
aws.arns.assume_role_arn = <your-assume-role-arn>
aws.arns.auth_http.ssl_options.cacertfile = <your-ca-cert-arn>

The following table describes each configuration setting.

Setting Purpose
auth_backends.1 = cache auth_backends.2 = http Enables the HTTP authentication backend with a cache layer in front, which reduces the number of calls to your authentication service
auth_http.user_path URL the broker calls to authenticate users
auth_http.vhost_path URL the broker calls to check vhost access
auth_http.resource_path URL the broker calls to check resource permissions (queues, exchanges)
auth_http.topic_path URL the broker calls to check topic-level permissions
auth_http.http_method HTTP method the broker uses to call the endpoints. Set to post
auth_http.ssl_options.verify Certificate verification mode for the HTTPS connection to the auth service. Verify_peer validates the server certificate
auth_http.ssl_options.sni Server Name Indication hostname sent during the TLS handshake with the auth service
aws.arns.assume_role_arn ARN of the IAM role the broker assumes to securely retrieve the CA certificate
aws.arns.auth_http.ssl_options.cacertfile ARN of the CA certificate the broker uses to validate the auth service’s TLS certificate

Important considerations

  1. The HTTP authentication service must be highly available. If the service is unreachable, all authentication attempts fail. Consider deploying it behind a load balancer with health checks.
  2. HTTPS is mandatory for all authentication endpoints. The broker rejects any endpoint configured with a plain http URL, ensuring credentials are always protected in transit.
  3. Front the HTTP backend with the cache backend (auth_backends.1 = cache) to reduce the number of calls to your authentication service and improve connection latency. Also keep your service’s response times low to avoid connection timeouts and degraded broker performance.
  4. The authentication service receives plaintext passwords. Make sure the service handles credentials securely and doesn’t log them.
  5. The broker connects to your authentication service over TLS. Configure certificate validation with auth_http.ssl_options.verify = verify_peer, and provide the CA certificate and the IAM role for retrieving it through the aws.arns.auth_http.ssl_options.cacertfile and aws.arns.assume_role_arn settings.

Implementation guides

For step-by-step deployment and validation instructions, see the following resources:

  1. Amazon MQ for RabbitMQ OAuth 2.0 authentication – Configure OAuth 2.0 token-based authentication for Amazon MQ.
  2. Amazon MQ for RabbitMQ LDAP integration – Configure LDAP directory integration for Amazon MQ.
  3. Amazon MQ for RabbitMQ HTTP authentication backend – Configure the HTTP authentication backend for Amazon MQ.
  4. Amazon MQ samples repository – AWS Cloud Development Kit (AWS CDK) stacks and sample code for LDAP and OAuth 2.0 integrations.

Conclusion

This post explained how OAuth 2.0, LDAP, and HTTP authentication backends work for Amazon MQ for RabbitMQ, and when to use each one. OAuth 2.0 provides token-based, passwordless authentication with automatic credential expiration. LDAP connects your broker to existing directory infrastructure for centralized user and group management. The HTTP backend offers maximum flexibility for custom identity integrations. Used individually or in combination, these approaches eliminate broker-local credential management and provide centralized access control through your existing identity infrastructure.

In the next post in this series, we cover IAM authentication and OAuth 2.0 authorization for Amazon MQ for RabbitMQ.

For more information about Amazon MQ security, see the following resources:

  1. Amazon MQ Developer Guide: Security
  2. RabbitMQ OAuth 2.0 plugin documentation
  3. RabbitMQ LDAP plugin documentation
  4. Amazon MQ samples repository

If you have questions or feedback about this post, leave a comment in the Comments section. For troubleshooting help, visit the AWS re:Post community for Amazon MQ.


About the authors

Vinodh Kannan Sadayamuthu

Vinodh Kannan Sadayamuthu

Vinodh is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.

Sarath Kumar Kallayil Sreedharan

Sarath Kumar Kallayil Sreedharan

Sarath Kumar K.S. is a Senior Technical Account Manager/Enterprise Support lead at Amazon Web Services. Sarath works with enterprise customers to help them architect and build highly reliable and cost-effective solutions on AWS. He specializes in serverless, messaging technologies, and AI services, and has a background in application development and architecture. In his spare time, he enjoys reading, traveling, playing cricket, and spending time with his family

Mutual TLS and SSL certificate authentication for Amazon MQ for RabbitMQ

Post Syndicated from Harshith Mithamar original https://aws.amazon.com/blogs/big-data/mutual-tls-and-ssl-certificate-authentication-for-amazon-mq-for-rabbitmq/

This is Part 1 of a three-part series on authentication and authorization for Amazon MQ for RabbitMQ. For an overview of all available methods, see Authentication and Authorization Options for Amazon MQ for RabbitMQ. For OAuth 2.0, LDAP, and HTTP authentication, see Part 2. For IAM authentication, see Part 3.

When you use Amazon MQ for RabbitMQ to handle sensitive data, standard TLS encryption alone might not meet your compliance requirements. Compliance frameworks like SOX, HIPAA, and PCI DSS often require verification of the identity of both parties in a connection. Features like mutual TLS (mTLS) and SSL certificate authentication can help support those requirements by adding certificate-based identity verification to your messaging infrastructure.

Amazon MQ for RabbitMQ version 4 or later supports two certificate-based security features that address these needs: SSL certificate authentication for passwordless certificate-only login, and mTLS for certificate-based peer verification with username and password authentication. This post explains how each approach works, highlights the key configuration options, and helps you decide which one fits your use case.

Overview

This post covers two certificate-based security features for Amazon MQ for RabbitMQ:

  1. SSL certificate authentication: Passwordless authentication where clients authenticate solely using X.509 client certificates through the EXTERNAL SASL mechanism. The broker extracts the username directly from the certificate, eliminating the need for passwords.
  2. Mutual TLS (mTLS): Certificate-based peer verification where both the client and broker prove their identities using certificates, while clients still authenticate with a username and password. This secures AMQP connections and the RabbitMQ management interface.

Both features are available for Amazon MQ for RabbitMQ version 4 and above, and both use AWS ARNs for certificate and credential references, integrating with AWS Certificate Manager (ACM), and AWS Identity and Access Management (IAM).

How SSL certificate authentication works

SSL certificate authentication eliminates the need to transmit credentials during connection. The broker extracts the client’s identity from the certificate, though the corresponding user must exist in RabbitMQ’s internal store for authorization. Instead of using certificates only for transport-layer verification, the broker uses the EXTERNAL SASL mechanism to extract the client’s identity directly from the X.509 certificate.

When a client connects to a broker configured with SSL certificate authentication:

  1. The client initiates a TLS connection and presents its client certificate.
  2. The Amazon MQ broker assumes an IAM role to retrieve the CA certificate from ACM.
  3. The broker validates the client certificate against the configured CA certificate.
  4. The broker extracts the username from the client certificate using the configured field (Common Name, Distinguished Name, or Subject Alternative Name).
  5. The broker authenticates the client using the extracted username. No password required.

The following diagram shows the SSL certificate authentication flow. On the left, the client application holds only an X.509 client certificate with no credentials. In the center, the arrows show the TLS handshake carrying the client certificate to the broker, and the return path confirming authentication with no password needed. On the right, the Amazon MQ for RabbitMQ broker performs certificate validation, assuming an IAM role to retrieve the CA certificate from ACM. It then uses the EXTERNAL SASL mechanism to extract the username from the certificate’s CN, DN, or SAN field and establishes the authenticated session.

Client authenticates to the Amazon MQ for RabbitMQ broker using only an X.509 certificate through the EXTERNAL SASL mechanism, with no password

Figure 1: SSL certificate authentication flow

Username extraction options

The broker can extract the client identity from different fields of the X.509 certificate:

ssl_cert_login_from value Certificate field used Example
common_name Common Name (CN) CN=myapp → username myapp
distinguished_name Full Distinguished Name CN=myapp,O=MyOrg → username CN=myapp,O=MyOrg
subject_alternative_name Subject Alternative Name (SAN) entry SAN dns:myapp.example.com → username myapp.example.com

When you use subject_alternative_name, you also configure ssl_cert_login_san_type (dns, ip, email, uri, or other_name) and ssl_cert_login_san_index to specify which SAN entry to use.

Note: The username extraction options for ssl_cert_login_from apply only to SSL certificate authentication. mTLS doesn’t extract identity from the client certificate.

Key configuration

The following rabbitmq.conf snippet shows the essential settings for SSL certificate authentication:

# Enable certificate-only authentication
auth_mechanisms.1 = EXTERNAL
ssl_cert_login_from = common_name
auth_backends.1 = internal
# Require client certificates
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = true
# AWS integration for certificate retrieval
aws.arns.assume_role_arn = ${AmazonMqAssumeRoleArn}
aws.arns.ssl_options.cacertfile = ${CaCertArn}

The following table describes what each setting controls:

Setting Purpose
auth_mechanisms.1 = EXTERNAL Enables the EXTERNAL SASL mechanism, authenticating clients using their X.509 certificate instead of a username and password
ssl_cert_login_from = common_name Tells the broker which certificate field to extract the username from
ssl_options.verify = verify_peer Enables client certificate verification
ssl_options.fail_if_no_peer_cert = true Rejects connections from clients that do not present a certificate
aws.arns.assume_role_arn IAM role ARN the broker assumes to retrieve certificates from ACM
aws.arns.ssl_options.cacertfile ARN of the CA certificate in ACM used to validate client certificates

Note: EXTERNAL and internal serve different purposes. EXTERNAL is the authentication mechanism that verifies client identity using the X.509 certificate. internal is the authorization backend that resolves permissions for the authenticated user from RabbitMQ’s built-in user store.

Important considerations

  1. Client certificates must be signed by a trusted Certificate Authority (CA). The broker validates the certificate chain during authentication.
  2. Amazon MQ enforces the use of AWS ARNs for certificate-related settings. Use aws.arns.ssl_options.cacertfile instead of ssl_options.cacertfile.
  3. Amazon MQ automatically creates a system user named monitoring-AWS-OWNED-DO-NOT-DELETE with monitoring-only permissions. This user uses RabbitMQ’s internal authentication system even on SSL certificate-enabled brokers and is restricted to loopback interface access only.
  4. If any setting requires the use of an AWS ARN, you must also provide aws.arns.assume_role_arn.
  5. Amazon MQ doesn’t currently support CRL or OCSP for certificate revocation. To revoke a client certificate that’s no longer trusted, replace the CA certificate on AWS Private Certificate Authority (AWS Private CA), re-issue valid client certificates, and apply a configuration update to the broker.
  6. To rotate certificates, update the CA certificate on AWS Private CA and update the broker configuration. Configuration changes don’t take effect immediately. To apply your changes, wait for the next maintenance window or reboot the broker.

How mutual TLS (mTLS) works

Standard TLS works like visiting a secure website: only the server proves its identity to your browser using a certificate. With mTLS, both your client application and the message broker must prove their identities using certificates. This two-way authentication helps verify that only authorized clients can connect to your broker. Unlike SSL certificate authentication, mTLS still requires a username and password at the application layer.

When your client connects to an Amazon MQ broker with mTLS enabled, the following authentication process occurs:

  1. The client initiates a TLS connection and presents its client certificate.
  2. The Amazon MQ broker assumes an IAM role to retrieve the CA certificate from ACM.
  3. The broker validates the client certificate against the CA certificate.
  4. The client authenticates with a username and password in the application layer.
  5. Authentication succeeds, and the broker establishes a secure, encrypted connection with the client.

Note: Unlike SSL certificate authentication, mTLS doesn’t extract the username from the certificate. The client certificate proves transport-layer trust only. The broker validates it against the CA certificate but does not use any certificate fields for application-level authentication. The username provided at login doesn’t need to match the client certificate’s CN.

The following diagram illustrates this two-layer flow. On the left, the client application holds both a client certificate and a username and password. In the center, the arrows show the TLS handshake carrying the client certificate to the broker, followed by the credentials. On the right, the Amazon MQ for RabbitMQ broker performs certificate validation at the transport layer, assuming an IAM role to retrieve the CA certificate from ACM. It then authenticates the username and password at the application layer before establishing the secure connection to the client.

Mutual TLS flow in which the broker validates the client certificate, then authenticates the username and password at the application layer

Figure 2: Mutual TLS authentication flow

With mTLS, you can secure:

  • Client connections to the AMQP endpoint.
  • The RabbitMQ management interface.
  • Connections to OAuth 2.0 identity providers.
  • HTTPS authentication server connections.
  • Lightweight Directory Access Protocol (LDAP) server communications.

Key configuration

The following rabbitmq.conf snippet shows the essential settings for mTLS:

auth_backends.1 = internal
# Require client certificates for AMQP and management
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = true
management.ssl.verify = verify_peer
# AWS integration for certificate retrieval
aws.arns.assume_role_arn = ${AmazonMqAssumeRoleArn}
aws.arns.ssl_options.cacertfile = ${CaCertArn}
aws.arns.management.ssl.cacertfile = ${CaCertArn}

The following table describes the mTLS-specific settings and their purpose:

Setting Purpose
ssl_options.verify = verify_peer Enables client certificate verification for AMQP connections
ssl_options.fail_if_no_peer_cert = true Rejects connections from clients that do not present a certificate
management.ssl.verify = verify_peer Enables client certificate verification for the RabbitMQ management interface
aws.arns.ssl_options.cacertfile ARN of the CA certificate in ACM used to validate client certificates for AMQP
aws.arns.management.ssl.cacertfile ARN of the CA certificate in ACM used to validate client certificates for the management interface

Certificate requirements

Both SSL certificate authentication and mTLS require three types of certificates:

  • Server certificate: Authenticates the broker to clients. Obtain from AWS Private Certificate Authority (AWS Private CA) and reference using an AWS ARN.
  • Client certificates: Authenticate each client application to the broker. Issue from your organization’s CA or AWS Private CA.
  • CA certificate: Validates client certificates on the broker side. Store in ACM and reference in the broker’s SSL configuration.

Comparing SSL certificate authentication and mTLS

Use the following table to decide which method fits your security requirements:

Aspect SSL certificate authentication Mutual TLS (mTLS)
Authentication mechanism EXTERNAL SASL — certificate is the sole credential Transport-layer cert verification + username/password at application layer
Password required No Yes
Username source Extracted from certificate (CN, DN, or SAN) Provided by client at login
SASL mechanism EXTERNAL PLAIN (default)
Management interface cert verification Not included by default Supported through management.ssl.verify
Key config directive auth_mechanisms.1 = EXTERNAL ssl_options.verify = verify_peer
Use case Passwordless environments, PKI-managed identities Adding cert verification to existing credential-based auth
Compliance fit Environments requiring no passwords on the wire Frameworks requiring two-factor (something you have + something you know)

Choose SSL certificate authentication when eliminating passwords entirely from your messaging layer, or when your PKI infrastructure already manages client identities. Choose mTLS when adding transport-layer certificate verification to an existing deployment that relies on username/password authentication, or when compliance frameworks mandate two-factor authentication.

Additional SSL options

Both methods support the following additional configuration options:

Configuration Description
ssl_options.depth Maximum certificate chain depth for verification
ssl_options.hostname_verification Hostname verification mode: wildcard or none
ssl_cert_login_san_type SAN type when using Subject Alternative Name: dns, ip, email, uri, or other_name
ssl_cert_login_san_index Zero-based index of the SAN entry to use

Implementation guides

For step-by-step deployment and validation instructions, see the following resources:

Both tutorials use AWS CDK for infrastructure deployment and include validation scripts to test connectivity.

Conclusion

SSL certificate authentication and mTLS each address different security requirements for Amazon MQ for RabbitMQ. SSL certificate authentication uses the X.509 certificate as the sole credential through the EXTERNAL SASL mechanism, eliminating passwords entirely. mTLS adds transport-layer certificate verification on top of existing username/password authentication, giving you two-factor security. If you are building a regulated environment, SSL certificate authentication removes passwords from the wire entirely, which might help support security requirements in frameworks that address credential management. If you’re incrementally hardening an existing deployment, mTLS lets you add transport-layer verification without changing how clients authenticate. In the next post in this series, we cover OAuth 2.0, LDAP, and HTTP authentication for Amazon MQ for RabbitMQ.

To get started with Amazon MQ for RabbitMQ, see the Amazon MQ service page.

Additional resources

For more information about Amazon MQ security, see the following resources:


About the authors

Harshith Mithamar

Harshith Mithamar

Harshith is a Technical Account Manager at AWS. He works with enterprise customers to help them build secure, scalable messaging solutions on AWS.

Vinodh Kannan Sadayamuthu

Vinodh Kannan Sadayamuthu

Vinodh Kannan is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.

Authentication and authorization options for Amazon MQ for RabbitMQ

Post Syndicated from Vinodh Kannan Sadayamuthu original https://aws.amazon.com/blogs/big-data/authentication-and-authorization-options-for-amazon-mq-for-rabbitmq/

Managing authentication for message brokers at scale is complex: credentials sprawl, audit requirements, and integration with existing identity providers create operational overhead. The default approach of creating RabbitMQ users with static usernames and passwords works for getting started, but it quickly becomes a liability at scale. Credentials must be distributed securely, rotated regularly, and revoked promptly when team members change roles or leave the organization. For regulated industries, auditors want to see that your messaging infrastructure enforces the same identity and access controls as the rest of your environment.

Different organizations have different identity infrastructures. Some manage users through Active Directory. Others have standardized on OAuth 2.0. Platform teams building on AWS want to use AWS Identity and Access Management (IAM) roles and policies they understand. Security-conscious environments might require certificate-based authentication where no passwords are transmitted over the network at all.

Amazon MQ for RabbitMQ supports multiple authentication and authorization methods, so you can connect your broker to the identity infrastructure you already use. This post introduces the available options and helps you choose the right one for your use case.

Authentication methods at a glance

Amazon MQ for RabbitMQ supports the following authentication and authorization methods:

Method Credential type User management Recommended for
Simple credentials Username / password Broker-local Getting started, development environments
OAuth 2.0 Bearer tokens from external identity provider External identity provider Workloads that need short-lived tokens from a third-party identity provider
IAM authentication Short-lived JSON Web Tokens (JWTs) from AWS Security Token Service (AWS STS) IAM AWS-native workloads, multi-tenant isolation, credential-free authentication
LDAP Directory credentials Active Directory or LDAP server Organizations with existing directory services
HTTP-based auth backend Username / password validated by external server External HTTP server Custom auth logic, centralized user management across brokers
SSL certificate authentication Certificate only (passwordless) Broker-local (username extracted from cert) Eliminating passwords entirely with certificate-only identity
Mutual TLS (mTLS) Certificate and username/password Broker-local Adding transport-layer certificate verification to existing credential-based auth

Choosing the right method

The right choice depends on your existing identity infrastructure, security requirements, and operational preferences.

Simple credentials

The default method. You create RabbitMQ users with usernames and passwords directly on the broker. This is a straightforward way to get started, but it requires you to manage credentials manually. Choose this for development, testing, or small-scale deployments where credential management overhead is acceptable.

OAuth 2.0

Clients obtain short-lived tokens from any OAuth 2.0-compatible identity provider and present them to the broker as bearer tokens. Choose this when you have an existing identity provider (other than IAM) that issues tokens for your applications, and you want automatic token expiration without managing broker-local credentials.

IAM authentication

IAM serves as an identity provider. Client applications use their IAM credentials to obtain a short-lived JWT from AWS Security Token Service (AWS STS) and present it as a bearer token. IAM policies control which roles can obtain tokens. RabbitMQ scope aliases on the broker map each role’s Amazon Resource Name (ARN) to specific resource permissions (read, write, configure, and administrator). AWS CloudTrail logs every token issuance for auditing. Choose this when your workloads run on AWS compute services with IAM roles, and you want credential-free, IAM-native authentication with broker-level authorization.

LDAP

Connect your broker to an existing directory service such as Active Directory. Users authenticate with their directory credentials, and RabbitMQ permissions map to LDAP group memberships. Choose this when your organization already manages users and groups through a directory service, and you want to apply existing password policies and group-based access control to broker access.

HTTP-based auth backend

Delegates authentication and authorization decisions to a custom HTTPS server. The broker sends HTTP requests to your server for user validation, virtual host access, resource permissions, and topic permissions. Choose this when you need custom authentication logic, want to centralize user management across multiple brokers, or need to integrate with an identity system that doesn’t support OAuth 2.0 or LDAP natively.

SSL certificate authentication

Removes passwords entirely. The broker uses the EXTERNAL Simple Authentication and Security Layer (SASL) mechanism to extract the client’s identity directly from the X.509 certificate (for example, from the Common Name field) and uses it as the RabbitMQ username. With this method, your application doesn’t transmit credentials over the network. Choose this when your security policy requires passwordless authentication, and you manage client identities through a public key infrastructure (PKI).

Mutual TLS (mTLS)

Adds certificate verification on top of existing username/password authentication. During the TLS handshake, the client validates the broker’s certificate and the broker validates the client’s certificate, then the client provides a username and password at the application layer. This gives you two-factor security: something you have (the certificate) plus something you know (the password). Choose this when compliance frameworks require mutual authentication, but you want to retain your existing username/password authentication flow.

Conclusion

Amazon MQ for RabbitMQ version 4 supports seven authentication and authorization methods. With these methods, you can align your message broker security with your existing identity infrastructure. Your organization might standardize on IAM, manage identities through Active Directory, federate access through a third-party identity providers like Okta or Microsoft Entra ID, or rely on PKI for certificate-based trust. In each case, you can eliminate the operational overhead of managing static credentials at scale.

Choose your implementation path:

For sample code and infrastructure templates, clone the
Amazon MQ samples repository and deploy the CDK stack for your chosen authentication method.

About the authors

Vinodh Kannan Sadayamuthu

Vinodh Kannan Sadayamuthu

Vinodh is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.

Vignesh Selvam

Vignesh Selvam

Vignesh is the Principal Product Manager for Amazon MQ at AWS. He works with customers to solve their messaging needs and with the open-source communities for innovating with message brokers. Prior to joining AWS, he built products for security and analytics.

Introducing Private Networking for Amazon MQ for RabbitMQ

Post Syndicated from Jean-Sébastien Dominique original https://aws.amazon.com/blogs/big-data/introducing-private-networking-for-amazon-mq-for-rabbitmq/

With Private Networking for Amazon MQ for RabbitMQ, your brokers can establish outbound connections to private resources in your VPC without exposing those resources publicly. This post explains how the feature works and walks you through setting it up.

Amazon MQ for RabbitMQ brokers could previously only reach external destinations over the public internet. If you used a private Lightweight Directory Access Protocol (LDAP) server for broker authentication, you had to expose that server publicly. If you wanted to federate messages between private brokers, you needed workarounds like Network Load Balancers with IP allowlisting, as described in Implementing Federation on Amazon MQ for RabbitMQ Private Brokers. Private Networking removes those constraints.

You can connect your broker to private identity providers, other Amazon MQ for RabbitMQ brokers, or self-hosted RabbitMQ brokers running in private subnets. Combined with cross-Region networking services like AWS Transit Gateway, you can extend these connections across AWS Regions and accounts, with traffic staying on the AWS private network.

How it works

Private Networking connects your broker to private destinations using three AWS services: Amazon VPC Lattice, AWS Resource Access Manager (AWS RAM), and AWS PrivateLink.

You create a VPC Lattice resource gateway in a VPC that can reach your private destination. You then create a VPC Lattice resource configuration that defines the destination, such as an IP address or Domain Name System (DNS) name. You add the resource configuration to a RAM resource share and associate the resource share with your broker through the UpdateBroker API operation. After rebooting the broker, the network path is active and your broker can reach the private destination.

The broker does not need to be private. A publicly accessible broker works the same way.

What you can connect to

Private Networking supports three use cases.

Private identity providers

If you use an LDAP server or other identity provider for RabbitMQ authentication, you no longer need to expose it publicly. Create a resource configuration pointing to your identity provider, associate it with your broker, and use the DNS name returned by the DescribeSharedResources API operation in place of the public endpoint. Follow the existing guidance for setting up an identity provider, substituting the private DNS name.

Self-hosted RabbitMQ brokers

You can use Shovel or Federation to connect your Amazon MQ for RabbitMQ broker to a self-hosted RabbitMQ broker running in a private subnet. Create a resource configuration pointing to the self-hosted broker and use the DNS name from the DescribeSharedResources API operation in your Shovel or Federation configuration.

This pattern is useful for hybrid cloud architectures where you run RabbitMQ on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), or on-premises infrastructure and want to exchange messages with Amazon MQ without exposing either side publicly.

Other Amazon MQ for RabbitMQ brokers

You can federate or shovel messages between two Amazon MQ for RabbitMQ brokers using Private Networking. Create a resource configuration pointing to the destination broker’s endpoint and specify that same endpoint as the custom domain name on the resource configuration. This helps to verify that the DNS name resolves correctly and Transport Layer Security (TLS) peer verification succeeds.

This extends to brokers in different AWS Regions and different AWS accounts. By combining Private Networking with cross-Region networking services like AWS Transit Gateway or VPC peering, you can build a fully private federation or shovel path between brokers, with no public endpoints required.

DNS names and custom domains

Each resource configuration can include a custom domain name. If you add a verified domain, that domain resolves to the private destination. If you do not add a verified domain, Amazon MQ provides a DNS name for the broker’s private connection. Retrieve this DNS name with the DescribeSharedResources API operation.

If you specify an unverified domain on a resource configuration, it is ignored. The broker’s private connection receives a private DNS name instead, which you can retrieve with the DescribeSharedResources API operation.

For more details on custom domain names and domain verification with VPC Lattice, see Custom domain names for VPC Lattice resources.

TLS peer verification in RabbitMQ 4

Note: If you are running RabbitMQ 4, review this section before configuring Shovel or Federation connections.

RabbitMQ 4 enforces TLS certificate peer verification by default for Shovel and Federation connections. RabbitMQ 3 does not enforce this by default. When using Private Networking, the DNS name that Amazon MQ assigns to the private connection will not match the TLS certificate of the destination, which causes peer verification to fail.

The recommended approach is to specify the destination broker’s endpoint (for example, b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111.mq.us-east-1.on.aws) as the custom domain name on the resource configuration. This exception only applies to Amazon MQ for RabbitMQ broker endpoints. You cannot use an unverified domain for self-hosted brokers. Specifying the Amazon MQ endpoint causes the DNS name to match the destination’s TLS certificate, and peer verification succeeds. This approach works regardless of your RabbitMQ version and avoids the issue entirely.

Getting started

To get started with Private Networking for Amazon MQ for RabbitMQ, follow these steps.

Prerequisites

Before you begin, verify you have the following:

  • An AWS account.
  • The AWS Command Line Interface (AWS CLI) installed and configured.
  • AWS Identity and Access Management (IAM) permissions to manage Amazon MQ, VPC Lattice, and AWS RAM resources.
  • An existing VPC with connectivity to your private destination.

Walkthrough

After you have the prerequisites, follow these steps:

  1. Create an Amazon MQ for RabbitMQ broker if you do not already have one.
  2. Create a VPC Lattice resource gateway in a VPC that can reach your private destination. Make sure the resource gateway’s security group allows outbound traffic to your destination on the required port (for example, port 5671 for AMQPS (AMQP over TLS) or port 636 for LDAPS (LDAP over TLS)). The resource gateway must share at least one Availability Zone with the broker. Cluster brokers cover multiple Availability Zones, so this is satisfied. For single-instance brokers, verify the Availability Zone overlap.
  3. Create a VPC Lattice resource configuration pointing to your private destination (IP address or DNS name). If you’re connecting to another Amazon MQ broker, specify the destination broker’s endpoint as the custom domain name on the resource configuration, as shown in the following figure.VPC Lattice resource configuration showing the custom domain name field and resource definition populated with the Amazon MQ broker endpointFigure 1: VPC Lattice resource configuration showing the custom domain name field and resource definition populated with the Amazon MQ broker endpoint.
  4. Add the resource configuration to a RAM resource share. The resource share must allow external principals, as shown in the following figure.RAM resource share configuration with the Allow external principals option selectedFigure 2: RAM resource share configuration with the Allow external principals option selected.
  5. Associate the resource share with your broker by editing the broker and adding the resource share. You can also do this using the update-broker command with the AWS CLI. You must pass the entire list of resource share ARNs you want on the broker. This is a put operation, not an add or remove operation.
    aws mq update-broker \
      --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \
      --resource-share-arns arn:aws:ram:us-east-1:111122223333:resource-share/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222

    The associated RAM resource share appears as shown in the following figure.

    Network settings view with associated RAM resource shares

    Figure 3: Network settings view with associated RAM resource shares.

    Select the resource share in the Associated RAM resource shares section. The network status of each shared resource is displayed in the Shared resources section, as shown in the following figure.

    RAM resource share selection showing the network status of each shared resource

    Figure 4: RAM resource share selection showing the network status of each shared resource.

  6. Reboot the broker from the AWS Management Console or the AWS CLI to create the network path:
    aws mq reboot-broker --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

  7. Retrieve the DNS names for your RabbitMQ configuration. This operation also surfaces issues encountered during setup:
    aws mq describe-shared-resources --broker-id b-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

  8. Use the DNS name returned in the output in your Shovel, Federation, or identity provider configuration. Adding new resource configurations to an existing RAM resource share does not automatically update the broker. You must call update-broker and reboot the broker for the new resource configurations to take effect.

Cleaning up

Private Networking uses VPC Lattice and PrivateLink resources that incur ongoing charges. If you no longer need the private connection:

  1. Call update-broker with the resource share removed from the list (or an empty list to remove all), then reboot the broker.
  2. After the broker reboot completes and the resources are no longer in use, delete the VPC Lattice resource configuration and resource gateway.
  3. Optionally, remove the Amazon MQ account principal from the RAM resource share. This principal may still be in use if other brokers are associated with the same resource share, so only remove it if no other brokers depend on it.
  4. If you created a new Amazon MQ for RabbitMQ broker for this walkthrough and no longer need it, delete the broker from the Amazon MQ console or with the delete-broker command.

Operational behavior: Resource access and reboots

Removing a VPC Lattice resource configuration from a RAM resource share while the broker is actively using it revokes access immediately, with no reboot required. Removing a principal from a RAM resource share has the same effect: brokers associated through that principal lose access to the resources in the share immediately. These are intentional security behaviors managed by RAM and VPC Lattice.

Adding new resource configurations to an existing resource share does not take effect automatically. You must call update-broker and reboot the broker for the new resource configurations to take effect. This is by design. It helps verify that changes to a resource share only reach the broker when someone with broker management permissions explicitly triggers the update, providing clear security separation between share management and broker management.

Private Networking is available for Amazon MQ for RabbitMQ brokers in all the AWS Regions where Amazon VPC Lattice is available. Amazon MQ for ActiveMQ brokers do not support this feature.

Pricing

Private Networking uses Amazon VPC Lattice and AWS PrivateLink. Data processing and data transfer charges apply to traffic sent through the private connection. There is an Amazon MQ pricing of $0.01 per GB of data processed through the resource endpoint. For details, see the Amazon MQ pricing page, VPC Lattice pricing page and AWS PrivateLink pricing page.

Conclusion

In this post, we explained how Private Networking for Amazon MQ for RabbitMQ works and walked through the setup process. Whether you’re securing a private identity provider, federating messages between brokers, or connecting to self-hosted RabbitMQ, your broker can now reach private destinations without exposing them publicly.

To learn more, see the Amazon MQ Private Networking documentation.

If you have questions or feedback, leave a comment on this post.


About the authors

Jean-Sébastien Dominique

Jean-Sébastien Dominique

Jean-Sébastien is a Software Development Engineer at Amazon Web Services with 20 years of experience across a wide range of software development domains. He’s interested in the intersection of systems design, human factors, and AI – how people and complex systems interact in practice.

Ishita Chakraborty

Ishita Chakraborty

Ishita is a Senior Technical Account Manager at Amazon Web Services with expertise in serverless and messaging architectures. She works with enterprise customers to deliver technical solutions and strategic guidance – from infrastructure optimization to AI/ML adoption.

Migrate JMS applications to Amazon MQ for RabbitMQ with minimal changes

Post Syndicated from Vinodh Kannan Sadayamuthu original https://aws.amazon.com/blogs/big-data/migrate-jms-applications-to-amazon-mq-for-rabbitmq-with-minimal-changes/

Running JMS applications on on-premises brokers or Apache ActiveMQ requires manual patching cycles, capacity planning for peak loads, and maintaining high availability across multiple data centers. With Amazon MQ version 4 and above, you can migrate your existing JMS applications without rewriting your messaging layer, removing weeks of rewrite work.

This post shows you how to migrate your JMS applications and walks through a complete setup, from creating the broker to sending and receiving messages. You will also see a real-world scenario: migrating an existing Apache ActiveMQ workload to an Amazon MQ broker running RabbitMQ. The post covers configuration changes, monitoring with Amazon CloudWatch, and validation steps to make sure that your migration succeeds.

Amazon MQ version 4 and above includes built-in support for the RabbitMQ JMS Client and the JMS Topic Exchange plugin. The RabbitMQ JMS Client and JMS Topic Exchange plugin work together, allowing your existing JMS applications to connect using familiar JMS APIs. You update the connection factory configuration and broker endpoint. Your business logic, message producers, consumers, and listeners stay exactly as written.

Understanding JMS and AMQP

How the RabbitMQ JMS Client works

Use the RabbitMQ JMS Client to connect your Java application to Amazon MQ. The client translates your JMS API calls (javax.jms or jakarta.jms) into AMQP 0-9-1 messages that the broker understands.

Advanced Message Queuing Protocol (AMQP) defines how messages are formatted and transmitted across the network at the wire level. This means that non-Java services can consume the same messages using native AMQP clients, making the protocol language-agnostic

Architecture diagram showing JMS to AMQP translation: Java and Spring applications use the JMS API and RabbitMQ JMS Client Library to communicate with an Amazon MQ broker running RabbitMQ via the AMQP 0-9-1 protocol.

JMS version support

Migrate at the JMS version that your application already uses. The client supports JMS 1.1, 2.0, and 3.1 (Jakarta Messaging), so you don’t need to upgrade your application code before migrating brokers. The client integrates with Spring Framework and Spring Boot applications without requiring custom bean factories or application context configuration.

Because the JMS abstraction layer sits between your application and the broker, most migrations require only a connection factory change, not a logic rewrite.

RabbitMQ JMS Topic Exchange plugin

Your existing publish/subscribe patterns work without client-side routing logic. The JMS Topic Exchange plugin adds server-side support for JMS topic semantics, handling topic routing and SQL-based message selection directly in the broker.

The plugin handles SQL-based message selection (JMS selectors like OrderType = `Electronics` AND Priority > 5) and topic hierarchies with wildcard pattern matching (* for single level, # for multiple levels). Your application uses standard JMS topic APIs (createTopic(), setMessageSelector()) without additional filtering logic.

Getting started

This walkthrough shows you how to set up Amazon MQ and connect your existing JMS application. You will create a broker, configure the connection factory, and send and receive messages.

Prerequisites

You need an existing JMS application built on Apache ActiveMQ or another JMS provider to migrate. If you don’t have one, you can still follow Steps 1–5 to create a broker and test the connection pattern. Before you begin, confirm that you have the following in place:

  • An active AWS account
  • AWS Command Line Interface (AWS CLI) installed. For instructions, see Installing the AWS CLI.
  • Java 11 or later installed on your local development environment.
  • An AWS Identity and Access Management (IAM) principal (user or role) with the AmazonMQFullAccess managed policy attached.
  • Maven or Gradle for dependency management.

Amazon MQ broker charges apply based on instance type and usage. Review the Amazon MQ pricing page before you start.

Step 1: Create an Amazon MQ for RabbitMQ broker

The following command creates a single-instance broker running RabbitMQ 4.2 on an mq.m7g.medium instance.

aws mq create-broker \ 
--broker-name my-rabbitmq-broker \ 
--engine-type rabbitmq \ 
--engine-version 4.2 \ 
--deployment-mode SINGLE_INSTANCE \ 
--host-instance-type mq.m7g.medium \ 
--auto-minor-version-upgrade \ 
--publicly-accessible \ 
--users "Username=admin,Password=[PASSWORD]" \ 
--region us-west-2

Replace <broker-name> with the name that you want to give to the broker. Replace <username> and <password> as described in the create-broker CLI documentation. After the command runs successfully, the command line displays the BrokerArn and BrokerId.

Note: This command creates a publicly accessible broker for demonstration purposes only. For production workloads, create brokers in private subnets within your VPC and restrict access using security groups. Don’t use the –publicly-accessible flag. For more information, see Security best practices for Amazon MQ.

The command returns output similar to:

{
    "BrokerArn": "arn:aws:mq:us-west-2:111122223333:broker:my-rabbitmq-broker:b-c8352341-ec91-4a78-ad9c-a57f23f235bb",
    "BrokerId": "b-c8352341-ec91-4a78-ad9c-a57f23f235bb"
}

Save the BrokerId value for the next step.

The broker takes approximately 15–20 minutes to reach the Running state. Run the following command every 2 minutes to check the status:

aws mq describe-broker --broker-id <BrokerId> --region us-west-2 --query 'BrokerState' 

Proceed to the next step after the broker state is RUNNING.

To get the broker endpoints, run:

aws mq describe-broker --broker-id <BrokerId> --region us-west-2 --query 'BrokerInstances'

Note the ConsoleURL and Endpoints from the output. The command returns output similar to:

[{
    "ConsoleURL": "https:// b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws",
    "Endpoints": ["amqps://b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws:5671"]
}]

Step 2: Add the RabbitMQ JMS Client dependency

Choose the dependency that matches your application’s current JMS version. If your imports reference javax.jms packages, use version 2.12.0. If your imports reference jakarta.jms packages (JMS 3.1 / Jakarta EE 9+), use version 3.4.0.

For JMS 1.1 and 2.0 (javax.jms):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>2.12.0</version>
</dependency>

For JMS 3.1 / Jakarta JMS 3.1 / Jakarta Messaging (jakarta.jms):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>3.4.0</version>
</dependency>

Step 3: Configure the connection factory

Store your broker credentials in AWS Secrets Manager before configuring the connection factory. This keeps credentials out of your source code and configuration files.

Create the secret:

aws secretsmanager create-secret \
--name dev-rabbitmq \
--description "Amazon MQ broker credentials" \
--secret-string '{"username":"admin","password":"[PASSWORD]"}' \ 
--region us-west-2

Add the AWS SDK for Secrets Manager to your pom.xml:

<dependency> 
<groupId>software.amazon.awssdk</groupId> 
<artifactId>secretsmanager</artifactId> 
<version>2.20.0</version> </dependency>

<dependency> 
<groupId>com.fasterxml.jackson.core</groupId> 
<artifactId>jackson-databind</artifactId> 
<version>2.15.0</version> 
</dependency>

Replace your existing broker URL with the Amazon MQ endpoint. In most cases, this is the only change required in your application configuration:

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import software.amazon.awssdk.regions.Region;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Map;
import javax.jms.*; 

// Retrieve credentials from AWS Secrets Manager
Map<String, String> creds;
try (SecretsManagerClient secretsClient = SecretsManagerClient.builder().region(Region.US_WEST_2).build()) {    
    GetSecretValueResponse response = secretsClient.getSecretValue(GetSecretValueRequest.builder().secretId("dev-rabbitmq").build());        
    ObjectMapper objectMapper = new ObjectMapper();    
    creds = objectMapper.readValue(response.secretString(),new TypeReference<Map<String, String>>() {});
    } 

// Create and configure the connection factory
RMQConnectionFactory connectionFactory = new RMQConnectionFactory();
connectionFactory.setHost("b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws");
connectionFactory.setPort(5671);connectionFactory.setVirtualHost("/");
connectionFactory.useSslProtocol();
connectionFactory.setUsername(creds.get("username"));
connectionFactory.setPassword(creds.get("password")); 

Replace the host value with your broker endpoint from Step 1.

The connection factory requires four parameters:

  • Host: Your broker endpoint from the describe-broker output
  • Port: 5671 for AMQP over TLS (Amazon MQ requires encryption in transit)
  • VirtualHost: “/” (the default RabbitMQ virtual host)
  • UseSslProtocol: true (required by Amazon MQ)

Step 4: Send messages

The following examples show how to send messages to a queue and a topic using the JMS 2.0 simplified API.

Point-to-point (queue) for one-to-one delivery:

try (JMSContext context = connectionFactory.createContext()) {
Queue queue = context.createQueue("orders");
context.createProducer().setProperty("OrderType", "Electronics").send(queue, "Order #12345");
System.out.println("Sent message to queue: orders");
}

Publish/subscribe (topic) for one-to-many broadcast:

try (JMSContext context = connectionFactory.createContext()) {
	Topic topic = context.createTopic("orders.electronics");
	context.createProducer().setProperty("MessageType", "Broadcast").send(topic, "New electronics order received!");
	System.out.println("Published message to topic: orders.electronics");
}

Message properties(OrderType, MessageType) are JMS headers that consumers can use for filtering. These properties become AMQP message headers when transmitted to the broker.

Step 5: Receive messages asynchronously

To receive messages asynchronously, attach a MessageListener to a consumer. The listener fires each time a message arrives.

Queue consumer:

Asynchronous consumers process messages in a background thread without blocking your main application logic. The MessageListener callback fires each time a message arrives, allowing your application to handle messages as they’re delivered rather than polling with receive().

try (JMSContext context = connectionFactory.createContext()) {
	Queue queue = context.createQueue("orders");
	JMSConsumer consumer = context.createConsumer(queue);
	consumer.setMessageListener(message -> {
		if (message instanceof TextMessage) {
			try {
				System.out.println("Received: " + ((TextMessage) message).getText());
			} catch (JMSException e) {
				e.printStackTrace();
			}
		}
	});

	System.out.println("Listening for messages on queue: orders");

	// Keep the consumer active for 30 seconds
	Thread.sleep(30000);
}

Topic subscriber:

try (JMSContext context = connectionFactory.createContext()) {
	Topic topic = context.createTopic("orders.electronics");
	JMSConsumer consumer = context.createConsumer(topic);
	consumer.setMessageListener(message -> {
		if (message instanceof TextMessage) {
			try {
				System.out.println("Subscriber received: " + ((TextMessage) message).getText());
			} catch (JMSException e) {
				e.printStackTrace();
			}
		}
	});

	System.out.println("Subscribed to topic: orders.electronics");

	// Keep the consumer active for 30 seconds
	Thread.sleep(30000);}

The Thread.sleep(30000) call keeps the consumer active for 30 seconds.

Use case: Migrating an ActiveMQ Workload to Amazon MQ for RabbitMQ

Migrate your Apache ActiveMQ applications to Amazon MQ by updating four configuration points. Your business logic, message producers, consumers, and listeners stay exactly as written. This walkthrough uses a real JMS 1.1 application with a centralized broker configuration class to show precisely which lines change and which remain identical.

Apache ActiveMQ powers messaging infrastructure for thousands of Java applications worldwide. If you run JMS applications on ActiveMQ, you can migrate to Amazon MQ for RabbitMQ with minimal code changes. The following steps demonstrate a complete migration using an application that includes a centralized broker configuration class, a message producer, and a message consumer.

Step 1: Update the Maven dependency

Replace the ActiveMQ client dependencies with the RabbitMQ JMS client in your pom.xml. The rabbitmq-jms artifact includes the RabbitMQ AMQP client and JMS API as transitive dependencies, so a single entry replaces both ActiveMQ artifacts.

Before (ActiveMQ):

<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-client</artifactId>
<version>5.18.6</version>
</dependency>

<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
<version>5.18.6</version>n>5.18.6</version>
</dependency>

After (Amazon MQ):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>2.12.0</version>
</dependency>

The rabbitmq-jms artifact pulls in the RabbitMQ AMQP client and the JMS API as transitive dependencies, so a single entry replaces both ActiveMQ artifacts.

Step 2: Update the broker configuration

If your application centralizes connection details in a shared configuration class, that class is the only file that needs to change. The queue name and everything else your application references remain the same.

Before (ActiveMQ):

// BrokerConfig.java - ActiveMQ version
public final class BrokerConfig {
	// OpenWire endpoint
	public static final String BROKER_URL = "tcp://localhost:61616";
	public static final String USERNAME = "[PASSWORD]";
	public static final String PASSWORD = "[PASSWORD]";
	public static final String QUEUE_NAME = "demo.queue";
	private BrokerConfig() {}}

After (Amazon MQ):

// BrokerConfig.java - Amazon MQ version

import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.ObjectMapper;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import java.util.Map;

public final class BrokerConfig {
	// AMQPS endpoint (TLS required by Amazon MQ)
	public static final String BROKER_URL = "amqps://b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws:5671";

	// Queue name carries over unchanged
	public static final String QUEUE_NAME = "demo.queue";

	// Secret name in AWS Secrets Manager
	private static final String SECRET_ID = "dev-rabbitmq";
	private static final Map<String, String> CREDENTIALS = loadCredentials();
	public static String getUsername() {return CREDENTIALS.get("username");}
	public static String getPassword() {return CREDENTIALS.get("password");}
	private static Map<String, String> loadCredentials() { 
		try (SecretsManagerClient client = SecretsManagerClient.builder().region(Region.US_WEST_2).build()) {
			GetSecretValueResponse response = client.getSecretValue(GetSecretValueRequest.builder().secretId(SECRET_ID).build());
			ObjectMapper mapper = new ObjectMapper();
			return mapper.readValue(response.secretString(), new TypeReference<Map<String, String>>() {});
		} catch (Exception e) {
			throw new RuntimeException("Failed to load broker credentials from Secrets Manager", e);
		}
	}
	private BrokerConfig() {}}

Two things changed in this file compared to the ActiveMQ version: the protocol prefix (tcp:// to amqps://) and the host and port (OpenWire on 61616 to AMQP over TLS on 5671). The queue name is identical. Credentials are no longer stored as static string constants. Instead, loadCredentials() retrieves them from AWS Secrets Manager at startup, and getUsername() and getPassword() expose them to the rest of the application. This follows AWS security best practices and streamlines credential rotation.

Step 3: Update the message producer

The producer requires two changes: the import statement and the factory instantiation. Every JMS API call after the factory (createConnection, createSession, createProducer, send) is identical to the ActiveMQ version.

Before (ActiveMQ):

import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;

public class MessageProducer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(BrokerConfig.USERNAME,BrokerConfig.PASSWORD,BrokerConfig.BROKER_URL);
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageProducer producer = session.createProducer(destination);
			producer.setDeliveryMode(DeliveryMode.PERSISTENT);
			for (int i = 1; i <= 5; i++) {
				TextMessage message = session.createTextMessage("Hello from ActiveMQ - message #" + i);
				producer.send(message);
				System.out.println("Sent: " + message.getText());
			}
			producer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close();
				} catch (JMSException ignored) {}}
			}
		}
	}

After (Amazon MQ):

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import javax.jms.*;

public class MessageProducer {

	public static void main(String[] args) {
		Connection connection = null;
		try {
			RMQConnectionFactory factory = new RMQConnectionFactory();
			factory.setUri(BrokerConfig.BROKER_URL);
			factory.setUsername(BrokerConfig.getUsername());
			factory.setPassword(BrokerConfig.getPassword());

			// Everything below this line is identical to the ActiveMQ version
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageProducer producer = session.createProducer(destination);
			producer.setDeliveryMode(DeliveryMode.PERSISTENT);
			for (int i = 1; i <= 5; i++) {
				TextMessage message = session.createTextMessage("Hello from Amazon MQ - message #" + i);
				producer.send(message);
				System.out.println("Sent: " + message.getText());
			}
			producer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try {
					connection.close();
				} catch (JMSException ignored) {}
			}
		}
	}
}

The import changes from org.apache.activemq.ActiveMQConnectionFactory to com.rabbitmq.jms.admin.RMQConnectionFactory. The factory construction switches from a constructor that accepts credentials and URL to a no-arg constructor with explicit setter calls. Credentials are now retrieved from AWS Secrets Manager through BrokerConfig.getUsername() and BrokerConfig.getPassword(). That is the complete change set for the producer.

Step 4: Update the message consumer

The consumer follows the same pattern as the producer. Swap the factory class and import, update the credential calls, and keep everything else.

Before (ActiveMQ):

import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
public class MessageConsumer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(BrokerConfig.USERNAME, BrokerConfig.PASSWORD, BrokerConfig.BROKER_URL);
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageConsumer consumer = session.createConsumer(destination);
			System.out.println("Waiting for messages on queue: " + BrokerConfig.QUEUE_NAME);
			Message message;
			while ((message = consumer.receive(10000)) != null) {
				if (message instanceof TextMessage) {
					TextMessage textMessage = (TextMessage) message;
					System.out.println("Received: " + textMessage.getText());
				}
			}
			consumer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close(); 
				} catch (JMSException ignored) {}
			}
		}
	}
}

After (Amazon MQ):

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import javax.jms.*;

public class MessageConsumer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			RMQConnectionFactory factory = new RMQConnectionFactory();
			factory.setUri(BrokerConfig.BROKER_URL);
			factory.setUsername(BrokerConfig.getUsername());
			factory.setPassword(BrokerConfig.getPassword());

			// Everything below this line is identical to the ActiveMQ version
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageConsumer consumer = session.createConsumer(destination);
			System.out.println("Waiting for messages on queue: " + BrokerConfig.QUEUE_NAME);
			Message message;
			while ((message = consumer.receive(10000)) != null) {
				if (message instanceof TextMessage) {
					TextMessage textMessage = (TextMessage) message;
					System.out.println("Received: " + textMessage.getText());
				}
			}
			consumer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close(); 
				} catch (JMSException ignored) {}
			}
		}
	}
}

The import changes from org.apache.activemq.ActiveMQConnectionFactory to com.rabbitmq.jms.admin.RMQConnectionFactory. The factory construction switches to a no-arg constructor with explicit setter calls, and BrokerConfig.USERNAME / BrokerConfig.PASSWORD are replaced with BrokerConfig.getUsername() / BrokerConfig.getPassword(). The session creation, queue lookup, consumer setup, and message processing loop are identical to the ActiveMQ version.

Configuration

The following table summarizes the changes required when migrating from Apache ActiveMQ.

 

ActiveMQ Amazon MQ for RabbitMQ
Maven dependency activemq-client 5.18.6 rabbitmq-jms 2.12.0
Connection factory class ActiveMQConnectionFactory RMQConnectionFactory
Import package org.apache.activemq com.rabbitmq.jms.admin
Broker URL format tcp://host:61616 amqps://broker-id.mq.region.on.aws:5671
Protocol OpenWire AMQP 0-9-1
Port 61616 (OpenWire) 5671 (AMQP over TLS)
TLS Optional Required
Credentials Plain text / JNDI AWS Secrets Manager (recommended)
Virtual host N/A / (default)
JMS version support JMS 1.1 JMS 1.1, 2.0, 3.1 (Jakarta)
Queue/Topic names demo.queue demo.queue (no change)
JMS API calls Standard JMS 1.1 Standard JMS 1.1 (no change)

Validating the migration

Run your application against Amazon MQ for RabbitMQ in a staging environment before directing production traffic to the new broker. Verify that messages flow correctly, consumers process as expected, and no data loss occurs during cutover.

The RabbitMQ management console provides real-time visibility into broker operations. Access it through the ConsoleURL from your broker details. The console shows queue depths, consumer counts, and message rates. Use it during testing to identify routing or throughput issues before production deployment

The console displays jms.durable.queues and jms.durable.topic exchanges. The JMS client creates these automatically when your application creates queues and topics, so no manual exchange configuration is required.

RabbitMQ management console showing jms.durable.queues and jms.durable.topic exchanges created automatically by the JMS client.

Monitoring with Amazon CloudWatch

Amazon MQ publishes broker metrics to Amazon CloudWatch with no additional configuration needed. This gives you persistent monitoring and alerting that works alongside the rest of your AWS observability setup, beyond what the RabbitMQ management console provides in real time.

After your JMS messages reach the Amazon MQ for RabbitMQ broker, they’re transported as AMQP messages, which means standard RabbitMQ operational best practices apply. Keep queue depth low to avoid memory pressure and consumer lag. Follow message durability and reliability guidelines to prevent message loss during broker restarts. For connection management, review broker setup and connection best practices to avoid connection churn.

Set Amazon CloudWatch alarms on MessageCount and ConnectionCount first. A rising queue depth with a stable or dropping consumer count is an early signal of a processing bottleneck. A sudden drop in connections can indicate a client configuration issue that’s more straightforward to catch before it affects production traffic.

Clean up

To avoid ongoing charges after testing, delete the Amazon MQ broker and Secrets Manager secret using the AWS CLI.

Delete the broker:

aws mq delete-broker --broker-id <your-broker-id> --region us-west-2

Delete the Secrets Manager secret:

aaws secretsmanager delete-secret \
--secret-id dev-rabbitmq \
--force-delete-without-recovery \
--region us-west-2

Broker deletion is permanent and can’t be undone. Amazon MQ removes all messages, configurations, and user credentials. Leaving the broker running incurs hourly charges based on the instance type, plus storage costs for message data retained on the broker.

Conclusion

In this post, we walked you through how to migrate your JMS applications. We also walked through a complete setup, from creating the broker to sending and receiving messages. Migrating the broker is the straightforward part. The more significant question is what you do next. After your JMS application is running on Amazon MQ for RabbitMQ, you have access to native AMQP clients, which means non-Java services can start consuming the same messages without a JMS layer. A Java-centric messaging system becomes a shared event backbone that service can participate in. The migration is a starting point, not just a lift-and-shift.

Next Steps

To get started with your migration, create your first Amazon MQ for RabbitMQ broker. For detailed technical guidance, see the Amazon MQ Developer Guide and explore the RabbitMQ JMS Client documentation.


About the authors


Vinodh Kannan is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.
Akhil Melakunta
Akhil Melakunta is a Senior Solutions Architect at Amazon Web Services (AWS) with over 12 years of industry experience. His expertise spans AI/ML, serverless, and messaging services, where he guides enterprise customers through large-scale cloud transformations on AWS.

Vignesh Selvam is the Principal Product Manager for Amazon MQ at AWS. He works with customers to solve their messaging needs and with the open-source communities for innovating with message brokers. Prior to joining AWS, he built products for security and analytics.

Build priority-based message processing with Amazon MQ and AWS App Runner

Post Syndicated from Aritra Nag original https://aws.amazon.com/blogs/architecture/build-priority-based-message-processing-with-amazon-mq-and-aws-app-runner/

Organizations need message processing systems that can prioritize critical business operations while handling routine tasks efficiently. When handling time-sensitive tasks like rush orders from key customers, critical system alerts, or multi-step business processes, you need to prioritize urgent messages while making sure other routine requests are processed reliably.

In this post, we show you how to build a priority-based message processing system using Amazon MQ for priority queuing, Amazon DynamoDB for data persistence, and AWS App Runner for serverless compute. We demonstrate how to implement application-level delays that high-priority messages can bypass, create real-time UIs with WebSocket connections, and configure dual-layer retry mechanisms for maximum reliability.

This solution addresses three critical challenges in modern data processing systems:

  • Implementing configurable delay processing at the application level
  • Supporting priority-based message routing that respects business requirements
  • Providing real-time feedback to users through WebSocket connections

The use of AWS managed services reduces operational complexity, so teams can focus on business logic rather than infrastructure management. Message handling with priority-based processing makes sure operations receive attention while routine tasks are processed in the background. Users will experience status updates that provide visibility into their requests, while retry mechanisms provide reliability during failures. The infrastructure as code (IaC) approach supports deployments across different environments, from development through production.

Solution overview

The solution consists of several AWS managed services to create a serverless, priority-based message processing system with real-time user feedback. The architecture implements intelligent routing based on three message priority levels, to make sure critical messages receive immediate processing:

  • High-priority path – Messages bypass delays and queue immediately with JMS priority 9
  • Standard-priority path – Messages undergo configured delays before queuing with JMS priority 4
  • Low-priority path – Messages process after all higher priority messages with JMS priority 0

The following diagram illustrates this architecture.

The solution uses the following AWS managed services to deliver a scalable, serverless architecture:

  • AWS App Runner is a fully managed container application service that automatically builds, deploys, and scales containerized applications. It provides automatic scaling based on traffic, built-in load balancing and HTTPS, seamless integration with container registries, and zero infrastructure management overhead.
  • Amazon MQ is a managed message broker service for Apache ActiveMQ that offers priority-based message queuing, automatic failover for high availability, message persistence and durability, and JMS protocol support for enterprise applications.
  • Amazon DynamoDB is a fully managed NoSQL database service providing single-digit millisecond performance at any scale, automatic scaling with on-demand pricing, built-in security and backup capabilities, and global tables for multi-Region deployments.

The system uses JMS priority levels with High=9, Medium=4, and Low=0 for automatic ordering, combined with conditional delay processing based on priority classification. Amazon MQ provides reliable message delivery and persistence with dead-letter queue (DLQ) configuration for failed message handling.

Asynchronous delay processing uses CompletableFuture implementation for non-blocking delays, thread pool management for concurrent processing, graceful error handling with retry mechanisms, and configurable delay periods per message type to optimize resource utilization. For real-time status updates, the solution provides WebSocket connections for bidirectional communication, Amazon DynamoDB Streams for change data capture (CDC), comprehensive status tracking throughout the processing lifecycle, and a React frontend integration for live updates, so users have complete visibility into their message processing status.

The standard priority messaging flow (shown in the following diagram) handles messages with configurable delays using JMS asynchronous processing capabilities. Messages wait for their specified delay period before entering the Amazon MQ queue, where they’re processed.

The high-priority messaging flow (shown in the following diagram) provides an express lane for critical messages. These messages skip the delay mechanism entirely and proceed directly to the queue, providing immediate processing for time-sensitive operations.

To make it even more straightforward to get started, we’ve prepared an example application that you can use to observe the Amazon MQ behavior with varying message volumes. You can find the source code repository, IaC implementation, and instructions to run the sample on GitHub.

In the following sections, we walk you through deploying the complete processing system.

Prerequisites

Make sure you have the following tools, permissions, and knowledge to successfully deploy the priority-based message processing system. You must have an active AWS account with the following configurations:

# JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
"apprunner:CreateService",
"apprunner:UpdateService",
"apprunner:DeleteService"
      ],
      "Resource": "arn:aws:apprunner:*:*:service/reactive-demo-*"
    },
    {
      "Effect": "Allow",
      "Action": [
"mq:SendMessage",
"mq:ReceiveMessage",
"mq:DeleteMessage"
      ],
      "Resource": "arn:aws:mq:*:*:broker/reactive-demo-broker/*"
    },
    {
      "Effect": "Allow",
      "Action": [
"dynamodb:PutItem",
"dynamodb:GetItem",
"dynamodb:UpdateItem",
"dynamodb:Query"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/reactive-items*"
    }
  ]
}

Install and configure the following development tools on your local machine:

To successfully implement this solution, you should have basic familiarity with the following:

  • Spring Boot applications
  • Message queue concepts
  • WebSocket protocols
  • React development

Configure the infrastructure stack

This step involves creating the core AWS services using the AWS Cloud Development Kit (AWS CDK). This modular approach enables independent stack management and environment-specific configurations.

  1. Create a new AWS CDK project:
# Bash
mkdir priority-processing && cd priority-processing
cdk init app --language python
pip install aws-cdk-lib constructs
  1. Create the infrastructure stack:
# Python
from aws_cdk import (
    Stack,
    aws_dynamodb as dynamodb,
    aws_amazonmq as mq,
    aws_kms as kms,
    Duration,
    RemovalPolicy,
    CfnOutput
)
from constructs import Construct

class MessageProcessingStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)

# Create KMS key for encryption
self.kms_key = kms.Key(
    self, "ProcessingKey",
    description="Key for message processing encryption",
    enable_key_rotation=True
)

# DynamoDB table with comprehensive configuration
self.items_table = dynamodb.Table(
    self, "ItemsTable",
    table_name="reactive-items",
    partition_key=dynamodb.Attribute(
name="id",
type=dynamodb.AttributeType.STRING
    ),
    stream=dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
    billing_mode=dynamodb.BillingMode.ON_DEMAND,
    encryption=dynamodb.TableEncryption.CUSTOMER_MANAGED,
    encryption_key=self.kms_key,
    point_in_time_recovery=True,
    removal_policy=RemovalPolicy.DESTROY
)

# Add Global Secondary Index for status queries
self.items_table.add_global_secondary_index(
    index_name="StatusIndex",
    partition_key=dynamodb.Attribute(
name="status",
type=dynamodb.AttributeType.STRING
    ),
    sort_key=dynamodb.Attribute(
name="createdAt",
type=dynamodb.AttributeType.STRING
    )
)

# Amazon MQ broker configuration
self.mq_broker = mq.CfnBroker(
    self, "MessageBroker",
    broker_name="reactive-demo-broker",
    engine_type="ACTIVEMQ",
    engine_version="5.18",
    host_instance_type="mq.t3.micro",
    deployment_mode="SINGLE_INSTANCE",
    publicly_accessible=False,
    logs=mq.CfnBroker.LogListProperty(
audit=True,
general=True
    ),
    encryption_options=mq.CfnBroker.EncryptionOptionsProperty(
use_aws_owned_key=False,
kms_key_id=self.kms_key.key_id
    ),
    users=[mq.CfnBroker.UserProperty(
username="admin",
password="SecurePassword123!",
console_access=True
    )]
)

# Output values for application configuration
CfnOutput(self, "TableName", 
    value=self.items_table.table_name,
    description="DynamoDB table name")
CfnOutput(self, "MQBrokerEndpoint",
    value=self.mq_broker.attr_amqp_endpoints[0],
    description="Amazon MQ broker endpoint")
  1. Run the following commands to deploy the stack:
# Bash
cdk bootstrap
cdk deploy MessageProcessingStack

You can verify the infrastructure on the AWS Management Console.

Configure the message processing application

In this step, we create the Spring Boot application with priority-based message processing capabilities. First, we configure the application.properties file to incorporate environment variables, including AWS credentials, AWS Regions, and other configuration parameters such as log levels into the application and business logic implementation. Next, we implement the message service using a JMS template with comprehensive error handling, followed by enhancing the JMS configuration with connection pooling for improved performance.

The following code illustrates an example message service implementation:

// Example message service implementation
@Service
public class MessageService {
    @Autowired
    private JmsTemplate jmsTemplate;
    
    public void sendPriorityMessage(Message message) {
jmsTemplate.send(session -> {
    Message jmsMessage = session.createTextMessage(message.getContent());
    jmsMessage.setJMSPriority(message.getPriority());
    return jmsMessage;
});
    }
}

For proper timestamp update implementation, we integrate the DynamoDB SDK service with caching capabilities. Finally, after implementing the REST controller for the API with asynchronous processing support, we can deploy the message processing application. This implementation includes Java code application-level delay processing for demonstration purposes. Although this approach effectively showcases the priority-based message routing capabilities and real-time WebSocket updates in our demo environment, AWS recommends using Amazon MQ delay processing features for production workloads. For production implementations, use Amazon MQ delay and scheduling capabilities instead of application-level delays through features like Amazon MQ delay queues, ActiveMQ scheduling features, and appropriate message Time-to-Live (TTL) configurations.

The following code is an example snippet showcasing the Amazon MQ feature:

// Create connection factory with Amazon MQ endpoint
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(brokerUrl);
factory.setUserName("admin");
factory.setPassword("your-password");
try (Connection connection = factory.createConnection();
     Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE)) {
    
    // Create destination and producer
    Destination destination = session.createQueue(queueName);
    MessageProducer producer = session.createProducer(destination);
    
    // Create message
    TextMessage message = session.createTextMessage(messageContent);
    
    // Set native delay using ActiveMQ scheduled delivery
    message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY, delayMillis);
    
    // Optionally set priority for delayed message
    message.setJMSPriority(4);
    
    // Send the message - it will be delivered after the specified delay
    producer.send(message);
}

Build and deploy the Spring Boot application to App Runner

In this step, we push the application to Amazon Elastic Container Registry (Amazon ECR) to run it in App Runner:

  1. Build and push the Docker image to Amazon ECR:
# Bash

# Build the Docker image
docker build -t reactive-demo .

# Create ECR repository
aws ecr create-repository --repository-name reactive-demo --region us-east-1

# Get login token and login to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI

# Tag and push image
ECR_URI=$(aws ecr describe-repositories --repository-names reactive-demo --query 'repositories[0].repositoryUri' --output text)
docker tag reactive-demo:latest $ECR_URI:latest
docker push $ECR_URI:latest
  1. Create the App Runner service with environment variables for the DynamoDB table and Amazon MQ broker endpoint:
# Python

from aws_cdk import (
    aws_apprunner as apprunner,
    aws_iam as iam
)

class AppRunnerStack(Stack):
    def __init__(self, scope: Construct, id: str, 
 table_name: str, mq_endpoint: str, **kwargs):
super().__init__(scope, id, **kwargs)

# Create IAM role for App Runner
app_runner_role = iam.Role(
    self, "AppRunnerRole",
    assumed_by=iam.ServicePrincipal("tasks.apprunner.amazonaws.com"),
    managed_policies=[
iam.ManagedPolicy.from_aws_managed_policy_name(
    "AmazonDynamoDBFullAccess"
),
iam.ManagedPolicy.from_aws_managed_policy_name(
    "AmazonMQFullAccess"
)
    ]
)

# Create App Runner service
self.service = apprunner.CfnService(
    self, "ReactiveProcessingService",
    service_name="reactive-processing-service",
    source_configuration=apprunner.CfnService.SourceConfigurationProperty(
authentication_configuration=apprunner.CfnService.AuthenticationConfigurationProperty(
    access_role_arn=app_runner_role.role_arn
),
image_repository=apprunner.CfnService.ImageRepositoryProperty(
    image_identifier=f"{ECR_URI}:latest",
    image_configuration=apprunner.CfnService.ImageConfigurationProperty(
port="8080",
runtime_environment_variables=[
    {"name": "DYNAMODB_TABLE_NAME", "value": table_name},
    {"name": "MQ_BROKER_URL", "value": mq_endpoint}
]
    ),
    image_repository_type="ECR"
)
    ),
    health_check_configuration=apprunner.CfnService.HealthCheckConfigurationProperty(
path="/actuator/health",
protocol="HTTP",
interval=10,
timeout=5,
healthy_threshold=1,
unhealthy_threshold=5
    ),
    instance_configuration=apprunner.CfnService.InstanceConfigurationProperty(
cpu="0.5 vCPU",
memory="1 GB"
    )
)

Set up real-time updates

For this step, we implement WebSocket support for real-time status updates using AWS Lambda to process DynamoDB streams and send updates to connected clients using Amazon API Gateway WebSocket connections. You can find the code snippet for this in this link

Deploy the React application to Amazon S3 and Amazon CloudFront

In this step, we create a frontend application to enable the WebSocket connection for seeing the messaging getting updated in the DynamoDB and API Gateway WebSocket connections.

Similar to the above section, here is the AWS cdk code for building the frontend for proceeding towards the validation of the solution

Validate the solution

This section provides comprehensive testing procedures to validate the priority-based message processing system.

Automated testing script

After you have completed the preceding steps, you can initiate a comprehensive testing script to validate priority processing and delay behavior:

# Bash
#!/bin/bash
curl -X POST "$API_URL/api/items" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "High Priority Task",
    "priority": "High",
    "delay": 10
  }'

Validation through the web interface

The following screenshot of the UI illustrates how the queueing mechanism can work with the real-time updates using WebSockets.

The web interface provides validation of the priority-based message processing system. Access the Amazon CloudFront URL to view the following information:

  • Real-time message processing with live status updates
  • Queue statistics showing message distribution by priority
  • Processing timeline demonstrating priority bypass behavior
  • WebSocket connection status indicating real-time connectivity

Amazon CloudWatch dashboards and alarms

AWS recommends creating Amazon CloudWatch dashboards to track your priority-based message processing system’s performance across multiple dimensions. Monitor message processing by priority levels to make sure high-priority messages are processed first and identify any bottlenecks in your priority routing logic. The following screenshot shows an example dashboard.

You can track queue depth and processing times to understand system load and latency patterns, helping you optimize resource allocation and identify when scaling is needed. Observe DynamoDB performance metrics including read/write capacity consumption, throttling events, and latency to make sure your database layer maintains optimal performance under varying loads.

Additionally, implement application-specific custom metrics such as message processing success rates, retry counts, and business-specific KPIs to gain deeper insights into your application’s behavior and make data-driven decisions for continuous improvement.

Security considerations

AWS recommends implementing comprehensive security measures to safeguard your message processing system. Start by implementing least privilege IAM policies that grant only the minimum permissions required for each component to function, making sure services like App Runner can only access the specific DynamoDB tables and Amazon MQ queues they need. Configure your network architecture using a virtual private cloud (VPC) with private subnets for Amazon MQ, isolating your message broker from direct internet access while maintaining connectivity through NAT gateways for necessary outbound connections.

Enable encryption at rest using AWS Key Management Service (AWS KMS) for DynamoDB tables and Amazon MQ data and enforce encryption in transit by configuring SSL/TLS connections for all service communications, particularly for ActiveMQ broker connections. Finally, configure security groups with minimal access rules that explicitly define allowed traffic between components, restricting inbound connections to only the ports and protocols required for your application to function, such as port 61617 for ActiveMQ SSL connections from App Runner instances.

Cost considerations

The following table contains cost estimates based on the US East (N. Virginia) Region. Actual costs might vary based on your Region, usage patterns, and pricing changes.

Service Small (1,000 msg/day) Medium (10,000 msg/day) Large (100,000 msg/day)
Amazon DynamoDB $5–10 $25–50 $200–400
Amazon MQ $15 (t3.micro) $30 (m5.large) $120 (m5.xlarge)
AWS App Runner $20–40 $50–150 $400–800
Amazon API Gateway WebSocket $3–5 $10–25 $50–100
Amazon CloudWatch Logs $5–10 $10–20 $30–50
Data Transfer $5 $10-20 $50-100
Total Estimated Cost $53–95 $135–295 $850–1,570

Troubleshooting

The following are common issues and their solutions when implementing the priority-based message processing system:

  • Messages not processing in priority order:
    • Verify JMS priority is configured correctly: message.setJMSPriority(priority)
    • Check ActiveMQ broker configuration for priority queue support
    • Confirm CLIENT_ACKNOWLEDGE mode is properly configured
    • Review queue consumer concurrency settings
  • WebSocket updates not working:
    • Verify DynamoDB Streams is enabled on the table
    • Check the Lambda function is triggered by stream events
    • Validate API Gateway WebSocket configuration and IAM permissions
    • Test the WebSocket connection using browser developer tools
  • Application scaling issues:
    • Monitor App Runner metrics in CloudWatch
    • Adjust auto scaling configuration based on traffic patterns
    • Consider Amazon MQ broker capacity and upgrade if needed
    • Review DynamoDB capacity settings and enable auto scaling

Clean up

To avoid incurring ongoing AWS charges, delete the resources you created in this walkthrough:

  1. Delete the CDK stacks:
cdk destroy MessageProcessingStack
cdk destroy FrontendStack
  1. Remove the App Runner service:
aws apprunner delete-service --service-arn <your-service-arn>
  1. Delete the ECR repositories and container images.
  2. Remove CloudWatch log groups if not set to auto-delete.
  3. Delete S3 buckets used for frontend hosting.

Next steps

To extend this solution and add additional capabilities, consider the following enhancements:

Conclusion

This solution demonstrates how to build a production-ready priority-based message processing system using AWS managed services. By combining Amazon MQ priority queuing with DynamoDB real-time streams and App Runner serverless compute, you create a resilient architecture that intelligently handles messages based on business priorities.The implementation of application-level delays with priority bypass makes sure critical messages receive immediate attention, and the dual-layer retry mechanism provides maximum reliability. Real-time WebSocket updates keep users informed of processing status, creating a responsive and transparent system.To learn more about the services and patterns used in this solution, explore the following resources:


About the authors

Implementing message prioritization with quorum queues on Amazon MQ for RabbitMQ

Post Syndicated from Akhil Melakunta original https://aws.amazon.com/blogs/compute/implementing-message-prioritization-with-quorum-queues-on-amazon-mq-for-rabbitmq/

Quorum queues are now available on Amazon MQ for RabbitMQ from version 3.13. Quorum queues are a replicated First-In, First-Out (FIFO) queue type that uses the Raft consensus algorithm to maintain data consistency. Quorum queues on RabbitMQ version 3.13 lack one key feature compared to classic queues: message prioritization. However, RabbitMQ version 4.0 introduced support for message priority, which behaves differently than classic queue message priorities. Migrating applications from classic queues with message priority to quorum queues on Amazon MQ for RabbitMQ presents challenges for customers. This post describes the different approaches to implementing message prioritization in quorum queues in Amazon MQ for RabbitMQ.

Amazon MQ is a managed message broker service for Apache ActiveMQ and RabbitMQ that simplifies setting up and operating message brokers on AWS.

Why message prioritization matters

Modern messaging systems require handling messages differently, depending on the business priority. Some messages are more time-sensitive or critical than others and prioritizing them can enhance the efficiency and responsiveness of applications. Message prioritization allows certain messages to be processed before others, aligning with business priorities and helping to ensure that high-value or time-critical messages receive the attention they need.

Message prioritization addresses critical business challenges across multiple industries. In insurance companies, it can expedite urgent claim processing by prioritizing high-priority messages over routine policy updates, reducing settlement times. Automotive manufacturers can make sure that critical production line alerts and safety notifications take precedence over standard telemetry data, preventing costly downtime. Energy utilities can prioritize real-time grid stability alerts and outage notifications, enabling faster responses to potential blackouts. By implementing message priority, industries can direct immediate attention to time-sensitive operations while efficiently managing routine processes within existing infrastructure. By using this approach to transform their communication strategies, organizations can respond more quickly and effectively to critical events.

Classic queues compared to quorum queues message prioritization

In this section, explore the fundamental differences between classic queues and quorum queues when it comes to message prioritization capabilities. Examine how each queue type handles message priority, the built-in features available, and key considerations.

Message prioritization with classic queues

In classic queues, RabbitMQ supports message priorities ranging from 1 to 255, with 1 being the lowest priority and 255 being the highest. However, it’s generally recommended to use a smaller range (for example, 1–5) for better performance, because RabbitMQ needs to maintain an internal sub-queue for each priority from 1 up to the maximum value configured for a given queue. A wider priority range adds more CPU and memory cost, which can impact broker performance.

Priority queue behavior in classic queues:

  • Classic queues require x-max-priority argument to define the maximum number of priorities for a given queue
  • A procedure sends a message with a priority property value
  • Consumers don’t need special configuration to handle priorities
  • Messages with higher priority are delivered before messages with lower priority
  • Within the same priority level, messages are delivered in FIFO order
  • Messages without a priority property are treated as if their priority is lowest
  • Messages with a priority that is higher than the queue’s maximum are treated as if they were published with the maximum priority

Example Python code for classic queue implementation with message priority:

#!/usr/bin/env python
import pika
import ssl
# Set up SSL context for secure connection
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
# Define credentials
credentials = pika.PlainCredentials('username', 'password') # Replace with actual credentials
# Set up connection parameters for Amazon MQ RabbitMQ broker
connection_parameters = pika.ConnectionParameters(
    host='b-example.mq.us-west-2.on.aws', # Replace with actual broker endpoint
    port=5671,
    credentials=credentials,
    ssl_options=pika.SSLOptions(context)
)
# Establish connection and create a channel
connection = pika.BlockingConnection(connection_parameters)
channel = connection.channel()
# Declare a direct exchange
# - direct exchanges route messages based on routing key
channel.exchange_declare(
    exchange='priority_exchange',
    exchange_type='direct',
)
# Declare a priority queue
# - x-max-priority=5 sets maximum priority level (0-5)
# - x-queue-type=classic specifies classic queue implementation
channel.queue_declare(
    queue='classic_priority_queue',
    arguments={
        'x-max-priority': 5,
        'x-queue-type': "classic"
    }
)
# Bind queue to exchange with routing key
# - This connects the queue to the exchange
# - Messages sent to the exchange with matching routing key will be routed to this queue
channel.queue_bind(
    queue='classic_priority_queue',
    exchange='priority_exchange',
    routing_key='priority_queue'
)
# Publish messages with different priorities
# Low priority message (priority=1)
channel.basic_publish(
    exchange='priority_exchange',
    routing_key='priority_queue',
    body='Low priority message',
    properties=pika.BasicProperties(priority=1)
)
print(" [x] Sent 'Low priority message'")
# Medium priority message (priority=2)
channel.basic_publish(
    exchange='priority_exchange',
    routing_key='priority_queue',
    body='Medium priority message',
    properties=pika.BasicProperties(priority=2)
)
print(" [x] Sent 'Medium priority message'")
# High priority message (priority=5)
channel.basic_publish(
    exchange='priority_exchange',
    routing_key='priority_queue',
    body='High priority message',
    properties=pika.BasicProperties(priority=5)
)
print(" [x] Sent 'High priority message'")
# Close the connection
connection.close()

The preceding code demonstrates message prioritization in RabbitMQ using a classic queue with built-in priority handling. The implementation connects to a RabbitMQ broker using the Python Pika library and declares a direct exchange, a classic queue with a maximum priority level of 5. Messages are then published to this single queue with explicitly assigned priority values (1 for low, 2 for medium, and 5 for high priority). When consumers fetch messages from this queue, RabbitMQ will deliver higher priority messages first.

Message prioritization with quorum queues

Unlike classic queues, quorum queues in Rabbit MQ 3.13 don’t support message prioritization natively. However, there are effective patterns that you can implement to achieve message priority with Quorum queues.

Using separate queues for different priorities

A straightforward method is to create multiple quorum queues, each dedicated to different priority levels. For example, you might have a high-priority queue and a low-priority queue. Using RabbitMQ exchange and binding key route messages to the appropriate queues based on their priority, allowing the system to process high-priority messages more promptly, as shown in the following figure.

Example to implement priority handling using separate quorum queues:

#!/usr/bin/env python
import pika
import ssl
# Set up SSL context for secure connection
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
# Define credentials
credentials = pika.PlainCredentials('username', 'password') #Replace with actual credentials
# Set up connection parameters for Amazon MQ RabbitMQ broker
connection_parameters = pika.ConnectionParameters(
    host='b-example.mq.us-west-2.on.aws',
    port=5671,
    credentials=credentials,
    ssl_options=pika.SSLOptions(context)
)
# Establish connection and create a channel
connection = pika.BlockingConnection(connection_parameters)
channel = connection.channel()
# Declare a direct exchange
# - Direct exchanges route messages based on routing key
channel.exchange_declare(
    exchange='priority_exchange_qq',
    exchange_type='direct'
)
# Create separate quorum queues for different priority levels
# Low priority queue
channel.queue_declare(
    queue='low_priority_queue',
    durable=True,
    arguments={
        'x-queue-type': "quorum" 
    }
)
# Bind the low priority queue to the exchange with a specific routing key
# - This creates a rule that messages sent to 'priority_exchange' with routing_key='low_priority_1'
# - will be routed to the 'low_priority_queue'
channel.queue_bind(
    queue='low_priority_queue',
    exchange='priority_exchange_qq',
    routing_key='low_priority_1'
)
# Medium priority queue
channel.queue_declare(
    queue='medium_priority_queue',
    durable=True,
    arguments={
        'x-queue-type': "quorum" 
    }
)
# Bind the medium priority queue to the exchange with a specific routing key
# - Messages with routing_key='medium_priority_2' will be directed to the 'medium_priority_queue'
channel.queue_bind(
    queue='medium_priority_queue',
    exchange='priority_exchange_qq',
    routing_key='medium_priority_2'
)
# High priority queue
channel.queue_declare(
    queue='high_priority_queue',
    durable=True,
    arguments={
        'x-queue-type': "quorum" 
    }
)
# Bind the high priority queue to the exchange with a specific routing key
# - Messages with routing_key='high_priority_2' will be directed to the 'high_priority_queue'
channel.queue_bind(
    queue='high_priority_queue',
    exchange='priority_exchange_qq',
    routing_key='high_priority_5'
)
# Publish messages to different priority queues
print(" [x] Publishing messages to different priority queues")
# Low priority message
channel.basic_publish(
    exchange='priority_exchange_qq',  
    routing_key='low_priority_1',
    body='Low priority message'
)
print(" [x] Sent 'Low priority message'")
# Medium priority message
channel.basic_publish(
    exchange='priority_exchange_qq', 
    routing_key='medium_priority_2',
    body='Medium priority message'
)
print(" [x] Sent 'Medium priority message'")
# High priority message
channel.basic_publish(
    exchange='priority_exchange_qq', 
    routing_key='high_priority_5',
    body='High priority message'
)
print(" [x] Sent 'High priority message'")
# Close the connection
connection.close()
print(" [x] Connection closed")

The preceding code demonstrates a message prioritization approach in RabbitMQ using separate quorum queues for different priority levels (low, medium, and high). The implementation uses the Python Pika library to connect to a RabbitMQ server, a direct exchange and three separate quorum queues for different priority levels, and publish messages to different routing keys with different priority.

Custom priority logic on consumers

Implement custom logic within your application to handle messages based on their priority. For example, you can use headers or metadata to determine the priority of a message and then use this information to route messages to different queues or handle them in a specific order.

Higher priority queues should use more consumers or consumers with higher resources allocated to process messages more quickly than lower priority queues. Use the basic.qos (prefetch) method in manual acknowledgement mode on your consumers to limit the number of messages that can be out for delivery at any time and allow messages to be prioritized. basic.qos is a value a consumer sets when connecting to a queue. It indicates how many messages the consumer can handle at one time. This method is shown in the following figure.

Note: This solution implements message priority on a best-effort basis. There is a possibility that low and medium priority messages may be processed before high priority messages.

Conclusion

Message prioritization in RabbitMQ brokers on Amazon MQ has different considerations for classic and quorum queues. Using quorum queues requires a thoughtful approach because of the lack of native support for message proritization in RabbitMQ. By employing separate queues and custom logic, you can achieve effective prioritization while maintaining the high availability and consistency that quorum queues offer. Embrace these strategies to optimize your messaging infrastructure, enhance application responsiveness, and make sure that critical messages are processed in a timely manner.

We recommend that you adopt quorum queues as the preferred replicated queue type on RabbitMQ 3.13 brokers. For more details, see Amazon MQ documentation. For more information, see quorum queues.

To learn more, see Amazon MQ for Rabbit MQ.

Improve RabbitMQ performance on Amazon MQ with AWS Graviton3-based M7g instances

Post Syndicated from Vignesh Selvam original https://aws.amazon.com/blogs/big-data/improve-rabbitmq-performance-on-amazon-mq-with-aws-graviton3-based-m7g-instances/

Amazon MQ is a fully managed service for open-source message brokers such as RabbitMQ and Apache ActiveMQ. Today, we are announcing the availability of AWS Graviton3-based Rabbit MQ brokers on Amazon MQ, which runs on Amazon EC2 M7g instances. AWS Graviton processors are custom designed server processors developed by AWS to provide the best price performance for cloud workloads running on Amazon EC2. It uses the Arm (arm64) instruction set. For example, when running an Amazon MQ for RabbitMQ cluster broker using M7g.4xlarge instances, you can achieve up to 50% higher workload capacity and up to 85% higher throughput compared to M5.4xlarge instances. Additionally, M7g brokers on Amazon MQ offer optimized disk sizes for clusters, providing reduction in storage cost savings over M5 brokers depending on the instance size chosen. To learn more, refer to Amazon EC2 M7g instances.

Amazon MQ helps you reduce the operational overhead of using open source message brokers like RabbitMQ while providing security, high availability, and durability. Many organizations use Amazon MQ to decouple applications, asynchronously process messages, and build event-driven architectures. We tested and validated M7g instances for RabbitMQ version 3.13, so you can run your critical messaging workloads on Amazon MQ brokers with improved performance characteristics, while also saving on costs. Amazon MQ supports M7g instances in a wide variety of sizes, ranging from medium to 16xlarge sizes, to suit your different messaging workloads. M7g instances support Amazon MQ for RabbitMQ features, making it straightforward for you to run your existing RabbitMQ workloads with minimal changes. You can get started by provisioning new brokers or upgrading your existing RabbitMQ brokers using Amazon EC2 M5 instances to Graviton3-based M7g instances as the broker type using the AWS Management Console, APIs using the AWS SDK, and the AWS Command Line Interface (AWS CLI).

The following table lists the specific characteristics of M7g instances on Amazon MQ.

M7g specs for Amazon MQ
Instance Name (MQ.m7g.*) vCPUs Memory (GiB) Network Bandwidth
medium 1 4 Up to 12.5 Gb
large 2 8 Up to 12.5 Gb
xlarge 4 16 Up to 12.5 Gb
2xlarge 8 32 Up to 15 Gb
4xlarge 16 64 Up to 15 Gb
8xlarge 32 128 15 Gb
12xlarge 48 192 22.5 Gb
16xlarge 64 256 30 Gb

M7g instances vs. M5 instances on Amazon MQ

Customers can see both performance improvements and cost savings for their RabbitMQ workloads when moving from M5 instances to M7g instances. In terms of performance, you can size your RabbitMQ brokers for workloads by measuring the workload capacity and throughput. Amazon MQ has improved the performance of RabbitMQ on both workload capacity and throughput for M7g instances. In terms of cost, you pay for the instance per hour, disk usage per Gb-month, and data transfer. Amazon MQ has optimized disk sizes to offer cost savings for customers on disk usage. Let’s first examine the performance improvements.

Workload capacity improvements

Workload capacity represents the total number of connections, channels, and queues that you can use without running into memory alarm. The actual usage of these resources is limited by the high memory watermark value. Every resource (for example, a queue) on creation uses up a small amount of memory, but when these resources are used, the memory used increases depending on the number and size of messages processed up until a memory threshold. The RabbitMQ broker goes into memory alarm when the memory used on a node reaches this pre-defined threshold known as high memory watermark. When a broker raises a memory alarm, it will block all connections that are publishing messages. After the memory alarm has cleared (for example, due to delivering some messages to clients that consume and acknowledge the deliveries), normal service resumes. The open source community guidance for RabbitMQ 3.13 is to configure the memory threshold at 40% of the available memory per node. M5 brokers have the memory threshold set at 40% on Amazon MQ.

We evaluated this recommendation across M7g instances and determined that the memory threshold can be increased for instances on Amazon MQ to more than 40% due to the operational improvements by the service, as illustrated in the following figure. This increase in available memory translates to a higher use of resources like queues, channels, and connections within the resource limits of the broker. The change in available memory results in up to 50% improvement in workload capacity for customers when compared to M5 brokers today.

Throughput improvements

The throughput of a broker varies widely with the queue type and usage pattern of customers. Amazon MQ evaluated the throughput capacity of a RabbitMQ three-node cluster broker by measuring the publish throughput in messages per second for 10 quorum queues with a message size of 1 KB and a ratio of 1:20 for connection to channels. We arrived at this benchmark test after evaluating multiple scenarios with the goal of providing you a simple way to estimate the average throughput you can expect from a RabbitMQ broker when following best practices. You can see up to 85% higher throughput compared to equivalent M5 brokers on Amazon MQ, as illustrated in the following figure.

The performance of a RabbitMQ broker depends on the version, queue type, and usage pattern in addition to the infrastructure used. You might see different performance improvements based on your specific usage patterns and resources used. We recommend using the Amazon MQ sizing guidance to size your broker and benchmarking the performance for your specific workload using M7g instances.

Cost savings on cluster disk usage

Customers using M7g brokers in cluster deployment mode are provisioned with a disk volume per node that varies in size depending on the instance size. For M5 brokers, the RabbitMQ brokers were provisioned with a fixed disk volume of 200 GB per node. The open source guidance around disk sizes is to use a size higher than twice the memory threshold. We tested various disk sizes and identified optimal disk sizes that would provide a better operational posture. With this change, customers using M7g cluster brokers on Amazon MQ will get cost savings due to the smaller disk size provisioned per node as compared to equivalent M5 brokers, as shown in the following table. Single-instance M7g brokers will continue to be provisioned with 200 GB of disk size.

Instance size Disk Volume M5 cluster(GB) Disk Volume M7g Cluster(GB) Cost savings for customersM5 vs. M7g (%)
medium 15
large 600 45 92.50%
xlarge 600 75 87.50%
2xlarge 600 135 77.50%
4xlarge 600 270 55.00%
8xlarge 525
12xlarge 780
16xlarge 1035

Pricing and Regional availability

M7g instances are available in AWS Regions where Amazon MQ is available at the time of writing except Africa (Cape Town), Canada West (Calgary), and Europe (Milan) Regions. Refer to Amazon MQ Pricing to learn about the availability of specific instance sizes by Region and the pricing for M7g instances.

Summary

In this post, we discussed the performance gains and cost savings achieved while using Graviton-based M7g instances. These instances can provide significant improvement in throughput and workload capacity compared to similar sized M5 instances for Amazon MQ workloads. To get started, create a new broker with M7g brokers using the console, and refer to the Amazon MQ Developer Guide for more information.


About the authors

Vignesh Selvam is the Principal Product Manager for Amazon MQ at AWS. He works with customers to solve their messaging needs and with the open-source communities for innovating with message brokers. Prior to joining AWS, he built products for security and analytics.

Samuel Massé is a Software Development Engineer at AWS. He has been leading the engineering effort to support M7g on the RabbitMQ team. In his free time he enjoys coding unfinished side projects.

Vinodh Kannan Sadayamuthu is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.

Networking of Amazon MQ for RabbitMQ event source mapping for AWS Lambda

Post Syndicated from Rafal Pawlaszek original https://aws.amazon.com/blogs/compute/networking-of-amazon-mq-for-rabbitmq-event-source-mapping-for-aws-lambda/

Event-driven architectures with message brokers need careful attention to security best practices. Amazon MQ for RabbitMQ combined with AWS Lambda enables serverless event processing. However, implementing defense in depth and least privilege principles necessitates a clear understanding of networking requirements. This is particularly important when working with different subnet types and their impact on service connectivity.

This post explores the networking aspects of Lambda event source mapping for Amazon MQ for RabbitMQ. Learn how deployment options influence your networking setup and security posture to make informed architectural decisions. These networking concepts are essential for building secure, scalable solutions, regardless of your experience level with message brokers.

For clarity in this post, when we refer to “RabbitMQ”, we mean Amazon MQ for RabbitMQ.

Prerequisites

The following prerequisites are necessary to complete this post:

  • An Amazon Web Services (AWS) account
  • Basic understanding of AWS networking concepts
  • Familiarity with Amazon MQ for RabbitMQ
  • Basic knowledge of Lambda

Furthermore, to enable setup of the discussed architectures, this post is accompanied by a GitHub repository that uses AWS Cloud Development Kit (AWS CDK).

Repository prerequisites

The following prerequisites are necessary for the repository:

Repository setup

Clone the https://github.com/aws-samples/sample-amazonmq-rabbitmq-lambda-esm repository. This repository contains all the necessary code and instructions to create relevant architectures using AWS CDK.

Install dependencies and build

Install the necessary NPM dependencies by running the following commands:

npm install
npm run build

Amazon MQ for RabbitMQ networking deployment options

Public accessibility is the primary networking differentiator when deploying a RabbitMQ broker in AWS. Although the broker operates in the Amazon MQ service account, the networking configuration varies based on this choice.

Public broker

When you deploy a publicly accessible broker, Amazon MQ provisions all networking components in the service account. The service provides a DNS name that resolves to an IP address of the Network Load Balancer (NLB) in that account. This configuration doesn’t support security groups. All security measures must be implemented through the RabbitMQ broker’s authentication and authorization mechanisms. The following diagram shows this communication flow.

Figure-1 DNS resolution of a public Amazon MQ for RabbitMQ broker.

Private broker

A private broker routes networking through a Amazon Virtual Private Cloud (Amazon VPC) in your account. Amazon MQ uses AWS PrivateLink to provision VPC Endpoints, which serve as entry points for broker communication.

The following diagram shows how client applications communicate with RabbitMQ:

  1. The client application connects to Amazon Route 53 Resolver
  2. Route 53 Resolver resolves the DNS name to the VPC Endpoint’s IP address
  3. The client communicates with the broker through PrivateLink
  4. Security groups protect the VPC Endpoint’s Elastic Network Interfaces (ENIs)

Figure-2 DNS resolution of a private Amazon MQ for RabbitMQ broker.

A private broker deployment offers two networking options:

  • Custom VPC configuration – Specify:
    • Subnets for VPC Endpoint creation
    • At least one security group to protect the VPC Endpoints
  • Default VPC configuration – Leave VPC options blank to use:
    • Default VPC
    • Default security group

Amazon MQ for RabbitMQ Lambda event source mapping building blocks

RabbitMQ solutions offer two approaches for message processing:

  • Create a custom client to read messages from broker queues
  • Use Lambda functions with event source mapping (ESM) for automated message retrieval

The ESM is a Lambda service resource that reads the messages from the broker and invokes the Lambda function synchronously. In the remainder of this post, we refer to this Lambda function as listener.

ESM connectivity depends on the following:

For public brokers, ESM uses public connectivity. For private brokers, ESM:

  • Assumes the listener’s IAM Role
  • Creates ENIs in the same subnets as the broker’s VPC Endpoints
  • Uses the same security groups that protect the VPC Endpoints

The listener’s IAM Role must include these Amazon Elastic Compute Cloud (Amazon EC2) permissions:

  • CreateNetworkInterface
  • DeleteNetworkInterface
  • DescribeNetworkInterfaces
  • DescribeSecurityGroups
  • DescribeSubnets
  • DescribeVpcs

To view ESM ENIs:

  1. Open the AWS Management Console
  2. Navigate to EC2 > Network Interfaces
  3. Look for ENIs with the following naming pattern:
    AWS Lambda VPC ENI-armq-<ACCOUNT_ID>-<ESM_ID>-<remainder>

    where:

    • ACCOUNT_ID – The AWS account number containing the ESM
    • ESM_ID – The unique identifier of the ESM

The following image shows example ESM ENIs.

Figure-3 An example list of interfaces that Amazon MQ for RabbitMQ creates for private brokers.

Disabling or deleting the ESM removes the ESM components.

An enabled ESM needs connectivity to the following:

Because the ESM queue polling process follows these steps:

  1. Assumes the listener’s IAM Role
  2. Retrieves RabbitMQ credentials from Secrets Manager
  3. Establishes broker communication
  4. Invokes the listener when messages are present

You have two options to enable private broker connectivity to support the queue polling process:

  1. Deploy VPC endpoints in ESM subnets for:
    • AWS Security Token Service (AWS STS)
    • Secrets Manager
    • Lambda
  2. Deploy NAT gateway in ESM subnets

ESM networking configuration options

The following sections detail ESM networking configurations for different deployment scenarios.

Option 1: Public broker

In this approach all network communication happens on the Amazon MQ service’s side. The ESM, when enabled, uses public connectivity.

To observe the architecture implemented in your account go to the cloned repository root location, make sure that you are signed in with AWS CLI and run the following:

cdk deploy PublicRabbitMqInstanceStack

Option 2: Private broker in a default VPC

Deploying a private RabbitMQ broker without specifying the VPC informs the Amazon MQ service to pick the default VPC for setting up the networking and then the public subnet(s) in that VPC. The default security group is used for securing the broker’s VPC Endpoints.

Creating the ESM provisions dedicated ENIs in the public subnets where the RabbitMQ broker’s VPC Endpoints reside with the default security group applied. The default security group allows itself for inbound traffic on all protocols and full port range, thus the ESM can route traffic through the VPC Endpoint.

Although the subnet is public with internet gateway access, the ESM ENIs operate in private address space, preventing direct communication with AWS services. To enable proper communication, create VPC Endpoints for AWS STS, Secrets Manager, and Lambda. These endpoints allow the ESM to communicate with AWS services through private IP addresses within your VPC. The following diagram shows the complete communication path from the ESM to the broker.

Figure-4 Networking configuration and request flow for a private broker provisioned in the default VPC.

To observe the architecture implemented in your account, go to the cloned repository root location, make sure that you are signed in with AWS CLI, and run the following

cdk deploy PrivateRabbitMqInstanceDefaultVpcStack

Option 3: Private broker in a Custom VPC with NAT

When deploying a private RabbitMQ broker in a custom VPC, specify either a single subnet for a standalone broker or multiple subnets for a cluster deployment. The deployment also needs a security group for the VPC Endpoint ENIs.

Configure the security group with a self-referencing inbound rule on the AMQP port. This configuration enables communication between the ESM and the RabbitMQ VPC Endpoints’ ENIs.

The following diagram shows how ESM resources communicate through networking components when deployed in a private subnet with NAT gateway. This architecture demonstrates the complete communication path from the ESM to the broker.

Figure-5 Networking configuration and request flow for a private broker provisioned in a private VPC subnet with NAT.

To observe the architecture implemented in your account, go to the cloned repository root location, make sure that you are signed in with AWS CLI, and run the following:

cdk deploy PrivateRabbitMqInstanceCustomVpcWithNatStack

Option 4: Private broker in a Custom VPC with isolated subnets

This configuration builds upon the previous architecture but introduces isolated subnets. These subnets restrict all internet connectivity, permitting only internal VPC network traffic. Although the broker networking components mirror Option 3, the isolation introduces more considerations.

The security group still needs an open AMQP port for queue operations, but the subnet isolation prevents the ESM from directly accessing AWS services. To address this limitation, deploy VPC Endpoints for AWS STS, Secrets Manager, and Lambda within the isolated subnets. These endpoints create a private communication path for the ESM to interact with essential AWS services without needing internet access.

The following diagram shows the communication architecture for ESM resources deployed in isolated subnets. It demonstrates how VPC Endpoints enable secure communication between the ESM and AWS services while maintaining network isolation. This architecture makes sure that the ESM can fulfill its message processing responsibilities without compromising security through internet exposure.

Figure-6 Networking configuration and request flow for a private broker provisioned in an isolated VPC subnet.

To observe the architecture implemented in your account, go to the cloned repository root location, make sure that you are signed in with AWS CLI, and run the following:

cdk deploy PrivateRabbitMqInstanceCustomVpcIsolatedSubnetStack

Option 5: Private broker in a Custom VPC with public subnets

The final configuration places the broker in public subnets while maintaining the core deployment requirements from the previous options. Despite the public subnet placement, the ESM’s networking behavior presents an important consideration: ESM ENIs operate in private address space, preventing direct internet communication even with an internet gateway present.

This architecture necessitates VPC Endpoints for AWS service communication, similar to Option 2. Any attempts to route ESM traffic through the internet gateway fail because the ENIs operate in private address space. Understanding this limitation is crucial for proper deployment planning.

The following diagram shows the ESM communication architecture in public subnets. Despite the different subnet type, this configuration mirrors the isolated subnet approach in its use of VPC Endpoints. These endpoints enable the ESM to communicate with AWS STS, Secrets Manager, and Lambda services through private, secure connections within the VPC.

Figure-7 Networking configuration and request flow for a private broker provisioned in a public VPC subnet.

To observe the architecture implemented in your account, go to the cloned repository root location, make sure that you are signed in with AWS CLI, and run the following:

cdk deploy PrivateRabbitMqInstanceCustomVpcPublicSubnetStack

Cleaning up

To prevent unexpected AWS charges, remove resources you’ve created. The following AWS CDK command helps you safely remove all deployed resources:

cdk destroy --all

Conclusion

This post explored the relationship between AWS Lambda event source mapping and RabbitMQ networking configurations. We examined various deployment scenarios, from public brokers to isolated subnets, each presenting unique considerations for secure and effective implementation.

Understanding these networking patterns enables you to make informed architectural decisions when deploying Amazon MQ for RabbitMQ with Lambda event source mapping. Whether choosing public accessibility or implementing private networking with VPC Endpoints, understanding the consequences of choosing specific networking configurations allows you to apply security best practices while meeting your application’s messaging needs. As you implement these patterns, consider your specific security requirements and operational needs to choose the most appropriate configuration for your use case.

Take the next step in optimizing your serverless messaging architecture. Dive in to the AWS documentation, experiment with the RabbitMQ and Lambda integration patterns discussed, and discover how these networking configurations can elevate the security and performance of your own applications. Start implementing these strategies today to build more robust, scalable solutions.

Implementing Federation on Amazon MQ for RabbitMQ Private Brokers

Post Syndicated from ISHITA CHAKRABORTY original https://aws.amazon.com/blogs/compute/implementing-federation-on-amazon-mq-for-rabbitmq-private-brokers/

Federation in RabbitMQ helps in message exchange and flow across multiple RabbitMQ brokers. Amazon MQ for RabbitMQ allows federated exchanges and queues via the Federation Plugin. The federation plugin enables a downstream broker to consume a message from an exchange or a queue on an upstream. This is used to connect multiple RabbitMQ brokers and provides multiple benefits like scalability, allowing to scale out the messaging infrastructure horizontally across multiple nodes or clusters. It also provides high availability for message replication across brokers for redundancy and the ability to segregate based on security or other criteria. These benefits allow federation to be used for the below use cases:

  1. Multi region deployments
  2. Hybrid cloud deployments
  3. Disaster recovery
  4. Migrating from on-premises to cloud

Currently, the federation plugin on Amazon MQ for RabbitMQ connects to publicly available upstream brokers only. This post explains how to implement federation for Amazon MQ RabbitMQ Private Brokers using Network Load Balancers (NLB). The steps allow private brokers to communicate with each other to create a distributed system.

Overview

In this solution, you will use two single-instance brokers to implement federation with private brokers.

  1. Create two Amazon Virtual Private Cloud (VPC) – one for upstream broker and one for downstream broker. Each VPC has a private and a public subnet along with internet gateway, security groups, route tables.
  2. Create Amazon MQ RabbitMQ private brokers in the private subnets of each VPC. The broker actually resides in an account that is owned by the Amazon MQ Service, in a private subnet with a Network Load Balancer (NLB) in front of it. The NLB is used to access the broker from your account using the Elastic Network Interface (ENI) associated with the VPC Endpoint for the NLB.
  3. Create a NLB pointing to the ENI for the upstream broker. The security group associated with the NLB is used to restrict traffic to only the NAT IPs associated with the downstream broker. The upstream broker that was accessible only privately will now be connected to the public internet with IP allow listing and messages will potentially transit the internet.
  4. Create an Amazon EC2 Instance in the downstream VPC in the public subnet to connect to it and setup the federation. You need the EC2 instance only for the setup and testing.
  5. Send a message to the upstream broker using the NLB endpoint, the message is also available to the downstream broker for consumption.

Prerequisites

The following are the prerequisites for this setup:

  • Access to an AWS account.
  • An AWS IAM user/Principal with the required permissions to deploy the infrastructure.

The stack creates two new VPC. Make sure that you have fewer than five VPCs in the selected region. You increase this limit using Quotas.

Deploying the solution

You will deploy the solution using AWS CloudFormation:

The high-level steps are the following:

  1. Deploy the broker CFN stack to create VPCs, subnets, internet gateway, security groups and route tables, along with the Amazon MQ RabbitMQ brokers
  2. Get the IP address of the private upstream broker created in the broker stack
  3. Open AWS support case to get the IP to allow for the NLB
  4. Create an NLB Stack with the Network Load Balancer and rules for accessing it using AWS CloudFormation
  5. Set up the federation between the Amazon MQ RabbitMQ brokers and testing the setup

This solution is available on GitHub in the AWS Samples repository.

Step 1: Deploy the AWS CloudFormation template for the broker stack

  1. Go to the CloudFormation Console and choose Create Stack. Choose With new resources (Standard) from the drop down.
  2. For Prepare template, choose an existing template and then for Specify template, choose Upload a template file and use this template file
  3. Provide a Stack name (such as BrokerStack).
  4. Update the username and CIDR Blocks provided as parameters to the stack or leave them as defaults. For ease of setup, this template uses EC2 with managed prefix lists for EC2 Instance Connect for five regions: us-east-1, us-west-1, us-west-2, eu-west-1 and ap-south-1. Add prefix lists for other regions in the template to run this cloud formation template in those regions.
  5. Choose Next and leave everything else as defaults.
  6. Choose Submit.

The broker stack deployment takes 10 -15 minutes.

The template creates two VPCs along with a private and public subnet on each VPC with internet gateway, security groups and route tables. It also creates two private brokers in each VPC along with an EC2 Instance (t2.micro) on the downstream VPC.

Step 2: Retrieve the IP Address for the private upstream broker

  1. Once the above stack creation is complete, navigate to the Outputs tab for the stack and copy the output for PrivateUpstreamBrokerEndpoints.
  2. Extract only the host name from the “PrivateUpstreamBrokerEndpoints” in the output from above.
  3. Resolve the hostname using the following commands.
    Linux or Mac

    $ dig +short {hostname}

    Windows

    C:\> nslookup {hostname}

Take note of the IP address. You will use it in later steps.

Step 3: Create a support case to get the Amazon MQ Rabbit MQ Downstream Broker NAT IPs

Create a support case with AWS Support to get the NAT IPs associated with the downstream MQ Broker. Provide the broker Amazon Resource Name (ARN) and explain your use case and the need to do federation allow listing in the description. Use this IP address to allow the Network Load Balancer to be accessed from particular IPs only.

Step 4: Deploy the AWS CloudFormation template for NLB Stack

  1. Go to the CloudFormation Console and choose Create Stack. Choose With new resources (Standard) from the drop down.
  2. For Prepare template, Choose an existing template. For Template source, choose Upload a template file and choose this template file.
  3. Choose Next.
  4. Under Specify stack details provide a Stack name (such as NLBStack).
  5. Use the IP Address from Step 2 and Step 3 above in the parameters and choose Next.
    Make sure that the NAT IP Address is a valid CIDR range like 52.0.0.1/32.
  6. Keep the rest as defaults and choose Next again
  7. Choose Submit.

The template creates a Network Load Balancer with 2 target groups and a Security Group for it and adds rules to the Upstream Default Security group.

Step 5: Configure Federation in the downstream broker

  1. Use the Upstream Broker NLB URL output from the NLBStack and replace it in the following export commands along with the Downstream Broker Uri from the output of the BrokerStack.
    export Upstream_Broker_NLB= <UpstreamBrokerNLBURL>
    export Downstream_Broker_Uri= <DownstreamBrokerURI> 

  2. From the AWS Console, search for AWS Secrets Manager and choose Secrets. You will find 2 secrets with names as DownstreamBrokerUsernamePassword and UpstreamBrokerUsernamePassword. Open one of them and choose Retrieve Secret value to get the passwords and usernames for the brokers. Repeat for the other one.
  3. Replace values for Upstream_Broker_Username, Upstream_Broker_Password, Downstream_Broker_Username and Downstream_Broker_Password in the following commands.
    ##creates federation on the private downstream broker
    curl -XPUT -d'{"value":{"uri":"amqps://Upstream_Broker_Username:Upstream_Broker_Password@'"$Upstream_Broker_NLB"':5671","expires":3600000}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/parameters/federation-upstream/%2f/my-upstream
    
    ##creates policy for federation on the private downstream broker with pattern for exchange with Test in its name
    curl -XPUT -d'{"pattern":"^Test", "definition":{"federation-upstream-set":"all"},"apply-to":"exchanges"}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/policies/%2f/federate-me

  4. From the EC2 Console, select the EC2 instance created as part of the Broker Stack in Step 1. Choose Connect and login to the instance using EC2 Instance Connect. Once connected to the terminal, paste the above lines with replaced values to create the federation upstream and the policy associated with it.

Step 6: Create TestExchange and Test Queue and Bind them

  1. Run the following steps to create a test exchange, a queue, and the binding for them. Replace values for Downstream_Broker_Username and Downstream_Broker_Password.
    ##creates a test exchange on the private downstream broker
    curl -H "content-type:application/json" -XPUT -d'{"type":"fanout","durable":true}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/exchanges/%2f/TestExchange
    
    ##creates a test queue on the private downstream broker
    curl -H "content-type:application/json" -XPUT -d'{"durable":true,"arguments":{"x-dead-letter-exchange":"", "x-dead-letter-routing-key": "my.queue.dead-letter"}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/queues/%2f/TestQueue
    
    ##Binds the queue to the exchange on the private downstream broker
    curl -H "content-type:application/json" -XPOST -d'{"routing_key":"","arguments":{}}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/bindings/%2f/e/TestExchange/q/TestQueue

Step 7: Validate Federation Status and Test Federation between brokers

  1. Check the Federation status by running the following command while still connected to the EC2 in the same session. Replace values for Downstream_Broker_Username and Downstream_Broker_Password.
    ##check federation status on the private downstream broker and format it as JSON
    curl -XGET https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/federation-links | python3 -m json.tool

    The output will look like the below with status as running.

    [
        {
            "node": "rabbit@localhost",
            "exchange": "TestExchange",
            "upstream_exchange": "TestExchange",
            "type": "exchange",
            "vhost": "/",
            "upstream": "my-upstream",
            "id": "5cd2293f",
            "status": "running",
            "local_connection": "<[email protected]>",
            "uri": "amqps://MyUpstreamNLB-XXXXXXXX.elb.us-east-1.amazonaws.com:5671",
    …
        }
    ]

  2. (Optional) Send a test message now. Since you restricted the Upstream Broker NLB to only receive traffic from the Downstream broker (via the IP Address received from the support case), you will need to manually allow the EC2 Public IP Address in the NLB Security Group that was created for port 443 to perform the below step. You will also need to allow the egress from EC2 to access the NLB.
    ##Send test message on the upstream broker
    curl -k -H "content-type:application/json" -XPOST -d'{"properties":{},"routing_key":"MYKEY","payload":"Hello World","payload_encoding":"string"}' https://Upstream_Broker_Username:Upstream_Broker_Password@{$Upstream_Broker_NLB}/api/exchanges/%2f/TestExchange/publish

    Once the message is sent it will show up as routed: true. This means that the message routed to the downstream broker successfully.

  3. Use the following command to validate the message on the downstream broker. This should show the payload that you sent earlier.
    ## Get message from queue on the downstream broker
    curl -H "content-type:application/json" -XPOST -d'{"ackmode":"ack_requeue_true","count":1,"encoding": "auto"}' https://Downstream_Broker_Username:Downstream_Broker_Password@{$Downstream_Broker_Uri}/api/queues/%2f/TestQueue/get

    Output:

    [
        {
            "payload_bytes": 11,
            "redelivered": true,
            "exchange": "TestExchange",
            "routing_key": "MYKEY",
            "message_count": 0,
             …
            "payload": "Hello World",
            "payload_encoding": "string"
        }
    ]

Cleanup

This section provides information for deleting various resources created as part of this post.

  1. Delete Stack NLBStack created as part of Step 4. For instructions, refer to Deleting a stack on the AWS CloudFormation console.
  2. Delete the BrokerStack created in Step 1.

Conclusion

This post explained how to implement federation for Amazon MQ RabbitMQ private brokers. You can extend this solution to RabbitMQ brokers in a cluster deployment, same as a single-instance broker. With federated exchanges, you can create a distributed system of RabbitMQ brokers to improve reliability and scalability of the messaging system. You can also use this as a template for hybrid architecture to move messages from a private on-premises broker to the cloud as explained in Migrating message driven applications to Amazon MQ for RabbitMQ. Get more details on Federation plugin from official documentation of RabbitMQ. Get more details on Amazon MQ for RabbitMQ in our developer guide.

Serverless ICYMI Q4 2024

Post Syndicated from Eric Johnson original https://aws.amazon.com/blogs/compute/serverless-icymi-q4-2024/

Welcome to the 27th edition of the AWS Serverless ICYMI (in case you missed it) quarterly recap. At the end of a quarter, we share the most recent product launches, feature enhancements, blog posts, webinars, live streams, and other interesting things that you might have missed!

In case you missed our last ICYMI, check out what happened in Q2 here.

Calendar showing October through December 2024

2024 Q4 calender

Serverless at re:Invent 2024

AWS re:Invent 2024 had 60,000 in-person attendees and 400,000 online viewers for the keynotes. The conference delivered 1,900 sessions from 3,500 speakers and included 546 AWS service and feature announcements.

The serverless content consisted of two tracks: Serverless (SVS) and App Integration (API). These tracks included 70 unique sessions and attracted nearly 11,000 attendees. Serverlesspresso, the coffee shop powered by serverless technology, operated in two locations during the event: the Expo Hall and the certification lounge.

Crowd of people standing around the AWS reI:nvent expo hall waiting to order coffee at the Serverlesspresso booth.

Serverlesspresso booth in the expo hall

Videos are available on Serverless Land YouTube.

AWS Lambda and Amazon Elastic Container Service (Amazon ECS) 10-year anniversary.

AWS marked significant milestones in serverless computing, celebrating 10 years of AWS Lambda and Amazon ECS. Lambda now serves over 1.5 million monthly customers and processes tens of trillions of requests each month. Amazon ECS launches more than 2.4 billion container tasks weekly and is used by over 65% of new AWS container customers.

AWS is commemorating this anniversary with insights from AWS Serverless Heroes, product leads, principal engineers, and AWS leadership sharing their perspectives on serverless evolution and future directions. These stories and insights are available at https://aws.amazon.com/serverless/10th-anniversary/.

AWS Lambda

The AWS Lambda team has spent a significant amount of time improving the Lambda development experience. Several enhancements have been made in the console as well as the local development experience.

Screen capture of the new AWS Lambda console with Code-OSS

Code-OSS as the new AWS Lambda inline editor

Lambda has launched a significant upgrade to its console by integrating Code-OSS, the open-source version of Visual Studio Code, delivering a familiar development experience directly in the cloud. The new Lambda Code Editor supports viewing larger function packages up to 50 MB, features a split-screen interface for simultaneous code editing and testing, and includes built-in Amazon Q Developer AI assistance for real-time coding suggestions. This enhancement comes at no additional cost and prioritizes accessibility with features like screen reader support and keyboard navigation. The update bridges the gap between cloud and local development by simplifying the process of downloading function code and AWS SAM templates, ultimately providing developers with a more streamlined and familiar serverless development experience. Watch the video explaining the changes in detail.

Additionally, the Lambda console enhances developer experience with two new features: a built-in CloudWatch Metrics Insights dashboard that surfaces key function metrics, and CloudWatch Logs Live Tail support for real-time log streaming and analysis, enabling faster troubleshooting without leaving the Lambda environment.

Screen capture of the new top 10 functions in the new AWS Lambda console

Top 10 Functions

Lambda now supports native JSON structured logging for .NET managed runtime applications, improving log searchability and analysis capabilities without requiring manual configuration of logging libraries.

Lambda has expanded its runtime support by adding Python 3.13 and Node.js 22 as both managed runtimes and container base images, providing access to the latest language features and ensuring long-term support through October 2029 and April 2027, respectively.

Lambda SnapStart capability is now available for Python and .NET runtimes, delivering sub-second startup performance for latency-sensitive applications by caching initialized execution environments.

Diagram of how SnapStart works compared to not having SnapStart

SnapStart support comparison

New CloudWatch metrics for Lambda Event Source Mappings provide enhanced visibility into event processing states for Amazon Simple Queue Service (SQS), Amazon Kinesis, and Amazon DynamoDB event sources, helping customers monitor and troubleshoot event processing issues.

Lambda introduces Provisioned Mode for Kafka event source mappings, allowing customers to optimize throughput by configuring dedicated event polling resources for applications with stringent performance requirements.

Finally, Lambda introduces an enhanced local development experience through the AWS Toolkit for Visual Studio Code, streamlining the serverless application development workflow. The update features a new Application Builder interface that guides developers through environment setup, offers sample applications, and provides quick-action buttons for common tasks like build, deploy, and invoke operations. Developers can now efficiently iterate on their code with features such as configurable build settings, step-through debugging, and the ability to sync local changes quickly to the cloud or perform full deployments. The toolkit integrates with AWS Infrastructure Composer for visual application building and includes comprehensive local testing capabilities with shareable test events. This enhancement simplifies the Lambda development process by enabling developers to author, test, debug, and deploy serverless applications without leaving their preferred IDE environment.

Screen capture of the getting started experience for serverless in a local IDE

Local IDE getting started

Amazon ECS and AWS Fargate

AWS enhances observability for containerized applications with CloudWatch Application Signals for Amazon ECS, adding infrastructure metrics correlation to existing traces and logs monitoring, enabling operators to identify and resolve performance issues across their application stack.

Amazon ECS adds service revision and deployment history tracking, allowing customers to monitor changes, track ongoing deployments, and debug deployment failures for long-running applications deployed after October 25, 2024.

A graph explaining the flow for service order and history

Service revisions and deployment history

Amazon ECS expands testing capabilities by supporting network fault injection experiments on AWS Fargate through AWS Fault Injection Service, enabling developers to verify application resilience using six different types of fault injection actions, including network disruptions and resource stress testing.

Amazon EventBridge

Amazon EventBridge announces significant performance improvements, reducing end-to-end latency by up to 94% from 2,235ms to 129.33ms at P99, enabling faster event processing for time-sensitive applications like fraud detection and gaming.

Amazon EventBridge and AWS Step Functions now integrate with private APIs through AWS PrivateLink and Amazon VPC Lattice, enabling secure connectivity between cloud and on-premises applications without custom networking code.

Screen capture of the Amazon EventBridge create connection screen showing the new Private option

Connections to Private APIs

EventBridge API destinations introduces proactive OAuth token refresh for public and private authorization endpoints, helping prevent delays and errors by automatically refreshing tokens before expiration.

AWS Step Functions

AWS Step Functions introduces the ability to export workflows as CloudFormation or SAM templates directly from the AWS console, enabling repeatable provisioning across accounts. Developers can export and customize templates from existing workflows, and use AWS Infrastructure Composer to visually connect workflows with other AWS resources.

Step Functions also adds Variables and JSONata support to enhance workflow development. Variables allow data assignment and reference between states, simplifying payload management, while JSONata provides advanced data transformation capabilities, including date formatting and mathematical operations. These features reduce the need for custom code and intermediate states, making it easier to build distributed serverless applications. Watch the in depth video to learn more.

Screen capture of AWS Step Function workflow studio using JSONata and variables in an example

JSONata and variables

Amazon Kinesis

Amazon Kinesis introduces significant updates to its client libraries. The new Kinesis Client Library (KCL) 3.0 reduces compute costs by up to 33% through enhanced load balancing, while the Kinesis Producer Library (KPL) 1.0 improves performance and security. Both libraries now support AWS SDK for Java 2.x and eliminate dependencies on SDK for Java 1.x, enabling seamless upgrades without requiring application code changes.

Screen capture of CPU usage metrics

KCL 3.0 metrics

Amazon MQ

Amazon MQ adds support for AWS PrivateLink, enabling customers to access Amazon MQ API endpoints directly from their VPC through interface VPC endpoints, eliminating the need for internet access and providing enhanced security through AWS’s internal network infrastructure.

Amazon Finch

AWS announces general availability of Linux support for Finch, an open source container development tool that simplifies building, running, and publishing Linux containers across all major operating systems. The release includes support for the Finch Daemon with Docker API compatibility and is available through RPM packages for Amazon Linux 2 and Amazon Linux 2023.

Amazon Simple Queue Service (SQS)

Amazon SQS increases the in-flight message limit for FIFO queues from 20,000 to 120,000 messages, enabling higher concurrent message processing. This enhancement allows customers to scale their receivers and process up to six times more messages simultaneously, provided they have sufficient publish throughput.

Amazon Managed Streaming for Apache Kafka(Amazon MSK)

Amazon MSK now introduces Managed Streaming for Apache Flink blueprints to simplify real-time AI application development. The service enables vector-embedding generation through Amazon Bedrock, streamlining the integration of streaming data with generative AI models. Using a straightforward configuration process, users can generate and index vector embeddings in Amazon OpenSearch, while leveraging LangChain’s data chunking capabilities for enhanced data retrieval efficiency. The service handles all integration aspects between MSK, embedding models, and Amazon OpenSearch vector stores.

AWS Amplify

AWS Amplify launches the Amplify AI kit for Amazon Bedrock, providing fullstack developers with tools to integrate AI capabilities into web applications. The kit includes a customizable React UI component, secure Bedrock access, and context-sharing features, enabling developers to implement chat, search, and summarization functionalities without machine learning expertise.

AWS AppSync

AWS AppSync launches AppSync Events, enabling developers to broadcast real-time data to multiple subscribers through serverless WebSocket APIs. The service eliminates the need to build and manage WebSocket infrastructure while providing secure, scalable event broadcasting capabilities. Developers can create APIs that automatically scale and integrate with services like Amazon EventBridge. The system supports features such as channel namespaces, event handlers, and multiple authorization modes, and is available in all regions where AWS AppSync operates. Users only pay for API operations and real-time connection minutes used.

Screen capture from the AWS AppSync console to create a new Event API.

Creating an AppSunc Event API

Amazon API Gateway

Amazon API Gateway released a significant enhancement to Amazon API Gateway, enabling customers to manage private REST APIs using custom private DNS names. This highly requested feature allows API providers to use user-friendly domain names like private.example.com, while maintaining TLS encryption for security. The implementation process involves creating a private custom domain, configuring certificates through AWS Certificate Manager (ACM), mapping private APIs, and setting resource policies. The feature supports cross-account sharing through AWS Resource Access Manager (AWS RAM) and is now available in all AWS Regions, including AWS GovCloud (US).

Serverless blog posts

October

November

Serverless Office Hours

Image from YouTube from the latest four Serverless Office Hours

Serverless office hours videos

October

November

Still looking for more?

The Serverless landing page has more information. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Serverless Developer Advocacy team on X (formerly Twitter) to see the latest news, follow conversations, and interact with the team.

And finally, visit the Serverless Land  for all your serverless needs.

Implementing transactions using JMS2.0 in Amazon MQ for ActiveMQ

Post Syndicated from Chris McPeek original https://aws.amazon.com/blogs/compute/implementing-transactions-using-jms2-0-in-amazon-mq-for-activemq/

This post is written by Paras Jain, Senior Technical Account Manager and Vinodh Kannan Sadayamuthu, Senior Specialist Solutions Architect

This post describes the transactional capabilities of the ActiveMQ broker in Amazon MQ by using a producer client application written using the Java Messaging System(JMS) 2.0 API. The JMS 2.0 APIs are easier to use and have fewer interfaces than the previous version. To learn about ActiveMQ’s JMS 2.0 support, refer to the ActiveMQ documentation on JMS2.0. Also check out What’s New in JMS 2.0 to learn more about features in JMS2.0.

Amazon MQ now supports ActiveMQ 5.18. Amazon MQ also introduces a new semantic versioning system that displays the minor version (e.g., 5.18) and keeps your broker up-to-date with new patches (e.g., 5.18.4) within the same minor version. ActiveMQ 5.18 adds support for JMS 2.0, Spring 5.3.x, and several dependency updates and bug fixes. For the complete details, see release notes for the Active MQ 5.18.x release series.

Overview

Messaging Patterns in Distributed Systems

Implementing messaging in a message-broker based distributed messaging often involves a fire-and-forget mechanism. Message producers send the messages to the broker and it is message broker’s responsibility to ensure that the messages are delivered to the consumers. In non-transactional use cases, the messages are independent of each other. However, in some situations, a group of messages needs to be delivered to consumers as part of a single transaction. This means either all the messages in the group are to be delivered to the consumer or none of those messages are delivered.

ActiveMQ 5.18 provides two levels of transaction support — JMS transactions and XA transactions.

JMS transactions are used when multiple messages need to be sent to the ActiveMQ broker as a single atomic unit. This transactional behavior is enabled by invoking the commit() and rollback() methods on a Session (for JMS 1.x) or JMSContext (for JMS 2.0) object. If all the messages are successfully sent, the transaction can be committed, ensuring that the messages are processed as a unit. If any issues occur during the sending process, the transaction can be rolled back, preventing the partial delivery of messages. This transactional capability is crucial when maintaining data integrity and ensuring that complex messaging operations are executed reliably. See ActiveMQ FAQ – How Do Transactions work FAQ for more details on how transactions work in ActiveMQ.

XA transactions are used when two or more messages need to be sent to ActiveMQ brokers and other distributed resources in a transactional manner. This is achieved by using an XA Session, which acts as an XA resource. See ActiveMQ FAQ – Should I use XA transactions FAQ for more details on XA transactions.

Transactional use case in an Order Management System

The example in this blog post shows the transactional capabilities in an Order Management System (OMS) application, using ActiveMQ as the message broker. Upon receiving an order, the OMS application sends a message (message 1) to the warehouse queue to start the packing process. Then the application runs an internal business process. If this process is successful, the application sends another message (message 2) to the shipping queue to start the package pickup process. In the event of internal business process failure, it is necessary to prevent message 2 from being sent to the shipping queue and rollback message 1 from the warehouse queue.

The flowchart below illustrates the logic behind the transactional use case featured in this example.

Flowchart illustrating the logic behind the transactional use case in the code example. Demonstrates flow for successful as-well-as failed transaction.

Flowchart describing transactional use case.

The JMS client stores both messages in-memory until the transaction is committed or rolled back. The client achieves this by maintaining a Transacted Session between the message producer client and the broker. A transacted session is a session that uses transactions to ensure message delivery. In our example, transacted session is created using the following statement.

JMSContext jmsContext = connectionFactory.createContext(adminUsername, adminPassword, Session.SESSION_TRANSACTED);

In the example for this post, we have shown a transacted session between the message producer and the broker. We are not showing transactions between the broker and the message consumer. You can implement it using the similar pattern.

Creating ActiveMQ broker

The following prerequisites are required to create and configure ActiveMQ broker in Amazon MQ.

Prerequisites:

To create a broker (AWS CLI):

  1. Run the following command to create the broker. This creates a publicly accessible broker for testing only. When creating brokers for production use, adhere to the Security best practices for Amazon MQ.
    aws mq create-broker \
        --broker-name <broker-name> \
        --engine-type activemq \
        --engine-version 5.18 \
        --deployment-mode SINGLE_INSTANCE \
        --host-instance-type mq.t3.micro \
        --auto-minor-version-upgrade \
        --publicly-accessible \
        --users Username=<username>,Password=<password>,ConsoleAccess=true
    

    Replace <broker-name> with the name you want to give to the broker. Replace <username> and <password> as per the create-broker CLI documentation. After the successful execution of the command the BrokerArn and the BrokerId is displayed on the command line. Note down these values.Creation of the broker takes about 15 minutes.

  2. Run the following command to get the status
    aws mq describe-broker --broker-id <BrokerId> --query 'BrokerState'

    Proceed to next step once the broker state is Running.

  3. Get the console URL and other broker endpoints by running the following command
    aws mq describe-broker --broker-id <BrokerId> --query 'BrokerInstances[0]’

    Note the ConsoleURL and ssl endpoint from the output.

Configuring the message producer client

The sample code in this post uses a sample message producer client written using JMS 2.0 API to send messages to the ActiveMQ broker.

  • In case of a successful transaction, the producer client sends a message to the first queue and waits for 15 seconds. Then it sends the message to the second queue and waits for another 15 seconds. Finally, it commits the transaction.
  • In case of a failed transaction, the producer client sends the first message and waits for 15 seconds. Then the code introduces an artificial failure, causing the transaction rollback.The 15 seconds wait time provides you the opportunity to verify the number of messages at broker side as the program progresses through the transaction flow. Until the producer client commits the transaction, none of the messages are sent to the broker, even for a successful transaction.

To download and configure the sample client:

  1. Get the Amazon MQ Transactions Sample Jar from the GitHub repository.
  2. To run the sample client, use the java command with -jar option which runs the program encapsulated in a jar file. The syntax for running the sample client is:
    java -jar <path-to-jar-file>/<jar-filename> <username> <password> <ssl-endpoint> <first-queue> <second-queue> <message> <is-transaction-successful> 

    Usage:
    <path-to-jar-file> – path in your local machine where you have downloaded the jar file.
    <jar-filename> – name of the jar file.
    <username> – username you selected while creating the broker.
    <password> – password you selected while creating the broker.
    <ssl-endpoint> – ssl endpoint you noted down in the step above.
    <first-queue> – name of the first queue in the transaction.
    <second-queue> – name of the second queue in the transaction.
    <message> – message text.
    <is-transaction-successful> – flag to tell the producer client if the transaction has to be successful or not.

Testing successful transactions

Following are the steps to test successful transactions with ActiveMQ:

  1. List queues and message counts in ActiveMQ console
    1. Navigate to the Amazon MQ console and choose your ActiveMQ broker.
    2. Login to ActiveMQ Web Console from URLs in Connections panel.
    3. Click on Manage ActiveMQ broker.
    4. Provide username and password used for the user created when you created the broker.
    5. Click on Queues on the top navigation bar.
    6. Check warehouse-queue and shipping-queue are not listed.
  2. Run the following command to send messages for order1 to both the queues successfully:
    java -jar <path-to-jar-file>/<jar-filename> <username> <password> <ssl-endpoint> warehouse-queue shipping-queue order1 true

    Replace the placeholders as mentioned in the command instructions above.With this command, the example producer client sends the first message to the warehouse-queue and prints the following message to the console and waits for 15 seconds.

    Sending message: order1 to the warehouse-queue
    Message: order1 is sent to the queue: warehouse-queue but not yet committed.

    During the 15 seconds wait, refresh the browser and verify that the warehouse-queue is now listed but has no pending or enqueued messages.

    After 15 seconds, the producer client sends the second message to the shipping-queue and prints the following message to the console and waits for 15 more seconds.

    Sending message: order1 to the shipping-queue
    Message: order1 is sent to the queue: shipping-queue but not yet committed.
    

    During this 15-second wait, refresh the browser window again and verify that the shipping-queue is now listed, but like the warehouse-queue, it has no pending or enqueued messages.

    Finally, the producer client commits both the messages and prints:

    Committing
    Transaction for Message: order1 is now completely committed.
    

  3. Refresh the browser and verify warehouse-queue and shipping-queue have 1 pending and enqueued message each. The list will look like below:Image shows example of queues with message count.Image showing the shipping and warehouse queues

Repeat this process for testing more successful transactions.

Testing failed transactions

  1. Note down the beginning number of pending and enqueued messages in each of the queues.
  2. Run the following command and pass false for <is-transaction-successful> to introduce an artificial failure.
    java -jar <path-to-jar-file>/<jar-filename> <username> <password> <ssl-endpoint> warehouse-queue shipping-queue failedorder1 false

    Replace the placeholders as mentioned in the initial command instructions above.With this command, the example producer client sends the first message to the warehouse-queue and prints the following message to the console and waits for 15 seconds.

    Sending message: failedorder1 to the warehouse-queue
    Message: failedorder1 is sent to the queue: warehouse-queue but not yet committed.
    

    During the 15 seconds wait, refresh the browser and verify that the counts in the warehouse-queue and shipping-queue are unchanged.

    Finally, the client artificially introduces a failure and rolls back the transaction and prints:

    Message: failedorder1 cannot be delivered because of an unknown error. Hence the transaction is rolled back.

  3. Refresh the browser to confirm that the counts for both the queues are unchanged. This example starts with 1 message each in each queue which remained unchanged after the failed transaction.Image shows example of shopping and warehouse queues with failed messages.Image showing shipping and warehouse queues with unchanged counts.

Note that for both the successful and unsuccessful scenarios, the messages that are sent to the queues as part of a transaction are stored in-memory at the client side. These messages are sent to the broker only when the transaction is committed.

Cleanup

  1. Delete the broker by running the following command
    aws mq delete-broker --broker-id <BrokerId>

Conclusion

In this post, you created an Amazon MQ broker for ActiveMQ for version 5.18. You also learned about the new semantic versioning introduced by Amazon MQ. ActiveMQ 5.18.x brings support for JMS 2.0, Spring 5.3.x and dependency updates. Finally, you created a sample application using JMS 2.0 API showing transactional capabilities of the ActiveMQ 5.18.x broker.

To learn more about Amazon MQ, visit https://aws.amazon.com/amazon-mq/.

Measuring Amazon MQ throughput using Maven 2 benchmark and AWS CDK

Post Syndicated from Chris Munns original https://aws.amazon.com/blogs/compute/measuring-amazon-mq-throughput-using-maven-2-benchmark-and-aws-cdk/

This post is written by Olajide Enigbokan, Senior Solutions Architect and Mohammed Atiq, Solutions Architect

In this post you will learn how to evaluate the throughput for Amazon MQ, a managed message broker service for ActiveMQ, by using the ActiveMQ Classic Maven Performance test plugin. This post will provide recommendations for configuring Amazon MQ to optimize throughput when leveraging ActiveMQ as a broker engine.

Overview on benchmarking throughput for Amazon MQ for ActiveMQ

To get a good balance of cost and performance while leveraging ActiveMQ on Amazon MQ, AWS recommends that customers benchmark during migration, instance type/size upgrade, or downgrade. Benchmarking can help you choose the correct instance type and size for your workload requirements. For common benchmark scenarios and benchmark figures for different instances types and sizes, see AmazonMQ for ActiveMQ Throughput benchmarks.

Performance of your ActiveMQ workload depends on the specifics of your use-case. For example, if you have a workload where durability is extremely important (meaning that messages cannot be lost), enabling persistence mode ensures that messages are persisted to disk before the broker informs the client that the message send action has completed. The faster the disk I/O capacity and the smaller the message size during these writes, the better the message throughput. For this reason, AWS recommends the mq.m5.* instance types for regular development, testing, and production workloads as described in Amazon MQ for ActiveMQ instance types. The mq.t2.micro and mq.t3.micro instance types are intended for product evaluation and are subject to burst CPU credits and baseline performance. Hence, they are not suitable for applications that require fixed performance. In the situation where a larger broker instance type is selected, AWS also recommends batching transactions for persistent store which allows you to send multiple messages per transaction while achieving an overall higher message throughput.

The next section describes the details of setting up your own benchmark for Amazon MQ using the open-source benchmarking tool: ActiveMQ Classic Maven Performance test plugin. The ActiveMQ Classic Maven Performance test plugin benchmark suite is highly recommended due to the ease in setup and deployment process.

Getting started

This walkthrough guides you through the steps for benchmarking your Amazon MQ brokers:

Step 1 – Build and push container image to Amazon ECR

Clone the mq-benchmarking-container-image-sample repository and follow the steps in the README file to build and push your image to an Amazon Elastic Container Registry (Amazon ECR) public repository. You will need this container image for Step 2.

Step 2 – Automate Your Benchmarking Setup with AWS CDK

Architecture of CDK deployment

Architecture of CDK deployment

To streamline the deployment of an active/standby ActiveMQ broker alongside Amazon Elastic Container Service (Amazon ECS) tasks for this walk-through, follow these steps below to set up the environment leveraging AWS Cloud Development Kit (AWS CDK). This will deploy the resources shown in the architecture diagram above.

2.1. Prerequisites:

Ensure the following packages are installed:

2.2 Repository Setup:

Clone the mq-benchmarking-sample repository. This repository contains all the necessary code and instructions to automate the benchmarking process using the AWS CDK.

2.3 Create a Virtual Environment:

Change directory (cd) to the cloned repository directory and create a Python virtual environment by running the following command:

cd mq-benchmarking-sample

python -m venv .venv

2.4 Activate Virtual Environment:

Run the following commands to activate your virtual environment:

# Linux
source .venv/bin/activate

# Windows
.\.venv\Scripts\activate

2.5 Install Dependencies:

Install the required Python packages using:

pip install -r requirements.txt

2.6 Customize and Deploy:

In this step, deploy the necessary stacks and their resources for benchmarking in your AWS account. The command ‘cdk deploy’ below deploys three stacks with resources for Amazon ECS, MQ and VPC. Deploy your application with AWS CDK using the command:

cdk deploy "*" -c container_repo_url=<YOUR CONTAINER REPO URL> -c container_repo_tag=<YOUR CONTAINER REPO TAG>

This command deploys your application with the specified Docker image. Replace <YOUR CONTAINER REPO URL> and <YOUR CONTAINER REPO TAG> with your specific Docker repo image details from Step 1. An example container repo URL would look like this: public.ecr.aws/xxxxxxxxx/xxxxxxxxxx.

The deployment of the stacks and their resources happen in three stages. Please select “yes” at each stage to deploy the stated changes as shown below:

First stage of the deploy

Select yes to deploy these changes

Deployed stacks and their resources

Deployed stacks and their resources

Optionally, you can include additional context variables in your command as seen below:

cdk deploy "*" -c vpc_cidr=10.0.0.0/16 -c mq_cidr=10.0.0.0/16 -c broker_instance_type=mq.m5.large -c mq_username=testuser -c tasks=2 -c container_repo_url=<YOUR CONTAINER REPO URL> -c container_repo_tag=<YOUR CONTAINER REPO TAG>

Note: In the example command above, the vpc_cidr specified is the same as mq_cidr. If you decide to use the above command, you will need to ensure that your vpc_cidr range is the same as your mq_cidr range. AWS recommends this as security best practice to ensure that your broker endpoint is only accessible from recognized IP ranges, see Security best practices for Amazon MQ.

More details on the above context variables:

  • broker_instance_type: Represents the instance type for the Amazon MQ Broker. You can start with the instance type mq.m5.large.
  • vpc_cidr: Allows you to customize the VPC’s CIDR block. The default CIDR is set to 10.42.0.0/16.
  • mq_cidr: Allows you set a specific security group CIDR range for the broker. This must be set to the vpc_cidr. From the sample command above, this is set to 10.0.0.0/16. For more flexibility with source IP ranges, you can edit the broker security group of your CDK deployment.
  • mq_username: Allows you to specify a username to access the ActiveMQ web console access and broker.
  • tasks: Determines the number of ECS tasks (1 or more) to run your Docker image. Since the OpenWire configuration file for both consumers and producers allow you to specify the number of clients that you want, all the clients in one ECS task will share the CPU and memory allocation for that task. You have the option to run multiple ECS tasks (with multiple clients) running the benchmark in parallel.

These adjustments allow for a more customized deployment to fit specific benchmarking needs and scenarios.

2.7 Benchmarking Execution

After deployment, you should see an output similar to the following:

Successful deployment of CDK application with output

Successful deployment of CDK application with output

1. Retrieve the TASK-ARN and access the Container

The above exec command in “outputs:” requires that you supply a <TASK-ARN> before the command can be run. To retrieve the <TASK-ARN> via the AWS CLI, you will need to do the following:

  • Run the below command and note down the Task ARN (needed later):
aws ecs list-tasks --cluster <cluster-name> --region <region>

You can also retrieve this value via the Amazon ECS console by going to your ECS Cluster and choosing Tasks.

  • Access the running ECS task using the ECS Exec feature with the command that is output from the CDK deployment. The command should look like the following:
aws ecs execute-command --region eu-central-1 --cluster arn:aws:ecs:eu-central-1:XXXXXXXX:cluster/ECS-Stack-ClusterEB0386A7-gRmSxC06y4ay --task <TASK-ARN> --container Benchmarking-Container --command "/bin/bash" --interactive

Before running the above command, replace the placeholder value of <TASK-ARN> with the value of the actual Task ARN noted earlier.

After retrieving the <TASK-ARN>, and running the exec command, you should have a directory structure as follows:

Directory Structure within ECS Task using ECS Exec

Directory Structure within ECS Task using ECS Exec

2. Configure the openwire-producer.properties and openwire-consumer.properties files.

Open both files. Shown below is the content of the openwire-producer.properties and openwire-consumer.properties files.

openwire-producer.properties:

sysTest.reportDir=./reports/
sysTest.samplers=tp
sysTest.spiClass=org.apache.activemq.tool.spi.ActiveMQReflectionSPI
sysTest.clientPrefix=JmsProducer
sysTest.numClients=25


producer.destName=queue://PERF.TEST
producer.deliveryMode=persistent
producer.messageSize=1024
producer.sendDuration=300000

factory.brokerURL=
factory.userName=
factory.password=

openwire-consumer.properties:

sysTest.reportDir=./reports/
sysTest.samplers=tp
sysTest.spiClass=org.apache.activemq.tool.spi.ActiveMQReflectionSPI
sysTest.destDistro=equal
sysTest.clientPrefix=JmsConsumer
sysTest.numClients=25

consumer.destName=queue://PERF.TEST

factory.brokerURL=
factory.userName=
factory.password=

In both files, provide the brokerURL, username and password as they are required before starting the benchmarking process. The brokerURL and username can be obtained from the Amazon MQ console

Amazon MQ broker

Amazon MQ broker

Once you click into the deployed broker, you will find the brokerURL under the Endpoints section for OpenWire.

Endpoints in Amazon MQ console

Endpoints in Amazon MQ console

The endpoint URL for OpenWire should be in this format:

failover:(ssl://b-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-1.mq.<aws region>.amazonaws.com:61617,ssl://b-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -2.mq.<aws region>.amazonaws.com:61617)
Retrieve username from Amazon MQ console

Retrieve username from Amazon MQ console

Since you are using an active/standby broker, the test would only leverage the active broker URL and not both. The failover protocol automatically manages this exchange. The password can be retrieved from the AWS Secrets Manager console or via the CLI.

The following parameters and values can be adjusted in both producer and consumer properties file to suite your use case:

  • sendDuration: The sendDuration which represents the time taken for the producer/consumer test to run. Default value is set to 300000ms.
  • messageSize: The messageSize which adjusts the size of messages sent is set to 1024KB by default.
  • deliveryMode: The deliveryMode is set to persistent by default.
  • numClients: numClients sets the number of concurrent consumers, influencing message processing speed. It is set to 25 by default.
  • destName: destName represents the name of your destination queue or topic. You can change the name to your preference.

For a more comprehensive guide, refer to the mq-benchmarking-sample documentation.

2.8 Benchmark Results

After populating both producer and consumer files with the required parameters, run the following maven commands (one after the other) in separate terminals to start the test:

Maven producer command:

mvn activemq-perf:producer -DsysTest.propsConfigFile=openwire-producer.properties

Maven consumer command:

mvn activemq-perf:consumer -DsysTest.propsConfigFile=openwire-consumer.properties

Once each of the above tests complete, they provide a summary of the tests in stdout as shown below:

#########################################
####    SYSTEM THROUGHPUT SUMMARY    ####
#########################################
System Total Throughput: 562020
System Total Clients: 25
System Average Throughput: 1873.4000000000003
System Average Throughput Excluding Min/Max: 1860.8333333333333
System Average Client Throughput: 74.936
System Average Client Throughput Excluding Min/Max: 74.43333333333334
Min Client Throughput Per Sample: clientName=JmsProducer19, value=2
Max Client Throughput Per Sample: clientName=JmsProducer13, value=169
Min Client Total Throughput: clientName=JmsProducer0, value=20224
Max Client Total Throughput: clientName=JmsProducer5, value=23917
Min Average Client Throughput: clientName=JmsProducer0, value=67.41333333333333
Max Average Client Throughput: clientName=JmsProducer5, value=79.72333333333333
Min Average Client Throughput Excluding Min/Max: clientName=JmsProducer0, value=67.04333333333334
Max Average Client Throughput Excluding Min/Max: clientName=JmsProducer8, value=78.91
[main] INFO org.apache.activemq.tool.reports.XmlFilePerfReportWriter - Created performance report: /app/activemq-perftest/./reports/JmsProducer_numClients25_numDests1_all.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:05.052s
[INFO] Finished at: Mon Apr 29 10:22:01 UTC 2024
[INFO] Final Memory: 15M/60M
[INFO] ------------------------------------------------------------------------
#########################################
####    SYSTEM THROUGHPUT SUMMARY    ####
#########################################
System Total Throughput: 562023
System Total Clients: 25
System Average Throughput: 1873.4100000000005
System Average Throughput Excluding Min/Max: 1864.6599999999996
System Average Client Throughput: 74.93640000000002
System Average Client Throughput Excluding Min/Max: 74.58639999999998
Min Client Throughput Per Sample: clientName=JmsConsumer13, value=0
Max Client Throughput Per Sample: clientName=JmsConsumer13, value=105
Min Client Total Throughput: clientName=JmsConsumer13, value=22475
Max Client Total Throughput: clientName=JmsConsumer14, value=22495
Min Average Client Throughput: clientName=JmsConsumer13, value=74.91666666666667
Max Average Client Throughput: clientName=JmsConsumer14, value=74.98333333333333
Min Average Client Throughput Excluding Min/Max: clientName=JmsConsumer13, value=74.56666666666666
Max Average Client Throughput Excluding Min/Max: clientName=JmsConsumer14, value=74.63333333333334
[main] INFO org.apache.activemq.tool.reports.XmlFilePerfReportWriter - Created performance report: /app/activemq-perftest/./reports/JmsConsumer_numClients25_numDests1_equal.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:02.434s
[INFO] Finished at: Mon Apr 29 10:22:02 UTC 2024
[INFO] Final Memory: 14M/68M
[INFO] ------------------------------------------------------------------------

The above output is from a test performed and shown as sample output.

System Average Throughput and System Total Clients are the most useful metrics.

In the reports directory look for two xml files with more detailed throughput metrics. In the JmsProducer_numClients25_numDests1_all.xml file for example, jmsClientSettings and jmsFactorySettings captures different broker switches.

Each of the report files captures exact test and broker environment. Keeping these files around will allow you to compare performance between different test cases and analyze how a set of configurations have impacted performance.

For this test, the average throughput for a producer is around 1873 messages per second for 25 clients. Keep in mind that the broker instance is an mq.m5.large. You can get higher throughput with more clients and a larger broker instance. This test demonstrates the concept of running fast consumers while producing messages.

More comprehensive information on the test output can be found in performance testing.

By following these guidelines and leveraging ECS Exec for direct access, you can deploy the ActiveMQ Classic Maven Performance test plugin, using AWS CDK. This setup allows you to customize and execute benchmark tests on Amazon MQ within an ECS task, facilitating an automated and efficient deployment and testing workflow.

Amazon MQ benchmarking architecture

Amazon MQ for ActiveMQ brokers can be deployed as a single-instance broker or as an active/standby broker. Amazon MQ is architected for high availability (HA) and durability. For HA and broker benchmarking, AWS recommends using the active/standby deployment. After a message is sent to Amazon MQ in persistent mode, the message is written to the highly durable message store which replicates the data across multiple nodes and Availability Zones.

Cleanup

To avoid incurring future charges for the resources deployed in this walkthrough, run the following command and follow the prompts to delete the CloudFormation stacks launched in 2.6 Customize and Deploy:

cdk destroy "*"

Conclusion

This post provides a detailed guide on performing benchmarking for Amazon MQ for ActiveMQ brokers leveraging the ActiveMQ Classic Maven Performance test plugin. Benchmarking plays a crucial role for customers migrating to Amazon MQ, as it offers insights into the broker’s performance under conditions that mirror their existing setup. This process enables customers to fine-tune their configurations and choose the appropriate instance type that aligns with their specific use case, ensuring optimal handling of their workloads’ throughput.

Get started with Amazon MQ by using the AWS Management Console, AWS CLI, AWS Software Development Kit (SDK), or AWS CloudFormation. For information on cost, see Amazon MQ pricing.

Introducing quorum queues on Amazon MQ for RabbitMQ

Post Syndicated from Chris Munns original https://aws.amazon.com/blogs/compute/introducing-quorum-queues-on-amazon-mq-for-rabbitmq/

This post is written by Vignesh Selvam (Senior Product Manager – Amazon MQ), Simon Unge (Senior software development engineer – Amazon MQ).

Amazon MQ for RabbitMQ announced support for quorum queues, a type of replicated queue designed for higher availability and data safety. This post presents an overview of this queue type, describes when you should use it, and best practices you can follow. The post also describes how Amazon MQ has also improved quorum queues in the open-source RabbitMQ community.

Overview of quorum queues

A quorum queue is a replicated first in, first out queue type offered by open-source RabbitMQ that uses the Raft consensus algorithm to maintain data consistency. Each quorum queue has a leader and multiple followers (replicas), which ensure that messages are replicated and persisted across a majority of nodes, thus providing resilience against node failures. Quorum queues only need a majority of member nodes (a quorum) to make decisions about data. If a RabbitMQ node hosting a leader becomes unavailable, another node hosting one of the followers is automatically elected as the leader. Once the node becomes available again, the node will become a follower for the quorum queue and catch up or synchronize with the new leader. Quorum queues can detect network failures faster and recover quicker than classic mirrored queues, thus improving the resiliency of the message broker as a whole.

Quorum queues share most of the fundamental features that are key to RabbitMQ replicated queue types such as consumption, consumer acknowledgements, cancelling consumers, purging and deletion. Poison message handling is a unique feature of quorum queues which help developers manage unprocessed messages more efficiently. A poison message is a message that cannot be processed and ends up being repeatedly requeued. Quorum queues keep track of the number of unsuccessful delivery attempts and expose it in the ‘x-delivery-count’ header that is included with any redelivered message. A delivery limit can be set using a policy argument for ’delivery-limit’. If the limit is reached, the message can be dropped or put in a dead-letter queue. This feature further improves the data reliability of a quorum queue.

You can get started with quorum queues by explicitly specifying the ‘x-queue-type’ parameter as ’quorum’ on a RabbitMQ broker running version 3.13 and above. We recommend that you change the default vhost queue type to ’quorum’ to ensure that all queues are created as quorum queues by default inside a vhost.

RabbitMQ queues console

RabbitMQ queues console

When should you use quorum queues?

You should use quorum queues when you need higher availability and consistency for their messaging infrastructure. Quorum queues are ideal for scenarios where data durability and fault tolerance are critical, such as financial transaction systems, e-commerce data processing systems, or any application requiring high reliability. They are particularly beneficial in environments where node failures are more likely or where maintaining data consistency across distributed systems is essential.

When should you NOT use quorum queues?

Quorum queues are not meant to be temporary. They do not support transient or exclusive queues and are not meant to be used in scenarios with high queue churn (declaration and deletion rates). They are also not recommended for unreplicated queues.

Best practices for quorum queues

Quorum queues perform better when the queues are short. You can set the maximum queue length using a policy or queue arguments to limit the total memory usage by queues (max-length, max-length-bytes).

Add a new queue dialog

Add a new queue dialog

Amazon MQ recommends publishers to use publisher confirms and consumers to use manual acknowledgements on quorum queues. Publisher confirms will only be issued once a published message has been successfully replicated to a quorum of nodes and is considered safe within the context of the system. Publisher confirms can also serve as a form of back pressure and protect the availability of the broker during periods of high workload. Manual acknowledgements are used to ensure messages that are not processed can be returned to the queue for reprocessing.

Open-source improvements by Amazon MQ

Amazon MQ contributed multiple improvements to the open-source RabbitMQ community to improve quorum queues for operators and users.

Automatic membership reconciliation
Quorum queues depend on a majority of replicas being available for the Raft consensus algorithm. Amazon MQ identified that many users and operators would prefer to maintain a certain minimal number of replicas (generally 3 or 5) at all times to ensure a majority always exists. The quorum queue replica management was also initially available only via CLI tools. Amazon MQ engineers introduced automatic membership reconciliation to improve this experience. Now, RabbitMQ can be configured to identify any queues that are below a target group member size, and automatically grow or add a node to the queue members. Thus ensuring a certain minimum number of replicas always exist.

Voter status
RabbitMQ considers a quorum queue member node to be a full member even if the member has not caught up or fully synced to the quorum. The CLI command rabbitmq-queues check_if_node_is_ quorum_critical can provide a false positive, and indicate a node is safe to remove, even though another node has queue members that are still synchronizing to the quorum. Amazon MQ introduced a new ‘non-voter’ state for a queue member node to indicate a member that is still catching up or synchronizing to the quorum. If a queue has a member in this state, it is not considered a full member. Once the member is fully synchronized, it is automatically promoted to the voter status, and is considered a full member. The command rabbitmq-queues check_if_node_is_quorum_critical now takes this into account and correctly reports if a node can be safely terminated without any queues becoming unavailable due to a loss of majority.

Inconsistent state management
When a broker is overloaded, a quorum queue can end up in an inconsistent state, where the quorum queue membership state stored in the Raft state machine differs from the RabbitMQ internal state for the queue. Amazon MQ introduced a periodic check per quorum queue that identifies if a queue has an inconsistent state and takes action to fix it.

Default queue type
The default queue type for a RabbitMQ broker vhost was classic queues. You could declare a different queue type by explicitly stating the ’x-queue-type’ as a queue creation argument. Amazon MQ introduced a global default queue type in the configuration file (rabbit.conf) that provides the ability to define a default queue type at the broker level. Now, an operator can change the default queue type to quorum queues if not specified during creation.

Membership management permissions
RabbitMQ users are able to configure the quorum queue membership using the management API. This can interfere with automatic membership reconciliation. Amazon MQ introduced the ability for an operator to turn off the membership management permissions available through the management API. Thus, preventing customers from accidentally affecting their broker.

Conclusion

Quorum queues on RabbitMQ provide a robust solution for scenarios requiring high availability and resilience. By leveraging the Raft consensus protocol, quorum queues ensure that messages are safely stored and replicated across a quorum of nodes, making them an excellent choice for modern, distributed message queuing systems.

Amazon MQ recommends that you adopt quorum queues as the preferred replicated queue type on RabbitMQ 3.13 brokers. For more details, see Amazon MQ documentation. To know more about the open-source feature, see quorum queues.

Get started with quorum queues on Amazon MQ for RabbitMQ 3.13 with a few clicks.

AWS Weekly Roundup: Global AWS Heroes Summit, AWS Lambda, Amazon Redshift, and more (July 22, 2024)

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-global-aws-heroes-summit-aws-lambda-amazon-redshift-and-more-july-22-2024/

Last week, AWS Heroes from around the world gathered to celebrate the 10th anniversary of the AWS Heroes program at Global AWS Heroes Summit. This program recognizes a select group of AWS experts worldwide who go above and beyond in sharing their knowledge and making an impact within developer communities.

Matt Garman, CEO of AWS and a long-time supporter of developer communities, made a special appearance for a Q&A session with the Heroes to listen to their feedback and respond to their questions.

Here’s an epic photo from the AWS Heroes Summit:

As Matt mentioned in his Linkedin post, “The developer community has been core to everything we have done since the beginning of AWS.” Thank you, Heroes, for all you do. Wishing you all a safe flight home.

Last week’s launches
Here are some launches that caught my attention last week:

Announcing the July 2024 updates to Amazon Corretto — The latest updates for the Corretto distribution of OpenJDK is now available. This includes security and critical updates for the Long-Term Supported (LTS) and Feature (FR) versions.

New open-source Advanced MYSQL ODBC Driver now available for Amazon Aurora and RDS — The new AWS ODBC Driver for MYSQL provides faster switchover and failover times, and authentication support for AWS Secrets Manager and AWS Identity and Access Management (IAM), making it a more efficient and secure option for connecting to Amazon RDS and Amazon Aurora MySQL-compatible edition databases.

Productionize Fine-tuned Foundation Models from SageMaker Canvas — Amazon SageMaker Canvas now allows you to deploy fine-tuned Foundation Models (FMs) to SageMaker real-time inference endpoints, making it easier to integrate generative AI capabilities into your applications outside the SageMaker Canvas workspace.

AWS Lambda now supports SnapStart for Java functions that use the ARM64 architecture — Lambda SnapStart for Java functions on ARM64 architecture delivers up to 10x faster function startup performance and up to 34% better price performance compared to x86, enabling the building of highly responsive and scalable Java applications using AWS Lambda.

Amazon QuickSight improves controls performance — Amazon QuickSight has improved the performance of controls, allowing readers to interact with them immediately without having to wait for all relevant controls to reload. This enhancement reduces the loading time experienced by readers.

Amazon OpenSearch Serverless levels up speed and efficiency with smart caching — The new smart caching feature for indexing in Amazon OpenSearch Serverless automatically fetches and manages data, leading to faster data retrieval, efficient storage usage, and cost savings.

Amazon Redshift Serverless with lower base capacity available in the Europe (London) Region — Amazon Redshift Serverless now allows you to start with a lower data warehouse base capacity of 8 Redshift Processing Units (RPUs) in the Europe (London) region, providing more flexibility and cost-effective options for small to large workloads.

AWS Lambda now supports Amazon MQ for ActiveMQ and RabbitMQ in five new regions — AWS Lambda now supports Amazon MQ for ActiveMQ and RabbitMQ in five new regions, enabling you to build serverless applications with Lambda functions that are invoked based on messages posted to Amazon MQ message brokers.

From community.aws
Here’s my top 5 personal favorites posts from community.aws:

Upcoming AWS events
Check your calendars and sign up for upcoming AWS events:

AWS Summits — Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. To learn more about future AWS Summit events, visit the AWS Summit page. Register in your nearest city: AWS Summit Taipei (July 23–24), AWS Summit Mexico City (Aug. 7), and AWS Summit Sao Paulo (Aug. 15).

AWS Community Days — Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world. Upcoming AWS Community Days are in Aotearoa (Aug. 15), Nigeria (Aug. 24), New York (Aug. 28), and Belfast (Sept. 6).

You can browse all upcoming in-person and virtual events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

Donnie

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Introducing Amazon MQ cross-Region data replication for ActiveMQ brokers

Post Syndicated from Pascal Vogel original https://aws.amazon.com/blogs/compute/introducing-amazon-mq-cross-region-data-replication-for-activemq-brokers/

This post is written by Dominic Gagné, Senior Software Development Engineer, and Vinodh Kannan Sadayamuthu, Senior Solutions Architect

Amazon MQ now supports cross-Region data replication for ActiveMQ brokers. This feature enables you to build regionally resilient messaging applications and makes it easier to set up cross-Region message replication between ActiveMQ brokers in Amazon MQ. This blog post explains how cross-Region data replication works in Amazon MQ, how to setup cross-Region replica brokers for ActiveMQ, and how to test promoting a replica broker.

Amazon MQ is a managed message broker service for Apache ActiveMQ and RabbitMQ that simplifies setting up and operating message brokers on AWS.

Cross-Region replication improves the resilience and disaster recovery capabilities of your systems. This new Amazon MQ feature makes it easier to increase resilience of your ActiveMQ messaging systems across AWS Regions.

How cross-Region data replication works in Amazon MQ for ActiveMQ

The Amazon MQ for ActiveMQ cross-Region data replication feature replicates broker state from the primary broker in one AWS Region to the replica broker in another Region. Broker state consists of messages that have been sent to a broker by a message producer. Additionally, message acknowledgments and transactions are replicated. Scheduled messages and broker XML configuration are not replicated from the primary to the replica broker.

State replication occurs asynchronously and runs in the background. When a message is sent to a cross-Region data replication enabled broker, the data is persisted both to the primary data store and also on a queue used to replicate data. The replica broker acts as a client of this queue and consumes data that represents broker state from the primary broker.

At any given moment, only the primary broker is available for client connections. The replica broker is a hot standby and passively replicates the primary broker’s state. However, it does not accept client connections. The following diagram shows a simplified version of a cross-Region data replication broker pair. All replication traffic is encrypted using TLS and remains within AWS’ private backbone.

Amazon MQ for ActiveMQ cross-region data replication architecture

Configuring cross-Region replica brokers for Amazon MQ for ActiveMQ

To set up a cross-Region replica broker, your Amazon MQ for ActiveMQ primary broker must meet the following eligibility criteria:

  • ActiveMQ version 5.17.6 or above
  • Instance size m5.large or higher
  • Active/standby broker deployment enabled
  • Be in the Running state

If you do not have an ActiveMQ broker that meets these criteria, see Creating and configuring an ActiveMQ broker for instructions on how to create a primary broker.

To configure cross-Region replication

  1. Navigate to the Amazon MQ console and choose Create replica broker.
    Amazon MQ console create replica broker
  2. Select a primary broker from the list of eligible primary brokers and choose Next.
    Amazon MQ console choose primary broker
  3. Under Replica broker details, select the Region for your replica broker and enter a Replica broker name.
    Amazon MQ console configure replica broker
  4. In the ActiveMQ console user for replica broker panel, enter a Username and Password for broker access.
    Amazon MQ console user for replica broker
  5. In the Data replication user to bridge access between brokers panel, enter a replication user Username and Password.
    Amazon MQ console user for replica broker
  6. In the Additional settings panel, keep the defaults and choose Next.
  7. Review the settings and choose Create replica broker.
    Note: The broker access type is automatically set based on the primary broker access type.
    Amazon MQ console create replica broker setting summary
  8. The creation process takes up to 25 minutes. Once the replica broker creation is complete, begin replication between the primary and the replica brokers by rebooting the primary broker.
  9. Once the primary broker is rebooted and its status is Running, you can see the replica details in the Data replication panel of the primary broker.
    Amazon MQ console broker replication details

Both brokers now synchronize with each other to establish an inter-Region network and connection through which broker state is replicated. Once both brokers are in the Running state, the primary broker accepts client connections and passes all broker state changes (messages, acknowledgments, transactions, etc.) to the replica broker.

The replica broker now asynchronously mirrors the state of the primary broker. However, it does not become available for client connections until it is promoted via a switchover or a failover. These operations are covered in the following section.

Testing data replication and promoting the replica broker

There are two ways to promote a replica broker: initiating a switchover or a failover.

Switchover Failover
  • Prioritizes consistency over availability.
  • Prioritizes availability over consistency.
  • Brokers are guaranteed to have identical states.
  • Brokers are not guaranteed to be in identical states.
  • Brokers may not be available immediately to serve client traffic.
  • Replica broker is immediately available to serve client traffic.

To initiate a failover or switchover

    1. Navigate to the Amazon MQ console, choose your primary broker, and log in to the ActiveMQ Web Console using the URLs located in the Connections panel.
    2. In the top menu, select Queues. You should be able to see four ActiveMQ.Plugin.Replication queues used by the replication feature.
      Active MQ console queues
    3. To test message replication from the primary to a replica broker, create a queue and send messages. To create the queue:
      • For Queue Name, enter TestQueue.
      • Choose Create.

      ActiveMQ console create queue

    4. Under Operations for the TestQueue, choose Send To and perform the following steps:
      • For Number of messages to send, enter 10 and keep the other defaults.
      • Under Message body, enter a test message.
      • Choose Send.

      ActiveMQ console send test message

    5. To promote the replica broker, navigate to the Amazon MQ console and change the Region to the AWS Region where the replica broker is located.
    6. Select the replica broker (in this example called Secondarybroker) and choose Promote replica.
      Amazon MQ console promote broker
    7. In the Promote replica broker pop-up window:
      • Select Failover or Switchover.
      • Enter confirm in text box.
      • Choose Confirm.

      Amazon MQ console confirm broker promotion

    8. While a replica broker is being promoted, its replication status changes to Promotion in progress. The corresponding primary broker’s replication status changes to Demotion in progress.

Replica Secondarybroker status – Promotion in progress:

Replica Secondarybroker status - Promotion in progress

Primary broker status – Demotion in progress:

Primary broker status - Demotion in progress

Secondarybroker status – Promoted to new primary broker:

Secondarybroker status – Promoted to new primary broker

  1. Once the Secondarybroker status is Running, log in to the ActiveMQ Web Console from the URLs located in the Connections panel. You can see the replicated messages sent from the former primary broker in Step 4 in the TestQueue:
    Replicated message from primary broker in TestQueue

Monitoring cross-Region data replication

To monitor cross-Region data replication progress, you can use the Amazon CloudWatch metrics TotalReplicationLag and ReplicationLag.

Amazon CloudWatch metrics TotalReplicationLag and ReplicationLag

You can use these two metrics to monitor the progress of a switchover. When their value reaches zero, the switchover will complete because the broker states have been synchronized and the replica broker begins accepting client connections. If the switchover does not progress fast enough, or if you need the replica broker to be immediately available to serve client traffic, you can request a failover at any time.

Note: A failover can interrupt an ongoing switchover. However, a switchover cannot interrupt an ongoing failover.

Issuing a failover request causes the replica broker to become immediately available, but does not provide any guarantees about what data has been replicated to the replica broker. This means that a failover can make data tracking and reconciliation more challenging for your client application than a switchover.

For this reason, we recommend that you always start with a switchover and interrupt it with a failover if necessary. To interrupt an ongoing switchover, follow the same steps as for promoting a replica broker, select the failover option, and confirm.

Note: If you fail back to the original primary broker, messages that are not replicated from the primary to the replica broker during the failover will still exist on the primary broker. Therefore, consumers must manage these messages. We recommend tracking the processed message IDs in a data store such as Amazon DynamoDB global tables and comparing the message to the processed message IDs.

If you no longer need to replicate broker data across Regions or if you need to delete a primary or replica broker, you must unpair the replica broker and reboot the primary broker. You can unpair the replica broker in the Amazon MQ console by following Delete a CRDR broker.

To unpair the broker using the AWS Command Line Interface (AWS CLI), run the following command, replacing the --broker-id with your primary broker ID:

aws mq update-broker --broker-id <primary broker ID> \
--data-replication-mode "NONE" \
--region us-east-1

Conclusion

Using the cross-Region data replication feature for Amazon MQ for ActiveMQ provides a straightforward way to implement cross-Region replication to improve the resilience of your architecture and meet your business continuity and disaster recovery requirements. This post explains how cross-Region data replication works in Amazon MQ, how to set up a cross-Region replica broker, and how to test and promote the replica broker.

For more details, see the Amazon MQ documentation.

For more serverless learning resources, visit Serverless Land.

Integrating IBM MQ with Amazon SQS and Amazon SNS using Apache Camel

Post Syndicated from Pascal Vogel original https://aws.amazon.com/blogs/compute/integrating-ibm-mq-with-amazon-sqs-and-amazon-sns-using-apache-camel/

This post is written by Joaquin Rinaudo, Principal Security Consultant and Gezim Musliaj, DevOps Consultant.

IBM MQ is a message-oriented middleware (MOM) product used by many enterprise organizations, including global banks, airlines, and healthcare and insurance companies.

Customers often ask us for guidance on how they can integrate their existing on-premises MOM systems with new applications running in the cloud. They’re looking for a cost-effective, scalable and low-effort solution that enables them to send and receive messages from their cloud applications to these messaging systems.

This blog post shows how to set up a bi-directional bridge from on-premises IBM MQ to Amazon MQ, Amazon Simple Queue Service (Amazon SQS), and Amazon Simple Notification Service (Amazon SNS).

This allows your producer and consumer applications to integrate using fully managed AWS messaging services and Apache Camel. Learn how to deploy such a solution and how to test the running integration using SNS, SQS, and a demo IBM MQ cluster environment running on Amazon Elastic Container Service (ECS) with AWS Fargate.

This solution can also be used as part of a step-by-step migration using the approach described in the blog post Migrating from IBM MQ to Amazon MQ using a phased approach.

Solution overview

The integration consists of an Apache Camel broker cluster that bi-directionally integrates an IBM MQ system and target systems, such as Amazon MQ running ActiveMQ, SNS topics, or SQS queues.

In the following example, AWS services, in this case AWS Lambda and SQS, receive messages published to IBM MQ via an SNS topic:

Solution architecture overview for sending messages

  1. The cloud message consumers (Lambda and SQS) subscribe to the solution’s target SNS topic.
  2. The Apache Camel broker connects to IBM MQ using secrets stored in AWS Secrets Manager and reads new messages from the queue using IBM MQ’s Java library. Only IBM MQ messages are supported as a source.
  3. The Apache Camel broker publishes these new messages to the target SNS topic. It uses the Amazon SNS Extended Client Library for Java to store any messages larger than 256 KB in an Amazon Simple Storage Service (Amazon S3) bucket.
  4. Apache Camel stores any message that cannot be delivered to SNS after two retries in an S3 dead letter queue bucket.

The next diagram demonstrates how the solution sends messages back from an SQS queue to IBM MQ:

Solution architecture overview for sending messages

  1. A sample message producer using Lambda sends messages to an SQS queue. It uses the Amazon SQS Extended Client Library for Java to send messages larger than 256 KB.
  2. The Apache Camel broker receives the messages published to SQS, using the SQS Extended Client Library if needed.
  3. The Apache Camel broker sends the message to the IBM MQ target queue.
  4. As before, the broker stores messages that cannot be delivered to IBM MQ in the S3 dead letter queue bucket.

A phased live migration consists of two steps:

  1. Deploy the broker service to allow reading messages from and writing to existing IBM MQ queues.
  2. Once the consumer or producer is migrated, migrate its counterpart to the newly selected service (SNS or SQS).

Next, you will learn how to set up the solution using the AWS Cloud Development Kit (AWS CDK).

Deploying the solution

Prerequisites

  • AWS CDK
  • TypeScript
  • Java
  • Docker
  • Git
  • Yarn

Step 1: Cloning the repository

Clone the repository using git:

git clone https://github.com/aws-samples/aws-ibm-mq-adapter

Step 2: Setting up test IBM MQ credentials

This demo uses IBM MQ’s mutual TLS authentication. To do this, you must generate X.509 certificates and store them in AWS Secrets Manager by running the following commands in the app folder:

  1. Generate X.509 certificates:
    ./deploy.sh generate_secrets
  2. Set up the secrets required for the Apache Camel broker (replace <integration-name> with, for example, dev):
    ./deploy.sh create_secrets broker <integration-name>
  3. Set up secrets for the mock IBM MQ system:
    ./deploy.sh create_secrets mock
  4. Update the cdk.json file with the secrets ARN output from the previous commands:
    • IBM_MOCK_PUBLIC_CERT_ARN
    • IBM_MOCK_PRIVATE_CERT_ARN
    • IBM_MOCK_CLIENT_PUBLIC_CERT_ARN
    • IBMMQ_TRUSTSTORE_ARN
    • IBMMQ_TRUSTSTORE_PASSWORD_ARN
    • IBMMQ_KEYSTORE_ARN
    • IBMMQ_KEYSTORE_PASSWORD_ARN

If you are using your own IBM MQ system and already have X.509 certificates available, you can use the script to upload those certificates to AWS Secrets Manager after running the script.

Step 3: Configuring the broker

The solution deploys two brokers, one to read messages from the test IBM MQ system and one to send messages back. A separate Apache Camel cluster is used per integration to support better use of Auto Scaling functionality and to avoid issues across different integration operations (consuming and reading messages).

Update the cdk.json file with the following values:

  • accountId: AWS account ID to deploy the solution to.
  • region: name of the AWS Region to deploy the solution to.
  • defaultVPCId: specify a VPC ID for an existing VPC in the AWS account where the broker and mock are deployed.
  • allowedPrincipals: add your account ARN (e.g., arn:aws:iam::123456789012:root) to allow this AWS account to send messages to and receive messages from the broker. You can use this parameter to set up cross-account relationships for both SQS and SNS integrations and support multiple consumers and producers.

Step 4: Bootstrapping and deploying the solution

  1. Make sure you have the correct AWS_PROFILE and AWS_REGION environment variables set for your development account.
  2. Run yarn cdk bootstrap –-qualifier mq <aws://<account-id>/<region> to bootstrap CDK.
  3. Run yarn install to install CDK dependencies.
  4. Finally, execute yarn cdk deploy '*-dev' –-qualifier mq --require-approval never to deploy the solution to the dev environment.

Step 5: Testing the integrations

Use AWS System Manager Session Manager and port forwarding to establish tunnels to the test IBM MQ instance to access the web console and send messages manually. For more information on port forwarding, see Amazon EC2 instance port forwarding with AWS System Manager.

  1. In a command line terminal, make sure you have the correct AWS_PROFILE and AWS_REGION environment variables set for your development account.
  2. In addition, set the following environment variables:
    • IBM_ENDPOINT: endpoint for IBM MQ. Example: network load balancer for IBM mock mqmoc-mqada-1234567890.elb.eu-west-1.amazonaws.com.
    • BASTION_ID: instance ID for the bastion host. You can retrieve this output from Step 4: Bootstrapping and deploying the solution listed after the mqBastionStack deployment.

    Use the following command to set the environment variables:

    export IBM_ENDPOINT=mqmoc-mqada-1234567890.elb.eu-west-1.amazonaws.com
    export BASTION_ID=i-0a1b2c3d4e5f67890
  3. Run the script test/connect.sh.
  4. Log in to the IBM web console via https://127.0.0.1:9443/admin using the default IBM user (admin) and the password stored in AWS Secrets Manager as mqAdapterIbmMockAdminPassword.

Sending data from IBM MQ and receiving it in SNS:

  1. In the IBM MQ console, access the local queue manager QM1 and DEV.QUEUE.1.
  2. Send a message with the content Hello AWS. This message will be processed by AWS Fargate and published to SNS.
  3. Access the SQS console and choose the snsIntegrationStack-dev-2 prefix queue. This is an SQS queue subscribed to the SNS topic for testing.
  4. Select Send and receive message.
  5. Select Poll for messages to see the Hello AWS message previously sent to IBM MQ.

Sending data back from Amazon SQS to IBM MQ:

  1. Access the SQS console and choose the queue with the prefix sqsPublishIntegrationStack-dev-3-dev.
  2. Select Send and receive messages.
  3. For Message Body, add Hello from AWS.
  4. Choose Send message.
  5. In the IBM MQ console, access the local queue manager QM1 and DEV.QUEUE.2 to find your message listed under this queue.

Step 6: Cleaning up

Run cdk destroy '*-dev' to destroy the resources deployed as part of this walkthrough.

Conclusion

In this blog, you learned how you can exchange messages between IBM MQ and your cloud applications using Amazon SQS and Amazon SNS.

If you’re interested in getting started with your own integration, follow the README file in the GitHub repository. If you’re migrating existing applications using industry-standard APIs and protocols such as JMS, NMS, or AMQP 1.0, consider integrating with Amazon MQ using the steps provided in the repository.

If you’re interested in running Apache Camel in Kubernetes, you can also adapt the architecture to use Apache Camel K instead.

For more serverless learning resources, visit Serverless Land.

How Munich Re Automation Solutions Ltd built a digital insurance platform on AWS

Post Syndicated from Sid Singh original https://aws.amazon.com/blogs/architecture/how-munich-re-automation-solutions-ltd-built-a-digital-insurance-platform-on-aws/

Underwriting for life insurance can be quite manual and often time-intensive with lots of re-keying by advisers before underwriting decisions can be made and policies finally issued. In the digital age, people purchasing life insurance want self-service interactions with their prospective insurer. People want speed of transaction with time to cover reduced from days to minutes. While this has been achieved in the general insurance space with online car and home insurance journeys, this is not always the case in the life insurance space. This is where Munich Re Automation Solutions Ltd (MRAS) offers its customers, a competitive edge to shrink the quote-to-fulfilment process using their ALLFINANZ solution.

ALLFINANZ is a cloud-based life insurance and analytics solution to underwrite new life insurance business. It is designed to transform the end consumer’s journey, delivering everything they need to become a policyholder. The core digital services offered to all ALLFINANZ customers include Rulebook Hub, Risk Assessment Interview delivery, Decision Engine, deep analytics (including predictive modeling capabilities), and technical integration services—for example, API integration and SSO integration.

Current state architecture

The ALLFINANZ application began as a traditional three-tier architecture deployed within a datacenter. As MRAS migrated their workload to the AWS cloud, they looked at their regulatory requirements and the technology stack, and decided on the silo model of the multi-tenant SaaS system. Each tenant is provided a dedicated Amazon Virtual Private Cloud (VPC) that holds network and application components, fully isolated from other primary insurers.

As an entry point into the ALLFINANZ environment, MRAS uses Amazon Route 53 to route incoming traffic to the appropriate Amazon VPC. The routing relies on a model where subdomains are assigned to each tenant, for example the subdomain allfinanz.tenant1.munichre.cloud is the subdomain for tenant 1. The diagram below shows the ALLFINANZ architecture. Note: not all links between components are shown here for simplicity.

Current high-level solution architecture for the ALLFINANZ solution

Figure 1. Current high-level solution architecture for the ALLFINANZ solution

  1. The solution uses Route 53 as the DNS service, which provides two entry points to the SaaS solution for MRAS customers:
    • The URL allfinanz.<tenant-id>.munichre.cloud allows user access to the ALLFINANZ Interview Screen (AIS). The AIS can exist as a standalone application, or can be integrated with a customer’s wider digital point-of -sale process.
    • The URL api.allfinanz.<tenant-id>.munichre.cloud is used for accessing the application’s Web services and REST APIs.
  2. Traffic from both entry points flows through the load balancers. While HTTP/S traffic from the application user access entry point flows through an Application Load Balancer (ALB), TCP traffic from the REST API clients flows through a Network Load Balancer (NLB). Transport Layer Security (TLS) termination for user traffic happens at the ALB using certificates provided by the AWS Certificate Manager.  Secure communication over the public network is enforced through TLS validation of the server’s identity.
  3. Unlike application user access traffic, REST API clients use mutual TLS authentication to authenticate a customer’s server. Since NLB doesn’t support mutual TLS, MRAS opted for a solution to pass this traffic to a backend NGINX server for the TLS termination. Mutual TLS is enforced by using self-signed client and server certificates issued by a certificate authority that both the client and the server trust.
  4. Authenticated traffic from ALB and NGINX servers is routed to EC2 instances hosting the application logic. These EC2 instances are hosted in an auto-scaling group spanning two Availability Zones (AZs) to provide high availability and elasticity, therefore, allowing the application to scale to meet fluctuating demand.
  5. Application transactions are persisted in the backend Amazon Relational Database Service MySQL instances. This database layer is configured across multi-AZs, providing high availability and automatic failover.
  6. The application requires the capability to integrate evidence from data sources external to the ALLFINANZ service. This message sharing is enabled through the Amazon MQ managed message broker service for Apache Active MQ.
  7. Amazon CloudWatch is used for end-to-end platform monitoring through logs collection and application and infrastructure metrics and alerts to support ongoing visibility of the health of the application.
  8. Software deployment and associated infrastructure provisioning is automated through infrastructure as code using a combination of Git, Amazon CodeCommit, Ansible, and Terraform.
  9. Amazon GuardDuty continuously monitors the application for malicious activity and delivers detailed security findings for visibility and remediation. GuardDuty also allows MRAS to provide evidence of the application’s strong security posture to meet audit and regulatory requirements.

High availability, resiliency, and security

MRAS deploys their solution across multiple AWS AZs to meet high-availability requirements and ensure operational resiliency. If one AZ has an ongoing event, the solution will remain operational, as there are instances receiving production traffic in another AZ. As described above, this is achieved using ALBs and NLBs to distribute requests to the application subnets across AZs.

The ALLFINANZ solution uses private subnets to segregate core application components and the database storage platform. Security groups provide networking security measures at the elastic network interface level. MRAS restrict access from incoming connection requests to ranges of IP addresses by attaching security groups to the ALBs. Amazon Inspector monitors workloads for software vulnerabilities and unintended network exposure. AWS WAF is integrated with the ALB to protect from SQL injection or cross-site scripting attacks on the application.

Optimizing the existing workload

One of the key benefits of this architecture is that now MRAS can standardize the infrastructure configuration and ensure consistent versioning of the workload across tenants. This makes onboarding new tenants as simple as provisioning another VPC with the same infrastructure footprint.

MRAS are continuing to optimize their architecture iteratively, examining components to modernize to cloud-native components and evolving towards the pool model of multi-tenant SaaS architecture wherever possible. For example, MRAS centralized their per-tenant NAT gateway deployment to a centralized outbound Internet routing design using AWS Transit Gateway, saving approximately 30% on their overall NAT gateway spend.

Conclusion

The AWS global infrastructure has allowed MRAS to serve more than 40 customers in five AWS regions around the world. This solution improves customers’ experience and workload maintainability by standardizing and automating the infrastructure and workload configuration within a SaaS model, compared with multiple versions for the on-premise deployments. SaaS customers are also freed up from the undifferentiated heavy lifting of infrastructure operations, allowing them to focus on their business of underwriting for life insurance.

MRAS used the AWS Well-Architected Framework to assess their architecture and list key recommendations. AWS also offers Well-Architected SaaS Lens and AWS SaaS Factory Program, with a collection of resources to empower and enable insurers at any stage of their SaaS on AWS journey.