Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/choosing-between-messaging-services-for-serverless-applications/
Most serverless application architectures use a combination of different AWS services, microservices, and AWS Lambda functions. Messaging services are important in allowing distributed applications to communicate with each other, and are fundamental to most production serverless workloads.
Messaging services can improve the resilience, availability, and scalability of applications, when used appropriately. They can also enable your applications to communicate beyond your workload or even the AWS Cloud, and provide extensibility for future service features and versions.
In this blog post, I compare the primary messaging services offered by AWS and how you can use these in your serverless application architectures. I also show how you use and deploy these integrations with the AWS Serverless Application Model (AWS SAM).
Examples in this post refer to code that can be downloaded from this GitHub repository. The README.md file explains how to deploy and run each example.
Overview
Three of the most useful messaging patterns for serverless developers are queues, publish/subscribe, and event buses. In AWS, these are provided by Amazon SQS, Amazon SNS, and Amazon EventBridge respectively. All of these services are fully managed and highly available, so there is no infrastructure to manage. All three integrate with Lambda, allowing you to publish messages via the AWS SDK and invoke functions as targets. Each of these services has an important role to play in serverless architectures.
SNS enables you to send messages reliably between parts of your infrastructure. It uses a robust retry mechanism for when downstream targets are unavailable. When the delivery policy is exhausted, it can optionally send those messages to a dead-letter queue for further processing. SNS uses topics to logically separate messages into channels, and your Lambda functions interact with these topics.
SQS provides queues for your serverless applications. You can use a queue to send, store, and receive messages between different services in your workload. Queues are an important mechanism for providing fault tolerance in distributed systems, and help decouple different parts of your application. SQS scales elastically, and there is no limit to the number of messages per queue. The service durably persists messages until they are processed by a downstream consumer.
EventBridge is a serverless event bus service, simplifying routing events between AWS services, software as a service (SaaS) providers, and your own applications. It logically separates routing using event buses, and you implement the routing logic using rules. You can filter and transform incoming messages at the service level, and route events to multiple targets, including Lambda functions.
Integrating an SQS queue with AWS SAM
The first example shows an AWS SAM template defining a serverless application with two Lambda functions and an SQS queue:
You can declare an SQS queue in an AWS SAM template with the AWS::SQS::Queue resource:
MySqsQueue:
Type: AWS::SQS::Queue
To publish to the queue, the publisher function must have permission to send messages. Using an AWS SAM policy template, you can apply policy that enables send messaging to one specific queue:
Policies:
- SQSSendMessagePolicy:
QueueName: !GetAtt MySqsQueue.QueueName
The AWS SAM template passes the queue name into the Lambda function as an environment variable. The function uses the sendMessage method of the AWS.SQS class to publish the message:
const AWS = require('aws-sdk')
AWS.config.region = process.env.AWS_REGION
const sqs = new AWS.SQS({apiVersion: '2012-11-05'})
// The Lambda handler
exports.handler = async (event) => {
// Params object for SQS
const params = {
MessageBody: `Message at ${Date()}`,
QueueUrl: process.env.SQSqueueName
}
// Send to SQS
const result = await sqs.sendMessage(params).promise()
console.log(result)
}
When the SQS queue receives the message, it publishes to the consuming Lambda function. To configure this integration in AWS SAM, the consumer function is granted the SQSPollerPolicy policy. The function’s event source is set to receive messages from the queue in batches of 10:
QueueConsumerFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: code/
Handler: consumer.handler
Runtime: nodejs12.x
Timeout: 3
MemorySize: 128
Policies:
- SQSPollerPolicy:
QueueName: !GetAtt MySqsQueue.QueueName
Events:
MySQSEvent:
Type: SQS
Properties:
Queue: !GetAtt MySqsQueue.Arn
BatchSize: 10
The payload for the consumer function is the message from SQS. This is an array of messages up to the batch size, containing a body attribute with the publishing function’s MessageBody. You can see this in the CloudWatch log for the function:
Integrating an SNS topic with AWS SAM
The second example shows an AWS SAM template defining a serverless application with three Lambda functions and an SNS topic:
You declare an SNS topic and the subscribing Lambda functions with the AWS::SNS:Topic resource:
MySnsTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Protocol: lambda
Endpoint: !GetAtt TopicConsumerFunction1.Arn
- Protocol: lambda
Endpoint: !GetAtt TopicConsumerFunction2.Arn
You provide the SNS service with permission to invoke the Lambda functions but defining an AWS::Lambda::Permission for each:
TopicConsumerFunction1Permission:
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref TopicConsumerFunction1
Principal: sns.amazonaws.com
The SNSPublishMessagePolicy policy template grants permission to the publishing function to send messages to the topic. In the function, the publish method of the AWS.SNS class handles publishing:
const AWS = require('aws-sdk')
AWS.config.region = process.env.AWS_REGION
const sns = new AWS.SNS({apiVersion: '2012-11-05'})
// The Lambda handler
exports.handler = async (event) => {
// Params object for SNS
const params = {
Message: `Message at ${Date()}`,
Subject: 'New message from publisher',
TopicArn: process.env.SNStopic
}
// Send to SQS
const result = await sns.publish(params).promise()
console.log(result)
}
The payload for the consumer functions is the message from SNS. This is an array of messages, containing subject and message attributes from the publishing function. You can see this in the CloudWatch log for the function:
Differences between SQS and SNS configurations
SQS queues and SNS topics offer different functionality, though both can publish to downstream Lambda functions.
An SQS message is stored on the queue for up to 14 days until it is successfully processed by a subscriber. SNS does not retain messages so if there are no subscribers for a topic, the message is discarded.
SNS topics may broadcast to multiple targets. This behavior is called fan-out. It can be used to parallelize work across Lambda functions or send messages to multiple environments (such as test or development). An SNS topic can have up to 12,500,000 subscribers, providing highly scalable fan-out capabilities. The targets may include HTTP/S endpoints, SMS text messaging, SNS mobile push, email, SQS, and Lambda functions.
In AWS SAM templates, you can retrieve properties such as ARNs and names of queues and topics, using the following intrinsic functions:
| Amazon SQS | Amazon SNS | |
| Channel type | Queue | Topic |
| Get ARN | !GetAtt MySqsQueue.Arn | !Ref MySnsTopic |
| Get name | !GetAtt MySqsQueue.QueueName | !GetAtt MySnsTopic.TopicName |
Integrating with EventBridge in AWS SAM
The third example shows the AWS SAM template defining a serverless application with two Lambda functions and an EventBridge rule:
The default event bus already exists in every AWS account. You declare a rule that filters events in the event bus using the AWS::Events::Rule resource:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
EventPattern:
source:
- "demo.event"
detail:
state:
- "new"
State: "ENABLED"
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: "ConsumerTarget"
The rule describes an event pattern specifying matching JSON attributes. Events that match this pattern are routed to the list of targets. You provide the EventBridge service with permission to invoke the Lambda functions in the target list:
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "EventConsumerFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn: !GetAtt EventRule.Arn
The AWS SAM template uses an IAM policy statement to grant permission to the publishing function to put events on the event bus:
EventPublisherFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: code/
Handler: publisher.handler
Timeout: 3
Runtime: nodejs12.x
Policies:
- Statement:
- Effect: Allow
Resource: '*'
Action:
- events:PutEvents
The publishing function then uses the putEvents method of the AWS.EventBridge class, which returns after the events have been durably stored in EventBridge:
const AWS = require('aws-sdk')
AWS.config.update({region: 'us-east-1'})
const eventbridge = new AWS.EventBridge()
exports.handler = async (event) => {
const params = {
Entries: [
{
Detail: JSON.stringify({
"message": "Hello from publisher",
"state": "new"
}),
DetailType: 'Message',
EventBusName: 'default',
Source: 'demo.event',
Time: new Date
}
]
}
const result = await eventbridge.putEvents(params).promise()
console.log(result)
}
The payload for the consumer function is the message from EventBridge. This is an array of messages, containing subject and message attributes from the publishing function. You can see this in the CloudWatch log for the function:
Comparing SNS with EventBridge
SNS and EventBridge have many similarities. Both can be used to decouple publishers and subscribers, filter messages or events, and provide fan-in or fan-out capabilities. However, there are differences in the list of targets and features for each service, and your choice of service depends on the needs of your use-case.
EventBridge offers two newer capabilities that are not available in SNS. The first is software as a service (SaaS) integration. This enables you to authorize supported SaaS providers to send events directly from their EventBridge event bus to partner event buses in your account. This replaces the need for polling or webhook configuration, and creates a highly scalable way to ingest SaaS events directly into your AWS account.
The second feature is the Schema Registry, which makes it easier to discover and manage OpenAPI schemas for events. EventBridge can infer schemas based on events routed through an event bus by using schema discovery. This can be used to generate code bindings directly to your IDE for type-safe languages like Python, Java, and TypeScript. This can help accelerate development by automating the generation of classes and code directly from events.
This table compares the major features of both services:
| Amazon SNS | Amazon EventBridge | |
| Number of targets | 10 million (soft) | 5 |
| Availability SLA | 99.9% | 99.99% |
| Limits | 100,000 topics. 12,500,000 subscriptions per topic. | 100 event buses. 300 rules per event bus. |
| Publish throughput | Varies by Region. Soft limits. | Varies by Region. Soft limits. |
| Input transformation | No | Yes – see details. |
| Message filtering | Yes – see details. | Yes, including IP address matching – see details. |
| Message size maximum | 256 KB | 256 KB |
| Billing | Per 64 KB | |
| Format | Raw or JSON | JSON |
| Receive events from AWS CloudTrail | No | Yes |
| Targets | HTTP(S), SMS, SNS Mobile Push, Email/Email-JSON, SQS, Lambda functions. | 15 targets including AWS Lambda, Amazon SQS, Amazon SNS, AWS Step Functions, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose. |
| SaaS integration | No | Yes – see integrations. |
| Schema Registry integration | No | Yes – see details. |
| Dead-letter queues supported | Yes | No |
| FIFO ordering available | No | No |
| Public visibility | Can create public topics | Cannot create public buses |
| Pricing | $0.50/million requests + variable delivery cost + data transfer out cost. SMS varies. | $1.00/million events. Free for AWS events. No charge for delivery. |
| Billable request size | 1 request = 64 KB | 1 event = 64 KB |
| AWS Free Tier eligible | Yes | No |
| Cross-Region | You can subscribe your AWS Lambda functions to an Amazon SNS topic in any Region. | Targets must be in the same Region. You can publish across Regions to another event bus. |
| Retry policy |
|
At-least-once event delivery to targets, including retry with exponential backoff for up to 24 hours. |
Conclusion
Messaging is an important part of serverless applications and AWS services provide queues, publish/subscribe, and event routing capabilities. This post reviews the main features of SNS, SQS, and EventBridge and how they provide different capabilities for your workloads.
I show three example applications that publish and consume events from the three services. I walk through AWS SAM syntax for deploying these resources in your applications. Finally, I compare differences between the services.
To learn more building decoupled architectures, see this Learning Path series on EventBridge. For more serverless learning resources, visit https://serverlessland.com.
