All posts by Anand Komandooru

MCP went stateless: Is your AWS MCP server deployment well-architected?

Post Syndicated from Anand Komandooru original https://aws.amazon.com/blogs/architecture/mcp-went-stateless-is-your-aws-mcp-server-deployment-well-architected/

On July 28, 2026, MCP published its largest revision since launch, making the protocol core stateless and bringing remote MCP servers into alignment with AWS Well-Architected Framework best practices. The initialize handshake is gone, and so is the Mcp-Session-Id header that clients had to echo on every later request. Every request now carries its own protocol version and client context. A client’s first message can be the actual tool call, and any server instance can respond to it. If your MCP server was built for the session-based protocol, the sticky sessions, shared session stores, and custom observability plumbing it required are no longer necessary. If you run behind Amazon Bedrock AgentCore Gateway, protocol management and backward compatibility are handled for you. This post is for teams managing the full deployment stack themselves.

If a client wants to know what a server supports before calling it, a new server/discover method returns the supported protocol versions, capabilities, and identity in a single response. Servers must implement it per the MCP 2026-07-28 specification, but calling it is optional for the client.

This matters on AWS because the old design fought horizontal scaling. A session lived on whichever instance issued it. Running more than one instance meant either pinning clients with sticky routing or externalizing session state to a shared store. Both were correct for that protocol. With the new protocol, neither is required. This post maps the MCP 2026-07-28 specification against the Well-Architected Agentic AI Lens and recommends migrating, because the new protocol achieves natively what the old one could only achieve through compensating infrastructure.

One thing to settle up front, because it drives everything else: stateless describes the protocol, not your application. Stateful use cases still work.

Think of it as a coat check. Under the old protocol the server was a valet who remembered your face, which meant you had to keep dealing with that same valet and nobody else could help you. Now you get a numbered ticket, and any attendant can serve you because the ticket carries the reference. When a server needs continuity across calls, a tool returns an identifier for the stored state. The model includes that identifier on the calls that follow. The state stays in your datastore. The model carries only the key. This is ordinary REST discipline. It has an advantage over the old model. The identifier sits in the model’s context rather than hidden in a header. The model can reason about it and thread it across tools.

What changes in your architecture

The following table compares the deployment patterns the session-based protocol required against the patterns the stateless core now supports.

Before (session-based) After (2026-07-28 stateless)
Elastic Load Balancing Application Load Balancer (ALB) stickiness so each session reaches the same instance. Plain round-robin. Delete the stickiness configuration.
Session state in Amazon DynamoDB or Amazon ElastiCache. No session store. Server-minted identifiers passed as tool arguments.
Parse request bodies at the gateway to route by method. Route and throttle on the Mcp-Method and Mcp-Name headers.
AWS Lambda required workarounds for the stateful handshake. AWS Lambda is a natural fit. Request in, response out.
Refetch tool lists per session. No caching story. Cache with ttlMs and cacheScope, the protocol’s built-in freshness fields.
Bolt-on tracing per implementation. Proprietary protocol logging channel. W3C Trace Context in _meta for distributed tracing. stderr or OpenTelemetry for logging. Protocol logging is deprecated.
Rely on stream resumption (Last-Event-ID) for broken responses. Make tools idempotent. Clients re-issue broken calls.

⚠ Don’t delete yet if you serve 2025-era clients. The 2026-07-28 spec includes a backward-compatible lane that preserves session semantics for older clients. Your ALB stickiness rules and session store (DynamoDB/ElastiCache) must remain in place until you stop serving pre-2026-07-28 clients.
Action: Instrument your gateway to log protocol version per request. Set a sunset date for the legacy lane and communicate it to client teams. Only decommission session infrastructure after traffic on the old version reaches zero. This guidance applies to session infrastructure built to compensate for the old protocol’s requirements. Managed hosts that offer session features by design for specific use cases are not in scope.

One behavioral change to plan for. Servers can no longer push a request to a client mid-call, which is how confirmations, sampling, and root queries used to work over a held-open stream. The spec replaces that pattern with Multi Round-Trip Requests (MRTR). A server that needs input returns an input_required result containing an inputRequests map. This map holds elicitations, sampling calls, or root queries, and an opaque requestState token. The client fulfills the requests, then re-sends the original call with inputResponses and the echoed requestState. Any instance can pick that up because requestState carries all the context the server needs to resume. No shared session store is required. The server does not hold the connection open. This is what makes the pattern work on AWS Lambda.

The Well-Architected view

The AWS Well-Architected Agentic AI Lens already prescribes standardized protocol-based integration as a best practice. For more detail, refer to Establish standardized tool integration protocols (MCP, A2A). What follows is not new guidance but a reading of how the MCP 2026-07-28 specification makes those best practices genuinely achievable for a remote MCP server, pillar by pillar.

Diagram mapping MCP 2026-07-28 protocol changes to the six Well-Architected Agentic AI Lens pillars

Figure 1: How the MCP 2026-07-28 specification maps to the Well-Architected Agentic AI Lens pillars

Operational excellence. The Lens identifies observability as the foundation for operating agents. If you cannot trace a decision end to end, you cannot debug, optimize, or audit it. The 2026-07-28 spec builds observability into the protocol itself. Three changes make this concrete:

  1. Tracing. Every request carries W3C Trace Context keys in _meta (traceparent, tracestate, baggage), so it traces end to end through any OpenTelemetry-compatible backend, including Amazon CloudWatch. The Lens prescribes end-to-end tracing and telemetry for agent operations.
  2. Operational signals without body parsing. The Mcp-Method and Mcp-Name headers expose the operation type on every POST, and every response carries a required resultType field (complete or input_required). Gateways and observability tools get unambiguous per-operation signals for metrics, alarms, and AWS WAF rules without inspecting payloads. The result directly addresses the Lens recommendation for implementing metrics and monitoring for agent-specific patterns.
  3. Standardized logging. MCP’s proprietary protocol logging is deprecated in favor of stderr and OpenTelemetry. The Lens makes the same recommendation: implement structured logging through standardized, queryable formats.

Security. The Lens treats agent security as harder than traditional service security: agents act autonomously with delegated credentials, and their inputs (including state identifiers) are visible to, and potentially manipulable by, the model. MCP’s 2026-07-28 spec hardens the protocol surface against these risks. Five changes strengthen the security posture:

  1. Issuer validation. Clients must validate the iss parameter per RFC 9207, confirming which authorization server produced a response. The Lens calls for the same discipline under strong authentication for agent identities.
  2. Client type declaration. Clients must declare application_type at registration so a desktop or CLI client is not mistaken for a web app, verifying authentication mechanisms match the client’s security profile. The same Lens best practice applies: strong authentication for agent identities. (Note: Dynamic Client Registration itself is now deprecated in favor of Client ID Metadata Documents.)
  3. Bounded human interaction. A server can prompt a user only while it is handling that user’s request, through the Multi Round-Trip Requests pattern. This is a protocol-enforced constraint that bounds when human interaction can occur, aligning with the Lens’s human-in-the-loop controls for critical decisions.
  4. Ownership enforcement. Because state identifiers are visible to the model, servers must enforce ownership on every call. The protocol will not stop a caller from presenting an identifier that is not theirs, so the Lens best practice for tool authorization at the gateway applies: validate that the requesting identity owns the resource it references. The same discipline applies to requestState tokens: the spec requires servers to treat them as untrusted input and protect their integrity with HMAC or AEAD, rejecting any token that fails verification.
  5. Schema validation. Tool input and output schemas are now validated against JSON Schema 2020-12, giving servers a formal contract for rejecting malformed or injected arguments before execution. This maps to the Lens requirement to validating tool inputs at the boundary.

Reliability. Agents hold multi-step context that is expensive to reconstruct after failure, making reliability harder than in traditional services. MCP’s 2026-07-28 spec addresses this at the protocol layer. Four changes reduce that fragility:

  1. Stateless transport. The spec removes protocol-level sessions, so any instance can serve any request. Instance loss is a non-event. Retries need no session affinity, and scale-in never drains sessions. The protocol embodies the failure-isolation philosophy at the protocol layer without additional infrastructure.
  2. Continuation tokens. Interrupted multi-step interactions resume through requestState, an opaque continuation token the server returns and the client echoes on retry. This embodies the Lens principle of designing workflows in stages with incremental recovery.
  3. Idempotent retry. Stream resumability was removed, so a broken response stream loses the in-flight payload and the client must re-issue the call. The mitigation is the same idempotent task execution pattern the Lens prescribes for retryable agent actions: make tools idempotent so re-issued requests produce no duplicate side effects.
  4. Standardized error codes. The spec allocates error code ranges (-32000 to -32019 implementation-defined, -32020 to -32099 reserved for MCP), giving clients and gateways a canonical signal set for retry, backoff, and circuit-breaking decisions. Gateways can now implement standardized communication protocols.

Performance efficiency. Redundant data fetches and per-interaction protocol overhead are the two main performance drags the Lens identifies in agentic workloads. MCP’s 2026-07-28 spec addresses both at the protocol layer. Three changes reduce that overhead:

  1. Protocol-declared caching. Two fields are now required on list and resource-read results: ttlMs (how many milliseconds a response stays fresh) and cacheScope (whether shared intermediaries can cache it or only the requesting client). Tool lists now return in deterministic order, allowing LLM prompt-cache hits across calls. The protocol now delivers what the Lens recommends under optimizing inference-time performance for agent workloads.
  2. Freshness semantics. Clients and MCP-aware gateways can cache responses using protocol-declared freshness (ttlMs + cacheScope), the same data-type-specific TTL discipline the Lens recommends under protocol-declared freshness semantics, without guessing at staleness.
  3. Header-based routing. Routing and throttling decisions now live in HTTP headers (Mcp-Method, Mcp-Name) rather than parsed message bodies, reducing per-interaction overhead in line with what the Lens prescribes for efficient protocol-based agent communications.

Cost optimization. The Lens identifies always-on infrastructure serving bursty agent traffic as the highest source of idle cost in an agent stack. MCP’s stateless architecture eliminates an entire category of that cost: session infrastructure.

  1. Delete session infrastructure. Audit for anything that exists only to preserve sessions (ElastiCache clusters, sticky-routing rules, session-replication logic) and delete it. This follows the same principle the Lens applies to cost-optimizing tool serving through serverless and resource sharing. Infrastructure that runs constantly to serve unpredictable traffic should be replaced with consumption-based patterns that scale to zero. A two-node Amazon ElastiCache (cache.t4g.micro) session store is about $23/month (AWS Pricing Calculator, July 2026). The larger saving is eliminating an entire class of infrastructure and the operational burden around it. Sticky routing costs capacity too by distributing load unevenly, and the savings scale with the size of your fleet.
  2. Serverless as first-class pattern. AWS Lambda has no sticky routing and no persistent connections. A session-based MCP server meant externalizing state to a shared store. Even a “session-free” mode still paid for the mandatory handshake. With the 2026-07-28 stateless core, request in, response out is exactly what AWS Lambda does natively. Serverless MCP moves from workaround to first-class pattern, delivering what the Lens recommends for cost-optimizing tool serving through serverless and resource sharing.

Sustainability. The Lens identifies static provisioning for bursty agent traffic as the primary source of wasted infrastructure capacity. The 2026-07-28 spec’s stateless architecture eliminates the structural reasons for that over-provisioning.

  1. No more pinned-session capacity. The spec’s stateless design means no instance holds a session, so no instance needs to stay warm for one. Right-size against your actual traffic pattern rather than a theoretical peak, the same principle the Lens applies to appropriately scaling compute, networking, and data dependencies for agent workloads. Instance-agnostic routing means the fleet you do keep can run closer to its real utilization, instead of padding for the instances that happened to hold long-lived sessions.

The AWS Well-Architected Agentic AI Lens articulated these best practices as general principles for agentic workloads. The fact that a major protocol revision, designed independently, converges on the same architectural shape is evidence that the framework captures something real about how reliable distributed systems need to work.

What to watch

The architectural shift creates its own operational surface. These are the areas where the new defaults need deliberate attention rather than passive adoption.

Long-lived streams did not disappear. The subscriptions/listen method consolidates change notifications into a single opt-in POST-response stream, so check idle timeouts across your load balancer, proxy, and compute tier if your servers use it.

Deprecations with a clock. The spec deprecated Roots, Sampling, Logging, and the HTTP+SSE transport with a twelve-month floor before removal. The earliest any of these can be removed is July 2027. It also removed ping, logging/setLevel, and notifications/roots/list_changed outright, and moved log level into per-request _meta. The suggested migration paths:

  • Pass directories through tool parameters or resource URIs instead of Roots.
  • Integrate directly with LLM provider APIs instead of Sampling.
  • Log to stderr or OpenTelemetry instead of protocol-level Logging.
  • Migrate HTTP+SSE to Streamable HTTP.

Plan the exits now rather than at the deadline.

MCP Apps puts server-supplied HTML inside your host. Pre-declared UI resource templates, mandatory iframe sandboxing, and auditable JSON-RPC communication between the iframe and host all help. But treat template review as mandatory before deployment, and decide deliberately which servers in your fleet can ship UI at all.

cacheScope is a multi-tenant disclosure risk. Setting cacheScope: "public" on a response that contains tenant-specific data lets shared intermediaries serve one tenant’s list to another. Default to "private" and widen deliberately only for responses that are genuinely identical across callers.

Built-in protection against future breaks

Three mechanisms shipped alongside the stateless core to prevent a repeat of this kind of breaking change.

A feature lifecycle policy gives every feature an Active, Deprecated, or Removed state. Nothing can be removed until at least twelve months after it is deprecated. An extensions framework lets new capabilities ship as opt-in extensions that prove themselves outside the core. That is where Tasks landed after its experimental version needed a redesign. And no Standards Track proposal can reach Final status without a matching scenario in the conformance suite. This is the same suite the official SDKs are validated against.

The handshake and session removal were a deliberate, one-time break to fix the foundation. From here, what you build against 2026-07-28 comes with documented notice periods.

Self-check

Run these ten questions against your own deployment before you decide whether, and how, to migrate.

  1. Can any instance of your server handle any request, with no session affinity at the load balancer?
  2. Have you deleted everything that existed only to preserve a protocol session?
  3. Do your list responses set ttlMs and cacheScope deliberately, and does your gateway route on headers rather than parsed bodies?
  4. Does every client validate iss, and does every server enforce ownership per identifier rather than trusting the identifier itself?
  5. Do you have a firm date to stop supporting 2025-11-25 clients?
  6. Have you replaced server-initiated pushes with Multi Round-Trip Requests so no instance holds a connection open for client input?
  7. Are your tools idempotent so clients can safely re-issue any broken call?
  8. Do you propagate W3C Trace Context end-to-end and emit logs through stderr or OpenTelemetry instead of MCP protocol logging?
  9. Are you still paying for session infrastructure (DynamoDB, ElastiCache, sticky routing) that nothing uses?
  10. Do you have a governance policy for MCP Apps before any server in your fleet exposes one?

A “no” to any of these is where the new spec pays off. Each maps to the pillar sections earlier in this post. Start with the migration path that follows, run your server against the official conformance suite, and use the related AWS resources at the end to plan the change.

Migration path

You do not need to move immediately. Protocol versions are frozen snapshots, and a client and server only need to share one, so 2025-11-25 servers keep working with clients that still speak it. But hosts retire old versions on their own timeline, the community is already moving (GitHub’s MCP Server shipped support ahead of the release), and 2025-11-25 is now frozen. Future capabilities and fixes land on 2026-07-28 or later.

For a new server, target 2026-07-28 directly: stateless from the start, explicit identifiers, and no dependence on Roots, Sampling, or MCP Logging.

For an existing server, work through these steps in order:

  1. Upgrade the SDK and opt in. Speaking the new revision is never automatic.
  2. Audit for session assumptions and migrate off the experimental Tasks API if you used it (Tasks is now an official extension with a redesigned interface).
  3. Plan the deprecation exits (Roots, Sampling, Logging, HTTP+SSE) and change the resource-not-found error code from -32002 to -32602.
  4. Collect the infrastructure savings by deleting session stores, sticky-routing rules, and handshake infrastructure.

For a platform or gateway team: add header-based routing and per-operation throttling on Mcp-Method, honor ttlMs and cacheScope in your caching layer. Also propagate W3C Trace Context, and set a policy for MCP Apps before the first server in your fleet ships one.

Validate before you ship. The official conformance suite covers the new behaviors, and protocol inspectors can pin 2026-07-28 to test your server against exactly what clients will send. Start in a test environment, then promote to production once the suite passes.

Conclusion

The session-based protocol was correct for the constraints it operated under, but those constraints are gone. If you are deploying MCP servers on AWS, the 2026-07-28 specification is the Well-Architected path forward. Migrate your servers, sunset your legacy lane, and delete the infrastructure that existed only to compensate for a protocol limitation that no longer applies.


About the authors

Implement a full stack serverless search application using AWS Amplify, Amazon Cognito, Amazon API Gateway, AWS Lambda, and Amazon OpenSearch Serverless

Post Syndicated from Anand Komandooru original https://aws.amazon.com/blogs/big-data/implement-a-full-stack-serverless-search-application-using-aws-amplify-amazon-cognito-amazon-api-gateway-aws-lambda-and-amazon-opensearch-serverless/

Designing a full stack search application requires addressing numerous challenges to provide a smooth and effective user experience. This encompasses tasks such as integrating diverse data from various sources with distinct formats and structures, optimizing the user experience for performance and security, providing multilingual support, and optimizing for cost, operations, and reliability.

Amazon OpenSearch Serverless is a powerful and scalable search and analytics engine that can significantly contribute to the development of search applications. It allows you to store, search, and analyze large volumes of data in real time, offering scalability, real-time capabilities, security, and integration with other AWS services. With OpenSearch Serverless, you can search and analyze a large volume of data without having to worry about the underlying infrastructure and data management. An OpenSearch Serverless collection is a group of OpenSearch indexes that work together to support a specific workload or use case. Collections have the same kind of high-capacity, distributed, and highly available storage volume that’s used by provisioned Amazon OpenSearch Service domains, but they remove complexity because they don’t require manual configuration and tuning. Each collection that you create is protected with encryption of data at rest, a security feature that helps prevent unauthorized access to your data. OpenSearch Serverless also supports OpenSearch Dashboards, which provides an intuitive interface for analyzing data.

OpenSearch Serverless supports three primary use cases:

  • Time series – The log analytics workloads that focus on analyzing large volumes of semi-structured, machine-generated data in real time for operational, security, user behavior, and business insights
  • Search – Full-text search that powers applications in your internal networks (content management systems, legal documents) and internet-facing applications, such as ecommerce website search and content search
  • Vector search – Semantic search on vector embeddings that simplifies vector data management and powers machine learning (ML) augmented search experiences and generative artificial intelligence (AI) applications, such as chatbots, personal assistants, and fraud detection

In this post, we walk you through a reference implementation of a full-stack cloud-centered serverless text search application designed to run using OpenSearch Serverless.

Solution overview

The following services are used in the solution:

  • AWS Amplify is a set of purpose-built tools and features that enables frontend web and mobile developers to quickly and effortlessly build full-stack applications on AWS. These tools have the flexibility to use the breadth of AWS services as your use cases evolve. This solution uses the Amplify CLI to build the serverless movie search web application. The Amplify backend is used to create resources such as the Amazon Cognito user pool, API Gateway, Lambda function, and Amazon S3 storage.
  • Amazon API Gateway is a fully managed service that makes it straightforward for developers to create, publish, maintain, monitor, and secure APIs at any scale. We use API Gateway as a “front door” for the movie search application for searching movies.
  • AWS CloudFront accelerates the delivery of web content such as static and dynamic web pages, video streams, and APIs to users across the globe by caching content at edge locations closer to the end-users. This solution uses CloudFront with Amazon S3 to deliver the search application user interface to the end users.
  • Amazon Cognito makes it straightforward for adding authentication, user management, and data synchronization without having to write backend code or manage any infrastructure. We use Amazon Cognito for creating a user pool so the end-user can log in to the movie search application through Amazon Cognito.
  • AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. Our solution uses a Lambda function to query OpenSearch Serverless. API Gateway forwards all requests to the Lambda function to serve up the requests.
  • Amazon OpenSearch Serverless is a serverless option for OpenSearch Service. In this post, you use common methods for searching documents in OpenSearch Service that improve the search experience, such as request body searches using domain-specific language (DSL) for queries. The query DSL lets you specify the full range of OpenSearch search options, including pagination and sorting the search results. Pagination and sorting are implemented on the server side using DSL as part of this implementation.
  • Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. The solution uses Amazon S3 as storage for storing movie trailers.
  • AWS WAF helps protects web applications from attacks by allowing you to configure rules that allow, block, or monitor (count) web requests based on conditions that you define. We use AWS WAF to allow access to the movie search app from only IP addresses on an allow list.

The following diagram illustrates the solution architecture.

The workflow includes the following steps:

  1. The end-user accesses the CloudFront and Amazon S3 hosted movie search web application from their browser or mobile device.
  2. The user signs in with their credentials.
  3. A request is made to an Amazon Cognito user pool for a login authentication token, and a token is received for a successful sign-in request.
  4. The search application calls the search API method with the token in the authorization header to API Gateway. API Gateway is protected by AWS WAF to enforce rate limiting and implement allow and deny lists.
  5. API Gateway passes the token for validation to the Amazon Cognito user pool. Amazon Cognito validates the token and sends a response to API Gateway.
  6. API Gateway invokes the Lambda function to process the request.
  7. The Lambda function queries OpenSearch Serverless and returns the metadata for the search.
  8. Based on metadata, content is returned from Amazon S3 to the user.

In the following sections, we walk you through the steps to deploy the solution, ingest data, and test the solution.

Prerequisites

Before you get started, make sure you complete the following prerequisites:

  1. Install Nodejs latest LTS version.
  2. Install and configure the AWS Command Line Interface (AWS CLI).
  3. Install awscurl for data ingestion.
  4. Install and configure the Amplify CLI. At the end of configuration, you should successfully set up the new user using the amplify-dev user’s AccessKeyId and SecretAccessKey in your local machine’s AWS profile.
  5. Amplify users need additional permissions in order to deploy AWS resources. Complete the following steps to create a new inline AWS Identity and Access Management (IAM) policy and attach it to the user:
    • On the IAM console, choose Users in the navigation pane.
    • Choose the user amplify-dev.
    • On the Permissions tab, choose the Add permissions dropdown menu, then choose Inline policy.
    • In the policy editor, choose JSON.

You should see the default IAM statement in JSON format.

This environment name needs to be used when performing amplify init when bringing up the backend. The actions in the IAM statement are largely open (*) but restricted or limited by the target resources; this is done to satisfy the maximum inline policy length (2,048 characters).

    • Enter the updated JSON into the policy editor, then choose Next.
    • For Policy name, enter a name (for this post, AddionalPermissions-Amplify).
    • Choose Create policy.

You should now see the new inline policy attached to the user.

Deploy the solution

Complete the following steps to deploy the solution:

  1. Clone the repository to a new folder on your desktop using the following command:
    git clone https://github.com/aws-samples/amazon-opensearchserverless-searchapp.git

  2. Deploy the movie search backend.
  3. Deploy the movie search frontend.

Ingest data

To ingest the sample movie data into the newly created OpenSearch Serverless collection, complete the following steps:

  • On the OpenSearch Service console, choose Ingestion: Pipelines in the navigation pane.
  • Choose the pipeline movie-ingestion and locate the ingestion URL.

  • Replace the ingestion endpoint and Region in the following snippet and run the awscurl command to save data into the collection:
awscurl --service osis --region <region> \
-X POST \
-H "Content-Type: application/json" \
-d "@project_assets/movies-data.json" \
https://<ingest_url>/movie-ingestion/data 

You should see a 200 OK response.

  • On the Amazon S3 console, open the trailer S3 bucket (created as part of the backend deployment.
  • Upload some movie trailers.

Storage

Make sure the file name matches the ID field in sample movie data (for example, tt1981115.mp4, tt0800369.mp4, and tt0172495.mp4). Uploading a trailer with ID tt0172495.mp4 is used as the default trailer for all movies, without having to upload one for each movie.

Test the solution

Access the application using the CloudFront distribution domain name. You can find this by opening the CloudFront console, choosing the distribution, and copying the distribution domain name into your browser.

Sign up for application access by entering your user name, password, and email address. The password should be at least eight characters in length, and should include at least one uppercase character and symbol.

Sign Up

After you’re logged in, you’re redirected to the Movie Finder home page.

Home Page

You can search using a movie name, actor, or director, as shown in the following example. The application returns results using OpenSearch DSL.

Search Results

If there’s a large number of search results, you can navigate through them using the pagination option at the bottom of the page. For more information about how the application uses pagination, see Paginating search results.

Pagination

You can choose movie tiles to get more details and watch the trailer if you took the optional step of uploading a movie trailer.

Movie Details

You can sort the search results using the Sort by feature. The application uses the sort functionality within OpenSearch.

Sort

There are many more DSL search patterns that allow for intricate searches. See Query DSL for complete details.

Monitoring OpenSearch Serverless

Monitoring is an important part of maintaining the reliability, availability, and performance of OpenSearch Serverless and your other AWS services. AWS provides Amazon CloudWatch and AWS CloudTrail to monitor OpenSearch Serverless, report when something is wrong, and take automatic actions when appropriate. For more information, see Monitoring Amazon OpenSearch Serverless.

Clean up

To avoid unnecessary charges, clean up the solution implementation by running the following command at the project root folder you created using the git clone command during deployment:

amplify delete

You can also clean up the solution by deleting the AWS CloudFormation stack you deployed as part of the setup. For instructions, see Deleting a stack on the AWS CloudFormation console.

Conclusion

In this post, we implemented a full-stack serverless search application using OpenSearch Serverless. This solution seamlessly integrates with various AWS services, such as Lambda for serverless computing, API Gateway for constructing RESTful APIs, IAM for robust security, Amazon Cognito for streamlined user management, and AWS WAF for safeguarding the web application against threats. By adopting a serverless architecture, this search application offers numerous advantages, including simplified deployment processes and effortless scalability, with the benefits of a managed infrastructure.

With OpenSearch Serverless, you get the same interactive millisecond response times as OpenSearch Service with the simplicity of a serverless environment. You pay only for what you use by automatically scaling resources to provide the right amount of capacity for your application without impacting performance and scale as needed. You can use OpenSearch Serverless and this reference implementation to build your own full-stack text search application.


About the Authors

Anand Komandooru is a Principal Cloud Architect at AWS. He joined AWS Professional Services organization in 2021 and helps customers build cloud-native applications on AWS cloud. He has over 20 years of experience building software and his favorite Amazon leadership principle is “Leaders are right a lot“.

Rama Krishna Ramaseshu is a Senior Application Architect at AWS. He joined AWS Professional Services in 2022 and with close to two decades of experience in application development and software architecture, he empowers customers to build well architected solutions within the AWS cloud. His favorite Amazon leadership principle is “Learn and Be Curious”.

Sachin Vighe is a Senior DevOps Architect at AWS. He joined AWS Professional Services in 2020, and specializes in designing and architecting solutions within the AWS cloud to guide customers through their DevOps and Cloud transformation journey. His favorite leadership principle is “Customer Obsession”.

Molly Wu is an Associate Cloud Developer at AWS. She joined AWS Professional Services in 2023 and specializes in assisting customers in building frontend technologies in AWS cloud. Her favorite leadership principle is “Bias for Action”.

Andrew Yankowsky is a Security Consultant at AWS. He joined AWS Professional Services in 2023, and helps customers build cloud security capabilities and follow security best practices on AWS. His favorite leadership principle is “Earn Trust”.

Automate the archive and purge data process for Amazon RDS for PostgreSQL using pg_partman, Amazon S3, and AWS Glue

Post Syndicated from Anand Komandooru original https://aws.amazon.com/blogs/big-data/automate-the-archive-and-purge-data-process-for-amazon-rds-for-postgresql-using-pg_partman-amazon-s3-and-aws-glue/

The post Archive and Purge Data for Amazon RDS for PostgreSQL and Amazon Aurora with PostgreSQL Compatibility using pg_partman and Amazon S3 proposes data archival as a critical part of data management and shows how to efficiently use PostgreSQL’s native range partition to partition current (hot) data with pg_partman and archive historical (cold) data in Amazon Simple Storage Service (Amazon S3). Customers need a cloud-native automated solution to archive historical data from their databases. Customers want the business logic to be maintained and run from outside the database to reduce the compute load on the database server. This post proposes an automated solution by using AWS Glue for automating the PostgreSQL data archiving and restoration process, thereby streamlining the entire procedure.

AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. There is no need to pre-provision, configure, or manage infrastructure. It can also automatically scale resources to meet the requirements of your data processing job, providing a high level of abstraction and convenience. AWS Glue integrates seamlessly with AWS services like Amazon S3, Amazon Relational Database Service (Amazon RDS), Amazon Redshift, Amazon DynamoDB, Amazon Kinesis Data Streams, and Amazon DocumentDB (with MongoDB compatibility) to offer a robust, cloud-native data integration solution.

The features of AWS Glue, which include a scheduler for automating tasks, code generation for ETL (extract, transform, and load) processes, notebook integration for interactive development and debugging, as well as robust security and compliance measures, make it a convenient and cost-effective solution for archival and restoration needs.

Solution overview

The solution combines PostgreSQL’s native range partitioning feature with pg_partman, the Amazon S3 export and import functions in Amazon RDS, and AWS Glue as an automation tool.

The solution involves the following steps:

  1. Provision the required AWS services and workflows using the provided AWS Cloud Development Kit (AWS CDK) project.
  2. Set up your database.
  3. Archive the older table partitions to Amazon S3 and purge them from the database with AWS Glue.
  4. Restore the archived data from Amazon S3 to the database with AWS Glue when there is a business need to reload the older table partitions.

The solution is based on AWS Glue, which takes care of archiving and restoring databases with Availability Zone redundancy. The solution is comprised of the following technical components:

  • An Amazon RDS for PostgreSQL Multi-AZ database runs in two private subnets.
  • AWS Secrets Manager stores database credentials.
  • An S3 bucket stores Python scripts and database archives.
  • An S3 Gateway endpoint allows Amazon RDS and AWS Glue to communicate privately with the Amazon S3.
  • AWS Glue uses a Secrets Manager interface endpoint to retrieve database secrets from Secrets Manager.
  • AWS Glue ETL jobs run in either private subnet. They use the S3 endpoint to retrieve Python scripts. The AWS Glue jobs read the database credentials from Secrets Manager to establish JDBC connections to the database.

You can create an AWS Cloud9 environment in one of the private subnets available in your AWS account to set up test data in Amazon RDS. The following diagram illustrates the solution architecture.

Solution Architecture

Prerequisites

For instructions to set up your environment for implementing the solution proposed in this post, refer to Deploy the application in the GitHub repo.

Provision the required AWS resources using AWS CDK

Complete the following steps to provision the necessary AWS resources:

  1. Clone the repository to a new folder on your local desktop.
  2. Create a virtual environment and install the project dependencies.
  3. Deploy the stacks to your AWS account.

The CDK project includes three stacks: vpcstack, dbstack, and gluestack, implemented in the vpc_stack.py, db_stack.py, and glue_stack.py modules, respectively.

These stacks have preconfigured dependencies to simplify the process for you. app.py declares Python modules as a set of nested stacks. It passes a reference from vpcstack to dbstack, and a reference from both vpcstack and dbstack to gluestack.

gluestack reads the following attributes from the parent stacks:

  • The S3 bucket, VPC, and subnets from vpcstack
  • The secret, security group, database endpoint, and database name from dbstack

The deployment of the three stacks creates the technical components listed earlier in this post.

Set up your database

Prepare the database using the information provided in Populate and configure the test data on GitHub.

Archive the historical table partition to Amazon S3 and purge it from the database with AWS Glue

The “Maintain and Archive” AWS Glue workflow created in the first step consists of two jobs: “Partman run maintenance” and “Archive Cold Tables.”

The “Partman run maintenance” job runs the Partman.run_maintenance_proc() procedure to create new partitions and detach old partitions based on the retention setup in the previous step for the configured table. The “Archive Cold Tables” job identifies the detached old partitions and exports the historical data to an Amazon S3 destination using aws_s3.query_export_to_s3. In the end, the job drops the archived partitions from the database, freeing up storage space. The following screenshot shows the results of running this workflow on demand from the AWS Glue console.

Archive job run result

Additionally, you can set up this AWS Glue workflow to be triggered on a schedule, on demand, or with an Amazon EventBridge event. You need to use your business requirement to select the right trigger.

Restore archived data from Amazon S3 to the database

The “Restore from S3” Glue workflow created in the first step consists of one job: “Restore from S3.”

This job initiates the run of the partman.create_partition_time procedure to create a new table partition based on your specified month. It subsequently calls aws_s3.table_import_from_s3 to restore the matched data from Amazon S3 to the newly created table partition.

To start the “Restore from S3” workflow, navigate to the workflow on the AWS Glue console and choose Run.

The following screenshot shows the “Restore from S3” workflow run details.

Restore job run result

Validate the results

The solution provided in this post automated the PostgreSQL data archival and restoration process using AWS Glue.

You can use the following steps to confirm that the historical data in the database is successfully archived after running the “Maintain and Archive” AWS Glue workflow:

  1. On the Amazon S3 console, navigate to your S3 bucket.
  2. Confirm the archived data is stored in an S3 object as shown in the following screenshot.
    Archived data in S3
  3. From a psql command line tool, use the \dt command to list the available tables and confirm the archived table ticket_purchase_hist_p2020_01 does not exist in the database.List table result after post archival

You can use the following steps to confirm that the archived data is restored to the database successfully after running the “Restore from S3” AWS Glue workflow.

  1. From a psql command line tool, use the \dt command to list the available tables and confirm the archived table ticket_history_hist_p2020_01 is restored to the database.List table results after restore

Clean up

Use the information provided in Cleanup to clean up your test environment created for testing the solution proposed in this post.

Summary

This post showed how to use AWS Glue workflows to automate the archive and restore process in RDS for PostgreSQL database table partitions using Amazon S3 as archive storage. The automation is run on demand but can be set up to be trigged on a recurring schedule. It allows you to define the sequence and dependencies of jobs, track the progress of each workflow job, view run logs, and monitor the overall health and performance of your tasks. Although we used Amazon RDS for PostgreSQL as an example, the same solution works for Amazon Aurora-PostgreSQL Compatible Edition as well. Modernize your database cron jobs using AWS Glue by using this post and the GitHub repo. Gain a high-level understanding of AWS Glue and its components by using the following hands-on workshop.


About the Authors

Anand Komandooru is a Senior Cloud Architect at AWS. He joined AWS Professional Services organization in 2021 and helps customers build cloud-native applications on AWS cloud. He has over 20 years of experience building software and his favorite Amazon leadership principle is “Leaders are right a lot.”

Li Liu is a Senior Database Specialty Architect with the Professional Services team at Amazon Web Services. She helps customers migrate traditional on-premise databases to the AWS Cloud. She specializes in database design, architecture, and performance tuning.

Neil Potter is a Senior Cloud Application Architect at AWS. He works with AWS customers to help them migrate their workloads to the AWS Cloud. He specializes in application modernization and cloud-native design and is based in New Jersey.

Vivek Shrivastava is a Principal Data Architect, Data Lake in AWS Professional Services. He is a big data enthusiast and holds 14 AWS Certifications. He is passionate about helping customers build scalable and high-performance data analytics solutions in the cloud. In his spare time, he loves reading and finds areas for home automation.