Post Syndicated from Siddhesh Tiwari original https://aws.amazon.com/blogs/architecture/building-resilient-real-time-streaming-workers-with-amazon-dynamodb-leases/

Consider a real-time transcription service processing 500 concurrent meetings. Each worker processing these meetings requires a dedicated outbound WebSocket connection to an upstream streaming source. When a single worker fails, it drops 100+ connections, causing 2 to 3 minutes of data loss per connection until operators manually restart services.

Building real-time streaming workers that maintain hundreds of persistent WebSocket connections presents a coordination challenge: when a worker stops unexpectedly, its connections become unmanaged and data stops flowing. Exactly one worker must own each connection, yet workers fail, redeploy, and scale independently. Without a mechanism to track ownership and automatically transfer connections that healthy workers can claim, operators must intervene manually for every failure.

This pattern reduces manual intervention during failures, reduces connection recovery time from minutes to seconds, and helps minimize downtime during deployments without requiring external coordination services.

In this post, you learn how to build a WebSocket fleet management system on Amazon Elastic Container Service (Amazon ECS) and AWS Fargate. Amazon DynamoDB is the primary service that manages distributed lease ownership, coordination, and failover in this solution. For the compute layer, this post uses Amazon ECS on AWS Fargate to run the worker fleet. However, you can adapt this pattern to any compute layer of your choice, such as Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Compute Cloud (Amazon EC2) with Auto Scaling groups, without changing the core lease logic. You learn how to implement lease-based ownership with conditional writes, automatic failover through orphan reconciliation, and low downtime deployments through graceful shutdown.

The challenge: managing long-lived WebSocket connections

WebSocket connections are fundamentally different from HTTP requests. An HTTP request arrives, gets processed, and returns a response. The server holds no state between requests. A WebSocket connection, by contrast, is a persistent bidirectional channel. The worker must maintain an open TCP connection, process messages the upstream source sends, and respond to keep-alive pings from the upstream source.

This statefulness introduces several operational challenges:

Worker failures. When a worker process stops unexpectedly or its container terminates, the worker drops its WebSocket connections. The upstream source might buffer data briefly, but without a mechanism to detect the failure and reassign the connection to a healthy worker, the system loses data.

Rolling deployments. ECS rolling deployments terminate old tasks and start new ones. Each terminated task drops its connections. Without coordination, there’s a window where connections have no owner.

Horizontal scaling. Adding workers is straightforward. New tasks start and pick up work. Removing workers is harder. You need to drain connections from departing workers and verify other workers take over before the task exits.

Double-claiming. If two workers both believe they own the same connection, they both attempt to connect to the same upstream source. This can cause duplicate data processing, protocol errors, or connection rejection by the upstream service.

Because the workers are WebSocket clients that initiate outbound connections to upstream sources, you need a coordination mechanism that operates at the application layer rather than the network layer.

Solution overview

The architecture uses six AWS services to coordinate a fleet of WebSocket workers:

Figure 1: WebSocket fleet management architecture

1. Amazon API Gateway: You use this to receive START and STOP events from external systems through a REST API. A START event signals that a new streaming session (for example, a meeting or live feed) has begun and requires a dedicated WebSocket connection. A STOP event signals that the streaming session has ended and the connection should be released.
2. AWS Lambda (event router): You use this to write connection state to Amazon DynamoDB and enqueue a notification to Amazon Simple Queue Service (Amazon SQS).
3. Amazon DynamoDB: You use this to store connection state and lease ownership. Conditional writes (atomic operations that succeed only if specified conditions are met) can provide distributed locking capabilities without external coordination services.
4. Amazon SQS: You use this to distribute work notifications to workers for fast pickup of new connections.
5. Amazon ECS on AWS Fargate: You use this to run the worker fleet. Each worker polls Amazon SQS, manages WebSocket connections, and renews leases through heartbeats.
6. Amazon CloudWatch: You use this to collect custom metrics (active connection count) that drive ECS automatic scaling.

The key insight is that DynamoDB conditional writes act as a distributed lock without requiring a separate coordination service. Each connection has a lease: a time-bounded ownership claim. Workers must continuously renew their lease. If a worker stops unexpectedly, the lease expires and another worker takes over.

Why not SQS alone or an existing lock client?

SQS plays an important role in this architecture as a fast notification channel, but it cannot serve as the sole coordination mechanism. SQS is designed for task execution, delivering a unit of work to one consumer. WebSocket connection ownership is not a one-time task. It is a continuous state that must be maintained and renewed for the lifetime of the connection. SQS has no mechanism to track who currently owns a connection, query for connections with no active owner, or represent the domain state (desired_state, ws_url, last_seq) needed to manage a connection. DynamoDB provides all these capabilities through persistent items, conditional writes, and secondary indexes.

The amazon-dynamodb-lock-client library published by AWS implements similar distributed locking primitives on DynamoDB. However, it is designed for Java environments and does not integrate domain-specific connection state into the lock record. This solution is implemented in async Python to match the worker architecture, combines lock ownership and connection metadata in a single DynamoDB item to reduce read operations, and uses a GSI to enable fleet-wide reconciliation queries that a general-purpose lock client does not provide.

The lease pattern

A lease is a row in DynamoDB that tracks who owns a connection and when that ownership expires. The table uses the following schema:

A global secondary index (GSI), a secondary lookup structure that you can use to query on non-primary-key attributes, on desired_state (partition key) and lease_expires_at_ms (sort key) allows efficient queries for unmanaged connections: those with desired_state = STARTED and an expired lease.

A note on clock accuracy

The lease expiration mechanism relies on epoch millisecond timestamps generated by worker processes using their local system clocks. DynamoDB evaluates lease expiration conditions against the now value supplied by the calling worker, not against a DynamoDB server-side clock. This means all workers must have reasonably synchronized clocks for the lease pattern to behave correctly.

AWS Fargate tasks running in the same AWS region receive clock synchronization through the Amazon Time Sync Service, which keeps clock skew between tasks to within a few milliseconds. This is well within the safety margin provided by the default 20-second lease duration and 5-second heartbeat interval. If you deploy this pattern on compute infrastructure outside of AWS Fargate, verify that NTP synchronization is configured and monitor for clock drift. For environments where clock accuracy cannot be guaranteed, increase the lease duration by the maximum expected clock skew to prevent false lease expirations.

The lease lifecycle has four states. Figure 2 shows the lease state machine.

Figure 2: Lease lifecycle

Acquire

A worker claims a connection by writing its worker ID (lease_owner) and a future expiration timestamp (lease_expires_at_ms) to the DynamoDB lease record. The conditional expression ensures that only one worker can succeed: it checks that either no lease exists yet (attribute_not_exists) or the existing lease has already expired (lease_expires_at_ms < :now). If two workers attempt to acquire the same connection simultaneously, DynamoDB evaluates this condition atomically and only one worker succeeds. The other receives a ConditionalCheckFailedException and gracefully backs off.

The following code example is from the worker application (worker.py), which initializes the Amazon DynamoDB table client, worker ID, and configuration at startup. The complete implementation is available in the GitHub repository.

async def try_acquire_lease(pk: str) -> Optional[dict]:
    """Attempt to acquire lease on a connection."""
    try:
        resp = table.update_item(
            Key={"pk": pk},
            UpdateExpression=(
                "SET lease_owner = :w, "
                "lease_expires_at_ms = :exp, "
                "updated_at_ms = :now"
            ),
            ConditionExpression=(
                "attribute_not_exists(lease_expires_at_ms) "
                "OR lease_expires_at_ms < :now"
            ),
            ExpressionAttributeValues={
                ":w": WORKER_ID,
                ":exp": now_ms() + LEASE_SECONDS * 1000,
                ":now": now_ms(),
            },
            ReturnValues="ALL_NEW",
        )
        return resp["Attributes"]
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return None  # Another worker already owns this connection
        raise

The ConditionExpression is the critical piece: it succeeds when the lease does not exist yet (attribute_not_exists) or has already expired (lease_expires_at_ms < :now).

Renew

The owning worker renews its lease every few seconds (the heartbeat). The conditional expression verifies the worker still owns the lease:

async def renew_lease(pk: str) -> bool:
    """Renew lease for owned connection."""
    try:
        table.update_item(
            Key={"pk": pk},
            UpdateExpression=(
                "SET lease_expires_at_ms = :exp, "
                "updated_at_ms = :now"
            ),
            ConditionExpression="lease_owner = :w",
            ExpressionAttributeValues={
                ":w": WORKER_ID,
                ":exp": now_ms() + LEASE_SECONDS * 1000,
                ":now": now_ms(),
            },
        )
        return True
    except ClientError:
        return False  # Lost ownership

If renewal returns False, the worker knows it has lost ownership (perhaps another worker acquired the expired lease) and exits cleanly.

Release

During graceful shutdown, the worker explicitly releases its leases so other workers can acquire them immediately rather than waiting for expiration:

async def release_lease(pk: str):
    """Release lease on connection."""
    try:
        table.update_item(
            Key={"pk": pk},
            UpdateExpression=(
                "SET lease_owner = :empty, "
                "lease_expires_at_ms = :zero"
            ),
            ConditionExpression="lease_owner = :w",
            ExpressionAttributeValues={
                ":w": WORKER_ID,
                ":empty": "",
                ":zero": 0,
            },
        )
    except ClientError:
        pass  # Already released or taken by another worker

Expired

When a worker stops unexpectedly, because of a container crash, network partition, or process failure, it can no longer renew its lease. Unlike graceful shutdown, the worker has no opportunity to explicitly release ownership. The lease remains in DynamoDB with the crashed worker’s lease_owner value, but the lease_expires_at_ms timestamp passes without renewal.

This expired lease represents a connection with no active owner: desired_state remains STARTED (the connection should be active) but no healthy worker is managing it. The connection is now an orphan.

The reconciliation loop detects this condition by querying the GSI for records where desired_state = STARTED and lease_expires_at_ms < now. Any healthy worker that finds such a record can attempt to acquire it using the same conditional write used during initial acquisition. Because lease_expires_at_ms < :now is one of the valid conditions for acquisition, the expired lease is treated identically to an unclaimed one.

The Expired state is transient: it exists between the moment a lease stops being renewed and the moment the reconciliation loop runs and a new worker successfully acquires it. The maximum time a connection spends in the Expired state is bounded by the reconciliation interval (default: 60 seconds).

Technical implementation

The following sections walk through each component of the system, starting with how events enter the pipeline and ending with how the fleet scales.

Event ingestion

When an external system needs to start or stop a streaming connection, it sends an event to the Lambda event router through API Gateway. The Lambda function writes the connection state to DynamoDB and enqueues a notification to SQS:

def handler(event, context):
    payload = json.loads(event.get("body", "{}"))
    event_type = payload["event_type"].upper()
    connection_id = payload["connection_id"]
    pk = f"CONN#{connection_id}"

    if event_type == "START":
        table.put_item(Item={
            "pk": pk,
            "desired_state": "STARTED",
            "ws_url": payload["ws_url"],
            "last_seq": 0,
            "lease_owner": "",
            "lease_expires_at_ms": 0,
            "updated_at_ms": now_ms(),
        })
        sqs.send_message(
            QueueUrl=QUEUE_URL,
            MessageBody=json.dumps({"pk": pk})
        )

    elif event_type == "STOP":
        table.update_item(
            Key={"pk": pk},
            UpdateExpression="SET desired_state = :s, updated_at_ms = :t",
            ExpressionAttributeValues={
                ":s": "STOPPED", ":t": now_ms()
            },
        )

    return {"statusCode": 200, "body": "OK"}

DynamoDB is the source of truth for connection state. Amazon SQS serves as a fast notification channel. When a START event arrives, the SQS message immediately notifies available workers that they can claim a new connection, so workers do not need to wait for the next reconciliation cycle (default: 60 seconds) to discover and acquire the new connection. Without SQS, new connections would only be picked up when the reconciliation loop queries the GSI for unmanaged connections on its next scheduled run.

Worker polling

Each ECS Fargate worker runs a continuous SQS polling loop to pick up new connection notifications. The loop follows four steps before starting a new WebSocket connection:

1. Capacity check

Before accepting any new work, the worker checks whether it has reached its maximum connection limit (MAX_CONNECTIONS). If the worker is at capacity, it pauses for 5 seconds and skips the current polling cycle. This prevents a single worker from being overwhelmed while other workers in the fleet remain underutilized.

2. Deduplication

If the worker already manages the connection referenced in the SQS message (tracked in its local connections dictionary), it deletes the message and moves on. This handles cases where the same connection generates multiple SQS notifications, for example during retries or redeliveries.

3. Lease acquisition before WebSocket start

The SQS message is a hint, not a guarantee of ownership. Before starting a WebSocket connection, the worker must successfully acquire the DynamoDB lease using try_acquire_lease. If another worker has already claimed the connection, try_acquire_lease returns None and this worker skips it. This ensures exactly one worker owns each connection at any time.

4. Task creation

If the lease is acquired and desired_state is STARTED, the worker creates an async task to manage the WebSocket connection. The SQS message is then deleted regardless of whether the lease was acquired, preventing repeated reprocessing of the same notification.

The following code shows the full polling loop implementation:

async def poll_sqs():
    while not shutdown_event.is_set():
        if len(connections) >= MAX_CONNECTIONS:
            await asyncio.sleep(5)
            continue

        resp = await asyncio.to_thread(
            sqs.receive_message,
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=1,
            WaitTimeSeconds=10,
            VisibilityTimeout=30,
        )

        for msg in resp.get("Messages", []):
            body = json.loads(msg["Body"])
            pk = body["pk"]

            if pk in connections:
                sqs.delete_message(
                    QueueUrl=QUEUE_URL,
                    ReceiptHandle=msg["ReceiptHandle"]
                )
                continue

            conn_data = await try_acquire_lease(pk)
            if conn_data and conn_data.get("desired_state") == "STARTED":
                asyncio.create_task(
                    manage_websocket(
                        pk, conn_data["ws_url"],
                        conn_data.get("last_seq", 0)
                    )
                )
            sqs.delete_message(
                QueueUrl=QUEUE_URL,
                ReceiptHandle=msg["ReceiptHandle"]
            )

Connection management

Once a worker acquires a lease, it opens a WebSocket connection to the upstream source and runs three concurrent async tasks for the lifetime of that connection. These three tasks work together to keep the connection alive, process incoming data, and detect when the connection should stop.

1. Heartbeat loop

The heartbeat loop calls renew_lease every HEARTBEAT_EVERY seconds. If renewal fails, meaning another worker has taken ownership or the lease record has changed, the loop exits immediately. This is the mechanism by which a worker detects that it has lost ownership of a connection mid-flight.

2. Receive loop

The receive loop processes every incoming message from the upstream WebSocket source. Each message is written to a separate DynamoDB messages table with the connection ID, a timestamp, the message data, and the worker ID. The loop runs continuously until the WebSocket connection closes or an error occurs.

3. Desired state checker

Every 10 seconds, the desired state checker reads the connection record from DynamoDB. If desired_state has been set to STOPPED, meaning an external system sent a STOP event through the API, the loop exits, signaling that this connection should be closed even though the WebSocket itself is still open.

How the three tasks interact

All three tasks run concurrently using asyncio.gather. When any one of the three tasks returns or raises an exception, asyncio.gather completes and execution moves to the finally block. This means a single trigger, lease loss, WebSocket closure, or a STOP event, is sufficient to cleanly end the connection regardless of the state of the other two tasks.

Cleanup

The finally block always runs, regardless of how the connection ended. It releases the DynamoDB lease so other workers can acquire the connection immediately and removes the connection from the worker’s local tracking dictionary.

The following code shows the full connection management implementation:

async def manage_websocket(pk: str, ws_url: str, last_seq: int):
    connections[pk] = {"pk": pk, "ws_url": ws_url, "ws": None}

    try:
        async with websockets.connect(ws_url) as ws:
            connections[pk]["ws"] = ws

            async def heartbeat_loop():
                while not shutdown_event.is_set():
                    await asyncio.sleep(HEARTBEAT_EVERY)
                    if not await renew_lease(pk):
                        print(f"[{pk}] Lost lease, closing")
                        return

            async def receive_loop():
                async for msg in ws:
                    data = json.loads(msg)
                    messages_table.put_item(Item={
                        "pk": pk,
                        "sk": str(now_ms()),
                        "message_data": data.get("data", str(data)),
                        "timestamp_ms": now_ms(),
                        "worker_id": WORKER_ID,
                    })

            async def check_desired_state():
                while not shutdown_event.is_set():
                    await asyncio.sleep(10)
                    resp = table.get_item(Key={"pk": pk})
                    if resp.get("Item", {}).get("desired_state") == "STOPPED":
                        return

            await asyncio.gather(
                heartbeat_loop(),
                receive_loop(),
                check_desired_state()
            )

    except Exception as e:
        print(f"[{pk}] WebSocket error: {e}")
    finally:
        await release_lease(pk)
        connections.pop(pk, None)

Production note: The code samples use print() for clarity. In production, replace these with structured logging (the Python logging module or Amazon CloudWatch Logs) and emit CloudWatch metrics for lease acquisition failures and reconnection events to support operational alerting.

Scaling note: The per-connection check_desired_state() loop shown here works for small fleets. At scale, replace individual GetItem calls with a single centralized loop that uses BatchGetItem to check the state of all active connections in one call, reducing DynamoDB reads from N calls every 10 seconds to 1 batched call.

Orphan reconciliation

The reconciliation loop is the safety net of the system. It runs on every worker periodically, independent of the SQS polling loop. Its sole purpose is to find connections that should be active but have no current owner, and reacquire them.

The loop queries the GSI for all records where desired_state = STARTED and lease_expires_at_ms is less than the current time. These are connections that an external system has requested as active, but whose lease has either never been claimed or has expired without renewal, indicating the previous owner is no longer running.

For each orphaned connection found, the worker calls try_acquire_lease. Because try_acquire_lease uses a DynamoDB conditional write, multiple workers can safely run reconciliation concurrently without risk of double-claiming. Exactly one worker succeeds for each connection. The others receive a ConditionalCheckFailedException and move on.

The reconciliation interval (default: 60 seconds) determines the maximum recovery time for unexpected worker terminations. A worker that crashes without running its graceful shutdown handler leaves its leases to expire naturally after LEASE_SECONDS (default: 20 seconds). The reconciliation loop then picks up those connections within the next 60-second cycle, giving a worst-case recovery time of approximately 80 seconds (20 seconds lease expiry plus up to 60 seconds reconciliation interval).

The following code shows the full implementation:

async def reconcile_orphaned_connections():
    while not shutdown_event.is_set():
        await asyncio.sleep(RECONCILE_EVERY)

        if len(connections) >= MAX_CONNECTIONS:
            continue

        resp = table.query(
            IndexName=GSI_NAME,
            KeyConditionExpression=(
                "desired_state = :state "
                "AND lease_expires_at_ms < :now"
            ),
            ExpressionAttributeValues={
                ":state": "STARTED",
                ":now": now_ms()
            },
            Limit=RECONCILE_PAGE_SIZE,
        )

        for item in resp.get("Items", []):
            pk = item["pk"]
            if pk not in connections and len(connections) < MAX_CONNECTIONS:
                conn_data = await try_acquire_lease(pk)
                if conn_data:
                    asyncio.create_task(
                        manage_websocket(
                            pk, conn_data["ws_url"],
                            conn_data.get("last_seq", 0)
                        )
                    )

Figure 3: Automatic failover through orphan reconciliation

Graceful shutdown

When ECS sends a SIGTERM signal during a rolling deployment or scale-in event, the worker has a limited window to clean up before the container is forcibly terminated. Rather than dropping connections abruptly and waiting for leases to expire naturally, the worker performs a coordinated shutdown in three steps.

Step 1: Signal propagation

The signal_handler function sets a shared shutdown_event when SIGTERM is received. This event is checked by every running loop across all active connections. The heartbeat loop, the desired state checker, and the reconciliation loop all exit their while not shutdown_event.is_set() loops as soon as the event is set. No additional per-connection shutdown logic is needed. The shared event propagates the shutdown signal automatically to all concurrent tasks.

Step 2: Parallel cleanup

Rather than closing connections and releasing leases sequentially, which would take longer as the number of active connections grows, the worker closes all WebSocket connections and releases all leases concurrently using asyncio.gather. For a worker managing hundreds of connections, this keeps the total shutdown time roughly constant regardless of connection count.

Step 3: Immediate lease release

During graceful shutdown, the worker sets lease_expires_at_ms = 0 for each released connection. A value of 0 means the lease appears already expired to any worker running a reconciliation query. Other workers in the fleet pick up the released connections on their next reconciliation cycle rather than waiting for the original lease duration (default: 20 seconds) to elapse naturally.

Contrast with unexpected termination

Graceful shutdown is the fast path. When a worker exits cleanly through SIGTERM, connections are available for reacquisition within one reconciliation cycle. When a worker crashes unexpectedly without running the shutdown handler, leases expire naturally after LEASE_SECONDS (default: 20 seconds) and are then picked up by the reconciliation loop. Both paths converge on the same outcome, another worker acquires the connection, but graceful shutdown is significantly faster.

The following code shows the full graceful shutdown implementation:

shutdown_event = asyncio.Event()

def signal_handler(signum, frame):
    shutdown_event.set()

async def graceful_shutdown():
    await shutdown_event.wait()
    tasks = []
    for pk, conn in list(connections.items()):
        if conn.get("ws"):
            tasks.append(conn["ws"].close())
        tasks.append(release_lease(pk))
    await asyncio.gather(*tasks, return_exceptions=True)

Setting shutdown_event causes the heartbeat loops and state checkers to exit their while not shutdown_event.is_set() loops. The graceful_shutdown function then closes the active WebSocket connections and releases its leases in parallel. Released leases have lease_expires_at_ms = 0, which means the reconciliation loop on other workers picks them up on its next cycle rather than waiting for the original lease to expire.

Scaling the fleet

Each worker publishes a custom CloudWatch metric with its active connection count:

async def publish_metrics():
    while not shutdown_event.is_set():
        await asyncio.sleep(30)
        cw.put_metric_data(
            Namespace="WsFleet",
            MetricData=[{
                "MetricName": "ActiveConnections",
                "Value": len(connections),
                "Unit": "Count",
                "Dimensions": [
                    {"Name": "ServiceName", "Value": SERVICE_NAME}
                ],
            }],
        )

An AWS Application Auto Scaling target tracking policy scales the fleet based on the average ActiveConnections metric across all workers. When the average exceeds the target (for example, 700 connections per task), ECS launches additional tasks. New tasks start their SQS polling and reconciliation loops, picking up new connections and rebalancing the fleet.

Figure 4: Automatic scaling based on active connection count

Scale-in is safe because of the lease pattern. When ECS terminates a task, the worker receives SIGTERM, releases its leases, and other workers acquire the freed connections through reconciliation.

Tuning guidance. These values represent a starting point. Adjust based on your requirements:

• Lease duration: Start with 20s. Reduce for faster failover, increase if network hiccups cause false expirations.
• Heartbeat interval: Keep below lease duration. A 4:1 ratio (lease:heartbeat) gives 4 renewal attempts before expiry.
• Reconciliation interval: Start with 60s. Reduce for faster recovery from unexpected terminations, increase to lower DynamoDB read cost.
• Max connections per task: Start with 100 and increase while monitoring memory and CPU utilization in CloudWatch Container Insights. Each WebSocket connection typically consumes 2-5 MB of memory depending on message throughput.

DynamoDB cost considerations

The dominant cost driver in this architecture is heartbeat writes. Each active connection generates one update_item call per heartbeat interval, consuming 1 WCU. At the default 5-second heartbeat interval:

For production deployments at sustained high connection counts, use provisioned capacity with Auto Scaling rather than on-demand pricing. Heartbeat writes are predictable and consistent, which makes them well-suited to provisioned throughput. Configure Auto Scaling on your provisioned capacity to track connection count changes as the fleet scales.

To reduce cost, consider the following adjustments:

1. Increase the heartbeat interval. Doubling the heartbeat interval from 5 seconds to 10 seconds halves WCU consumption. Maintain the 4:1 lease-to-heartbeat ratio by also doubling the lease duration. This increases the failover window proportionally.
2. Increase the reconciliation interval. Increasing from 60 seconds to 120 seconds halves RCU consumption from reconciliation queries. This slows recovery from unexpected terminations.
3. Use BatchGetItem for desired state checks. Replace the per-connection get_item calls in the check_desired_state loop with a single BatchGetItem call covering all active connections. This reduces RCU consumption from N reads per cycle to 1 batched read per cycle.

   GSI queries during reconciliation use eventually consistent reads by default, which halves the RCU cost compared to strongly consistent reads. Monitor your GSI read consumption in the DynamoDB console and adjust the reconciliation page size and interval to stay within your cost targets.

Conclusion

Managing long-lived WebSocket connections at scale requires explicit ownership tracking, automatic failover, and coordination across a fleet of workers. This post showed you a pattern that addresses these challenges using DynamoDB conditional writes as a distributed lease mechanism.

Key takeaways:

• You can use DynamoDB conditional writes for atomic distributed coordination without external lock services. The ConditionExpression on update_item helps confirm one worker owns each connection at a time.
• The heartbeat and reconciliation pattern handles the full failure spectrum. Lease expiration detects unexpected worker terminations. Graceful shutdown handles rolling deployments. New workers acquire leases and departing workers release them, making scaling safe.
• This pattern applies to systems that manage long-lived WebSocket connections at scale: real-time transcription, IoT data ingestion, financial feed processing, or live event streaming.

Getting started

The complete implementation, including the worker application, Lambda event router, and Terraform templates for the DynamoDB table, SQS queue, and ECS cluster, is available in the GitHub repository. Follow the instructions in the repository README to deploy the infrastructure and validate the lease lifecycle with a small set of test connections.

For further enhancements, add distributed tracing with AWS X-Ray for end-to-end visibility across workers, and implement reconnection logic with upstream replay or offset-based resumption to handle data gaps between worker failure and recovery.

Further reading

• Amazon DynamoDB conditional writes – Explore condition expressions for implementing distributed locks.
• Amazon DynamoDB Global Secondary Indexes – Understand how to query for expired leases efficiently.
• Amazon ECS on AWS Fargate – Learn how to run containerized workers without managing servers.
• Application Auto Scaling – Configure target tracking policies to scale your fleet based on custom metrics.
• Amazon CloudWatch custom metrics – Publish application-level metrics for scaling decisions.