Post Syndicated from HoSeong Lee original https://aws.amazon.com/blogs/devops/optimize-eks-operations-with-agents-reduce-mttr-with-aws-devops-agent-and-a-kubernetes-operator/

Introduction

Running workloads on Amazon Elastic Kubernetes Service (Amazon EKS) can involve managing failures like OOMKilled or IP exhaustion. Engineers must repeatedly collect pod logs, trace events, and check node logs—a process that slows at night/weekends, with critical data lost when pods are deleted or nodes become unhealthy. This collection phase is pure overhead on mean time to resolution (MTTR): the incident stays open while an engineer gathers data that a machine could have captured the moment the failure occurred. Automating it shortens MTTR and lets the on-call engineer start at the analysis step instead of the data-gathering step.

Existing AI tools have limitations: K8sGPT only analyzes current resource state, and Amazon Bedrock Agents requires manual tool integration and pipeline setup. Neither provides end-to-end automated incident investigation.

AWS DevOps Agent addresses these gaps—a frontier agent that connects code repositories, observability tools, CI/CD pipelines, and skills to autonomously analyze root causes. This post shows how to build an automated incident response pipeline using the DevOps Agent Operator, a Kubernetes Operator that detects EKS failures and triggers DevOps Agent investigations automatically.

Solution overview

AWS DevOps Agent provides powerful incident analysis. However, it does not detect pod failures inside an EKS cluster on its own. To start an investigation, an external source must trigger DevOps Agent through a webhook. When this trigger occurs, two conditions must be met:

1. Immediate failure detection: You must detect the failure before the pod is rescheduled or deleted.
2. Sufficient context: You must send the data that the analysis needs, such as the manifest, logs, events, and node information.

The DevOps Agent Operator is a Kubernetes Operator that meets both conditions automatically.

Why use an Operator?

DevOps Agent runs only when something calls it through a webhook or a manual trigger. In 24/7 operations, doing this manually is not practical. Kubernetes keeps events for only about an hour, restarted containers overwrite their logs, and deleted pods lose them entirely. If you do not collect data right after a failure, the key evidence is gone for good.

DevOps Agent can already run describe and logs with kubectl, and tools like Datadog can detect failures and trigger it.

A separate Operator still adds value for three reasons:

1. Proactive preservation of volatile data: The Operator detects state changes in milliseconds via watch and preserves data to S3/CloudWatch instantly—before external tool delays (metric collection, alert evaluation, webhook delivery) let evidence disappear.
2. Selective collection of node-level data: kubectl exposes only container-level and event data, but root causes often live deeper in the node—for example, OOMKilled traces to node dmesg, and IP exhaustion details are in IPAMD introspection. Because the Operator knows the real-time pod-to-node mapping, it collects only what each failure type needs from the exact node.
3. Encoding operational knowledge in code: The Operator pattern captures human expertise in code, applying different strategies per failure type—dmesg/memory for OOMKilled, previous logs/restart history for CrashLoopBackOff, IPAMD/ENI mappings for IP exhaustion—directly improving analysis accuracy.

In short, the Operator captures evidence at the failure site before it disappears and collects data beyond the reach of kubectl, giving DevOps Agent the best possible material to analyze.

Note: If data collection or an upload to Amazon S3 or CloudWatch Logs fails, the reconcile returns an error and the pod is requeued with exponential backoff rather than dropped, and throttled AWS API requests are retried automatically. The Operator also runs a single reconcile worker and marks each pod with a processed annotation, so a mass failure—for example, 100 replicas crashing at once—is handled one pod at a time and each pod is reported only once. For noisy clusters, WEBHOOK_MIN_SEVERITY and WEBHOOK_SKIP_CATEGORIES let you narrow which failures trigger an investigation.

Architecture

The preceding diagram shows the full flow. The DevOps Agent Operator detects a failure inside the EKS cluster and sends the context to AWS DevOps Agent.

Getting started

Prerequisites

• Region availability: AWS DevOps Agent is available in six AWS Regions—US East (N. Virginia), US West (Oregon), Europe (Frankfurt), Europe (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo). Create your Agent Space in one of these Regions.
• Node type: Node-level log collection uses AWS Systems Manager Run Command against the EC2 instance that ran the failed pod, so it requires Amazon EKS managed node groups or self-managed EC2 nodes. On AWS Fargate, the Operator still collects Kubernetes-level data—the pod manifest, events, and container logs—but node-level data such as dmesg output and IPAMD introspection is not available.
• Systems Manager registration: Attach the AmazonSSMManagedInstanceCore policy to your node group’s IAM role so the nodes appear as managed nodes. Without it, node-level collection is skipped and only Kubernetes-level data is collected.

Setting up this solution involves two steps.

The first step is to configure the Agent Space for DevOps Agent. You connect the sources that DevOps Agent needs to analyze an incident, such as code repositories and observability tools. You also set up a generic webhook to receive failure information from the Operator.

The second step is to deploy the DevOps Agent Operator to the EKS cluster. When the Operator detects a pod failure, it collects the context and sends it automatically to the webhook that you set up in the first step.

After you complete these steps, you have an end-to-end pipeline. When a pod failure occurs, DevOps Agent starts an investigation automatically.

Step 1: Configure the Agent Space for DevOps Agent

Configure the webhook

DevOps Agent supports two types of webhooks:

• Integration-specific webhooks: Created automatically when you set up an integration with an external solution, such as Slack or Datadog.
• Generic webhooks: Created manually to trigger an investigation from sources that an external integration does not cover.

The DevOps Agent Operator uses a generic webhook. It maintains security through HMAC-SHA256 authentication.

For detailed setup instructions, see the following documentation. This post creates a generic webhook as an example.

• Configuring webhook access

Configure the pipeline

You connect GitHub or GitLab so that DevOps Agent can track deployment events and correlate code changes with failures.

1. Register GitHub or GitLab at the AWS account level.
2. Connect the repositories that you want to monitor to the Agent Space.

With this connection, DevOps Agent can analyze the recent deployment history and code changes when a failure occurs. DevOps Agent currently supports GitHub and GitLab. For GitLab, you can use both the managed instance and a self-managed instance that is reachable from outside.

For detailed setup instructions, see the following documentation. This post uses GitHub as an example.

• Connecting to CI/CD pipelines

Configure communication

DevOps Agent joins your team’s existing communication channels to share its investigation activity. When you connect Slack, you can follow the full process in real time, from failure detection to completed analysis.

For detailed setup instructions, see the following documentation. This post uses Slack as an example.

• Connecting to ticketing and chat

Step 2: Deploy the DevOps Agent Operator

To install the DevOps Agent Operator, complete the prerequisite steps and the Operator deployment steps in order.

Before you continue, download the source code. You can find the source code at the following link: DevOps Agent Operator source code

1. Prerequisite steps

Before you deploy the Operator to an existing EKS cluster, complete the following prerequisite steps.

1.1. Create an IAM policy for SSM, Amazon S3, and CloudWatch

cat >devops-agent-operator-permission.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SSMCommandExecution",
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand",
        "ssm:GetCommandInvocation"
      ],
      "Resource": [
        "arn:aws:ec2:<aws-region>:*:instance/*",
        "arn:aws:ssm:<aws-region>:*:*"
      ]
    },
    {
      "Sid": "S3LogStorage",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>/*"
    },
    {
      "Sid": "S3BucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>"
    },
    {
      "Sid": "CloudWatchLogsIncidentStorage",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<aws-region>:*:log-group:/<cloudwatch-log-group-name>:*"
    }
  ]
}
EOF

Note: To keep this example readable, the policy allows ssm:SendCommand on EC2 instances in the account. In production, restrict it to your cluster’s nodes with an IAM condition key—for example, a StringEquals condition on ssm:resourceTag/eks:cluster-name in a statement that targets only the instance ARN—so that the Operator cannot run commands on unrelated instances. Keep the AWS-RunShellScript document ARN in a separate statement without the condition, because a document carries no instance tags and a single combined statement would deny the call.

Next, create the policy from this file.

aws iam create-policy \
    --policy-name devops-agent-operator-policy \
    --policy-document file://devops-agent-operator-permission.json

1.2. Create a trust policy

cat >devops-agent-operator-trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
EOF

1.3. Create an IAM role

aws iam create-role \
    --role-name devops-agent-operator-role \
    --assume-role-policy-document file://devops-agent-operator-trust-policy.json

aws iam attach-role-policy --role-name devops-agent-operator-role --policy-arn=arn:aws:iam::<aws-account-id>:policy/devops-agent-operator-policy

1.4. Associate Pod Identity

EKS Pod Identity associates Kubernetes service accounts directly with IAM roles, enabling pods to access AWS services like Amazon CloudWatch under the principle of least privilege. For more information, see Learn how EKS Pod Identity grants pods access to AWS services.

Pod Identity requires the eks-pod-identity-agent add-on, which is not installed on existing clusters by default. If your cluster does not have it yet, add it first:

aws eks create-addon \
    --cluster-name <eks-cluster-name> \
    --addon-name eks-pod-identity-agent

Then create the association:

aws eks create-pod-identity-association
  --cluster-name <eks-cluster-name>
  --namespace devops-agent-operator-system
  --service-account devops-agent-operator
  --role-arn arn:aws:iam::<aws-account-id>:role/devops-agent-operator-role

2. Build the image

Because the Operator is a reference implementation, the sample provides source code only—no prebuilt container image.
Build the image with the Dockerfile at the following location and push it to a registry that you control, which also keeps the image that runs in your cluster inside your own supply chain. Then use that image to deploy the Operator.
Building the image locally requires Go 1.25 or later. The Operator is built against the Kubernetes 1.35 client libraries and uses only the core Pod, Node, and Event APIs.

• Dockerfile location

For example, suppose that you create a separate repository from all the files under Devops Agent Operator – Sample

You can then build the image through CI/CD with the following GitHub Action as a reference.

name: Build and Push container images to GitHub Container Registry
jobs:
  ...
  build-and-push:
    name: Build and Push Image
    runs-on: ubuntu-latest
    needs: create-tag
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.WRITE_REGISTRY_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Build and Push
        uses: docker/build-push-action@v6
        with:
          context: .
          file: Dockerfile
          push: true
          provenance: false
          no-cache: true
          tags: |
            "ghcr.io/${{ github.repository_owner }}/devops-agent-operator:${{ needs.create-tag.outputs.sha_short }}"
            "ghcr.io/${{ github.repository_owner }}/devops-agent-operator:latest"

3. Deploy the Operator

The following steps are based on the example YAML files for the DevOps Agent Operator. Download the repository, or run the following command to download the files, and then continue.

curl -s https://api.github.com/repos/aws-samples/kr-tech-blog-sample-code/contents/containers/devops-agent-operator/examples?ref=main | jq -r '.[].download_url' | xargs -n1 curl -O

3.1. Set the environment variables

Open the 05-deployment.yaml file, and then change the following variables to values that match your environment.

containers:
- name: manager
    # Use the image that you built in step 2
    image: <operator-image>:latest
    ...
    env:
    # Required settings
    - name: DEVOPS_AGENT_WEBHOOK_URL
        value: "<devops-agent-webhook-url>"
    ...
    - name: EKS_CLUSTER_NAME
        value: "<eks-cluster-name>"
    - name: AWS_REGION
        value: "<aws-region>"
    - name: AWS_ACCOUNT_ID
        value: "<aws-account-id>"
    # Optional settings
    - name: ENABLE_SSM_COLLECTION
        value: "true"
    - name: CLOUDWATCH_LOG_GROUP
        value: "<cloudwatch-log-group-name>"

Also change the 04-configmap.yaml file to values that match your environment.

data:
  # Comma-separated list of namespaces to watch (empty = all namespaces)
  WATCH_NAMESPACES: ""
  # Comma-separated list of namespaces to exclude
  EXCLUDE_NAMESPACES: "kube-system,kube-public,kube-node-lease"
  # Enable AWS SSM node log collection (requires IAM permissions)
  ENABLE_SSM_COLLECTION: "true"
  # AWS region for SSM and S3
  AWS_REGION: "<aws-region>"
  ...

In a shared or multi-tenant cluster, set WATCH_NAMESPACES to the namespaces that your team owns so that the Operator does not collect data from other teams’ workloads. If you leave it empty, the Operator watches every namespace except those listed in EXCLUDE_NAMESPACES.

Note: DevOps Agent references the collected data only while it investigates the incident, so you do not need to retain it long-term. Keeping a short retention period on the CloudWatch log group—and a matching S3 Lifecycle expiration rule on the bucket—keeps the storage cost of this solution minimal.

# Expire the incident logs in CloudWatch Logs after 14 days
aws logs put-retention-policy \
    --log-group-name <cloudwatch-log-group-name> \
    --retention-in-days 14

# Expire the incident objects in Amazon S3 after 14 days
aws s3api put-bucket-lifecycle-configuration \
    --bucket <s3-bucket-name> \
    --lifecycle-configuration '{"Rules":[{"ID":"expire-incident-data","Status":"Enabled","Filter":{"Prefix":"incidents/"},"Expiration":{"Days":14}}]}'

3.2. Create the webhook secret

Edit the 06-webhook-secret.yaml file:

stringData:
  webhook-secret: "<webhook-secret>"

3.3. Deploy the Kubernetes resources

kubectl apply -f .

The example deployment runs a single replica with leader election enabled, so you can raise the replica count for availability without two Operators processing the same failure.

3.4. Verify the deployment

# Check the pod status
kubectl get pods -n devops-agent-operator-system

# Check the logs
kubectl logs -f deployment/devops-agent-operator \
  -n devops-agent-operator-system

When the Operator works correctly, it produces the following logs:

Configuration loaded
Log collector initialized (sinceMinutes: 15)
Webhook client initialized
CloudWatch Logs client initialized
S3 client initialized
Starting workers (worker count: 1)

Use case: Automated analysis of an OOMKilled failure

The following scenario shows how the DevOps Agent Operator and DevOps Agent work together. In this environment, Slack is connected as the notification channel for DevOps Agent, and GitHub is connected as the pipeline.

Scenario

In this scenario, a developer pushed a code change to add a new feature to the web-python service and built a new container image. The developer then updated the running web-python deployment in the EKS cluster with the newly built image.

After the new version rolled out successfully, the developer verified that other services were unaffected. Shortly after, a Slack notification arrived. DevOps Agent reported that the pod that was just deployed had terminated with an OOMKilled status, and that it was investigating the related incident.

kubectl get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,IMAGE:.spec.containers[*].image'
NAME READY STATUS RESTARTS IMAGE
web-python-56b9874b88-tdljd true Running 0 <your-registry>/web-python:sha-96cd2b0

# Deploy the new version
kubectl get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,IMAGE:.spec.containers[*].image'
NAME READY STATUS RESTARTS IMAGE
web-python-645b4f7867-lvqgr true Running 0 <your-registry>/web-python:sha-15d1398

The following steps describe what happens after the pod with the new image is deployed.

Step-by-step flow

1. Failure detection

The kubelet detects the OOM termination of the web-python container and updates the pod status. The informer in the DevOps Agent Operator receives this change in real time. It detects the change from the previous state (Running) to the current failure state (OOMKilled).

kubectl describe po web-python-645b4f7867-lvqgr
Name: web-python-645b4f7867-lvqgr
Namespace: default
...
Annotations: devops-agent.io/failure-type: OOMKilled
                  devops-agent.io/processed: true
                  devops-agent.io/processed-at: 2026-05-30T07:24:43Z

2. Kubernetes-level data collection

As soon as the Operator detects the failure, it collects Kubernetes-level data including pod manifests, pod logs, previous crash logs, and OOM-related event timelines.

3. Node-level data collection

It then gathers node-level data such as kubelet, containerd, and ipamd logs, disk/memory/network usage, and the kernel OOM killer log from dmesg output.

4. Data storage

Based on your configuration, the Operator stores the collected data in CloudWatch Logs and Amazon S3. DevOps Agent can reference the data in CloudWatch Logs during the investigation when it needs to.

5. DevOps Agent trigger

The Operator sends a webhook request that includes an HMAC-SHA256 signature to DevOps Agent. The payload includes investigation instructions for the AI agent.

The DevOps Agent Operator handles steps 1 through 5. You can also see these steps in the logs of the Operator pod.

# 1. Failure detection
2026-05-30T07:24:05Z INFO Failure detected {"controller": "pod", ... "pod": {"name":"web-python-645b4f7867-lvqgr","namespace":"default"}, "failureType": "OOMKilled", "container": "web-python", "exitCode": 137}

# 2-3. Data collection
2026-05-30T07:24:06Z INFO ssm-collector Collecting node logs via SSM {"node": "ip-192-168-1-10.ec2.internal", "instanceID": "i-0123456789abcdef0"}
...

# 4. Data storage
2026-05-30T07:24:08Z INFO cloudwatch CloudWatch Logs upload completed {"logGroup": "cw-log-group-devops-agent-operator", "logStream": "incidents/2026-05-30T07-24-05Z/default/web-python-645b4f7867-lvqgr", "eventsCount": 13}
...

# 5. DevOps Agent trigger
...
2026-05-30T07:24:08Z INFO webhook Webhook request with S3 reference successful {"incidentId": "2026-05-30T07-24-05Z/default/web-python-645b4f7867-lvqgr", "status": 200}

6-7. DevOps Agent Investigation

As DevOps Agent starts the investigation, it shares the incident and its investigation status in the Slack channel that you configured for communication. Through this notification, the engineer can open the Agent Space and follow the investigation in real time.

Skill-based investigation: DevOps Agent automatically selects the skill that matches the incident type. Following the OOMKilled skill, it systematically performs the steps to check the memory configuration, analyze usage patterns, and review the code change history.

Correlation analysis: In addition to the troubleshooting data that it receives, DevOps Agent connects the following sources for its analysis:

• GitHub: Checks recent code changes for memory-related modifications.
• CloudWatch: Checks memory usage trends in Container Insights.

In this scenario, you can see that DevOps Agent starts its analysis from the data that the Operator uploaded to CloudWatch Logs, as the skill specifies.

The skill also specifies the relationship between the GitHub repository that you connected as a pipeline and the container image. DevOps Agent uses this information to review the code changes that occurred recently.

This information helps DevOps Agent identify the root cause of the incident.

8. Analysis results

DevOps Agent organizes the analysis results:

• Investigation Timeline: This tab shows the agent’s investigation steps—which skills it referenced and what data it analyzed.
  This view helps you optimize the skill to guide investigations more efficiently.
• Root causes: This section summarizes the root cause from the overall investigation.

Unbounded `processed_records` list in web-python application causes memory leak at ~20Mi/min
The Python Flask application in image `<your-registry>/web-python:sha-15d1398` contains a background worker thread (`_cache_worker`) that generates 500 records every 2 seconds and appends processed results to an in-memory list called `processed_records`. Unlike the `cache` list which has eviction logic capped at 80MB (`CACHE_SIZE_MB`), the `processed_records` list has NO eviction or size limit — it grows unboundedly. With Python/Flask overhead (~30MB) + the cache growing toward its 80MB cap, the remaining headroom within the 200Mi container memory limit is exhausted in approximately 10 minutes. This was confirmed by two consecutive pod instances (lvqgr and 7rwdj) both being OOMKilled after exactly ~10 minutes of runtime.

With the investigation from DevOps Agent, the engineer can identify the cause of the problem.

In the preceding example, you can see how the agent identifies a critical memory leak in the recently changed service code. It then reasons about the cause of the OOM event together with the commit ID.

9. Analysis and mitigation plan through chat

The engineer reviews the results and, when needed, can ask DevOps Agent follow-up questions:

• “Check whether other services show a similar memory growth pattern.”
• “Will fixing it with approach A help solve the problem?”

In the following example, the engineer asks whether increasing the pod memory limit will help solve the problem. The agent responds based on its investigation.

As this shows, DevOps Agent goes beyond simple problem analysis. It uses the context that it accumulated during the investigation to respond to the engineer’s follow-up questions with detailed explanations.

In this scenario, the problem is a logic issue in the source code. For that reason, DevOps Agent could not provide a clear plan at the Kubernetes or AWS infrastructure level. However, based on the root cause, you can receive a mitigation plan related to a rollback.

Conclusion

In this post, we introduced the DevOps Agent Operator – a Kubernetes Operator that automatically detects EKS workload failures, collects diagnostic data, and triggers AWS DevOps Agent for root cause analysis.

By combining these two tools, engineers gain the following benefits:

• Faster response: Automatic data collection and analysis as soon as a failure occurs, even during nights and weekends.
• No loss of information: Immediate preservation of all troubleshooting data before a pod is rescheduled or deleted.
• Comprehensive analysis: DevOps Agent analyzes code repositories, observability tools, and CI/CD pipelines together to trace root causes that are hard to find with a single tool.
• Organizational knowledge: Through skills, the solution reflects your team’s operational knowledge, enabling incident response with consistent quality.
• Continuous improvement: Proactive recommendations based on accumulated incident data help prevent future incidents.

Looking ahead, there are several ways to extend this solution:

• Support for more resource types: Extend monitoring beyond pods to Job, CronJob, Deployment, and StatefulSet.
• MCP server integration: DevOps Agent supports Model Context Protocol (MCP) servers, enabling advanced workflows such as querying additional resources during analysis or performing pattern analysis on past incidents.
• Proactive pattern analysis: As incident data accumulates in Amazon S3 and CloudWatch Logs, DevOps Agent can identify recurring patterns – such as “OOMKilled repeats every Monday morning” – and recommend preventive measures.

The DevOps Agent Operator project is open source on GitHub. It is a reference implementation rather than a supported product: use it as a working example of how to encode your own detection conditions and collection strategy for the failures your team actually sees.

To try it yourself, clone the repository, follow the deployment steps in this post, and point the Operator at your own Agent Space webhook. Start with a non-production cluster and a narrow WATCH_NAMESPACES list, then widen the scope once you see the investigations that DevOps Agent produces.

References

• AWS DevOps Agent official page
• AWS DevOps Agent announcement blog
• DevOps Agent skill guide
• Best Practices for Deploying AWS DevOps Agent in Production
• DevOps Agent Operator source code