Post Syndicated from Nick Beall original https://aws.amazon.com/blogs/compute/building-an-aeo-scoring-pipeline-with-aws-lambda-durable-functions/

When an AI agent crawls your documentation looking for answers, it doesn’t browse like a human. It scores structure, extracts entities, checks factual consistency, and moves on, often in under a second. If your content doesn’t surface clearly in those evaluations, it won’t appear in AI-generated answers regardless of its current search ranking. This problem is called Agent Engine Optimization (AEO), and solving it requires running multiple evaluation steps that are expensive, stateful, and failure-prone.

In this post, you build a five-step AEO scoring pipeline using AWS Lambda durable functions. Each step (fetch, score, analyze, publish, record) runs as a checkpointed operation that suspends without incurring additional costs between stages, retries without re-spending on prior work, and reads like ordinary Python. You score content against five AEO signals, see a real audit example, and walk away with a deployable AWS Serverless Application Model (AWS SAM) template.

A note on fetching: The pipeline uses a plain HTTP GET, mimicking how non-rendering AI crawlers (GPTBot, ClaudeBot, PerplexityBot) see pages today. Sites that rely on client-side JavaScript may score lower because most AI agents do not execute JavaScript.

Solution overview

The pipeline runs as a single Lambda durable function that chains five checkpointed steps. Scoring is fully deterministic (pure Python, no large language model (LLM)), so the same page consistently produces the same score. Amazon Bedrock generates prose analysis explaining the results. For production deployments that surface generated content to end users, consider adding Amazon Bedrock Guardrails for responsible AI controls. Because the function suspends between steps without incurring additional costs, a failure at step 3 resumes from step 3. The deterministic scores from step 2 are already checkpointed.

The following architecture diagram shows the five pipeline stages:

1. Fetch – Retrieve the target page through an HTTP GET and extract structured signals (headings, token count, robots.txt rules, llms.txt, AGENTS.md).
2. Score – Apply the deterministic rubric: threshold-based checks produce a reproducible 0–100 score with no LLM involvement.
3. Analyze – Send the scores and raw signals to Amazon Bedrock, which generates prose findings and fix recommendations (temperature 0 for consistency).
4. Publish – Render the audit report as HTML and JSON, write to Amazon Simple Storage Service (Amazon S3).
5. Record – Persist the score and metadata to Amazon DynamoDB for trend tracking.

Figure 1: The five checkpointed stages of the AEO scoring pipeline, from fetch through record.

This gives you retry safety, cost predictability, and simpler debugging. The pipeline runs fully automated by default, with an optional callback gate for workflows requiring human approval.

Why Lambda durable functions

An AEO audit is five operations that depend on each other, any of which can fail mid-flight. Without checkpointing, a retry means re-running every step from the beginning, including Bedrock calls that already returned valid results.

Lambda durable functions solve this with checkpointed steps. Each step’s output is persisted before the next begins. When a Bedrock call gets throttled at step 3, only step 3 retries. The pipeline doesn’t re-run steps 1 and 2 or re-spend on their inference tokens.

Between steps, the function suspends with no environment held open, so you pay nothing while waiting. The pipeline reads as a single Python file, top to bottom. Each step’s checkpoint is independently inspectable, so debugging a failed run means looking at one step’s result rather than replaying the full execution log.

Prerequisites

• An AWS account with Amazon Bedrock model access enabled.
• AWS SAM CLI v1.100+ installed and configured.
• Python 3.12+.
• AWS Identity and Access Management (IAM) permissions to deploy AWS CloudFormation stacks (Lambda, DynamoDB, Amazon S3, and Amazon Bedrock invoke)

A SAM template in the GitHub repository provisions the DynamoDB table, S3 bucket, Lambda durable function, and required IAM roles. You deploy the full stack with sam build && sam deploy --guided.

Estimated time: 20–30 minutes. Each audit run uses a single Bedrock inference call plus brief Lambda durable function and DynamoDB usage. See the AWS Pricing Calculator for estimates based on your Region and usage volume.

The five AEO signals

Scoring evaluates content against five signals drawn from Addy Osmani’s Agent Engine Optimization framework. Each signal maps to a capability an AI agent needs when deciding whether to use your content. Scoring is deterministic: every point maps to a measurable threshold (file exists, token count under ceiling, heading hierarchy valid). No LLM is involved in producing the number.

1. Discoverability (0–25)

This signal checks whether the URL returns server-rendered HTML, whether an llms.txt file exists at the domain root, and whether AI crawlers receive a successful response. A page returning HTTP 200 with parseable content passes. A page returning 403 or 429 to AI user agents fails.

2. Parsability (0–25)

This signal scores heading hierarchy (H1 through H6), presence of code blocks and tables, and whether the page has semantic landmarks. Because agents parse structurally rather than linearly, it’s essential for sites to be well organized and logical. Osmani suggests serving Markdown and HTML, leading each section with the outcome, and using tables for parameter references, among other tips that make your site easier to parse.

3. Token efficiency (0–25)

It measures total token count against a 30,000-token ceiling and a 15,000-token quick-start target. Pages bloated with navigation elements or repeated boilerplate score lower.

4. Capability signaling (0–15)

This signal checks for AGENTS.md or skill.md files describing available actions, required inputs, authentication, and rate limits. Without these, agents discover capabilities through trial and error.

5. Access control (0–10)

It checks robots.txt for AI-specific rules (GPTBot, ClaudeBot, PerplexityBot). Pages with explicit Allow or Disallow rules for AI user agents score higher than pages with no AI-specific rules, where access is open by default. Either choice is fine. The signal rewards making the decision on purpose.

Deterministic scores feed into the analyze step, where Bedrock generates prose explanations and actionable fix recommendations.

Example: auditing a real page

To see the pipeline in action, here are the results from a technical documentation page: a conceptual overview for a cloud computing service.

Target: A technical documentation page (conceptual overview, 958 words)

Overall score: 58/100

Top three fixes:

1. Create a skill.md declaring what agents can do with this service (available actions, required inputs, constraints).
2. Fix the heading hierarchy to follow valid nesting (H1, H2, H3) and add fenced code blocks with language tags.
3. Add an AGENTS.md at the domain root and explicit AI crawler rules to robots.txt.

The scoring is deterministic, and every point maps to a measurable condition. Amazon Bedrock then runs LLM inference on the results to generate the prose findings.

Under the hood: Steps and callbacks

Every step in the preceding audit, from fetch through record, runs as a checkpointed step. The callback is optional and sits after the analyze step, if you want a human to review the report before it’s published.

Lambda durable functions expose two core primitives: steps and callbacks. This pipeline runs fully automated by default (five steps, no human gate), but you can opt into a callback-based approval gate when the use case calls for it.

Steps

A step is a unit of work. The runtime checkpoints its return value before the next step begins. You declare one with the @step decorator:

@durable_step
def score_content(step_context: StepContext, page_content: str) -> dict:
    response = bedrock.invoke_model(
        modelId=os.environ["BEDROCK_MODEL_ID"],
        body=json.dumps({"prompt": build_scoring_prompt(page_content)})
    )
    return json.loads(response["body"].read())

If this step succeeds, its return value is persisted. On retry, completed steps are skipped and their cached results passed forward. Two rules:

• Keep return values small. Return structured results, not raw API responses.
• One side effect per step. A step that calls Bedrock and also writes to DynamoDB is harder to reason about on retry. Make one external call per step.

Callbacks

By default, the pipeline runs all five steps without stopping. To add a human review gate, turn on the callback. The function suspends without incurring additional costs until an external signal arrives. In practice, you wire this to a Slack bot or approval dashboard rather than a CLI command. Resumption happens through the SendDurableExecutionCallbackSuccess AWS API call with the callback ID. Because this is an authenticated API call, only principals with the appropriate IAM permissions can signal completion.

# In the handler (called on DurableContext, not inside a step):
callback = context.create_callback(name="review_report")
send_to_reviewer(callback.callback_id, report)

# Execution suspends here until the external system calls
# SendDurableExecutionCallbackSuccess with the callback ID.
result = callback.result()

The function suspends at callback.result() without incurring additional costs. The external system calls SendDurableExecutionCallbackSuccess with the callback ID and the approval payload. Timeouts raise TimeoutError for fallback routing.

When to add a callback

By default, the pipeline skips the callback entirely. To turn it on, set APPROVAL_GATE=true in the function’s environment variables.

Clean up

To avoid ongoing charges, delete the stack when you’re done experimenting:

sam delete --stack-name aeo-scoring-pipeline

This removes the Lambda durable function, DynamoDB table, S3 bucket (if empty), and the associated IAM roles. If your S3 bucket contains audit reports you want to keep, download them first or remove the DeletionPolicy: Delete setting from the template before deploying.

Conclusion

You now have a five-step AEO scoring pipeline in a single Lambda function that checkpoints between steps without incurring additional costs. Some steps involve LLM inference you don’t want to pay for twice. Durable checkpoints help make sure you do not.

To get started:

1. Clone the repository: github.com/aws-samples/sample-aeo-scoring-pipeline
2. Deploy with sam build && sam deploy --guided
3. Run your first audit: aws lambda invoke --function-name AeoScoringPipeline --payload '{"url": "https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html"}'

Next steps: schedule audits using Amazon EventBridge, add Slack alerts when scores drop, or extend the rubric with domain-specific signals.

For more on Lambda durable functions, see the AWS Lambda durable functions Developer Guide. For the AEO framework, see Addy Osmani’s original post on agent-optimized content.