Post Syndicated from Doug Perkes original https://aws.amazon.com/blogs/compute/introducing-self-managed-amazon-s3-buckets-for-aws-lambda-function-code/

If you manage Lambda functions at scale, you’ve likely hit the 75 GB code storage limit or explained to your security team why deployment artifacts live in an S3 bucket you don’t control. Today, we’re announcing self-managed Amazon S3 buckets for AWS Lambda deployment packages. Lambda reads your code directly from your bucket, eliminating quota pressure and giving you full security control.

Previously, the default AWS-managed code storage created three challenges at scale. First, all copies count toward your 75 GB code storage quota. Second, you cannot apply your own encryption, access controls, or compliance tags to the internal bucket. Third, the copy cannot be incorporated into your disaster recovery strategies.

With self-managed S3 buckets, Lambda reads your function code directly from your bucket. No copy, no duplication. Your S3 object becomes the single source of truth for your functions. Deployment packages no longer count against your account’s code storage limit. You manage the bucket’s security and compliance posture: choosing the encryption, defining the access policies, managing lifecycle transitions, and maintaining the audit trail. And because you own the bucket, you can use S3 Cross-Region Replication to maintain fallback copies of your code in a secondary Region, so that your functions remain deployable even if your primary Region experiences an issue. Using self-managed S3 buckets also results in a faster time to first invoke for new functions and after function updates, because Lambda no longer needs to copy your zip package to a Lambda-managed S3 bucket.

You can use this feature today in all AWS standard regions where Lambda is available, at no additional charge beyond your standard Amazon S3 storage and request costs. Let’s look at some use cases, how it works, and how to use it at scale.

Use cases

Here are a few patterns where owning your deployment bucket makes a real difference.

CI/CD pipelines and artifact management

With self-managed storage, your CI/CD pipeline uploads once, and Lambda references the same object. One set of lifecycle rules and access controls covers all artifacts, and rollbacks mean pointing the function to a previous S3 object version.

Multi-account and multi-team architectures

Organizations using AWS Organizations often separate workloads into multiple accounts: a development account, a staging account, and a production account. They centralize shared resources in a tooling or shared-services account.

Self-managed buckets integrate naturally with this pattern:

• Store all deployment artifacts in a central “artifact account” bucket.
• Grant cross-account s3:GetObject access to Lambda execution roles in each workload account through bucket policies.
• Maintain a single inventory of what code is deployed where, managed by your platform or DevOps team.
• Enforce consistent encryption, versioning, and retention policies from one place.

Disaster recovery and business continuity

Because your deployment artifacts live in a bucket you own, you can use the built-in replication features of S3, Cross-Region Replication (CRR) or Same-Region Replication (SRR), to maintain copies of your code artifacts in backup locations. Combined with S3 Versioning and Object Lock, this gives you a durable, tamper-proof code archive that supports rapid recovery if a deployment is accidentally corrupted or deleted.

How it worked before

When you deploy a Lambda function using a .zip deployment package stored in Amazon S3, the process has traditionally worked like this:

1. You upload your .zip deployment package to your S3 bucket.
2. You call CreateFunction or UpdateFunctionCode, specifying the S3 bucket and S3 key.
3. Lambda copies the .zip artifact from your bucket into an internal, service-managed bucket.
4. Lambda uses this copy to create the optimized version of your function that runs at invocation time.
5. The copied artifact counts toward your account’s 75 GB code storage quota.

Figure 1 — Standard Lambda deployment

This model is straightforward and works well for most workloads. However, it creates three friction points at scale:

• Storage quota pressure: Every deployment package copy counts toward your account’s 75 GB total code storage limit. Organizations with hundreds of functions and multiple published versions can exhaust this quota.
• No control over stored artifacts: You cannot configure encryption (beyond the service default), access logging, lifecycle policies, Object Lock, or compliance tags on the internal bucket.
• Redundant storage: Your original artifact remains in your bucket while a copy lives in the Lambda bucket used for provisioning new instances of your Lambda function.

What’s new: REFERENCE mode

This launch introduced a new function configuration setting, S3ObjectStorageMode. The default value is COPY, which provides the existing behavior described in the preceding section. To enable self-managed S3 buckets, set S3ObjectStorageMode to REFERENCE when creating or updating a function. In this mode, Lambda no longer copies your deployment package. Instead, it stores a reference to your S3 object and reads the code directly from your bucket when needed. If you do not specify S3ObjectStorageMode, Lambda still takes a copy by default.

Figure 2 — Lambda deployment with self-managed storage

This gives you:

• No quota consumption. Deployment packages in your bucket don’t count against the 75 GB Function and layer storage account limit.
• Improved performance. Lambda no longer copies the code to an internal bucket, so function creation and updates are faster.
• Full security and compliance control. Apply your own bucket policies, encryption, Object Lock, versioning, access logging, and compliance tags.
• Single source of truth. Your S3 object is the canonical artifact with no additional copies and no drift.
• Disaster recovery options. Use S3 Cross-Region Replication to maintain fallback copies in a secondary Region.

How it works

To use this feature, specify the S3ObjectStorageMode parameter when creating or updating your function.

Creating a new function (AWS CLI):

aws lambda create-function \
  --function-name my-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/my-lambda-role \
  --handler app.handler \
  --code S3Bucket=amzn-s3-demo-bucket,S3Key=deployments/my-function.zip,S3ObjectVersion=abc123,S3ObjectStorageMode=REFERENCE

Updating an existing function:

aws lambda update-function-code \
  --function-name my-function \
  --s3-bucket amzn-s3-demo-bucket \
  --s3-key deployments/my-function.zip \
  --s3-object-version def456 \
  --s3-object-storage-mode REFERENCE

AWS Identity and Access Management (IAM) permissions

Lambda needs permission to read the deployment package from your bucket. You can grant access through an S3 bucket policy.

S3 bucket policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LambdaSelfManagedCodeAccess",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/deployments/my-function.zip",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:lambda:us-east-1:123456789012:function:my-function"
        }
      }
    }
  ]
}

We recommend including the aws:SourceArn condition key scoped to your specific function ARN to allow for least-privileged access. Note the Resource is scoped to the exact S3 key rather than a wildcard prefix. This follows least-privilege and matches how the aws:SourceArn condition locks down which function can access which object.

Bucket requirements

Your S3 bucket must meet the following requirements:

• Versioning (required). You must enable S3 versioning to make sure that Lambda references a specific, immutable artifact and to protect against accidental overwrites.
• Encryption. The following encryption types are supported: SSE-S3, SSE-KMS (including customer-managed KMS keys), and DSSE-KMS. If you use SSE-KMS, the Lambda principal must have kms:Decrypt permission on the key.
• Object Lock. Supported. You can apply Object Lock in Compliance or Governance mode to prevent accidental deletion of deployment artifacts.
• Access logging. You can enable S3 server access logging or AWS CloudTrail data events to audit every time Lambda reads your code.

What happens when the object is unavailable

Lambda periodically accesses the source object from your S3 bucket to reoptimize your function code. You must maintain access to the source object for your function to remain active.

If Lambda loses access to the source object for a function, the function transitions to the Inactive state. To restore the function, restore access to the source object and then update the function.

Performance considerations

Lambda functions with self-managed code storage behave the same as standard Lambda functions with one difference during function creation and update. Lambda does not copy your deployment package to a Lambda-managed S3 bucket. In our testing with a 200MB Python 3.13 function, functions using self-managed storage showed function creation times approximately 5s less than the default COPY mode. Reading directly from your S3 bucket without an intermediate copy step can provide a modest advantage, particularly for larger deployment packages.

Getting started with infrastructure as code

Self-managed code storage can be implemented using infrastructure-as-code tooling with either the AWS CLI or AWS CloudFormation today.

AWS CLI

aws lambda create-function \
  --function-name my-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/my-lambda-role \
  --handler app.handler \
  --code S3Bucket=amzn-s3-demo-bucket,S3Key=deployments/my-function.zip,S3ObjectVersion=abc123,S3ObjectStorageMode=REFERENCE \
  --region us-east-1

AWS CloudFormation

MyFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: my-function
    Runtime: python3.13
    Handler: app.handler
    Role: !GetAtt MyLambdaRole.Arn
    Code:
      S3Bucket: amzn-s3-demo-bucket
      S3Key: deployments/my-function.zip
      S3ObjectVersion: abc123
      S3ObjectStorageMode: REFERENCE

Using it at scale

Once you adopt self-managed S3 buckets for your Lambda deployment packages, your artifact bucket grows over time as you deploy new versions of your functions. This section covers strategies for managing that growth efficiently, keeping your versions organized, and planning for cross-Region deployments.

Managing artifact lifecycle with S3 lifecycle policies

Every time you update a function’s code, S3 creates a new object version in your bucket. The previous objects don’t disappear. They accumulate. Without a cleanup strategy, your storage grows indefinitely and old artifacts clutter your bucket.

S3 Lifecycle policies let you automate this entirely. You define rules that transition or delete objects based on age, and S3 executes them on your behalf: no scripts, no cron jobs, no manual intervention.

Strategy 1: Archive old versions to Glacier

If compliance or audit requirements mandate that you retain all historical deployment packages, but you rarely need to access them, transition old object versions to a lower-cost storage class:

{
  "Rules": [
    {
      "ID": "ArchiveOldDeploymentPackages",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "deployments/"
      },
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "GLACIER_FLEXIBLE_RETRIEVAL"
        }
      ]
    }
  ]
}

This rule transitions any non-current object version to S3 Glacier Flexible Retrieval after 30 days. Your active deployment packages remain in S3 Standard for fast access, while historical versions move to archival storage at a fraction of the cost.

For artifacts you need to retain for years but will rarely access again, consider a tiered approach: moving to Glacier Flexible Retrieval first, then to Deep Archive:

"NoncurrentVersionTransitions": [
  {
    "NoncurrentDays": 30,
    "StorageClass": "GLACIER_FLEXIBLE_RETRIEVAL"
  },
  {
    "NoncurrentDays": 365,
    "StorageClass": "DEEP_ARCHIVE"
  }
]

Strategy 2: Delete old versions you no longer need

If you don’t have a compliance requirement to retain every historical artifact, you can expire old versions outright:

{
  "Rules": [
    {
      "ID": "DeleteOldDeploymentPackages",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "deployments/"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 14,
        "NewerNoncurrentVersions": 2
      }
    }
  ]
}

This rule keeps the 2 most recent non-current versions of each object (giving you a rollback path) and deletes anything older than 14 days beyond that. This aligns well with a deployment strategy where you want the ability to quickly roll back to your previous one or two releases, but don’t need to retain anything older.

Tracking object and function versions

With REFERENCE mode, there is a direct relationship between your S3 object version and your Lambda function version. We recommend the following practices:

• Tag your objects with metadata from your CI/CD pipeline (commit SHA, build ID, pipeline run ID) so you can trace any deployed function back to the exact source that produced it.
• Document the mapping between Lambda function versions (or aliases) and S3 object version IDs. This makes rollbacks straightforward: update the function to reference the previous object version.

Cross-account considerations

How you organize your artifact buckets across AWS accounts depends on your operational model:

• Centralized artifact account: A single bucket in a shared-services or tooling account, with bucket policies granting cross-account s3:GetObject access to Lambda execution roles in workload accounts. This gives your platform team a single inventory of all deployment artifacts with consistent lifecycle, encryption, and access policies.
• Per-account buckets: Each workload account owns its own artifact bucket. Requires less effort to set up, but harder to enforce consistent governance across many accounts.

Either pattern works with self-managed storage. Choose based on how your organization balances centralized control against team autonomy.

Cross-Region considerations

With REFERENCE mode, your S3 object is the authoritative copy for your function. Self-managed code storage supports cross-Region function creation within a partition for all default Regions (non-opt-in Regions). You can store your code packages in one Region and deploy your functions in another. This makes cross-Region planning critical. There are four factors to balance:

Disaster recovery

This is the most critical consideration. Because your S3 object is the single source of truth in REFERENCE mode, you do not want all your deployment artifacts in a single Region with no fallback. A recommended pattern:

1. Primary Region: Your main artifact bucket where CI/CD pipelines deposit new deployment packages.
2. Fallback Region: A secondary bucket populated via S3 Cross-Region Replication (CRR). If your primary Region becomes unavailable, you can update your Lambda functions to reference the replicated objects in the fallback Region.

Enable S3 Replication Time Control (RTC) if you need a guaranteed SLA (15 minutes) for replication completion.

Cost

Weigh replication + storage costs against per-deploy cross-Region data transfer. If you deploy frequently, storing replicated copies in each target Region is usually cheaper. For infrequent deployments, a one-time transfer may suffice.

Governance and data residency

Some organizations, particularly in regulated industries, have strict requirements about where code artifacts can reside. Before configuring cross-Region replication, confirm that your data is permitted to leave its current Region. Certain regulatory frameworks (for example, data sovereignty laws, FedRAMP boundaries) may restrict replication to specific Region pairs.

Performance

If your workload requires fast function creation and activation times, for example, in a CI/CD pipeline where deployment speed is critical, keep your S3 objects in the same Region where you are creating your Lambda functions. Cross-Region reads add latency to the initial code download, which directly impacts how quickly a new function version becomes active after deployment.

For workloads where creation speed is less critical (for example, batch processing functions that are updated infrequently), the latency of a cross-Region read might be acceptable and can simplify your architecture.

Things to know

Before adopting self-managed S3 buckets for your Lambda functions, keep the following in mind:

• Availability: You can use this feature today in all AWS standard regions where Lambda is supported.
• Pricing: There is no additional Lambda charge. You pay standard S3 costs for storage, and any cross-Region data transfer.
• Maximum deployment package size: The existing limits apply: 250 MB unzipped.
• Supported runtimes: All Lambda runtimes that support .zip deployment packages are compatible. Container image deployments are not affected by this feature.
• Migration: You can switch an existing function from service-managed to self-managed storage by calling UpdateFunctionCode with the --s3-object-storage-mode REFERENCE parameter. Lambda recreates the function by referencing the object in your S3 bucket and deletes the saved copy.
• Reverting: You can switch back to service-managed storage at any time by updating the function with --s3-object-storage-mode COPY. Lambda resumes copying the artifact to its internal bucket.
• Object availability is your responsibility: In REFERENCE mode, Lambda depends on your S3 object being accessible. If the object is deleted, the bucket policy changes, or the KMS key is disabled, new invocations requiring a cold start will fail.

Conclusion

In this post, we showed how self-managed S3 buckets for Lambda give you more capacity, more control, and simpler compliance, all without changing how you write or invoke your functions. Your deployment packages no longer count against account quotas, your security team can apply the same policies to code artifacts that they apply everywhere else, and your disaster recovery story is as strong as the replication capabilities of S3.

To get started:

• Read the Lambda Developer Guide: Self-managed S3 code storage for full documentation.
• Try it in the AWS Lambda Console. Choose Reference Mode under Code storage mode when creating your next function.
• Review S3 Lifecycle Configuration examples to plan your artifact retention strategy.
• Explore S3 Cross-Region Replication for disaster recovery planning.

Post Syndicated from Sahithi Ginjupalli original https://aws.amazon.com/blogs/compute/modernizing-lambda-s3-workloads-with-amazon-s3-files/

Learn how Amazon S3 Files simplifies Lambda functions by eliminating transfer code and /tmp constraints. See three modernization patterns with code examples for image processing, ETL pipelines, and multi-agent AI workloads.

AWS Lambda functions that interact with Amazon Simple Storage Service (Amazon S3) typically follow a familiar pattern: download an object to /tmp, process it locally, and upload the result back to S3. This pattern is well-understood and reliable, but it requires you to write code for managing transfers, monitoring /tmp capacity, and cleaning up ephemeral storage alongside your actual processing logic.

Amazon S3 Files changes this by letting your Lambda function mount an S3 bucket as a file system. Your function reads and writes files at a local mount path (such as /mnt/data), and the file system handles synchronization with S3 automatically. The transfer and storage management code goes away, and what remains is your processing logic working directly with files.

In this post, we walk through three common Lambda + S3 workloads and show how to modernize each one by using S3 Files. You will see how the code gets shorter, the /tmp size constraint disappears, and the developer experience improves.

Walkthrough

Prerequisites

Before you begin, make sure you have:

• An AWS account with permissions to create Lambda functions, S3 file systems, and VPC resources.
• An existing VPC with private subnets and appropriate security groups.

Getting started

To integrate a Lambda function with S3 Files, you can follow these three steps:

1. Create an S3 file system for your bucket. You can do this through the S3 console, AWS Command Line Interface (AWS CLI), or AWS CloudFormation. This single operation creates the file system, mount targets in your Amazon Virtual Private Cloud (Amazon VPC), and an access point.
2. Add the file system configuration to your Lambda function. Specify the access point ARN and local mount path (for example, /mnt/data). Your function must be in a VPC with access to the mount target. For optimal throughput on large files, configure your function with 512 MB or more of memory to enable direct reads from S3.
3. If you are modernizing your existing Lambda function’s code, replace boto3 transfer code with file paths. Change s3.download_file(bucket, key, '/tmp/file') to open('/mnt/data/' + key) and remove upload and cleanup logic.

Your function’s execution role needs s3files:ClientMount and s3files:ClientWrite permissions (included in the AmazonS3FilesClientReadWriteAccess managed policy). For direct S3 reads on large files, also add s3:GetObject and s3:GetObjectVersion.

Pattern 1: Multi-agent shared workspace

Agentic AI workloads, where multiple autonomous agents collaborate on a task, require shared mutable state. Agents need to read each other’s outputs, write intermediate artifacts, and coordinate without tight coupling. With Lambda today, this typically means serializing state to S3 objects or Amazon DynamoDB between every step, adding latency and code for each handoff.

S3 Files gives multiple Lambda functions a shared file system. Agents communicate through the file system itself, with no S3 API calls and no serialization overhead.

Example: Collaborative research agents

Three Lambda functions mount the same S3 bucket at /mnt/workspace. An orchestrator prepares the task, research agents work in parallel, and a synthesis agent combines their findings:

import os
import json

WORKSPACE = "/mnt/workspace"

# --- Orchestrator Agent ---
def orchestrator_handler(event, context):
    session_id = event["session_id"]
    session_dir = f"{WORKSPACE}/sessions/{session_id}"

    os.makedirs(f"{session_dir}/research", exist_ok=True)
    os.makedirs(f"{session_dir}/output", exist_ok=True)

    # Write task assignments directly to shared workspace
    with open(f"{session_dir}/manifest.json", "w") as f:
        json.dump({
            "query": event["research_query"],
            "agents": ["market_analysis", "technical_review", "competitor_scan"],
            "status": "in_progress"
        }, f)

    return {"session_dir": session_dir}


# --- Research Agent (one of many, running in parallel) ---
def research_agent_handler(event, context):
    session_dir = event["session_dir"]
    agent_name = event["agent_name"]

    # Read task from shared workspace (no S3 GET call)
    manifest = json.load(open(f"{session_dir}/manifest.json"))

    # Perform research (invoke Amazon Bedrock, search, etc.)
    # TODO: Implement perform_research() for your use case
    findings = perform_research(manifest["query"], agent_name)

    # Write results to shared workspace (no S3 PUT call)
    with open(f"{session_dir}/research/{agent_name}.json", "w") as f:
        json.dump(findings, f, indent=2)

    return {"status": "complete", "agent": agent_name}


# --- Synthesis Agent ---
def synthesis_handler(event, context):
    session_dir = event["session_dir"]

    # Read all research outputs from shared directory
    all_findings = {}
    for f in os.listdir(f"{session_dir}/research"):
        with open(f"{session_dir}/research/{f}") as fh:
            all_findings[f.replace(".json", "")] = json.load(fh)

    # Synthesize and write final report
    # TODO: Implement synthesize_findings() for your use case
    report = synthesize_findings(all_findings)
    with open(f"{session_dir}/output/report.md", "w") as f:
        f.write(report)

    return {"report_path": f"{session_dir}/output/report.md"}

In the traditional approach, each agent would need to call s3.get_object() to read the manifest, s3.put_object() to write findings, and the synthesis agent would need to call s3.list_objects() then s3.get_object() for each result. That’s eight or more S3 API calls per workflow run replaced by file I/O.

What the shared workspace pattern gives you:

• Agents discover each other’s outputs by listing a directory (no coordination logic needed).
• Sessions, agents, and outputs map to directories, not flat object key conventions.
• Close-to-open consistency means that when an agent closes a file after writing, the next agent to open it sees the complete content.
• No need to marshal state into S3 PutObject calls between steps.

Pattern 2: Image thumbnail generation

The S3 thumbnail generator is a common Lambda + S3 pattern. An image is uploaded to S3, a Lambda function is triggered, it downloads the image, resizes it with Pillow, and uploads the thumbnail to a destination bucket.

The traditional approach

import boto3
import os
import uuid
from urllib.parse import unquote_plus
from PIL import Image

s3_client = boto3.client('s3')

def resize_image(image_path, resized_path):
    with Image.open(image_path) as image:
        image.thumbnail(tuple(x / 2 for x in image.size))
        image.save(resized_path)

def handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = unquote_plus(record['s3']['object']['key'])
        tmpkey = key.replace('/', '')
        download_path = '/tmp/{}{}'.format(uuid.uuid4(), tmpkey)
        upload_path = '/tmp/resized-{}'.format(tmpkey)

        s3_client.download_file(bucket, key, download_path)
        resize_image(download_path, upload_path)
        s3_client.upload_file(
            upload_path, '{}-resized'.format(bucket), 'resized-{}'.format(key)
        )

What this approach requires you to manage beyond the core resize logic:

• Transfer orchestration: Downloading the source, uploading the result, and handling partial transfer failures.
• Storage capacity: Both the source and resized image must fit in /tmp simultaneously.
• Ephemeral storage cleanup: If the function fails mid-execution or is reused across invocations, orphaned files can accumulate in /tmp.
• Redundant downloads: If the same image triggers a retry, it must be downloaded again.

With the file system approach

import os
from urllib.parse import unquote_plus
from PIL import Image

MOUNT = "/mnt/images"

def resize_image(image_path, resized_path):
    with Image.open(image_path) as image:
        image.thumbnail(tuple(x / 2 for x in image.size))
        image.save(resized_path)

def handler(event, context):
    for record in event['Records']:
        key = unquote_plus(record['s3']['object']['key'])
        input_path = f"{MOUNT}/source/{key}"
        output_path = f"{MOUNT}/resized/resized-{os.path.basename(key)}"

        os.makedirs(os.path.dirname(output_path), exist_ok=True)
        resize_image(input_path, output_path)

What changed

The function moves from a download-process-upload pipeline to direct file I/O. No boto3 client, no /tmp management, no upload step. The resize_image function is unchanged because it always worked with file paths. The difference is that those paths now point to a mounted S3 file system instead of ephemeral local storage.

You still handle errors in your processing logic (for example, invalid image formats). What you no longer need to handle are transfer-specific failure modes like partial downloads, failed uploads, or /tmp capacity checks.

Pattern 3: CSV-to-Parquet ETL pipeline

Another commonly used serverless ETL pattern is an S3 event triggers a Lambda function when CSV files land in a bucket. The function downloads the CSV, transforms it to Parquet by using pandas and pyarrow, and uploads the result.

The traditional approach

import boto3
import pandas as pd
import os

s3 = boto3.client("s3")
BUCKET = "data-pipeline-bucket"

def handler(event, context):
    key = event["Records"][0]["s3"]["object"]["key"]
    filename = os.path.basename(key)
    local_input = f"/tmp/{filename}"
    local_output = f"/tmp/{filename.replace('.csv', '.parquet')}"

    try:
        # Download from S3
        s3.download_file(BUCKET, key, local_input)

        # Check /tmp space (10 GB limit)
        tmp_usage = sum(
            os.path.getsize(f"/tmp/{f}")
            for f in os.listdir("/tmp") if os.path.isfile(f"/tmp/{f}")
        )
        if tmp_usage > 9 * 1024**3:  # 9 GB safety margin
            raise RuntimeError("Approaching /tmp storage limit")

        # Transform
        df = pd.read_csv(local_input)
        df["processed_at"] = pd.Timestamp.now()
        df.to_parquet(local_output, engine="pyarrow", compression="snappy")

        # Upload result back to S3
        output_key = key.replace("raw/", "processed/").replace(".csv", ".parquet")
        s3.upload_file(local_output, BUCKET, output_key)

        return {"status": "success", "output_key": output_key}

    finally:
        # Clean up /tmp
        for f in [local_input, local_output]:
            if os.path.exists(f):
                os.remove(f)

What this approach requires you to manage beyond the core transform logic:

• Storage capacity: Source and output files share /tmp, limiting practical file size.
• Cold start cost: Initializing the boto3 client adds startup latency.
• Transfer failure modes: Partial downloads, failed uploads, and orphaned /tmp files need their own handling.
• Redundant downloads: Retries or reprocessing require downloading the same file again.

With the file system approach

import pandas as pd
import os

MOUNT = "/mnt/data"

def handler(event, context):
    key = event["Records"][0]["s3"]["object"]["key"]
    input_path = f"{MOUNT}/{key}"
    output_path = f"{MOUNT}/{key.replace('raw/', 'processed/').replace('.csv', '.parquet')}"

    # Ensure output directory exists
    os.makedirs(os.path.dirname(output_path), exist_ok=True)

    # Transform: direct file access, no download/upload
    df = pd.read_csv(input_path)
    df["processed_at"] = pd.Timestamp.now()
    df.to_parquet(output_path, engine="pyarrow", compression="snappy")

    return {"status": "success", "output_path": output_path}

What changed

With this change, a developer reading this code sees only the transform logic (read CSV, add column, write Parquet). The storage mechanics are handled by the file system.

Choosing the right approach: file system mounts vs. traditional access

Cleaning up

If you created resources while following along with this post, delete them to avoid incurring future costs. Start by removing the file system configuration from your Lambda function settings. Next, remove the S3 file system, which also deletes its associated mount targets and access points. Then delete the S3 buckets used for source and output data, along with the Lambda functions created for the examples. Finally, remove the IAM roles and policies created for Lambda execution or, if you added the S3 Files permissions (s3files:ClientMount, s3files:ClientWrite, s3:GetObject, s3:GetObjectVersion) to an existing role, remove these permissions. Additionally, If you created a new VPC for this tutorial, delete the VPC, which will also remove the associated private subnets, security groups, and route tables. If you used an existing VPC, remove the security groups and subnets created for this testing.

Warning: Deletion of an S3 bucket and its contents permanently deletes all objects in the buckets and cannot be undone. Make sure you have backed up any data you need to retain before proceeding.

Conclusion

In this post, we demonstrated how to modernize three common Lambda + S3 workloads by using Amazon S3 Files. Across image thumbnail generation, ETL pipelines, and multi-agent AI workloads, the migration follows the same principle: replace S3 API transfer logic with native file I/O and let the file system handle synchronization.

The improvements are consistent:

• Less code: Transfer and cleanup logic goes away, leaving only your processing logic.
• No /tmp size constraint: Process large files without local storage limits.
• Zero S3 API calls for data access: Reads and writes go through the file system mount.
• Fewer failure modes to handle: Transfer-specific issues (partial downloads, failed uploads, orphaned temp files) no longer apply.

For teams running Lambda + S3 workloads today, S3 Files isn’t a new architecture to learn. It’s transfer code you can remove. To learn more, see the S3 Files section in the Lambda documentation. To track upcoming features on the AWS Lambda roadmap, you can refer to the AWS Lambda roadmap.