AWS Lambda functions can now mount Amazon S3 buckets as file systems with S3 Files (2 minute read)

Tech awsserverlessinfrastructureai Read original

AWS Lambda now supports mounting S3 buckets as file systems, letting functions perform file operations without downloading data first.

What: Amazon S3 Files is a new integration that allows Lambda functions to mount S3 buckets as file systems using Amazon EFS under the hood. Multiple Lambda functions can access the same mounted file system simultaneously, reading and writing files directly without managing downloads or ephemeral storage limits.

Why it matters: This removes a major friction point in serverless architectures where functions previously had to download S3 objects to local storage, process them, and upload results back. The shared file system capability is especially valuable for AI workflows where multiple agent functions need to share state and memory across pipeline steps.

Takeaway: Configure S3 Files mounting through the Lambda console, AWS CLI, or infrastructure-as-code tools like CloudFormation or SAM to start using the feature.

Deep dive

Lambda functions can now mount S3 buckets as local file systems and perform standard file I/O operations directly on S3 data
Built on Amazon EFS infrastructure, combining EFS performance with S3's scalability, durability, and cost model
Multiple Lambda functions can mount the same S3 Files file system concurrently, creating a shared workspace without custom synchronization code
Eliminates the traditional pattern of downloading S3 objects to ephemeral storage, processing, then uploading results back
Removes ephemeral storage limit concerns since data stays in S3 rather than consuming Lambda's temporary disk space
Particularly suited for AI and machine learning pipelines where agent functions need persistent memory and shared state
Works with Lambda durable functions for orchestrating multi-step workflows with automatic checkpointing
Example use case: orchestrator clones a repository to shared workspace while parallel agent functions analyze different parts of the code simultaneously
Available in all regions where both Lambda and S3 Files exist, for functions not using capacity providers
No additional charge beyond standard Lambda compute and S3 storage/access pricing

Decoder

S3 Files: A new AWS service that presents S3 buckets as mountable file systems, built on EFS infrastructure
Amazon EFS: Elastic File System, AWS's managed network file system service that can be mounted by multiple compute instances
Lambda durable functions: A feature that orchestrates multi-step workflows in Lambda with automatic state checkpointing and parallel execution
Ephemeral storage: Temporary disk space available to Lambda functions during execution, which is lost when the function completes

Original article

AWS Lambda functions can now mount Amazon S3 buckets as file systems with S3 Files

AWS Lambda now supports Amazon S3 Files, enabling your Lambda functions to mount Amazon S3 buckets as file systems and perform standard file operations without downloading data for processing. Built using Amazon EFS, S3 Files gives you the performance and simplicity of a file system with the scalability, durability, and cost-effectiveness of S3. Multiple Lambda functions can connect to the same S3 Files file system simultaneously, sharing data through a common workspace without building custom synchronization logic.

The S3 Files integration simplifies stateful workloads in Lambda by eliminating the overhead of downloading objects, uploading results, and managing ephemeral storage limits. This is particularly valuable for AI and machine learning workloads where agents need to persist memory and share state across pipeline steps. Lambda durable functions make these multi-step AI workflows possible by orchestrating parallel execution with automatic checkpointing. For example, an orchestrator function can clone a repository to a shared workspace while multiple agent functions analyze the code in parallel. The durable function handles checkpointing of execution state while S3 Files provides seamless data sharing across all steps.

To use S3 Files with Lambda, configure your function to mount an S3 bucket through the Lambda console, AWS CLI, AWS SDKs, AWS CloudFormation, or AWS Serverless Application Model (SAM). To learn more about how to use S3 Files with your Lambda function, visit the Lambda developer guide.

S3 Files is supported for Lambda functions not configured with a capacity provider, in all AWS Regions where both Lambda and S3 Files are available, at no additional charge beyond standard Lambda and S3 pricing.