lambda-ecs-worker-pattern
Extends AWS Lambda functionality using Amazon SQS and ECS to process long-running tasks beyond Lambda's time limits.
What is lambda-ecs-worker-pattern?
Lambda ECS Worker Pattern is a serverless architecture example that combines AWS Lambda with Amazon ECS to handle workloads exceeding Lambda's execution time limits. It demonstrates how to offload intensive processing tasks from Lambda functions to ECS containers while maintaining an event-driven workflow, using S3 uploads as triggers.
AWS developers and architects building serverless applications that require long-running or computationally intensive tasks beyond Lambda's 60-second timeout limit.
Developers choose this pattern to extend Lambda's capabilities with containerized workloads, enabling them to use the right compute option for each application component while maintaining a serverless-first approach with automated deployment scripts.
Overview
This example code illustrates how to extend AWS Lambda functionality using Amazon SQS and the Amazon EC2 Container Service (ECS).
Use Cases
Best For
- Processing large file uploads in S3 that require more than 60 seconds of computation
- Running CPU-intensive batch jobs like image rendering or video encoding in a serverless architecture
- Building event-driven pipelines where Lambda handles orchestration and ECS handles heavy processing
- Implementing scalable background job processing with SQS as a durable message buffer
- Creating hybrid serverless solutions that combine Lambda's quick startup with ECS's long-running capabilities
- Demonstrating AWS integration patterns between Lambda, S3, SQS, and ECS for educational purposes
Not Ideal For
- Applications where all processing consistently completes within AWS Lambda's 60-second timeout limit, avoiding unnecessary ECS overhead
- Teams prioritizing minimal infrastructure management over serverless benefits, as ECS requires cluster maintenance and scaling
- Projects needing cross-cloud portability, due to heavy reliance on AWS-specific services like S3, SQS, and ECS
- Real-time systems requiring sub-second latency, as ECS task startup and processing can introduce significant delays
Pros & Cons
Pros
Effectively extends Lambda's capabilities by offloading intensive work to ECS, demonstrated with POV-Ray rendering that exceeds Lambda's 60-second limit.
Seamlessly combines S3 triggers, SQS queues, and ECS tasks into a cohesive workflow, as shown in the architecture diagram and demo.
Includes a full POV-Ray image rendering demo with automated Fabric scripts, providing a tangible implementation to learn from and adapt.
Fabric scripts handle setup of all AWS resources, reducing manual configuration errors and speeding up deployment, as detailed in the setup instructions.
Cons
Deeply integrated with AWS services (S3, SQS, ECS), making migration to other clouds complex and costly, with no built-in abstraction layers.
Requires configuring AWS CLI, DockerHub, IAM roles, and running Fabric commands, with prerequisites noting familiarity with multiple AWS and Docker concepts.
The demo assumes a test ECS cluster and builds images on nodes, admitting it's not suitable for critical applications without modifications for scaling or error handling.
Frequently Asked Questions
Related Projects
serverless⚡ Serverless Framework – Effortlessly build apps that auto-scale, incur zero costs when idle, and require minimal maintenance using AWS Lambda and other managed cloud services.
chromeless🖥 Chrome automation made simple. Runs locally or headless on AWS Lambda.
ZappaServerless Python
aws-sam-localCLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.