Question 1

How to adapt this architecture for PDF processing instead of Markdown?

Accepted Answer

Replace the Markdown conversion Lambda function with a PDF processing library like PyPDF2 or AWS Textract, update IAM permissions for new services, and modify the S3 event triggers in the SAM template. The sentiment analysis workflow can remain if text extraction is integrated.

Question 2

AWS Lambda file processing vs using AWS Batch for large files?

Accepted Answer

Lambda is best for real-time, event-driven processing of files up to payload limits (6MB via S3 triggers), while AWS Batch suits batch jobs on larger datasets or compute-intensive tasks. This architecture uses Lambda for scalability but may need adjustments for files over 100MB, as hinted in the testing section.

Question 3

How to handle files larger than 100MB in this setup?

Accepted Answer

The README suggests testing with oversized files and using X-Ray for error tracing. For production, consider S3 multipart uploads, streaming processing with Lambda, or integrating AWS Step Functions for chunked processing, as Lambda has payload and timeout limitations.

Question 4

What's the cost of using Amazon Comprehend for sentiment analysis here?

Accepted Answer

Costs scale with the number of characters processed; Comprehend charges per unit, which can add up for high-volume file processing. Alternatives like open-source NLP libraries might reduce expenses but require more maintenance.

Question 5

How to add a third processing workflow, like image thumbnail generation?

Accepted Answer

Extend the SAM template by adding another SQS queue, Lambda function for image processing, and SNS subscription. Configure S3 events to trigger this workflow, similar to the conversion and sentiment queues, and update error handling and monitoring resources.

Question 6

How to monitor errors in dead-letter queues effectively?

Accepted Answer

Use the pre-configured CloudWatch alarms that send email notifications via SNS when messages hit the DLQs. Additionally, inspect queues manually with AWS CLI commands or the Console, as shown in the testing and cleanup scripts.

lambda-refarch-fileprocessing

What is lambda-refarch-fileprocessing?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions