AWS pattern for processing 100k+ DynamoDB records when overlapping events require immediate processing
22:02 28 May 2026

I have a DynamoDB-backed system where a parent entity can have more than 100,000 associated records.

When a change event occurs, all associated records must be reprocessed asynchronously.

Current idea:

- Query DynamoDB using a GSI

- Paginate results using LastEvaluatedKey

- Publish batches of record IDs to SQS

- Process batches using Lambda

The complication is that multiple independent event sources can target the same parent entity at arbitrary times.

For example:

10:00 - Event A affects Parent X

10:02 - Event B affects Parent X

10:05 - Event C affects Parent X

I cannot delay processing or introduce an aggregation window because each event must be handled immediately.

Questions:

1. Is paginated DynamoDB query + chunked SQS fanout the recommended approach at this scale?

2. How do large-scale systems avoid excessive duplicate processing when multiple events target the same set of records?

3. Would a job/orchestration table be the typical solution here, and if so, what state is usually tracked?

I'm interested in AWS/DynamoDB/SQS architectural patterns rather than implementation details.

amazon-web-services amazon-dynamodb amazon-sqs