I have a DynamoDB-backed system where a parent entity can have more than 100,000 associated records.
When a change event occurs, all associated records must be reprocessed asynchronously.
Current idea:
- Query DynamoDB using a GSI
- Paginate results using LastEvaluatedKey
- Publish batches of record IDs to SQS
- Process batches using Lambda
The complication is that multiple independent event sources can target the same parent entity at arbitrary times.
For example:
10:00 - Event A affects Parent X
10:02 - Event B affects Parent X
10:05 - Event C affects Parent X
I cannot delay processing or introduce an aggregation window because each event must be handled immediately.
Questions:
1. Is paginated DynamoDB query + chunked SQS fanout the recommended approach at this scale?
2. How do large-scale systems avoid excessive duplicate processing when multiple events target the same set of records?
3. Would a job/orchestration table be the typical solution here, and if so, what state is usually tracked?
I'm interested in AWS/DynamoDB/SQS architectural patterns rather than implementation details.