I'm designing a system where events are consumed from a message broker and processed by multiple instances of the same service.

For example, an event might look like:

{
  "eventId": "evt_12345",
  "userId": "user_42",
  "type": "PAYMENT_COMPLETED",
  "amount": 1000
}

The broker provides at-least-once delivery, so the same event can be delivered to a consumer more than once, especially when a consumer crashes after processing the event but before acknowledging the message.

For example:

Consumer receives evt_123
        |
        v
Process payment
        |
        v
Consumer crashes
        |
        v
Message is redelivered
        |
        v
Payment gets processed again

The system has around 10 million events per day and multiple consumer instances running concurrently.

I'm considering several approaches:

1. Store processed eventIds in the database and ignore events that already exist.

2. Store event IDs in Redis with a TTL.

3. Make the business operation itself idempotent using a unique constraint/idempotency key.

4. Use the message broker's exactly-once features, if available.

5. Combine Redis with database-level idempotency.

I'm particularly unsure about where the responsibility for deduplication should live.

For example, would something like this be sufficient?

Consumer
   |
   v
Check Redis: "Have I processed eventId?"
   |
   +---- Yes ---> Ignore
   |
   +---- No ----> Process
                    |
                    v
                 Set Redis key

My concern is that this introduces a race condition if two consumers receive the same event at approximately the same time.

I'm also worried about what happens if Redis becomes unavailable or loses its data.

What would be the recommended design for this kind of system?

Specifically:

• Where should idempotency/deduplication be implemented?

• Is Redis a good solution for this, or should the database be the source of truth?

• How should race conditions between consumers be handled?

• What should happen when the consumer crashes halfway through processing?

• Is "exactly once" actually achievable, or should the system be designed around at-least-once delivery and idempotent processing?

• What trade-offs would you consider at higher scale?