I'm working on a Spring Boot service that provisions PostgreSQL clusters through a cloud provider's API.
The provider's API is asynchronous, so my endpoint roughly works like this:
POST /clusters
|
v
Spring Boot
|
v
Cloud API
|
v
operation ID
I return 202 Accepted to the client and keep track of the provisioning operation in PostgreSQL.
The part I'm struggling with is retries.
Suppose I receive:
POST /clusters
Idempotency-Key: uuid
{
"name": "customer-db",
"cpu": 4,
"storage": 100
}
I save the request in my database and then call the cloud provider.
But what happens if the provider successfully creates the cluster and my application crashes before I receive/save the response?
When the client retries uuid, my database only tells me that the request was started. I don't have the provider's operation ID, so calling the provider again could potentially create a second cluster.
I've been thinking about storing something like:
idempotency_key
request_hash
operation_id
cluster_id
status
and using the idempotency key to return the existing operation for retries.
But that doesn't really solve the crash window between:
Cloud provider creates resource
↓
Spring Boot receives response
↓
Spring Boot saves operation ID
The provider I'm integrating with doesn't guarantee that it will treat my idempotency key as an idempotency key either.
What's the usual way to handle this case?