Romulus documentation
Retries and idempotency
Track the intended effect separately from the requests that try to apply it.
{
"operation_id": "op_4c01",
"run_id": "run_7f21",
"step": "update_crm",
"outcome": "unknown",
"last_attempt": "attempt_02"
}Classify what failed
Retry transient computation failures within a defined attempt and time budget. Surface permanent errors for review. For an external write with an ambiguous result, preserve the unknown state and choose a recovery path based on the provider’s contract.
Use backoff and jitter to spread retries. A growing retry queue can put extra load on a recovering provider, so apply a concurrency limit and keep the retry window within the workflow deadline.
Reuse the same operation ID
Persist the operation ID before sending the request. Keep the intended payload or a reference to it. A second attempt should retrieve this record and use the same provider idempotency key when the provider supports that behaviour.
Confirm key scope, retention and how changed request bodies are treated. A retry after the deduplication window can still create a second effect. Use an authoritative operation lookup if it exists and stop for review if the outcome remains unresolved.
Exercise the difficult outcomes
Test a dropped response after a successful write and a timeout before the provider receives anything. Also test a provider that cannot perform lookup temporarily. The execution record should distinguish reconciliation, a safe retry and a case requiring human review.
Track write attempts and confirmed effects separately. A valid recovery can record several attempts and a single effect, provided the provider’s behaviour supports that conclusion.