Romulus product team
Start from the point of failure
Imagine a support agent that reads a customer record, classifies a ticket and writes the result to a CRM. The worker stops after the classification. If the classification only existed in memory, the next worker has to repeat it. If the checkpoint only says that classification finished, the next worker still has no result to write.
The checkpoint needs to preserve the output required by the next step and the position from which the run should continue. Keep a reference to the workflow version alongside that position. A checkpoint for version three can be difficult to interpret after a deployment changes the step order or output shape.
Commit the output with the transition
Save the output reference and advance the run position in one durable transaction. With separate commits, a crash can leave an output with no corresponding advance, or an advance with no readable output. If a result lives in object storage, write and verify the object first. Commit the reference after it becomes readable and plan to collect unreferenced objects later.
For large or sensitive results, store only what a continuation requires and make the retention boundary explicit. Deleting an output reference from the run record is insufficient if the object and any copies still contain the underlying data.
Handle competing attempts
An expired worker lease can allow a replacement to start while the original attempt is still alive. When committing progress, check that the current owner and attempt version still match. Otherwise a stale worker could overwrite a newer result or move the run backwards.
That ownership check protects the execution record. External writes need their own controls because a stale worker might already have sent the request. Use a stable operation ID and a provider-supported deduplication or reconciliation path when the step changes an external system.
Exercise the boundary
Terminate a worker immediately before the checkpoint and confirm that the uncommitted step can run again. Then terminate it immediately after the commit and confirm that the saved result is reused. Finally, interrupt two competing attempts and check that only the current owner can advance the run.
Inspect the resulting trace as if you were on call. It should be possible to identify which outputs were committed, which attempts ran, and which worker owns the next transition. A trace that says only ‘retried’ leaves the operator guessing what actually happened.