Romulus documentation
Execution model
Model the current position, committed outputs and attempt ownership in a run record.
{
"run_id": "run_7f21",
"workflow_version": 3,
"state": "runnable",
"next_step": "score_priority",
"checkpoint_version": 2,
"outputs": {
"load_context": "ref_context_09"
}
}Separate the run from an attempt
A run represents one workflow invocation and persists across process restarts. Each execution attempt belongs to a specific step in that run. Use a new attempt identifier when retrying, while retaining the run identity and workflow version.
Lease a runnable step to a worker for a bounded period. A replacement can claim it after the lease expires. Reject stale progress updates with an ownership or version check. A lease expiring does not mean the original process has stopped.
Commit progress atomically
Persist the step result reference and the next execution position in the same transaction. Advance only if the expected checkpoint version is still current. If the new output lives outside the run store, make it readable before committing the reference.
When a step starts an external action, preserve the action intent before making the call. A checkpoint protects internal state. Reconciliation is what lets the run recover an uncertain write.
Keep a recoverable version boundary
Before deploying a workflow revision, decide how existing runs resume. Keep their original definition available or migrate their checkpoints with an explicit mapping. Test the upgrade with a run paused before a step whose input shape changed.
Expose the run position and attempt history to operators. A recovery trace should indicate whether an output was restored from a checkpoint or recomputed by an attempt.