Romulus documentation
Production readiness
Review recovery, operational limits and data handling before deploying a workflow.
Verify the boundary
Terminate a worker before and after each important commit. Check that unfinished computation can retry and that completed results are reused. Run a stale-worker test where the original attempt returns after a replacement claims the step.
For external effects, test provider acceptance followed by a dropped response. Identify which endpoints support stable idempotency keys or operation lookup. Document the operator procedure for unknown outcomes that cannot be resolved automatically.
Keep the workload bounded
Choose limits for attempt count, concurrency, run duration and event wait duration. Watch the age of the oldest runnable and waiting runs alongside counts. A small backlog with an old head can reveal a stalled scheduler or a signal that never arrived.
Define what happens when capacity is exhausted. Backpressure at intake and controlled replay can protect both the run store and its downstream providers during recovery.
Practise the incident response
Make run IDs, attempt IDs and operation IDs searchable. Keep only the data needed to recover or investigate, apply access controls, and decide how the workflow behaves when referenced data expires.
Rehearse a restore with interrupted and waiting runs in the same workload. Verify that scheduler recovery can find pending work and that duplicate deliveries and uncertain external writes follow the same rules after a restore.