Romulus product team
Turn waiting into run state
An agent that starts an external job may wait minutes or hours for the result. Keeping a process alive during that interval ties recovery to a worker that could be recycled at any time. Store a wait condition on the run and release the worker after the state is committed.
The wait record needs the run position, expected event type, correlation ID and any deadline. Capture the external job ID early enough to reconcile the job if the worker fails immediately after submitting it. A launch request can have the same uncertain outcome as other external writes.
Give incoming events a durable home
A completion event can arrive before the workflow records its wait, especially when the external task completes quickly. A receiver that checks only current waits can lose the signal. Record validated events durably and arrange for the matcher to consider stored events when a wait becomes active.
Match on the event type and a scoped correlation ID. Where several tenants share infrastructure, include the tenant or equivalent ownership boundary. A payload supplied by an external source needs authentication and validation before it can advance a run.
Advance the wait once
An event source may redeliver the same event after a network interruption. Deduplicate by an event identifier scoped to its producer. Several legitimate events can refer to the same external job, so job ID alone usually makes a poor deduplication key.
A timeout and a completion event can also race. Define which transition the workflow accepts, then commit the resolution with a conditional update against the current wait. A late event may still be useful for investigation even after the deadline has passed.
Make recovery inspectable
Record when the wait started, which correlation ID it expects, and which event or deadline resolved it. When the workflow remains pending, operators need to distinguish a missing signal from a mismatch or an event processing backlog.
Test delivery before the wait, duplicate delivery, and the race between a timeout and a valid completion. Restart the matcher after it records an event but before it schedules a continuation. Verify that the run can recover and advance once, with the next worker able to explain which event it consumed.