Romulus product team
Separate a timeout from a rejection
A client-side timeout means the client stopped waiting for a response. The request could have been dropped before delivery, still be processing, or have committed before the response disappeared. Those cases look the same from the worker. Treat the outcome as unknown until there is evidence about what the provider did.
A definitive rejection can follow a different path. For example, a documented validation response may establish that the provider did not accept the request. Use the service contract to decide whether that distinction is available, especially when proxies or intermediaries are involved.
Keep the operation ID stable
Before sending the write, store an operation ID tied to the intended effect and the run step. A replacement worker must be able to load the same ID after a crash. Generating a fresh identifier for every retry makes attempts easier to trace but breaks deduplication. Keep the operation ID and attempt ID as separate fields.
Record enough information to distinguish this action from another legitimate action on the same resource. A support ticket could have several separate updates. Using only the ticket ID as the key can accidentally suppress later writes.
Reconcile using the provider’s contract
If the provider supports operation lookup, use the stored identifier and confirm that the returned action matches the intended request. If the provider guarantees idempotent retries for an identical request and key, a retry with the original key can be appropriate. Check how long the provider retains the key and how it handles requests with the same key and a changed payload.
If neither reliable lookup nor idempotency is available, pause the run for review. Give the operator the operation ID, the attempt history, and the information they need to check the provider. Record the resolution before allowing dependent steps to run.
Test after acceptance and before response
A mock that fails before the provider receives the request does not exercise the ambiguous case. Simulate the provider applying the write and then dropping the response. Resume the run with a replacement worker and inspect both the workflow trace and the external record.
Repeat with a provider that never applied the request and one that is temporarily unable to answer a lookup. Confirm that the system can distinguish a safe retry from a case that needs to remain open. These checks establish what your specific integration can recover.