Two-Way Sync Error Handling: Retries and Replay

Classify the failure before deciding to retry

An HTTP error alone rarely explains the business outcome. Combine the status with the endpoint’s response, the record identity, and what happened at the destination. A timeout is ambiguous: the target may have committed the write before the response was lost.

Avoid repeatedly sending a value that a validation rule will always reject. Retrying such a record consumes capacity and delays healthy records. Preserve the error context in a visible exception queue and assign an owner who can decide whether to correct the data, alter the mapping, or intentionally exclude the field.

Prevent duplicate business effects

Idempotency means repeating the same logical operation does not add another effect. For a sync, stable record matching is necessary, but it is not always sufficient. Updating a contact twice may be harmless while firing an automation twice can create two orders or send two messages.

Use a stable operation identifier when the receiving API supports one. Keep that identifier unchanged across attempts for the same operation; use a different identifier for a genuinely new action. Record the source entity, logical operation, target identity, and known result so that recovery can answer “did this already happen?” AWS’s idempotent API guidance explains why caller-provided request identifiers help make retries safe.

Do not deduplicate all changes to a record using the record ID alone. A second legitimate update has the same record ID but different intent. Conversely, do not generate a new random operation ID every time the same request is retried. Test both cases: repeat the same update, then make a new update to the same entity.

A destination upsert can prevent a second record when its unique key is correct; it does not automatically prevent downstream side effects. Include destination workflows, triggers, notifications, and financial actions in your duplicate test. Document what is protected by the sync layer and what requires protection in the receiving application.

Bound retries so an outage does not become overload

Choose timeouts, a maximum retry budget, and increasing delays with jitter. Jitter spreads retries over time so many blocked records do not hit a recovering API together. Honor a valid Retry-After instruction when provided, and coordinate retries with the connector’s quota and concurrent requests.

Retry at one owned layer where possible. If a client, worker, and workflow each repeat a call, their attempts can multiply. AWS’s retry-control guidance recommends limiting retry pressure and verifying idempotency. Your acceptance test should include a sustained outage, not just a single artificial error.

• Measure both new traffic and backlog traffic against the same destination limits.
• Keep retries for one rejected record from starving unrelated healthy records.
• Escalate when retry age exceeds the workflow’s freshness target.
• Record the last error and next action; a retry counter alone is insufficient.
• Require a deliberate decision before retrying a non-idempotent create with an unknown result.

There is no universal retry count that makes every connector reliable. A short-lived rate limit and a missing administrator permission need different responses. Tune the policy to the endpoint, the recovery objective, and the cost of repeating the business action.

Distinguish historical replay from current-state repair

Historical replay requires the original change, a retained history, and a policy for ordering it against newer changes. Current-state repair reads what the source says now and attempts to bring the destination into agreement. Both can be useful; they solve different problems.

For example, a shipment may move from packed to shipped to delivered during an outage. A customer-facing status field may only need “delivered.” A billing workflow that must act on the “shipped” transition may require preserved event history. Decide which contract applies before choosing a recovery action.

Keep replay scope narrow: select the time window, entities, mapping version, and expected target state. Test a small representative set before expanding. A successful replay job is not proof of correctness; compare affected record identities and critical values afterward.

A recovery procedure your team can rehearse

• 01Identify the affected system, objects, first failure time, stable record IDs, and current business impact.
• 02Classify the failure and determine whether the destination may already have applied the operation.
• 03Choose the authoritative state and the intended recovery method: retry, current-state repair, historical replay, or reconciliation.
• 04Fix the underlying permission, validation, dependency, capacity, or mapping problem.
• 05Test one representative record and inspect both systems plus downstream side effects.
• 06Process a bounded batch while watching destination pressure and unresolved issue age.
• 07Reconcile affected identities, values, relationships, and business totals. Record exceptions and owners.
• 08Close the incident only after the outcome is verified; add the failure to the release test set.

Download this recovery runbook. Pair it with the production-readiness checklist and field mapping workbook. For a concrete connector review, bring a failing-record example to a Stacksync demo using test data and the exact write behavior you need to verify.