Appearance

Recovery and Troubleshooting ​

Bridge incidents usually fall into one of three buckets:

  1. the source-side deposit happened, but no destination-side progress is visible yet
  2. the destination-side withdraw entry exists, but is not finalized or not executable
  3. the withdraw completed, but the source-side deposit has not finalized or cleaned up yet

The most important thing is to identify which phase the transfer is stuck in.

Mental model for debugging ​

A transfer does not finish in one step.

You need to reason about these stages separately:

  • deposit
  • ack_deposit
  • withdraw
  • ack_withdraw
  • cleanup_deposit

Bridge-specific

It is completely possible for one phase to succeed while a later phase is still pending. For example, destination-side withdrawal may be available while source-side cleanup has not happened yet.

Symptom: deposit succeeded, but nothing shows up on the destination chain ​

What this usually means ​

The source-side DepositEntry exists, but authorities have not yet driven ack_deposit to threshold finalization on the destination-side WithdrawTracker.

Operator checks ​

  • confirm the deposit entry exists on the source-side tracker
  • confirm the bridge server is running
  • confirm source and target RPC endpoints are healthy
  • check whether the authority signer has submitted recent crank actions
  • inspect /health and /metrics

Useful places to look:

  • bridge server logs
  • source DepositTracker
  • target WithdrawTracker
  • /health.last_error
  • /health.last_action

Common causes ​

  • bridge server or crank offline
  • bad source or target RPC connectivity
  • authority signer unavailable
  • destination-side finalization still waiting on threshold agreement
  • pair frozen, missing, or over its active limit

Symptom: withdraw entry exists, but withdraw is not executable ​

What this usually means ​

The target-side WithdrawEntry has not finalized to DepositResult::Confirmed, or it finalized to a rejected result instead.

What to check ​

  • is the entry still in Voting?
  • did it finalize to Rejected with one of the known reasons?
  • did it finalize to Confirmed and is created_token_account still Pending?

Relevant rejection reasons include:

  • InternalError
  • InsufficientFunds
  • LimitExceeded
  • TokenPairFrozen
  • TokenPairNotFound

Symptom: withdrawal happened, but source-side state still looks unfinished ​

What this usually means ​

The destination-side result still needs to be relayed back by ack_withdraw, then the source-side deposit must finalize, then cleanup_deposit can run.

Important nuance ​

ack_withdraw is only ready when the destination-side result is:

  • Rejected, or
  • Confirmed and created_token_account != Pending

That means a confirmed destination-side entry is not ready to relay back if the withdraw path has not finished updating that field yet.

Symptom: finalized deposit still has not been cleaned up ​

What this usually means ​

The source-side deposit entry is finalized, but cleanup_deposit has not run yet.

Cleanup is permissionless, but the crank is expected to automate it.

What cleanup can do ​

Depending on the final result, it may:

  • refund the full amount on rejection
  • refund leftover entry.amount - confirmed.amount on partial confirmation
  • mark cleanup_completed=true

Manual recovery concepts for operators ​

ack_deposit ​

Use this when source deposits exist and destination-side withdraw entries have not been created or finalized.

ack_withdraw ​

Use this when destination-side results exist, but source-side deposit entries have not finalized.

cleanup_deposit ​

Use this when the source-side deposit entry is finalized but refund/cleanup has not happened.

Zink recommendation

Do not treat manual CLI intervention as a substitute for understanding tracker state. Always inspect which side is ahead before replaying operator actions.

Failure patterns worth recognizing ​

Threshold-no behavior ​

If authorities disagree strongly enough to reach a threshold no:

  • destination-side withdraw entries may be removed/reset during ack_deposit
  • source-side deposit entries may be reset to Initialized during ack_withdraw

That is not necessarily corruption — it is the protocol resolving disagreement.

Cursor / vote-map issues ​

Authorities track their own next expected vote positions and last-vote direction. If one authority lands in a “pending no” path, forward progress can look strange until that disagreement is resolved.

Governance changes during pending work ​

Proposal-driven changes can reset non-finalized work:

  • WithdrawTracker removes non-finalized entries
  • DepositTracker resets non-finalized entries back to Initialized

If operators changed token-pair config mid-incident, check whether state was intentionally reset.

Server-side checks ​

Useful questions for the bridge server:

  • Is crank enabled?
  • When did it last run?
  • When did it last succeed?
  • What was the last action?
  • What was the last error?
  • Are retries exhausting?

Relevant endpoints:

  • GET /health
  • GET /metrics
  • POST /crank/once (auth-gated)

Escalation bundle ​

When escalating a bridge issue internally, collect:

  • source chain ID
  • target chain ID
  • source deposit entry ID if known
  • destination withdraw entry ID if known
  • source and target transaction signatures
  • current DepositStatus
  • current WithdrawStatus
  • latest /health payload
  • recent server logs
  • whether cleanup has already run

Zink is a general-purpose SVM network for programs, operators, and bridge integrations.

Zink