Scoutinternal

Webhook Notifications: Suspension, Missing Fires, and Configuration Issues

Webhook Notifications: Suspension, Missing Fires, and Configuration Issues

Problem

Customers using BitGo webhooks report a range of issues: webhooks are suspended due to repeated delivery failures, webhook notifications are not fired for incoming deposits or transfers, webhook test functionality is unavailable in the UI, token transactions are missed because of misconfiguration, or the customer's receiving server blocks BitGo's requests. These issues affect all coin types (BTC, ETH, LTC, XRP, TRX, SOL, POLYGON, TON, DOGE, AVAXC, XLM, and their tokens) and can be observed on both self-custody and custodial wallets across production and test environments.

Diagnostics

  • Get the wallet ID. Always request the wallet ID from the customer before investigating.
  • List webhooks on the wallet. Use the internal admin tool (bga wallet webhooks or equivalent) or the API endpoint GET /api/v2/{coin}/wallet/{walletId}/webhooks to retrieve all webhooks configured on the wallet. Inspect the "state" field for each webhook — values are "active" or "suspended".
  • Check allToken flag. If the customer is missing notifications for ERC-20, TRC-20, SPL, or other token transactions on an account-based coin (ETH, TRX, SOL, POLYGON, OPETH, BSC, AVAXC, BASEETH), confirm that "allToken": true is set on the webhook. If "allToken": false, only native coin transfers will trigger notifications — token transfers will be silently skipped.
  • Check successiveFailedAttempts and failingSince. High values indicate the webhook URL has been unreachable or returning errors. Look at the "response" block in webhook notifications for HTTP error codes (403, 404, 500, 422, etc.) and response bodies.
  • Inspect webhook notification history for a specific transaction. Use bga wallet webhooknotifications -id <txHash> or the List Webhook Notifications endpoint to see whether a notification was created, its state (processed, pending, failed), the response code, and the response body.
  • Check for platform-wide outages. Review https://status.bitgo.com/ and internal Slack channels for known indexer delays or webhook processing incidents. Multiple customers reporting missing webhooks simultaneously points to a BitGo-side issue rather than customer configuration.
  • Verify the webhook URL is reachable. Look at the "response" object in webhook notifications. A 403 with body containing "Enable JavaScript and cookies to continue" indicates Cloudflare or similar CDN/WAF is blocking BitGo. A 500 or empty body may indicate the customer's server is down or misconfigured. A 404 with "Token not found" indicates the URL path has changed or is invalid.
  • Confirm webhook type matches intended use. A "type": "transfer" webhook fires for transactions. A "type": "pendingapproval" webhook fires for approval events. Customers sometimes create the wrong type. For OFC (trading) wallets, the webhook should use coin=ofc with allToken=True.
  • Check API token permissions for webhook secret creation. For the POST /api/v2/webhook/secret endpoint, the requester must be an admin of the specified enterprise or organization. A non-admin user will receive an "unauthorized" error.

Resolution

Scenario: webhook-webhooks-we-suspension#suspended-due-to-failures

Trigger: Customer receives a "Webhook Suspension Notification" email stating a webhook has been suspended due to too many failures.

Signals: Webhook Suspension Notification, suspended, too many failures, successiveFailedAttempts, failingSince, state suspended

Steps:

  1. Retrieve the webhook list for the wallet and confirm the webhook shows "state": "suspended".
  2. Inform the customer that a webhook cannot be "unsuspended" once suspended. They must delete the suspended webhook and create a new one.
  3. Check the "response" block in the webhook notification history to identify why deliveries were failing (e.g., HTTP 500, 403, 404). Share this information with the customer so they can fix the underlying issue on their server before recreating.
  4. Once the customer has fixed their server-side issue, instruct them to delete the old suspended webhook and recreate it via the UI (Wallet → Settings → Webhooks) or API (POST /api/v2/{coin}/wallet/{walletId}/webhooks).
  5. Advise the customer to use the Simulate Webhook API to test the new webhook: https://developers.bitgo.com/api/v2.wallet.simulatewebhook

Notes: BitGo support cannot unsuspend or recreate webhooks on behalf of the customer. The customer's team must perform this action. Suspension thresholds cannot be configured or disabled by the customer.

"Please note that a webhook cannot be 'unsuspended' once suspended. Please create a new webhook if required." "Upon checking the wallet webhook (walletId: [WALLET_ID]), we see the webhook is suspended due to continuous code 500 errors. Code 500 errors generally indicates that something's wrong on the webhook server side... As webhook cannot be unsuspended, please create a new webhook after fixing any issue on the server end. You can use the Simulate Webhook API for testing if required - https://developers.bitgo.com/api/v2.wallet.simulatewebhook" "This message means that your webhook has been failing to connect to the URL that it was set up with and has suspended transferring data after many failed attempts. To fix this please remove this webhook and remake it."

Scenario: webhook-webhooks-we-suspension#alltoken-false-missing-token-notifications

Trigger: Customer reports not receiving webhook notifications for token transactions (e.g., USDT, USDC, ERC-20, TRC-20, SPL tokens) but native coin transfers work fine.

Signals: allToken false, token transfers missing, USDT not triggering, ERC-20 missing, TRC-20 webhook, sol token, opeth token, OP:USDC, OP:USDT, baseeth token, BASE token

Steps:

  1. List the webhooks on the wallet and check the "allToken" field. If it is false, this is the root cause.
  2. Inform the customer that "allToken" must be set to true to receive notifications for token transactions. When false, only native coin transactions (e.g., ETH, TRX, SOL) trigger the webhook.
  3. The webhook cannot be edited to change this flag. The customer must delete the existing webhook and recreate it with allToken set to true.
  4. For API creation, reference: https://developers.bitgo.com/api/v2.wallet.addwebhook — include the allToken: true parameter.
  5. Note: In some UI versions, the option to set allToken may not be available. If the customer encounters this, advise them to revert to the "classic view" or use the API to create the webhook.

Notes: This is one of the most common webhook misconfiguration issues. It affects all account-based coins with tokens: ETH, TRX, SOL, POLYGON, OPETH, BSC, AVAXC, BASEETH (Base chain), and others. For OFC (trading) wallets, engineering advises using coin=ofc with allToken=True to receive transfer notifications for all currencies.

"I checked the webhook for the wallet and the allToken is set to be false, this needs to be true so you get notified for erc20 transfers, when it's false you will only get notified only for eth transactions." "Can you recreate this webhook on the wallet with the 'allTokens' flag set to True? This should trigger the notifications for the trc20 token transactions. The current webhook has this set to 'false'." "We see that allToken": false, which means this won't trigger the webhooks to the coin's tokens. You need to recreate it and set the allToken to True." "I checked if allToken flag is set to true or false, it was false, so asked client to set it to true." (baseeth wallet, type: transfer, allToken: false)

Scenario: webhook-webhooks-we-suspension#cloudflare-or-cdn-blocking

Trigger: Webhook notification shows "state": "processed" or "failed" with a response code of 403 and body containing "Enable JavaScript and cookies to continue" or "Forbidden", or the customer's webhook URL is behind Cloudflare or similar WAF/CDN.

Signals: 403, Forbidden, Enable JavaScript and cookies to continue, Cloudflare, CDN, WAF, blocked, bot protection

Steps:

  1. Retrieve the webhook notification for the relevant transaction and check the "response" block for a 403 code.
  2. Inform the customer that their CDN or WAF (commonly Cloudflare) is intercepting BitGo webhook requests before they reach the customer's server.
  3. Provide the customer with the following solutions:
    • Add a page rule on Cloudflare to exempt the BitGo webhook URL path (e.g., https://www.example.com/api/webhooks/bitgo).
    • Whitelist BitGo IP addresses (note: these may change due to platform maintenance — BitGo discourages relying solely on IP whitelisting):
      • Prod: 52.43.255.93, 35.164.184.184, 54.214.57.139, 18.220.107.76/32, 18.222.87.170/32, 3.137.109.219/32
      • Test: 34.218.55.207, 34.215.209.72, 34.216.199.10, 18.220.168.234, 13.58.253.95, 18.216.193.244
    • Create a separate endpoint that is not behind Cloudflare/WAF.
  4. Reference Cloudflare's documentation for exempting API traffic from bot protection: https://developers.cloudflare.com/waf/custom-rules/use-cases/challenge-bad-bots/#exempt-api-traffic
  5. Remind the customer that after 2+ days of continuous failures, the webhook will be automatically suspended and will need to be deleted and recreated.

Notes: This scenario is distinct from a customer server error (5xx). The 403 with JavaScript/cookies challenge is specifically a CDN interception pattern. The customer may confirm via Postman that the URL works, but BitGo's server-to-server requests do not execute JavaScript challenges.

"Sometimes customers use our webhooks but their cdn (e.g. cloudflare) filters our requests. Then they don't get the webhook... Add a page rule on Cloudflare to exempt the bitgo webhook url... Whitelist bitgo IP addresses: Prod: 52.43.255.93, 35.164.184.184, 54.214.57.139, 18.220.107.76/32, 18.222.87.170/32, 3.137.109.219/32" "we see the webhook processed the transfer you mentioned but saw a 403 error when relaying this to the webhook server... "response": { "code": 403, "type": "text/html", "body": "Enable JavaScript and cookies to continue" }... Please kindly check the access and set up of your webhook server and try again."

Scenario: webhook-webhooks-we-suspension#platform-wide-webhook-outage

Trigger: Multiple customers simultaneously report missing webhook notifications across different coins and wallets, or there is a known platform incident affecting indexers or webhook processing.

Signals: webhooks not firing, all deposits down, indexer delay, platform outage, multiple coins affected, engineering fix, status.bitgo.com

Steps:

  1. Check https://status.bitgo.com/ and internal Slack channels for any active incidents affecting webhook delivery or blockchain indexers.
  2. If a known incident is confirmed, inform the customer that the engineering team is aware and working to resolve the issue.
  3. For missed webhooks during the outage, advise the customer to use the Simulate Webhook API to retrigger notifications: https://developers.bitgo.com/api/v2.wallet.simulatewebhook. This can also be done from the UI.
  4. Once engineering confirms the issue is resolved, follow up with the customer and ask them to monitor for continued problems.
  5. In some cases, the engineering team may retrigger missed notifications in bulk on the backend. Confirm with engineering whether this has been done before advising the customer to simulate all missed transactions individually.

Notes: Platform-wide outages have historically affected BTC/UTXO indexers (causing delayed webhooks), webhook processing queues (causing missed notifications across all coins), and authentication systems (which can also halt webhook delivery). During recovery, duplicate webhook sends may occur — warn customers to implement idempotency in their webhook handler.

"We had an indexer delay affecting BTC / UTXO yesterday which might have affected the webhook notifications. Our Engineering team has completed the fix for the above issue, please help to monitor today onwards if you are still experiencing the same issue as reported." "Our engineering team has resolved the issue with webhooks not being triggered. Please let us know if they continue not to be sent. They are working on a method to retrigger notifications not sent during this issue." "Yes, yesterday our platform has triggered an issue for not sending out the webhook notifications which is fixed now. If you have missed any webhooks you can use our API to simulate webhooks: https://developers.bitgo.com/api/v2.wallet.simulatewebhook"

Scenario: webhook-webhooks-we-suspension#customer-server-returning-errors

Trigger: Webhook notifications show "state": "failed" or "pending" with response codes 4xx or 5xx originating from the customer's server (not Cloudflare).

Signals: 500, 422, 404, server error, customer server, webhook failing, failingSince, successiveFailedAttempts

Steps:

  1. Retrieve the webhook notification history for the transaction in question and share the "response" block with the customer, including the HTTP code and body.
  2. Explain what the error code means:
    • 500: Server-side error on the customer's webhook endpoint.
    • 422: The customer's server could not process the webhook payload (e.g., missing expected fields).
    • 404: The webhook URL path is invalid or no longer exists.
  3. Ask the customer to fix the issue on their server side.
  4. Once fixed, the customer can use the Simulate Webhook API to resend missed notifications: https://developers.bitgo.com/api/v2.wallet.simulatewebhook
  5. If the webhook has already been suspended due to accumulated failures, the customer must delete and recreate it.

Notes: BitGo retries failed webhook notifications with backoff. After enough consecutive failures, the webhook is suspended. The customer should implement proper error handling and return HTTP 200 for successfully received webhooks.

Scenario: webhook-webhooks-we-suspension#no-webhook-configured

Trigger: Customer reports not receiving webhook notifications, and the webhook list for the wallet returns an empty array or shows no webhooks.

Signals: no webhooks, empty array, webhook not configured, no webhook on wallet

Steps:

  1. Confirm with the customer that no webhooks are configured on the wallet by listing webhooks.
  2. Instruct the customer to create a webhook on the wallet via the UI (Wallet → Settings → Webhooks) or via API: https://developers.bitgo.com/api/v2.wallet.addwebhook
  3. Ensure the customer sets the correct type (transfer for transaction notifications, pendingapproval for approval events), the correct numConfirmations, and allToken: true if they need token notifications.
  4. BitGo support cannot create webhooks on behalf of the customer.

Notes: Customers sometimes confuse enterprise-level webhooks with wallet-level webhooks. Enterprise webhook endpoints only return enterprise-level webhooks. Wallet webhooks must be created per-wallet.

"Additionally, we are unable to create webhooks for you. Your team will need to recreate this webhook on the wallet and remove the old suspended webhook."

Scenario: webhook-webhooks-we-suspension#webhook-test-button-not-working

Trigger: Customer reports that the webhook "Test" button in the UI is greyed out or does not allow entering a Transfer ID.

Signals: test webhook, greyed out, Transfer ID disabled, test not working, pending approval required

Steps:

  1. Confirm with the customer whether the webhook type is pendingapproval. For pending approval webhooks, testing requires an active pending approval to exist on the wallet.
  2. For transfer type webhooks, the customer can test using the Simulate Webhook API endpoint instead: https://developers.bitgo.com/api/v2.wallet.simulatewebhook
  3. If the UI test functionality appears broken for transfer webhooks, this may be a known UI limitation. Advise the customer to use the API endpoint as a workaround.

Notes: Engineering has confirmed that pending approval webhook testing requires an existing pending approval on the wallet. This is the expected current behavior.

"Our engineering team is reporting this error response is expected at this time. We are working on a major upcoming release where this particular function will be fixed and allow your team to test the Webhook without a pending approval on the wallet. For now, to successfully test this webhook, a Pending Approval will be required on the wallet." "Our Engineering had checked this and confirmed that the test will work if there is a pending approval in the wallet. We have since fixed the error handling to return a better message when there is no pending approval."

Scenario: webhook-webhooks-we-suspension#webhook-secret-creation-unauthorized

Trigger: Customer receives an "unauthorized" error when calling POST /api/v2/webhook/secret to create a webhook secret.

Signals: unauthorized, webhook secret, forbidden, API Access Token, enterprise admin, owner

Steps:

  1. Confirm that the API request is being made by a user who is an admin of the specified enterprise or organization. Non-admin users will receive an "unauthorized" error.
  2. Ask the customer to verify which user account generated the API access token being used.
  3. Instruct them to retry using an access token generated by an enterprise admin/owner account.
  4. Reference documentation: https://developers.bitgo.com/api/v2.webhook.secret.create

Notes: The webhook secret endpoint requires enterprise admin or owner credentials. This is not a wallet-level permission — it is an enterprise-level operation.

"The requester must be an admin of the specified enterprise or organization. It looks like the API request was made by User ID : [EMAIL]. Please try again via User ID : [EMAIL]"

Scenario: webhook-webhooks-we-suspension#wrong-api-endpoint-for-listing

Trigger: Customer uses the block-level webhook list endpoint (GET /api/v2/{coin}/webhooks) to list wallet webhooks and receives an empty array or 500 error.

Signals: empty array, 500 Internal Server Error, list webhooks, wrong endpoint, block webhooks vs wallet webhooks

Steps:

  1. Clarify the difference between endpoints:
    • GET /api/v2/{coin}/webhooks — lists block-level webhooks, not wallet webhooks.
    • GET /api/v2/{coin}/wallet/{walletId}/webhooks — lists wallet-level webhooks.
    • GET /api/v2/enterprise/{enterpriseId}/webhooks — lists enterprise-level webhooks (requires enterprise owner token).
  2. Instruct the customer to use the correct wallet-level endpoint: https://developers.bitgo.com/api/v2.wallet.listwebhooks
  3. Ensure the access token used belongs to a user who is an admin on the wallet.

"You created a wallet webhook using this endpoint https://developers.bitgo.com/api/v2.wallet.addwebhook with the type 'Pending approval' thus you should use this endpoint to retrieve this webhook https://developers.bitgo.com/api/v2.wallet.listwebhooks" "This endpoint will only return Enterprise Level Webhooks that have been created against the Enterprise. This will not return a list of webhooks generated at the Wallet level... For a List of Wallet webhooks, the following can be used: https://developers.bitgo.com/api/v2.wallet.listwebhooks"

Scenario: webhook-webhooks-we-suspension#duplicate-webhook-notifications

Trigger: Customer reports receiving duplicate webhook notifications for the same transaction.

Signals: duplicate, duplicate webhook, double notification, idempotency

Steps:

  1. Retrieve the webhook notification history for the transaction. Check if there are multiple notification entries — look at the "state" field to determine if one was "processed" and another is "pending" (retry).
  2. BitGo may send separate notifications for "unconfirmed" and "confirmed" states of the same transaction — this is expected behavior if the webhook has numConfirmations set to 0.
  3. During platform recovery from outages, previously queued notifications may be resent, causing duplicates.
  4. Advise the customer to implement idempotency in their webhook handler — use the transaction hash and state as a unique key to deduplicate incoming notifications.
  5. If the customer has both a numConfirmations: 0 webhook and a separate higher-confirmation webhook on the same wallet, they will naturally receive multiple notifications per transaction.

Notes: BitGo's webhook system is designed for at-least-once delivery. Customers should always handle potential duplicates.

Scenario: webhook-webhooks-we-suspension#webhook-authentication-security

Trigger: Customer asks how to authenticate or verify that incoming webhook requests are genuinely from BitGo.

Signals: webhook authentication, security, token, secret, verify, identity, HMAC

Steps:

  1. BitGo provides a webhook secret feature. Direct the customer to the documentation: https://developers.bitgo.com/api/v2.webhook.secret.create
  2. The API call to create a webhook secret requires the enterprise admin or owner credentials.
  3. If the customer needs additional identity verification beyond the webhook secret, note that BitGo does not currently support adding custom authentication headers or tokens to outgoing webhook requests.
  4. Advise alternative approaches the customer can implement on their side:
    • Use the webhook secret for HMAC verification.
    • Whitelist BitGo IP addresses (see Cloudflare scenario above for current IPs, with caveat that these may change).
    • Upon receiving a webhook, call the BitGo Get Transfer API to independently verify the transaction rather than relying solely on the webhook payload.

Notes: Historically, BitGo did not support adding identity information to webhook requests. The webhook secret feature was introduced to address this need. Customers with strict security requirements should verify transaction details via the API after receiving a webhook.

"The API call can done only through the owner credentials so as we can see that Priyanka is not the owner so please try to use the API from the owner account."

Scenario: webhook-webhooks-we-suspension#missing-transaction-not-indexed

Trigger: Customer reports that a deposit does not appear in the wallet transaction history or UI, and no webhook was fired — the transaction exists on-chain but BitGo has not indexed it.

Signals: missing transaction, not in UI, deposit not showing, balance mismatch, not indexed, rebroadcast

Steps:

  1. Verify the transaction on-chain using the hash provided by the customer.
  2. Check the wallet transfer records using the admin tool (bga t <txhash>). If no transfer record exists, the transaction was not indexed.
  3. Escalate to engineering to rebroadcast/reindex the transaction. Provide the wallet ID, coin, and transaction hash.
  4. Once engineering resolves the indexing, confirm the transfer record is created and the webhook notification fires.
  5. If the wallet balance is also incorrect, engineering may need to correct it manually (as seen with TON wallet balance discrepancies).

Notes: This can happen with newer or less-established chains (e.g., TON), batch transactions involving unsupported tokens, or during indexer outages. The customer cannot self-serve this — it requires engineering intervention.

"We have corrected the wallet balance and continuing our investigation into your webhook issue." "We have rebroadcast this transaction. Can you check to see if this triggered the needed webhook for you? We are still investigating for a root cause."

Related

  • setting-up-a-wallet — Covers wallet settings including webhook configuration in the UI.
  • wallet-settings — Documents webhook setup under wallet settings tab, including trigger event types and the webhook dialog.
  • bitcoin-special-call-outs — Relevant for UTXO-based coin transaction fee and confirmation behavior that can affect webhook timing.