Payment failed but money deducted: how idempotency keys stop double charges

The app shows "Payment failed". A second later your phone gets an SMS: ₹2,000 debited.

You tap Pay again. Now ₹4,000 is gone, and you still have no order.

This post covers why this happens, the fixes that don't work, and the fix real payment systems use.

https://www.youtube.com/watch?v=A6hrGCQidgM

TL;DR

• The bank debited the money. Only the reply to your server got lost.
• A timeout means "I don't know", not "failed".
• A blind retry charges twice. An idempotency key makes retries safe.
• Webhooks and status polling find the real result. A daily reconciliation job catches anything they miss.

The problem

A payment goes through four hops:

App ──▶ Your server ──▶ Gateway ──▶ Bank      ✔ ₹2,000 debited
App ◀── Your server ◀─✘─ Gateway ◀── Bank      reply lost / timed out

The money moved. Only the reply got lost.

The bank did its job. But the network dropped, or your server stopped waiting after 30 seconds. Your server never heard "success", so it marked the payment failed.

Now two bad things can happen:

1. The user pays again, and gets charged twice.
2. Nobody follows up. ₹2,000 is debited and there's no order.

(If the user just kills the app, that's the easy case. Your server still got the reply; the app asks for the status when it reopens.)

Fixes that sound right

1. "Just retry"

Retrying is right. A blind retry is wrong: the first debit already went through, so the retry debits again.

2. "Wrap it in a DB transaction"

A transaction can roll back rows in your database. It can't pull money back from the bank.

3. "Retry with exponential backoff"

Backoff decides when to retry. Without a key, every retry still looks like a new payment.

4. "Send a new key on every tap"

Then every tap looks like a new payment again. The key must be one per checkout.

The real fix

Step 1: a timeout is PENDING, not FAILED

CREATED ──▶ PENDING ──▶ SUCCESS
                   └──▶ FAILED

On a timeout, save PENDING and show the user "Payment processing". Move to SUCCESS or FAILED only when you have proof.

Step 2: an idempotency key

The client creates a unique ID once per checkout and sends it with every retry:

POST /payments
Idempotency-Key: 7f3c9a2e-1b4d-4e8a-9c2f-5d6e7a8b9c0d

{ "orderId": 981, "amount": 2000 }

The server stores every key with a unique constraint:

CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  status       TEXT NOT NULL,      -- IN_PROGRESS / COMPLETED
  response     JSONB,
  created_at   TIMESTAMP DEFAULT now()
);

try {
    keys.insert(key, hash(request), "IN_PROGRESS");    // unique key
} catch (DuplicateKeyException e) {
    Saved saved = keys.find(key);
    if (!saved.hash().equals(hash(request))) throw new UnprocessableEntity(); // same key, different amount
    if (saved.isCompleted()) return saved.response();  // no new charge
    throw new Conflict("still processing");            // 409
}
Response r = gateway.charge(request, key);              // pass the key down too
keys.complete(key, r);
return r;

One key per checkout. The unique constraint lets one insert win.

Three rules:

• Same key again: return the saved response. No second charge.
• Same key, different amount: reject it. It's a bug, not a retry.
• Pass the key to the gateway too, so the gateway refuses a second charge as well. Stripe takes an Idempotency-Key header directly.

Step 3: find out what really happened

• Webhook: the gateway calls your URL with the final status. Verify its signature.
• Status poll: a job asks the gateway "what happened to payment 981?" for anything still PENDING after a few minutes.

Step 4: reconciliation, the safety net

Once a day, match your payments table against the gateway's or bank's settlement report:

Webhooks and polls miss things. The daily match doesn't.

Beyond the short

Two retries at the same moment

"Check if the key exists, then insert it" lets both retries through, and both charge. Let the unique constraint decide (or Redis SET key NX): only one insert can win. It's the same check-then-act gap as Day 2.

The same webhook arrives twice

Gateways retry webhooks. If the payment is already SUCCESS, ignore the event, or dedupe by the event ID.

How long do you keep keys?

Long enough to cover every retry. Stripe says keys can be removed once they're at least 24 hours old.

All the options

A timeout isn't a failure. It's a question you haven't answered yet.

This is Day 5 of my system design shorts. Next: the API works in Postman, but the browser shows a CORS error. Same request, so why?

Subscribe on YouTube or follow this blog so you don't miss it.

References

• Stripe: Idempotent requests
• Stripe blog: Designing robust and predictable APIs with idempotency
• IETF draft: The Idempotency-Key HTTP header