# Payment failed but money deducted: how idempotency keys stop double charges

The app shows **"Payment failed"**. A second later your phone gets an SMS: **₹2,000 debited**.

You tap Pay again. Now ₹4,000 is gone, and you still have no order.

This post covers why this happens, the fixes that don't work, and the fix real payment systems use.

%[https://www.youtube.com/watch?v=A6hrGCQidgM]

---

## TL;DR

- The bank debited the money. Only the **reply** to your server got lost.
- A timeout means **"I don't know"**, not "failed".
- A blind retry charges twice. An **idempotency key** makes retries safe.
- Webhooks and status polling find the real result. A daily **reconciliation** job catches anything they miss.

---

## The problem

A payment goes through four hops:

```plaintext
App ──▶ Your server ──▶ Gateway ──▶ Bank      ✔ ₹2,000 debited
App ◀── Your server ◀─✘─ Gateway ◀── Bank      reply lost / timed out
```

![The bank debits ₹2,000, but the reply between the gateway and your server is lost, so the app shows Payment failed](https://raw.githubusercontent.com/param4/system-design-with-param/main/shorts/05-payment-idempotency/boards/01-the-money-moved-the-reply-didn-t-final.png align="center")

*The money moved. Only the reply got lost.*

The bank did its job. But the network dropped, or your server stopped waiting after 30 seconds. Your server never heard "success", so it marked the payment **failed**.

Now two bad things can happen:

1. The user pays again, and gets **charged twice**.
2. Nobody follows up. **₹2,000 is debited and there's no order.**

(If the user just kills the app, that's the easy case. Your server still got the reply; the app asks for the status when it reopens.)

---

## Fixes that sound right

### 1\. "Just retry"

Retrying is right. A **blind** retry is wrong: the first debit already went through, so the retry debits again.

### 2\. "Wrap it in a DB transaction"

A transaction can roll back rows in **your** database. It can't pull money back from the bank.

### 3\. "Retry with exponential backoff"

Backoff decides **when** to retry. Without a key, every retry still looks like a new payment.

### 4\. "Send a new key on every tap"

Then every tap looks like a new payment again. The key must be **one per checkout**.

---

## The real fix

### Step 1: a timeout is PENDING, not FAILED

```plaintext
CREATED ──▶ PENDING ──▶ SUCCESS
                   └──▶ FAILED
```

On a timeout, save **PENDING** and show the user "Payment processing". Move to SUCCESS or FAILED only when you have proof.

### Step 2: an idempotency key

The **client** creates a unique ID once per checkout and sends it with every retry:

```http
POST /payments
Idempotency-Key: 7f3c9a2e-1b4d-4e8a-9c2f-5d6e7a8b9c0d

{ "orderId": 981, "amount": 2000 }
```

The server stores every key with a **unique** constraint:

```sql
CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,
  request_hash TEXT NOT NULL,
  status       TEXT NOT NULL,      -- IN_PROGRESS / COMPLETED
  response     JSONB,
  created_at   TIMESTAMP DEFAULT now()
);
```

```java
try {
    keys.insert(key, hash(request), "IN_PROGRESS");    // unique key
} catch (DuplicateKeyException e) {
    Saved saved = keys.find(key);
    if (!saved.hash().equals(hash(request))) throw new UnprocessableEntity(); // same key, different amount
    if (saved.isCompleted()) return saved.response();  // no new charge
    throw new Conflict("still processing");            // 409
}
Response r = gateway.charge(request, key);              // pass the key down too
keys.complete(key, r);
return r;
```

![Request 1 with key K1 charges, a retry with K1 gets the saved reply, and K1 with a new amount is rejected](https://raw.githubusercontent.com/param4/system-design-with-param/main/shorts/05-payment-idempotency/boards/02-same-key-same-answer-final.png align="center")

*One key per checkout. The unique constraint lets one insert win.*

Three rules:

- **Same key again:** return the saved response. No second charge.
- **Same key, different amount:** reject it. It's a bug, not a retry.
- **Pass the key to the gateway too**, so the gateway refuses a second charge as well. Stripe takes an `Idempotency-Key` header directly.

### Step 3: find out what really happened

- **Webhook:** the gateway calls your URL with the final status. Verify its signature.
- **Status poll:** a job asks the gateway "what happened to payment 981?" for anything still PENDING after a few minutes.

### Step 4: reconciliation, the safety net

Once a day, match your payments table against the gateway's or bank's settlement report:

| Bank says | You say | Action |
| --- | --- | --- |
| Debited | PENDING / FAILED | Complete the order, or refund |
| Not debited | SUCCESS | Flag it for someone to check |

![A daily job matches the bank settlement report against your payments table and acts on each mismatch](https://raw.githubusercontent.com/param4/system-design-with-param/main/shorts/05-payment-idempotency/boards/03-daily-reconciliation-final.png align="center")

*Webhooks and polls miss things. The daily match doesn't.*

---

## Beyond the short

### Two retries at the same moment

"Check if the key exists, then insert it" lets both retries through, and both charge. Let the **unique constraint** decide (or Redis `SET key NX`): only one insert can win. It's the same check-then-act gap as [Day 2](https://systemdesignwithparam.hashnode.dev/fix-overselling-race-condition).

### The same webhook arrives twice

Gateways retry webhooks. If the payment is already SUCCESS, ignore the event, or dedupe by the event ID.

### How long do you keep keys?

Long enough to cover every retry. Stripe says keys can be removed once they're at least 24 hours old.

---

## All the options

| Fix | Stops double charge? | Finds lost payments? |
| --- | --- | --- |
| Blind retry | No | No |
| DB transaction | No | No |
| Backoff only | No | No |
| Idempotency key | Yes | No |
| Key + PENDING + webhook / poll + reconciliation | Yes | Yes |

**A timeout isn't a failure. It's a question you haven't answered yet.**

This is Day 5 of my system design shorts. Next: the API works in Postman, but the browser shows a CORS error. Same request, so why?

[*Subscribe on YouTube*](https://www.youtube.com/@SystemDesignWithParam) *or follow this blog so you don't miss it.*

## References

- [Stripe: Idempotent requests](https://docs.stripe.com/api/idempotent_requests)
- [Stripe blog: Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency)
- [IETF draft: The Idempotency-Key HTTP header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/)