Idempotency in Payment Systems: What They Don't Tell You

Every payment engineer learns about idempotency keys on day one. Client generates a UUID, sends it with the request, and the server uses it to deduplicate. Simple, right?

Not quite. After implementing reconciliation at a payment gateway, here are the edges you actually hit.

The thirty-second window

Most idempotency implementations use Redis with a TTL equal to the idempotency key’s expiry window — say 24 hours. If a client retries within that window, they get the cached response.

But what happens when a bank callback arrives 48 hours later? Your idempotency key is gone. Redis evicted it. Now you process what looks like a fresh payment.

The fix: Layer your idempotency. Use Redis for the hot path (fast lookups on retries), but keep a persistent record in your database. When Redis returns nothing, check the DB.

func ProcessPayment(ctx context.Context, key string, req PaymentRequest) (*PaymentResult, error) {
    // Check Redis first (hot path)
    cached, err := redis.Get(ctx, key)
    if err == nil {
        return unmarshal(cached)
    }

    // Fall back to database
    existing, err := db.FindByIdempotencyKey(ctx, key)
    if err == nil {
        return existing, nil
    }

    // Genuinely new request
    return executePayment(ctx, key, req)
}

The partial success problem

A payment request involves multiple steps: validate, reserve funds, debit, notify. If the debit succeeds but the notification fails, what does the idempotency response say?

You can’t say “success” — the notification hasn’t gone out. You can’t say “failure” — the money has moved.

This is where most systems write a compensating transaction. But in practice, the simplest approach is to make downstream operations idempotent too:

func executePayment(ctx context.Context, key string, req PaymentRequest) (*PaymentResult, error) {
    result, err := gateway.Debit(ctx, req)
    if err != nil {
        return nil, err
    }

    // Notification is retry-safe — it checks if a notification for this
    // payment ID was already sent before sending a duplicate.
    go sendNotification(ctx, req.ID, result)

    return result, nil
}

The goroutine keeps retrying until the notification is confirmed sent. If the process dies mid-retry, the idempotency layer (Redis + DB) ensures the next request gets the correct response — and the notification retry picks up where it left off.

The key

The key isn’t your idempotency key. It’s your operational model. Every operation must be designed for “at-least-once” delivery, with idempotency as the safety net, not the strategy.