Design a payment system

Design the payment backend for an e-commerce site: when a customer checks out, charge their card through a payment provider, record the money movement correctly, and pay out to sellers.

1. Clarify the requirements

Before drawing anything, ask questions. These are the answers an interviewer would typically give:

Volume?

1 million payments a day.

Do we process cards ourselves?

No. We use a payment service provider (like Stripe or Adyen); we never store raw card numbers.

Consistency?

Money must never be lost or charged twice. Correctness beats latency.

Refunds?

Yes, refunds must be supported. Reconciliation with the provider is required.

Latency?

A few seconds at checkout is acceptable.

2. Estimate the scale

Assumptions: 1M payments per day.

• Average payments per second: about 11.6 payments/s1M ÷ 86,400 ≈ 12 TPS: low volume, so correctness (not throughput) is the challenge.

3. The high-level design

1. Checkout calls a payment serviceCheckout calls the payment service through an API gateway that authenticates, rate limits and requires an idempotency key on every payment.
2. A transactional ledgerMoney needs ACID transactions: payments and a double-entry ledger (every debit has a credit) in a relational database, with a synchronous replica.
3. Never charge twiceThe idempotency key is stored with the result, so a retried request returns the first result instead of charging again.
4. Execute payments asynchronouslyCalls to the payment provider can be slow or fail, so executors take jobs from a queue and retry with backoff, using the same idempotency key.
5. Watch it in productionMetrics and alerts close the loop: success rate per provider, retries, and reconciliation mismatches.

4. The components and what each one does

API gateway API gateway

Auth, rate limits; an idempotency key on every payment request.

Payment service Service

Creates the payment event, records it, hands execution to the payment executors. Stateless: scales out horizontally.

Payments & ledger SQL database · PostgreSQL

ACID transactions; double-entry ledger; primary + synchronous replica with failover; sharded by merchant as it grows. Nightly reconciliation against the provider's settlement file.

Idempotency keys Cache

idempotency key → result.

Payment executors Workers

Call the provider; retry with exponential backoff; idempotent.

Payment provider (Stripe) 3rd-party service

Card networks; sends webhooks when a payment settles.

Metrics & alerts Metrics & alerts

Dashboards and alerts on p99 latency, error rate, queue depth and cache hit ratio, so a bottleneck shows up before it becomes an outage.

5. Deep dive: correctness: idempotency, ledger & reconciliation

How do you guarantee a customer is never charged twice, even with retries, timeouts and crashes? How do you know your books match the provider's?

• Every payment request carries an idempotency key; the server and the PSP dedupe by it.
• Record money movements in an append-only double-entry ledger.
• Reconcile nightly against the PSP's settlement files; handle webhooks for async results.

6. What interviewers listen for

• Payment API with idempotency keyClients retry; the API must accept an idempotency key so a retry never charges twice.
• Double-entry, append-only ledgerAn immutable ledger makes every money movement auditable and balances checkable.
• Idempotency and safe retriesTimeouts are ambiguous: did the charge happen? Idempotency keys make retries safe.
• Reconciliation with the providerNightly reconciliation catches mismatches between your ledger and the PSP's records.
• Asynchronous results (webhooks / queue)PSPs often confirm payments asynchronously; handle webhooks idempotently.
• Payment state machinePayments move through states (created → pending → succeeded/failed → refunded).
• Security: PCI, tokenization, no raw card dataLet the PSP hold card data; store only tokens.
• Database replication and failoverA single database is a single point of failure. Replicas give redundancy and read capacity; failover promotes a replica when the primary dies.
• Partition (shard) data as it growsOne machine eventually can't hold the data or the write load. Say how you'd split it and by which key.

7. Follow-up questions

1. Your call to the PSP times out after 30 seconds. Did the customer get charged? What do you do?
2. The payment service crashes after the PSP charged the card but before your database recorded it. How do you recover?

The design in a nutshell

• Checkout → payment service (idempotency key) writes a pending payment row → calls PSP with the same key.
• PSP result (sync or webhook) updates the state machine; double-entry ledger records the movement in one DB transaction.
• Wallet/payout service pays sellers; nightly reconciliation against PSP settlement files.
• Strongly consistent SQL with replication; no raw card data (PSP tokens); monitoring of failed/pending payments.

Related lessons: Reliability: no data loss, no duplicates · The CAP theorem