Make the payment race happen. Then prove your fix holds.
raceproof drives your gateway against a fake PSP and lands the shopper's result, your own timeout and the PSP's webhook at exactly the wrong moment, hundreds of times in a row.
It runs on your machine against your test stack. PSP test data only, no real card data, no telemetry.
What the sandbox never does
It answers once, quickly, in order
In production, callbacks arrive twice, late, out of order, or in the same millisecond as your own timeout. A sandbox lets you click through a flow; it can't place two messages 5 ms apart.
The bug looks like bad luck
A payment refused although the shopper was charged. A merchant told FAILED, then COMPLETED. An authorisation left open at the PSP. Rare per payment, expensive at volume, and nearly impossible to reproduce from a log.
Mocks don't have timing
A unit test can mock the PSP's answer. It can't mock the moment the answer arrives, or what the PSP does with a second call that carries the same idempotency key.
Your gateway, a fake PSP, and a runner that controls the clock
Point your gateway at the fake PSP
Change the PSP base URL and shorten your timeouts in a test profile. Your gateway code stays as it is.
Describe your system once
lab.yamlholds your URLs, how to read a payment's status from your database, your status values and your timeouts. Scenarios refer to them by name, so the same scenario runs against any gateway.Run each scenario a hundred times
The runner scripts the fake, fires the shopper's and the issuer's messages at chosen moments with jitter, and checks your database and the PSP's call log after every run.
# C3: the CRes arrives as the watchdog fires
repeat: 300
jitter_ms: [0, 700]
steps:
- http: {url: "${gateway.start_payment_url}",
json_file: "${payloads.start_payment}"}
- wait_status: {is: pending_challenge}
- sleep_ms: "${timing.near_watchdog_ms}"
- http: {name: cres, url: "${fake}/_control/acs/challenge",
json: {outcome: "Y", url: "${gateway.challenge_url}"}}
assert:
- tx_status: {is: completed}
- refusal_marker: {count: 0}
- sql_equals: {named: merchant_final_count, value: 1}A green result has to mean something
A race test that passes because the race never happened is worse than no test. raceproof reports what it actually exercised.
Inconclusive is its own result
If your gateway never reached the state under test (it stalled, the fake was slow), the run counts as inconclusive, not passed. Too many inconclusive runs and the whole result is marked untrusted.
It shows whether the race was hit
The report counts how many runs landed inside the race window and in which order the two calls reached the PSP. Green with zero runs in the window is reported as proving nothing.
It finds the timing for you
--sweep timing.near_watchdog_ms=3000:4800:200 runs every value, interleaved in random order so that
drift in your environment hits all values alike.
The control run comes first
Run a scenario against the unfixed code before you trust it on the fixed code. A test that can't fail on the bug it is about proves nothing; the demo shop and paygate-demo have switches for exactly this.
Measured on paygate-demo, an open-source gateway
paygate-demo is a small Spring Boot gateway for Adyen and Stripe with two of these races behind switches you flip at runtime. The same scenario, the same timing, the fix off and then on:
| Scenario | Race | Fix off | Fix on |
|---|---|---|---|
| C3 | Adyen: the shopper's challenge result arrives as the gateway's challenge timeout fires | 82 of 100 runs fail | 0 of 100 |
| S1 | Stripe: the browser's return (reloaded) and the webhook report the outcome together | 100 of 100 runs fail | 0 of 100 |
Measured 2026-11-14 on a laptop, with paygate-demo's timeouts shortened to seconds. Your numbers will differ with your machine and timing; the fix-on column should not.
Watch two races fail, then pass
The scenario menu, C3 and M1 each with the fix off and then on, and the C3 race in a shop's checkout page. Recorded against the reference gateway that paygate-demo is cut from.
- 0:04
- 0:17
- 1:39
- 3:01
- 4:01
- 5:04
Fakes built from sandbox measurements
A race only shows up if the fake answers the way the real PSP does: its idempotency replays, its "already in progress" errors, its webhook delays and retries. Each pack's fake is built from measurements in the PSP's test environment and lists what was measured and what is assumed.
| Pack | What the fake covers | Scenarios | Status |
|---|---|---|---|
| Adyen | Native 3DS2: /payments, /payments/details, the issuer's method
and challenge pages, signed webhooks | M1, C1–C5, W1–W4 | available |
| Stripe | PaymentIntents with Stripe's own 3DS page, manual capture, signed webhooks with retries | S1–S9 | available |
| Your PSP | Tell us which PSP and flow you integrate | – | on request |
Payment providers: a pack for your API lets the merchants who integrate you find these races in their own code before they turn into support tickets about payments that were charged but refused. Talk to us about coverage for your flows.
Measured, for example
Adyen, 2026-11-14: a second challenge submission with the same idempotency key while the first is in progress gets 409 with error code 704; after the first has answered, the same key gets that answer replayed.
Stripe, 2026-11-14: webhook retries after an error answer come about 1 h and 3 h later, with the same event id, in a different order.
Assumed, and listed as such
Adyen: whether a retry after a failed (5xx or timed-out) first call also gets 704.
Stripe: the exact set of statuses from which a PaymentIntent can be cancelled.
Try it in two minutes
The core (runner, menu, report and the pack contract) is open source under Apache-2.0. It comes with a demo PSP and a demo shop with one race, R1, of the same shape as C3: the shopper's result and the shop's own timeout. No PSP account needed.
Run R1 with the shop's fix on: every run passes. Switch the fix off and run it again: some runs fail, and the report shows how many hit the race window.
pip install raceproof
raceproof demo # terminal 1
raceproof menu --config raceproof-demo.yaml # terminal 2: pick R1
# the shop's fix off, then run R1 again
curl -X PUT -H 'Content-Type: application/json' \
-d '{"race": false}' http://127.0.0.1:9301/_demo/fixes