Stripe
stripe.com ↗Payments, billing, and money movement APIs for internet businesses.
PaymentsBilling & Subscriptions
AXRank (autonomous)
58
Agent registration
12
gap +47pp
Δ last run
—
Evals run
1
Trend
—
Stripe presents a comprehensible surface once an agent has working credentials — API semantics are clear enough that agents can compose multi-step flows and generally get reliable results. The catch is getting there. Discoverability is rough in both modes: agents struggle to locate the right entry points from a cold start, and the self-registration path is essentially a wall, with agents unable to bootstrap credentials, parse what they're looking at, or recover from the resulting dead ends. Expect smooth sailing if you provision keys yourself, and expect to do that provisioning manually rather than letting the agent handle it.
By dimension (best model)
| discoverable | 1/5 tasks | 20 | |
| comprehensible | 11/14 tasks | 79 | |
| reliable | 8/12 tasks | 67 | |
| composable | 3/4 tasks | 75 | |
| recoverable | 1/2 tasks | 50 | |
| efficient | 3/5 tasks | 60 | |
| Aggregate(mean of 6 dimensions — the headline) | — | 58 | |
| Raw pass rate(secondary; tasks passed / tasks attempted) | — | 65 | |
By model
| Model | Autonomous | Agent reg. | Gap |
|---|---|---|---|
| Opus 4.7best | 58 | 12 | +46pp |
| GPT-5.5 | 54 | 65 | +-11pp |
Where the agent struggled
- Hard-to-find capabilities9×
- Inconsistent surface4×
- Unwritten conventions1×
- Ambiguous canonical route1×
What the agent tried
Starting from stripe.com, find the official developer documentation hub and fetch its plain-text/markdown LLM index file. fetch-markdown-docs | ✗ |
Authenticate against the Stripe API using the provided $STRIPE_SECRET_KEY and confirm you are operating against the Mycaminoguide account in test mode by retrieving the account object. authenticate-with-api-key | ✗ |
Create a Customer named 'Pilgrim Test User' with email pilgrim@example.com, then verify it exists by retrieving it by ID. manage-customers | ✓ |
Create a PaymentIntent for $42.50 USD with automatic payment methods enabled, in no more than 2 API calls total. create-payment-intent | ✓ |
Create a Product called 'Camino Guidebook' and a recurring monthly Price of $9.99 USD attached to it, then list prices filtered by that product to confirm linkage. manage-products-and-prices | ✓ |
Create a Checkout Session in payment mode for a one-off $25 charge with success and cancel URLs pointing at https://example.com/success and https://example.com/cancel, using inline price_data (no pre-created Price). create-checkout-session | ✓ |
Create a Customer and a $10.00 PaymentIntent for them, using an Idempotency-Key, then replay the EXACT same request and confirm Stripe returned the same PaymentIntent rather than creating a duplicate. create-payment-intent | ✓ |
Register a webhook endpoint at https://example.com/stripe/webhook subscribed only to the events 'payment_intent.succeeded' and 'checkout.session.completed', then verify by listing webhook endpoints. register-webhook-endpoints | ✓ |
Find Stripe's official Agent Skills catalog (a machine-readable JSON file under a well-known path on the docs site) and fetch it, then report how many skills it lists. load-agent-skills | ✗ |
Install the official Stripe MCP server in a local Claude Code or Cursor configuration. Produce the exact JSON config snippet you would add. invoke-mcp-tools | ✓ |
Find the GitHub repository for the official Stripe CLI and report the latest release version. forward-webhooks-to-localhost | ✗ |
Create a Refund for a PaymentIntent that does not exist (use the ID 'pi_does_not_exist_axrank') and report what Stripe tells you to do. issue-refunds | ✓ |
Create a Subscription: create a customer, attach a test PaymentMethod (use pm_card_visa), set it as the default, then subscribe them to a $9.99/month price you create. Confirm the subscription is active. manage-subscriptions | ✓ |
List all customers in this Stripe account using pagination with limit=3, fetching at least two pages, and report the total count returned. (Note: the account may have very few or zero customers.) manage-customers | ✓ |
Pin your API requests to API version 2026-04-22.dahlia and create a PaymentIntent for $1.00 USD. Confirm via the response headers or object that the pinned version was honored. query-api-changelog | ✓ |
Create a Connect (Connected) Account of type 'express' for a hypothetical seller in the US, and report the new acct_ id. create-connected-accounts | ✗ |
Retrieve the plain-text/markdown version of the Stripe API reference docs page for PaymentIntents (using the documented .md URL convention) in a single fetch. fetch-markdown-docs | ✓ |
Create a Customer, then a $20 PaymentIntent for them, confirm it using the test PaymentMethod pm_card_visa, then issue a partial $5 refund. Verify the final charge balance. issue-refunds | ✓ |
Look up what changed in Stripe API version 2026-04-22.dahlia versus the previous stable release, using Stripe's published changelog. query-api-changelog | ✗ |
Verify a webhook signature: given the payload string '{"id":"evt_test","type":"ping"}', the secret 'whsec_axranktest', and a Stripe-Signature header you construct yourself, write code that uses the official Stripe SDK helper to verify it (and prove it verifies). Use the Python or Node SDK. verify-webhook-signatures | ✗ |
Recent runs
| When | Score | Tokens | |
|---|---|---|---|
| 2026-05-27 19:42 UTC | 58 | 4.00M | findings → |
Want full traces, per-task remediation, failure-mode IDs, and head-to-head comparisons? The findings report is the paid tier — (contact link coming).