Postman
postman.com ↗API development platform for building, testing, mocking, and running HTTP collections.
Postman is reasonably discoverable and recoverable when an agent shows up with credentials in hand — entry points are findable and error paths give agents enough signal to retry intelligently. Beyond that, expect friction: workflows that chain calls together are where things fall apart, and the surface is only partly comprehensible, so agents spend cycles guessing at shapes and semantics. Reliability is middling under pre-provisioned creds and collapses entirely in self-registration, where almost nothing past the front door works. If you can hand your agent a working token up front, you'll have a much better time.
By dimension (best model)
| discoverable | 5/6 tasks | 83 | |
| comprehensible | 5/8 tasks | 63 | |
| reliable | 4/10 tasks | 40 | |
| composable | 2/7 tasks | 29 | |
| recoverable | 3/3 tasks | 100 | |
| efficient | 4/6 tasks | 67 | |
| Aggregate(mean of 6 dimensions — the headline) | — | 64 | |
| Raw pass rate(secondary; tasks passed / tasks attempted) | — | 55 | |
By model
| Model | Autonomous | Agent reg. | Gap |
|---|---|---|---|
| Opus 4.7best | 64 | 27 | +37pp |
| GPT-5.5 | 53 | 27 | +26pp |
Where the agent struggled
- Inconsistent surface5×
- Vague documentation4×
- Docs don't match behavior2×
- Ambiguous canonical route2×
- Missing examples2×
What the agent tried
Authenticate to the Postman API and confirm you're acting as user 'quintonwall' (user id 13455252). authenticate-with-api-key | ✓ |
List all workspaces on my account and tell me how many there are. manage-workspaces | ✓ |
Find the official Postman MCP server endpoint URL and the three tool configuration profiles it offers, starting from postman.com. Don't guess URLs. invoke-mcp-tools | ✗ |
Use the unauthenticated docs MCP server to search the Postman docs for how rate limits apply to MCP tool calls. search-docs-via-mcp | ✓ |
Create a new workspace called 'axrank-probe' of type 'personal'. manage-workspaces | ✗ |
Delete one of the two duplicate 'AXRank Test Collection' entries from my account, keeping exactly one. manage-collections | ✓ |
Create a mock server for the 'Hello API' collection in my account. manage-mock-servers | ✗ |
Fetch the 'Polygon.io Dev' environment and show me its variable keys (not values). manage-environments | ✓ |
Run my 'Polygon API - Hourly (Dev)' monitor now and report the run status. manage-monitors | ✓ |
Convert this OpenAPI 3 spec to a Postman Collection v2 using the official Postman OSS tooling: {"openapi":"3.0.0","info":{"title":"Tiny","version":"1.0.0"},"paths":{"/ping":{"get":{"responses":{"200":{"description":"ok"}}}}}} convert-openapi-to-collection | ✓ |
Use Newman to run my 'Hello API' collection and capture the exit code. run-collections-via-cli | ✗ |
Query the audit logs for my team for the last 7 days. query-audit-logs | ✓ |
Fork the 'Polygon API' collection into my 'sandbox-2026' workspace with the fork label 'axrank-fork-test'. manage-forks-and-pull-requests | ✗ |
How many APIs/specs are in my 'sandbox-2026' workspace? manage-api-specs | ✗ |
List my team members. (Note: try the /users endpoint first.) manage-users-groups-and-roles | ✓ |
Get the raw markdown version of the Postman API authentication docs page using the documented AI-agent convention. search-docs-via-mcp | ✓ |
Using the postman-collection JS SDK, load my 'Hello API' collection and add a new GET request named 'health' at path '/health' to its root. manipulate-collections-via-sdk | ✗ |
Create a new collection in my Team Workspace by sending just a name string in the POST body. manage-collections | ✗ |
Delete the mock server tied to the 'Hello API' collection. manage-mock-servers | ✗ |
I got a 429 from the Postman API. What does the rate-limit response actually tell me, and does the same quota apply to MCP tool calls? authenticate-with-api-key | ✓ |
Recent runs
| When | Score | Tokens | |
|---|---|---|---|
| 2026-05-24 22:45 UTC | 64 | 7.24M | findings → |
| 2026-05-23 22:53 UTC | 52 | 4.16M | findings → |
| 2026-05-23 21:29 UTC | 40 | 760.5k | findings → |
| 2026-05-23 21:06 UTC | 100 | 590.0k | findings → |
Want full traces, per-task remediation, failure-mode IDs, and head-to-head comparisons? The findings report is the paid tier — (contact link coming).