Build a fair API test
Choose an API on the work it completes. Start with eight prompts, explicit checks and a blank results sheet.
This kit contains original synthetic prompts for common application tasks. It is provider-neutral: there are no model rankings, generated scores or claimed test results. Downloading it makes no API requests. Running it against a paid endpoint is a separate action using your own account and budget.
Fix the test conditions
Record the exact endpoint, model ID and version if available, date, generation parameters, output limit, client version and concurrency. Keep these settings consistent between providers. Use the same prompts, then add private held-out tasks from your application so the public examples do not become the entire test.
The downloadable set covers JSON extraction, grounded summarisation, missing information, executable code, strict formatting, Unicode, multi-turn context and error classification. Each case includes acceptance checks. A small sample can expose integration problems; it cannot establish a universal refusal rate or prove which model is best.
Label the outcome first
| Label | When to use it |
|---|---|
| accepted | The completed answer meets the case’s stated checks. |
| refusal | The model explicitly declines the task rather than answering it. |
| incorrect | The model attempts the task but fails the acceptance checks. |
| transport_error | The request fails at the HTTP or service layer. |
| empty | The request completes without an assessable answer. |
Correctly admitting that the supplied text lacks an answer is not a refusal. Review ambiguous cases manually and keep the rule you used. The CSV has no pre-filled results.
Measure the right things
Time to first content token: elapsed time from sending the request to receiving the first usable content. Declare whether you include queuing and connection setup. Generation throughput: generated model tokens divided by generation time; do not substitute network fragments for tokenizer counts. If the API does not provide enough information, leave the metric unreported.
Cost per accepted answer: total measured spend, including failed attempts and retries, divided by accepted answers. If no answer is accepted, the ratio is undefined. Refusal share: declare a denominator. One useful definition is refusals divided by accepted + incorrect + refusal responses, with transport errors and empty responses reported separately. Always include the counts and sample size.
Make the result reproducible
Preserve the input cases, complete outputs, response IDs, usage data, timing method and labels. Remove API keys and personal data before sharing logs. Report repeated runs and uncertainty where appropriate. Zero observed refusals on a finite set never proves zero refusals for every possible request.
For implementation details, read the guides to measuring refusals, streaming events and bounded retries. Use the API cost calculator for an arithmetic estimate before running a workload.