Prompt Regression & Quality Benchmark Suite (150 cases)

150-case regression suite across 5 test types, for catching when a model or prompt change breaks something that worked.

Price
$3.00 USD A payment-processing fee of about 1% is added at checkout.
Availability
Available to buy on Harashsuq. One-time payment, one download per version.
Category
Evaluation / Benchmark Set
License
Commercial
Size
70.0 KB
Seller
Matthew Sullivan

A 150-case regression suite (regression.jsonl) for detecting when a change to a model version, prompt template, system prompt, or pipeline component breaks something that used to work — distinct from a general capability benchmark. 30 cases each across 5 test types: before/after comparison (a specific characteristic that must survive a change), consistency (run-to-run stability of a narrowly-bounded answer), edge cases (unusual but legitimate inputs, graded on graceful handling), adversarial-but-safe cases (leading questions, false premises, and conflicting instructions — judgment tests, not safety or jailbreak content), and formatting stability (a literal output-format spec to validate against). The same 30 task domains (JSON extraction, SQL generation, customer support, code review, and more) are used across all 5 test types. The README documents a full regression-pass procedure — baseline vs. candidate runs, which cases are exact-match, structural-validation, or LLM-as-judge, and how to weight and triage a failure by test type. Intended to be run before and after any pipeline change, comparing candidate against baseline rather than against an abstract notion of 'good.'

More Evaluation / Benchmark Sets