Practice Is the Only Standard for Testing AI

"A Beginner's Guide to Building Your Own Benchmark"

A few weeks ago I met up with a friend who works at a model company. He told me he maintains a personal benchmark — one with extremely niche questions that gives him a precise feel for how newly released models actually perform.

Obviously, in an era where Chinese models have gamed the public leaderboards into irrelevance, that kind of benchmark is worth its weight in gold.

So I asked if he'd be open to open-sourcing it. He said, "Any public benchmark becomes scrap paper for models to game. This thing is only useful for your own private use."

Though he wouldn't share his personal benchmark, he did want to pass on his experience and methods — to teach more people how to build their own using existing tools.

So he wrote this guide.

Here's what he wrote:

I built my own benchmark for three main reasons:

1. To troll with confidence

My personal benchmark gave me serious backbone when I called out GLM's weaknesses early this year, and when I criticized DeepSeek's coding capabilities.

Netizens are terrifying — saying anything negative about GLM or DeepSeek is treated like a capital offense.

2. Curiosity about when models will replace most of my work

3. Building my personal knowledge base

01

Good Benchmarks Grow Out of Real Scenarios

Anyone who works with large models has probably experienced this:

You tweak a prompt slightly and it feels like the output improved, but you're not sure if it's just placebo;

You switch to some new model billed as "Ultra," "SOTA," or "the pride of domestic AI," only to find it collapses on routine tasks;

You run a few classic test cases, everything looks fine, then you deploy or batch-process and bizarre errors start popping up everywhere.

These problems have plagued NLP since long before LLMs. That's why benchmarks matter:

To properly validate whether a model works in a given scenario, you can't just cherry-pick a few examples and call it. Evaluation needs to simulate the actual conditions of that scenario as closely as possible.

The point of building a benchmark is twofold: to have a position, and to define that position. Raw scores alone are meaningless. Humans need attribution — to look at the distribution of data — to know whether to revise the prompt, switch models, or patch the workflow next.

And good benchmarks always grow out of real scenarios.

Back in the industrial NLP era, many benchmarks started with specific cases from clients, then were manually generalized.

Now with large models, many benchmarks are just copied from public datasets, or synthesized in batches of hundreds from a single case.

In my view, any benchmark detached from real-world scenarios is built on sand.

A personal benchmark should be a cumulative collection of problems you've actually encountered. At minimum, the first version must be entirely distilled from real scenarios — not fabricated from scratch.

If you really must batch-generate cases, use a smart model to test a dumb one.

02

How to Design Benchmarks and Cases

Some common-sense questions in my benchmark have survived all the way to the GPT-5 era. When GPT-4 had its "aha moment," plenty of traditional NLP hard cases could still stump it.

First, pay attention to data distribution.

My benchmark has stayed relevant because I gave it broad scenario coverage.

I double-majored in physics and computer science, with a minor in literature.

So my benchmark covers math, physics, computer science, literature, history, interview questions I've encountered, and business problems from my work.

My rough distribution: math, physics, and CS theory at 10% each; history 5%; life commonsense 5%; literature 20%; coding and work problems 40%.

This distribution is a map of my entire cognition and experience.

Though GLM solved many of my coding cases early this year, it still couldn't handle questions about Limits of Computation from my CS coursework.

Some obscure history questions remain unanswerable by domestic models.

For example, I darkly included details about the Qing entry into the pass (from late-Ming historical sources), Soviet strategic and armament decisions in WWII (from a college course called "Russia's Today"), even whether a KV-1 could penetrate a Centurion's frontal armor.

On literature: models overcorrect on run-on sentences and sentence fragments, and misread scrambled chronology. The biggest pitfalls are transitional word abuse and misidentified rhetorical devices.

First, models struggle to spot overused transitions in human writing because they do the same thing themselves.

Second, when you use hyperbole, the model will flag it as illogical and forcibly "correct" jokes or expressive intent into bland mediocrity.

When a case fails, the problem can be attributed to one of four types (from Bloom's Taxonomy):

• Factual knowledge: Isolated basic knowledge nodes, e.g. "Paris is the capital of France"

• Conceptual knowledge: Relationships and structures between components, e.g. causal chains of "because A, therefore B." Since LLMs emerged, I've further broken this down into semantic, logical, and temporal subtypes

• Procedural knowledge: Methods, steps, algorithms, and skills for how to do things

• Metacognitive knowledge: Awareness and monitoring of one's own cognition, e.g. "I'm not sure" or "I might be wrong"

The biggest advantage of this attribution method is that it lets non-technical colleagues pinpoint exactly where a model falls short, then optimize the model/agent accordingly. If you're not doing optimization work, you don't need to learn this.

03

How to Build Your Own Benchmark

With the above in mind, let's get hands-on.

For non-technical people, a spreadsheet is the best database. We can flatten complex nested JSON into a single table.

First, collect cases.

Don't force yourself to brainstorm cases. Record things as they come to you. Whatever's in your documents, have a model help you organize it.

Every time AI screws you over, bank one case. Save that conversation verbatim, drop it in a fixed place — like an "AI Fails" folder in your notes app.

Every few days, copy them into your question bank, and add one sentence describing how you wanted it to respond. That sentence becomes the foundation for your evaluation rules later.

Starting with twenty or thirty cases is plenty. No need to launch with hundreds. Distribute them according to what you actually use AI for — more cases in scenarios you use more.

Next, set up questions and evaluation rules in your bank.

Split your spreadsheet into two tabs (copyable to Excel / Notion / Lark Sheets): one for questions, one for scores. The question bank stays fixed; add a new score sheet every time you run a test.

As shown above, don't write Column D evaluation rules as "must contain X word."

My most common methods: the answer must mention something; must not include a certain approach; missing a specific point counts as failure.

Take CASE-002: if you write "must not contain 'guaranteed returns,'" a competent model will likely say something like "no stock can guarantee returns" as investment risk disclosure — and get flagged as failing.

Finally, the score sheet. Model under test, prompt version, and runtime environment are your variables. Change only one at a time.

For example, CASE-001 only changed the prompt from v1.0 to v1.1, model and environment held constant, and flipped from fail to pass — indicating the prompt change worked.

Your benchmark will inevitably get cracked by models faster and faster, especially on coding.

But one scenario being solved doesn't matter — there are always more domains to test model boundaries.

Getting beaten by SOTA models isn't embarrassing. These days my benchmark is more about seeing the gaps between different models.

04

The Dark Arts: Letting an Agent Run It Automatically

If you have an AI assistant on hand (like Workbuddy / Qwen Office / Doubao Work), the most unorthodox move is to feed your JSON structure directly as a task specification, letting the AI test itself against its own rules.

You don't even need to write Python to parse JSON.

Once your spreadsheet is ready, hand it to an Agent and have it convert each row into a case following that JSON format, saved to a file called my_cases.json.

Then send the Agent this instruction (switch to whichever model you want to test first):

Spin up a separate subtask (sub-agent or new session) for answering. Only give it the system_prompt and user_input from input — don't give expected or evaluation. Bring the raw answer back, and you do the grading.

Check against must_include and must_not_include item by item, noting hit or miss for each.

Score by score_weights with cited answer excerpts as evidence. Total against pass_score.

Regardless of pass/fail, append to execution_history: model, prompt_version, score, passed, failure_reason, tested_at. Don't touch old records. For failures, specify the category in failure_reason: Facts / Relations / Procedures / Meta.

Finally give me a table: pass rate, which error category clusters, what flipped from fail to pass vs. pass to fail since last run.

Five years ago, building a chatbot meant manually cranking out 2,000 rows in Excel. Now it takes one sentence.

Models are absurdly good at decomposing unstructured content into structured information. Automated pipeline operations no longer require technical intervention.

I can't open-source my benchmark, but following this guide to build your own isn't much work.

Next time you criticize some model and someone accuses you of blasphemy, just slam the results on the table for public execution.

(Cover image generated by ChatGPT; article entirely human-written)

Subscribe to our Substack: funeralai.substack.com