Benchmarks and evaluation6 min read

LLM evaluation: how to build a test you can trust, and what Artificial Analysis changed in its v5 index

LLM evaluation decides which model you use. What Artificial Analysis changed in Intelligence Index v5, and seven design rules for your own marketing eval.

By overads Inc., the team building gloofy

LLM evaluation is the process of measuring how well a language model does a defined job, on fixed questions, under fixed settings, so that two models or two versions can be compared fairly. Public indexes are a starting point, but they measure general ability, not your task. The useful version for a marketing team is a small private test built from its own work. Artificial Analysis has just shown, in rebuilding its own index, which design choices matter most.

What Artificial Analysis changed in Intelligence Index v5

Artificial Analysis runs one of the most cited public model rankings. It announced version 5 of its Intelligence Index at the end of October, and its methodology page now lists v5.0.1, a November update that regraded every model after the authors of one component test fixed its grading.

The current index has four equally weighted areas, each 25%: knowledge work, coding, science and general knowledge. Each area combines two or three evaluations. Three design choices stand out:

  • Private questions. Adding up the components marked private on the methodology page gives 60% of the index, a share Gizmodo also reports. Questions that are not published cannot leak into a model's training data.
  • Repeats. Several components run three or five times per model and average the results, because one run of a model that samples its answers is a noisy reading.
  • Stated settings. Temperature 0 for non-reasoning models and 0.6 for reasoning models, unless the lab recommends otherwise, with output limits and retry rules published.

The rankings moved with the test. According to Gizmodo, Moonshot's Kimi K3.1 Pro fell from fourth on v4 to ninth on v5, and Google's Gemini 4 Carbon took first place with 59. Neither model became better or worse on the day; the questions changed. That is the first thing to remember about any LLM evaluation: a ranking belongs to a version of a test.

Why a public index is not your LLM evaluation

A general index answers "which model is most capable overall". A marketing team usually needs to know "which model tags our creative, writes our captions or scores our leads best, at a price we can pay". Those can have different answers.

Our own benchmark shows the gap. On exam v3.3, which scores 150 real published ads by exact label match, our 4B model scores 0.796, ahead of 70B and 120B general models on that task and below Mistral Large 3. No general index would predict that order, and our exam says nothing about general ability. The full board, including where we lose, is on our public marketing benchmark.

Seven rules for building your own LLM evaluation

These come from Artificial Analysis' published choices and from our own mistakes, each recorded in our build log.

  1. Use real inputs. Take 50 to 150 examples from your own work: real ads, briefs, leads. Invented examples test what models already write.
  2. Keep some of it private. Never paste the test into a prompt library, a fine-tuning set or a vendor's demo. A test a model has seen measures memory. Our post on benchmark contamination covers the other ways a score misleads.
  3. Give every model the same instructions. Our older 4B model scored 0.745 with the prompt it was trained on and 0.675 on the shared specification every rival got. That seven-point gap was prompt familiarity, not skill.
  4. Fix and publish the settings. Temperature, output limit, retries and the exact model version. If a model cannot run at temperature 0, say so; one entry on our earlier exam could not, and we marked it.
  5. Run it more than once. Three training runs of one of our models, differing only by random seed, spread 0.055 on the whole exam and up to 0.34 on ten-item slices. A difference smaller than the spread is not a difference.
  6. Version the answer key. Keys have errors. When ours was corrected, re-scoring Mistral Large 3 lifted it by 0.052. We ship both key versions with the rules that produced each, so an old score can always be traced.
  7. Score the file you will run. If you will use a quantised model, test that file. Quantising our current 4B cost 0.004; an older version lost 0.045.

How to judge open-ended outputs

Exact match works for labels. For copy, captions and hooks there is no single right answer, so most evaluations, including several components of Artificial Analysis' index, use language models as judges, often in panels of three.

Judges bring their own biases, chiefly preferring the first answer they see and preferring text from their own model family. We show every pair in both orders and count a win only when the judge agrees with itself. Our only significant blind result, on video hooks, held in both orders on all 49 items. The full protocol is in our guide to using an LLM as a judge.

What you are testingScoring methodMain risk
Labels, tags, categoriesExact match against a keyErrors in the key
Numbers and arithmeticRecompute with codeNone, if the code is right
Copy, captions, hooksPairwise blind judging, both ordersJudge bias, especially self-preference
Rule compliance (lengths, margins)Check with codeRules that change on the platform

Where gloofy fits, and where it does not

marketing-bench, the evaluation behind our numbers, is public at gloofy_bench, so anyone can run it, check our scores or use its structure for their own test. It is narrow: it measures marketing tagging and a few writing tasks, not general ability, and Gemini, GPT-5.6 and current Claude models were not measured on exam v3.3. There is also a caveat on our top rows: our two best models were trained on Claude Opus labels, and Claude Opus re-adjudicated the current key, so student and key share a teacher.

We do not offer evaluation as a service. The models themselves are listed under the gloofy models, and more on measuring models collects in the benchmarks and evaluation hub.

What to do this week

  • Pick the one task you use a model for most and collect 50 real examples with agreed answers.
  • Run your current model and one cheaper alternative on them, same instructions, same settings, three times each.
  • Have a second person check every item where the models and the key disagree.
  • Write down the test version, settings and date next to every score you share.

FAQ

What is LLM evaluation?

LLM evaluation is measuring a language model's performance on a defined task with fixed questions, settings and scoring, so models or versions can be compared fairly. It ranges from public indexes of general ability, such as Artificial Analysis' Intelligence Index, to small private tests built from a team's own work.

Why did model rankings change in Artificial Analysis' v5 index?

Because the test changed. Version 5 replaced several component evaluations, and about 60% of its weight now comes from questions marked private. According to Gizmodo, Kimi K3.1 Pro fell from fourth to ninth and Gemini 4 Carbon took first. A ranking describes one version of a test, not a permanent order.

How many examples does an LLM evaluation need?

For a team comparing models on one task, 50 to 150 real examples is a practical start. Smaller sets are noisy: our own runs differing only by random seed spread up to 0.34 on ten-item slices. Run each model several times and treat differences smaller than that spread as ties.

Should I trust public LLM benchmarks?

Use them to shortlist, not to decide. Public indexes measure general ability with their own task mix, which may not match your work. Check whether questions are private, whether settings are published and which version produced the score, then test the shortlist on your own examples before choosing.

Sources

  1. Artificial Analysis, "Intelligence Benchmarking Methodology"
  2. Gizmodo, "The Biggest Benchmark in AI Just Updated, Dealing a Major Blow to a Chinese Contender", 29 Oct 2026
  3. overads Inc., "marketing-bench leaderboard", gloofy_bench