Useful Beats Optimized: How SN121 Scores a Skill

Most evaluations ask one question: did it work. SN121 asks a harder one: did it work because the skill is good, or because the skill was built to beat the test. Getting that distinction right is the entire reason emissions can flow to the right place.

When you submit a skill to a challenge, it does not get a single grade from a single judge. It runs through a stack, and each layer exists because a previous version of the system got gamed without it.

First: the gates

Before any quality judgment happens, deterministic checks run. A skill-integrity check catches hardcoded answers. Additional gates reject submissions built to sidestep the task entirely. Fail a gate and you never reach the judge. This keeps the obvious exploits out of the scoring entirely and means the quality graders are only ever looking at submissions that at least tried.

Then: two graders, weighted evenly

The split between them matters more than anything else in the stack.

The first is a quality judge. It reads the skill's output against a rubric and scores it on the dimensions that matter for that scenario. But the rubric is not a fixed checklist. Each scenario carries its own weight distribution, so a skill tuned to one weighting falls short on the next. The rubric also penalizes template-like answers on purpose: a response that mechanically hits every criterion without showing real understanding is instructed to score lower, not higher.

The second grader is the one that closes the gap. It checks whether the output was actually shaped by the procedures in your submitted skill, rather than by the rubric. This is the difference between a skill that teaches an agent to do the work and a skill that just encodes what the grader wants to see.

When we added this second grader, skills that had been gaming their way to the top dropped. Genuine skills separated out by roughly fifteen points. Same submissions, honest ranking.

The score is visible

Every piece of that score is open. Each submission produces a downloadable results file with the judge's rationale for every test case. The dataset, the rubric, and the suite are public on each challenge page. A score is not a verdict handed down from a black box. It is a weighted, documented result you can open and read — which is exactly what has to be true when that score is what earns on-chain emissions.

What this means for how you build

Under this stack, building for the brief and building for the rubric are finally the same instruction. The skill that genuinely solves the brief is the skill that scores. That was not always true. The last ten challenges were the work of closing that gap, and it is what makes the reward mean something.

So here is what to bring into a challenge: build the skill that holds up when the scenario changes. Not the flashiest, not the one that studied the rubric hardest. The one that works.