Useful Beats Optimized: How SN121 Scores a Skill

Most evaluations ask one question: did it work. SN121 asks a harder one: did it work because the skill is good, or because the skill was built to beat the test. Getting that distinction right is the entire reason emissions can flow to the right place.

When you submit a skill to a challenge, it does not get a single grade from a single judge. It runs through a stack, and each layer exists because a previous version of the system got gamed without it.

First: the gates

Before any quality judgment happens, weight-0 pre-gates run (skill_alignment and dataset_derived). They add nothing to the quality score. Their job is to detect a skill built to win the Challenge or exploit the scoring system rather than teach a genuine capability, and to stop the run early. Fail either gate and the entire submission is zeroed; you never reach the paid quality graders. We do not publish how those gates work.

Then: the quality graders

The live stack is published per Challenge in suite.yaml. A typical mix (weights always sum to 100%) is:

• skill_use, 30%. Did the agent follow the skill’s procedure, or freelance and pretend.
• scenario_quality, 30%. How good is the output against that scenario’s quality bar, independent of whether the skill was followed.
• rubric (model-judge), 30%. Holistic judgment across the scenario’s dimensions, with an anti-template instruction.
• novelty_check, 10%. Deterministic. Near-identical boilerplate across scenarios drops toward 0.

Confirm the weights on the Challenge you are scoring. They are in suite.yaml and on the results JSON.

The split that still matters most is skill_use versus the quality judges. A skill that only encodes what the rubric wants will look strong on quality and weak on skill_use. A skill that actually teaches the procedure holds up across scenarios. Harness and judge models are chosen per Challenge to match a realistic stack for that use case; they are not a subnet-wide default.

The score is visible

The quality score is open. Each submission produces a downloadable results file with the judge's rationale for every test case. The dataset, the rubric, and suite.yaml are public on each challenge page. To audit a row: take each grader’s published weight, multiply by that grader’s score, sum them, then average across scenarios. A score is not a verdict from a black box. It is a weighted, documented result you can recompute, which is what has to be true when that score earns on-chain emissions. Pre-gate mechanics stay unpublished; they do not change the quality math on a passing submission.

What this means for how you build

Under this stack, building for the brief and building for the rubric are finally the same instruction. The skill that genuinely solves the brief is the skill that scores. That was not always true. The last ten challenges were the work of closing that gap, and it is what makes the reward mean something.

So here is what to bring into a challenge: build the skill that holds up when the scenario changes. Not the flashiest, not the one that studied the rubric hardest. The one that works.