Research Brief Citation Audit Skill
Overview
This skill teaches the agent to audit a research brief against a provided source set, classify every claim by how well the sources actually support it, and return a cleaned brief alongside a log of every claim that was removed or rewritten.
Given a draft brief and a set of labelled sources, a good result is two clearly separated artifacts: a revised brief in which every retained claim is bound to a specific source pointer inline, and an unsupported-claims log with one entry per removed or rewritten claim recording the verbatim original, its tier classification, the specific reason it was flagged, and the action taken.
The submission is a single SKILL.md that teaches the agent the procedure for doing this work. Strong skills teach the agent to recognise what kind of claim it is looking at, apply a consistent classification procedure, and handle edge cases — compound sentences, hedged inferences, vendor-authored data, conflicting sources, and missing numbers — rather than providing templated responses or lists of memorised claim patterns.
A skill built to memorise specific scenarios will not hold up as challenges evolve.
You submit a SKILL.md file. A fixed base agent loads it and answers all 10 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.
We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.
Requirements (10 Tests)
Your skill is tested across 7 capability types, with 10 scenarios in total. Every submission is evaluated against all 10 scenarios — there is no sampling, so consistent quality across every capability type counts.
| Category | Overview | Test cases |
|---|---|---|
| Type A — Compound Claim Decomposition | Tests whether the agent correctly splits multi-part sentences into atomic assertions before applying tier classification. The skill must teach the agent to identify sentence boundaries where a single grammatical unit bundles more than one factual claim, and to treat each sub-claim independently rather than inheriting the support status of its neighbours. | 1 |
| Type B — Confidence Laundering Detection | Tests whether the agent catches cases where the language of the draft brief is stronger, broader, or more certain than what the cited source actually asserts. The skill must teach the agent to compare the precise wording of a claim against the source text and to rewrite Tier B claims into hedged form that faithfully reflects the source's own confidence level and scope. | 1 |
| Type C — Source Conflict Escalation | Tests whether the agent escalates contradictions or ambiguities across sources rather than silently resolving them. The skill must teach the agent to recognise when two or more sources make incompatible assertions about the same fact, and to surface these as explicit open questions in the revised brief's Open Questions section. | 1 |
| Type D — High-Stakes Quantitative Audit | Tests whether the agent applies heightened scrutiny to numerical claims in briefs that carry explicit decision stakes. The skill must teach the agent to require a named source, a specific date or period, and methodology or scope information before classifying any number as Tier A, and to flag or downgrade numbers that fail any of these checks. | 1 |
| Type E — Quantitative Claim Audit | Tests whether the agent applies the full source-grounding procedure to quantitative claims in standard research briefs. The skill must teach the agent to treat the absence of a named source, time period, or methodological scope as a disqualifying condition for Tier A classification of any statistic or numerical assertion. | 1 |
| Type F — Quantitative Scope Mismatch | Tests whether the agent catches cases where a number is real and sourced but applied to a population, geography, or time period different from what the source actually measured. The skill must teach the agent to check that the scope of a quantitative claim in the brief matches the scope described in the source, and to flag or downgrade any claim where there is a mismatch. | 1 |
| Type G — Scope Drift Detection | Tests whether the agent correctly classifies claims that are plausible or even accurate as general world-knowledge but are not present in the provided source set. The skill must teach the agent to treat the source set as the evidentiary boundary and to log such claims as Tier C scope-drift rather than accepting them as implicitly supported. | 1 |
Surface details — including domain, organisation names, specific statistics, source identifiers, and brief topic — rotate across scenario variants, while the underlying claim-audit capabilities being tested remain the same.
Datasets
Available Datasets
When you submit, we run your SKILL.md through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your skill, and follows it to answer each scenario.
Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.
Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.
Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill. You submit only a SKILL.md; you cannot supply your own agent, add tools, or alter the base agent's configuration.
Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.
Evaluation Criteria
Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.
| Criteria | Weight | |
|---|---|---|
Response quality An LLM judge scores each response against a task-specific quality bar for this challenge — how correct, complete, and usable the output is on its own terms, independent of which procedure produced it. | 30 % | |
Skill use An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill. | 30 % | |
Scenario quality An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar. | 30 % | |
Output originality A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here. | 10 % |
Scenarios are scored and then averaged up to capability-type scores, which roll up to the final challenge score shown on the leaderboard. Deterministic penalties apply when agent outputs damage brief quality — for example, by retaining unsupported claims, incorrectly stripping supported ones, silently resolving conflicts that should be escalated, or producing an incomplete or malformed unsupported-claims log. The skill should equip the agent to avoid these failure modes across the full range of scenario types.
Submission
How to submit
Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.
Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.
Draft your SKILL.md. Focus on teaching procedure rather than providing templates.
Validate locally using the published dataset variants to sanity-check output quality.
Submit through the sundae_bar challenge portal by uploading your SKILL.md file.
Review your scored breakdown and resubmit.
Constraints
Submission format: a single SKILL.md file with YAML frontmatter (name, description) and Markdown body.
Base agent: fixed. Provided by sundae_bar.
Runtime: a fixed base-agent harness loads your SKILL.md and answers all 10 scenarios per submission with it.
Tests per submission: 10 (the full scenario set — no sampling).
Submission limits: 3 submissions per day. A rolling challenge limit also applies.
Rewards
Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.
Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.
Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.
Your SKILL.md is scanned at submission time by a skill integrity gate. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool.
A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.