Regulatory Obligation Delta Analysis Skill
Overview
This skill teaches the agent to compare two versions of a regulatory instrument and produce a structured, clause-by-clause table of what has changed, what has been added, and what has been removed, so a legal or compliance team knows exactly which obligations require action.
Given a prior version and an amended version of a regulatory text, plus any regulator-issued change summary or explanatory memorandum, a good result is a delta table covering every clause in both texts, each row labelled with a change type and an action trigger, followed by a summary block that counts changes by type and lists every clause flagged for human review alongside a specific question.
The submission is a SKILL.md file, with optional reference files the agent opens by name when SKILL.md directs it to. Strong skills teach the agent a reasoning procedure: how to align clause versions, how to apply a change-type taxonomy, and how to decide when a regulator summary controls versus when the clause text overrides it. Weak skills paste in template tables or pre-filled examples and break the moment clause numbering, thresholds, or entity names differ from what was memorised.
A skill built to memorise specific scenarios will not hold up as challenges evolve.
You submit a skill: a SKILL.md and, if you need them, reference files it points at. A fixed base agent loads it and answers all 10 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.
We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.
Requirements (10 Tests)
Your skill is tested across 6 capability types, with 10 scenarios in total. Every submission is evaluated against all 10 scenarios. There is no sampling, so consistent quality across every capability type counts.
| Category | Overview | Test cases |
|---|---|---|
| Type A: Clarification recognition | This type tests whether the agent correctly identifies changes that elaborate, enumerate, or define without adding new substantive obligations, and refuses to upgrade them to net-new or scope-expansion labels. The skill must teach the agent to recognise the specific signals that mark a change as clarification, including enumeration of a general standard that already applied, definition of a term already operative in the prior text, a regulator summary that is confirmed by the clause texts themselves, and cross-references to instruments that already required the action. | 1, 2 |
| Type B: Scope expansion detection | This type tests whether the agent correctly identifies genuine expansions of existing obligations and distinguishes them from clarifications. The skill must teach the agent to spot the concrete textual signals that mark a real expansion: a new noun appearing in the obligation's object, a new verb of action, a changed threshold or deadline, a new class of entity brought within scope, or the removal of an exemption that previously limited the obligation. | 1, 2 |
| Type C: Multi-change delta construction | This type tests whether the agent can handle a regulatory instrument with several different change types present simultaneously, assigning the correct label and action trigger to each clause without cross-contaminating labels or omitting rows. The skill must teach the agent to apply both expansion detection and action-trigger assignment across a full instrument, distinguishing substantive operational changes that require a Yes trigger from documentation-only updates that warrant a Yes-doc trigger. | 1, 2 |
| Type D: Regulator summary override | This type tests whether the agent applies independent judgment when a regulator-issued summary characterises a change differently from what the clause texts actually say. The skill must teach the agent to treat the summary as evidence of intent, compare it against the clause texts, and override the summary's characterisation when the texts resolve the question in a different direction, explaining in the notes column which clause element drives the override. | 1, 2 |
| Type E: Renumbering with insertion | This type tests whether the agent correctly handles instruments where new clauses have been inserted and existing clauses renumbered, without generating phantom deletions or spurious net-new additions for provisions that simply moved. The skill must teach the agent to complete a clause-alignment pass before any classification, and to render renumbered clauses in the format 'new-ref (was: prior-ref)' so readers can trace the source location. | 1, 2 |
| Type F: Systematic scope expansion | This type tests whether the agent can identify and correctly classify scope expansions that run across multiple clauses of an instrument in a consistent pattern, for example a new entity class introduced in a definition and then carried through several obligation clauses. The skill must teach the agent to track expansion signals across the full clause map and classify each affected clause individually, rather than absorbing the pattern into a single summary note. | 1, 2 |
Surface details such as jurisdiction names, regulator acronyms, clause reference formats, threshold values, and entity class labels rotate between variants while the underlying obligation-delta capability being tested remains the same.
Datasets
Available Datasets
When you submit, we run your skill through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your SKILL.md and the list of your reference files, opens a reference with read_reference when your SKILL.md tells it to, and follows the skill to answer each scenario.
Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.
Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.
What you submit: a folder or a .zip with SKILL.md at the top level and an optional references/ folder. A single SKILL.md is also accepted. All files are the submission.
Limits: SKILL.md is at most 10,000 bytes and is always loaded. references/ holds up to 30 files, each at most 32,000 characters, 128,000 characters in total. Text formats only: .md, .txt, .json, .yaml, .yml and .csv. Lower-case file names using letters, digits, dots, dashes and underscores. The current values are published by the platform API at GET /api/v2/submissions/limits; the submission form reads them from there.
How references are read: the agent always sees SKILL.md plus a list of your reference file names. It opens a reference only when SKILL.md tells it to, by name. Each read is one harness step within a cap of 10 steps per scenario, and the file's text counts against the scenario's token budget of 32,000 tokens. A reference the agent never opens does nothing for the score.
Rejected at upload: HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types, and anything over a limit. Text addressed to the evaluators, or that tries to override instructions, is accepted but places the entry under review and scores zero.
Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill, read_reference. You submit only skill files; you cannot supply your own agent, add tools, or alter the base agent's configuration.
Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.
Evaluation Criteria
Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.
| Criteria | Weight | |
|---|---|---|
Response quality An LLM judge scores each response against a task-specific quality bar for this challenge: how correct, complete, and usable the output is on its own terms, independent of which procedure produced it. | 30 % | |
Skill use An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill. | 30 % | |
Scenario quality An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar. | 30 % | |
Output originality A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here. | 10 % |
Scenarios are scored and those scores are averaged within each capability type, then combined into the final challenge score shown on the leaderboard. Deterministic penalties apply when the agent's output contains moves that damage quality in ways a reviewer could identify without ambiguity, for example misclassifying a clarification as a net-new obligation, omitting a clause from the delta table, or generating an action trigger that is inconsistent with the assigned change type. The penalty logic is designed to reflect the practical cost of those errors in a real compliance workflow, not to reward partial credit for structurally broken outputs.
Submission
How to submit
Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.
Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.
Draft your SKILL.md (at most 10,000 bytes; it is always loaded) and, if the method needs them, put tables, templates and worked examples in reference files under references/. Tell the agent in SKILL.md which reference to open and when, by file name. Focus on teaching procedure rather than providing templates.
Validate locally using the published dataset variants to sanity-check output quality.
Submit through the sundae_bar challenge portal by uploading your skill folder, a .zip of it, or a single SKILL.md. The folder has SKILL.md at the top level and an optional references/ folder beside it.
Review your scored breakdown and resubmit.
Constraints
Submission format: a folder or .zip with SKILL.md (YAML frontmatter with name and description, then a Markdown body) at the top level and an optional references/ folder. A single SKILL.md is also accepted.
Limits: SKILL.md at most 10,000 bytes. Up to 30 reference files, each at most 32,000 characters and 128,000 characters in total; text formats only (.md, .txt, .json, .yaml, .yml and .csv); lower-case file names.
Reference reads: the agent opens a reference only when SKILL.md names it. Each read is one of 10 steps per scenario and its text counts against the scenario's 32,000-token budget.
Base agent: fixed. Provided by sundae_bar.
Runtime: a fixed base-agent harness loads your skill and answers all 10 scenarios per submission with it.
Tests per submission: 10 (the full scenario set, no sampling).
Submission limits: 3 submissions per day. A rolling challenge limit also applies.
Rewards
Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.
Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.
Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.
Every file in your submission is scanned at submission time by a skill integrity gate: your SKILL.md and every reference file, whether or not your skill points at it. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool. Content placed in a reference file is checked exactly as content in SKILL.md.
A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.
Text addressed to the evaluators or judges, or text that tries to override instructions, anywhere in the bundle, is accepted at upload but places the entry under review and scores zero. Files that are not plain text (HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types) or that exceed a limit are rejected at upload. The copy and similarity checks run over the whole bundle, chunk by chunk, so padding a copied skill with extra reference files does not pass them.