Sundae Bar Logo
Past Challenge
Regulatory Compliance Triage Skill
SBC33

Regulatory Compliance Triage Skill

Overview

This skill teaches the agent to read a regulatory document and an organisation profile together, then deliver a structured verdict on whether the regulation applies to that organisation.

Given a regulatory artefact and an organisation profile, a good result is a per-criterion verdict table covering every scoping, reach, threshold, and exemption clause, followed by exactly one top-line verdict label, a concise rationale citing the specific clause and profile value that decided it, and a single routing pointer when the organisation is in scope or conditionally in scope.

The submission is a SKILL.md, with optional reference files the agent opens on demand. Strong skills teach the agent a reasoning procedure it can apply to any regulatory instrument and any organisation profile: how to sequence the checks, how to handle missing information, how to classify what it finds, and when to stop rather than speculate. Weak skills paste template verdicts or memorised clause patterns that break the moment the instrument, the profile details, or the jurisdiction changes.

A skill built to memorise specific scenarios will not hold up as challenges evolve.

You submit a skill: a SKILL.md and, if you need them, reference files it points at. A fixed base agent loads it and answers all 10 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.

We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.

Requirements (10 Tests)

Your skill is tested across 9 capability types, with 10 scenarios in total. Every submission is evaluated against all 10 scenarios. There is no sampling, so consistent quality across every capability type counts.

CategoryOverviewTest cases
Type A: Compliance gap versus applicabilityThis type tests whether the agent correctly treats an organisation's failure to meet an obligation as an IN-SCOPE finding rather than elevating to CONDITIONAL or NEEDS-LEGAL-REVIEW. The skill must teach the agent to keep applicability and compliance status strictly separate: the regulation either applies or it does not, and the organisation's non-compliance with its requirements is evidence of scope, not a reason for a softer verdict.1
Type B: Instrument-level versus provision-delimiting thresholdsThis type tests whether the agent correctly classifies each threshold it encounters as one that determines whether the instrument applies at all, or one that determines which provisions within the instrument apply. The skill must teach the agent to identify the measurement basis and window stated in the clause and to test the profile against those exact terms, producing the right verdict and confidence rating for each.1
Type C: Exemption with cap or nullification clauseThis type tests whether the agent evaluates an exemption clause together with any paired cap, nullification, or exclusion clause rather than treating the exemption in isolation. The skill must teach the agent to issue CONDITIONAL when an exemption currently holds but the profile shows a threshold is being approached that would switch it off, and to state the trigger precisely in the rationale.1
Type D: Exemption carve-out as dispositive elementThis type tests whether the agent identifies when a carve-out or exclusion within an exemption is itself the deciding element, rather than the exemption's main condition. The skill must teach the agent to evaluate carve-outs with the same rigour as primary scoping clauses, issuing CONDITIONAL when the carve-out is the hinge on which the exemption stands or falls.1
Type E: Group aggregation nullifying an exemptionThis type tests whether the agent correctly applies group-level aggregation rules when the instrument specifies that figures must be summed across the corporate group. The skill must teach the agent to show the arithmetic in the relevant verdict row and to reflect how the aggregated figure affects the exemption or threshold outcome, rather than reading the organisation's standalone figures in isolation.1
Type F: Indicator subordinate to a failed predicateThis type tests whether the agent applies the precedence rule correctly: a sub-criterion or indicator list is only evaluated if the predicate it qualifies has already passed. The skill must teach the agent to stop at a failed predicate and mark downstream indicators as Not-evaluated, preventing over-elevation of the verdict caused by treating subordinate factors as independent evidence.1
Type G: Peripheral ambiguity that does not elevate the verdictThis type tests whether the agent reserves NEEDS-LEGAL-REVIEW for ambiguity in a term that is genuinely decisive for today's obligation, rather than applying it whenever any ambiguity appears. The skill must teach the agent to check whether the ambiguous or unadopted term sits in a clause the profile already fails, and if so, to proceed to the appropriate verdict without elevation.1
Type H: Reach-clause predicate failureThis type tests whether the agent correctly short-circuits to OUT-OF-SCOPE only after confirming that no reach clause, effects-based clause, targeting clause, or group-reach clause could pull the organisation back into scope. The skill must teach the agent to check every reach mechanism before issuing OUT-OF-SCOPE, and to mark the predicate failure clearly in the verdict table.1
Type I: Scope delimitation distinguished from exemptionThis type tests whether the agent correctly identifies a threshold as one that narrows which provisions apply, rather than one that excludes the organisation from the instrument entirely. The skill must teach the agent to keep the organisation IN-SCOPE for surviving provisions when a delimiting threshold is not met, rather than issuing OUT-OF-SCOPE as if a full exemption applied.1

Surface details such as jurisdiction names, entity names, numeric thresholds, sector codes, and revenue figures rotate across scenario variants, while the underlying capability being tested remains the same.

Datasets

Available Datasets

When you submit, we run your skill through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your SKILL.md and the list of your reference files, opens a reference with read_reference when your SKILL.md tells it to, and follows the skill to answer each scenario.

Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.

Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.

What you submit: a folder or a .zip with SKILL.md at the top level and an optional references/ folder. A single SKILL.md is also accepted. All files are the submission.

Limits: SKILL.md is at most 10,000 bytes and is always loaded. references/ holds up to 30 files, each at most 32,000 characters, 128,000 characters in total. Text formats only: .md, .txt, .json, .yaml, .yml and .csv. Lower-case file names using letters, digits, dots, dashes and underscores. The current values are published by the platform API at GET /api/v2/submissions/limits; the submission form reads them from there.

How references are read: the agent always sees SKILL.md plus a list of your reference file names. It opens a reference only when SKILL.md tells it to, by name. Each read is one harness step within a cap of 10 steps per scenario, and the file's text counts against the scenario's token budget of 29,000 tokens. A reference the agent never opens does nothing for the score.

Rejected at upload: HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types, and anything over a limit. Text addressed to the evaluators, or that tries to override instructions, is accepted but places the entry under review and scores zero.

Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill, read_reference. You submit only skill files; you cannot supply your own agent, add tools, or alter the base agent's configuration.

Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.

Evaluation Criteria

Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.

CriteriaWeight

Response quality

An LLM judge scores each response against a task-specific quality bar for this challenge: how correct, complete, and usable the output is on its own terms, independent of which procedure produced it.

30 %

Skill use

An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill.

30 %

Scenario quality

An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar.

30 %

Output originality

A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here.

10 %

Scenarios are scored automatically. Deterministic penalties apply to moves that damage output quality, such as issuing a verdict before the criterion table is complete, producing a verdict label that does not follow from the table, adding content the format explicitly excludes, or misclassifying a profile attribute in a way that changes the outcome. Individual scenario scores feed into per-capability-type averages, which are combined into the final challenge score shown on the leaderboard.

Submission

How to submit

1
Login or create a sundaebar.ai account

Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.

2
Review the requirements

Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.

3
Build your skill

Draft your SKILL.md (at most 10,000 bytes; it is always loaded) and, if the method needs them, put tables, templates and worked examples in reference files under references/. Tell the agent in SKILL.md which reference to open and when, by file name. Focus on teaching procedure rather than providing templates.

4
Test locally

Validate locally using the published dataset variants to sanity-check output quality.

5
Submit via the platform

Submit through the sundae_bar challenge portal by uploading your skill folder, a .zip of it, or a single SKILL.md. The folder has SKILL.md at the top level and an optional references/ folder beside it.

6
Iterate

Review your scored breakdown and resubmit.

Constraints

Submission format: a folder or .zip with SKILL.md (YAML frontmatter with name and description, then a Markdown body) at the top level and an optional references/ folder. A single SKILL.md is also accepted.

Limits: SKILL.md at most 10,000 bytes. Up to 30 reference files, each at most 32,000 characters and 128,000 characters in total; text formats only (.md, .txt, .json, .yaml, .yml and .csv); lower-case file names.

Reference reads: the agent opens a reference only when SKILL.md names it. Each read is one of 10 steps per scenario and its text counts against the scenario's 29,000-token budget.

Base agent: fixed. Provided by sundae_bar.

Runtime: a fixed base-agent harness loads your skill and answers all 10 scenarios per submission with it.

Tests per submission: 10 (the full scenario set, no sampling).

Submission limits: 3 submissions per day. A rolling challenge limit also applies.

Rewards

Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.

Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.

Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.

Every file in your submission is scanned at submission time by a skill integrity gate: your SKILL.md and every reference file, whether or not your skill points at it. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool. Content placed in a reference file is checked exactly as content in SKILL.md.

A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.

Text addressed to the evaluators or judges, or text that tries to override instructions, anywhere in the bundle, is accepted at upload but places the entry under review and scores zero. Files that are not plain text (HTML comments, images, scripts, hidden or invisible characters, nested archives, disallowed file types) or that exceed a limit are rejected at upload. The copy and similarity checks run over the whole bundle, chunk by chunk, so padding a copied skill with extra reference files does not pass them.