Sales Forecast Integrity Analysis Skill
Overview
This skill teaches the agent to scrub a sales forecast — reading a CRM pipeline export, spotting deals that could slip or mislead, and producing a structured risk report a VP can act on before a forecast call.
Given a tabular CRM export (deal names, owners, amounts, stages, close dates, and commit categories, plus optional prior-period snapshots and quota data), a good response delivers three ordered artifacts: a coverage waterfall table with ratios and a structural verdict, a ranked anomaly table cut at the deals that drive approximately 80% of dollar-weighted exposure, and a ≤150-word VP-ready narrative paragraph that leads with the structural coverage verdict and names the single biggest risk with deal-level evidence.
The submission is a single SKILL.md file. Strong skills teach the agent a reasoning procedure it can apply to any forecast it has not seen before — how to build the waterfall first, how to weight anomalies by dollar exposure, how to calibrate materiality against coverage tier, and how to handle missing or inconsistent inputs without silently papering over them. Weak skills supply fill-in-the-blank templates or memorised signal lists that break the moment deal names, amounts, or quarter boundaries change.
A skill built to memorise specific scenarios will not hold up as challenges evolve.
You submit a SKILL.md file. A fixed base agent loads it and answers all 10 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.
We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.
Requirements (10 Tests)
Your skill is tested across 6 capability types, with 10 scenarios in total. Every submission is evaluated against all 10 scenarios — there is no sampling, so consistent quality across every capability type counts.
| Category | Overview | Test cases |
|---|---|---|
| Type A — Concentration Risk & Structural vs. Deal-Level Distinction | Tests whether the agent can distinguish a structural pipeline health problem from an isolated deal-level anomaly and communicate that distinction clearly in its narrative. The skill must teach the agent to recognise when a single large deal dominates total pipeline coverage, name that concentration risk explicitly, and prevent the narrative from treating a deal-level blip as the headline risk when the structural gap is the dominant story. | 1–2 |
| Type B — Coverage Calibration Against Data Contradictions | Tests whether the agent can hold the coverage-tier verdict and deal-level data signals in tension simultaneously — using the structural picture to decide which contradictions are material enough to surface. The skill must teach the agent to weigh commit-category overrides, stage mismatches, and activity gaps against the backdrop of whether the forecast is over- or under-covered before deciding what rises to the anomaly table. | 1–2 |
| Type C — Coverage Calibration & Slip Prioritisation | Tests whether the agent ranks and presents slippages in proportion to their actual impact on the forecast outcome, filtered through the coverage-tier lens. The skill must teach the agent to suppress low-exposure slippages in over-covered forecasts and elevate even modest slippages in under-covered ones, producing an anomaly table that reflects materiality rather than a raw list of every signal detected. | 1–2 |
| Type D — Delta Comparison & Promotion Analysis | Tests whether the agent can compare a current snapshot against a prior-period snapshot to identify silent material changes, stage regressions, close-date slips, and late-period commit promotions — and then calibrate which of those deltas are material given the coverage tier. The skill must teach the agent to surface the delta signals that move the forecast needle while suppressing noise, and to disclaim slippage analysis clearly when no prior snapshot is present. | 1–2 |
| Type E — Missing Input Discipline | Tests whether the agent handles incomplete or ambiguous inputs without silently papering over gaps. The skill must teach the agent to refuse coverage-ratio computation when quota is absent (asking for it rather than deriving a proxy from pipeline size), to request an exchange rate when currencies are mixed rather than summing them silently, and to normalise inconsistent commit-category labels with a brief note rather than halting or asking unnecessary clarifying questions. | 1–2 |
| Type F — Structural Gap Analysis & VP-Ready Narrative | Tests whether the agent can produce a narrative paragraph that a VP can read in under a minute and act on — leading with the structural coverage verdict, naming the single biggest risk with specific deal-level evidence, and refusing to headline a small anomaly when a coverage gap is the dominant problem. The skill must teach the agent to subordinate deal-level colour to the structural story when the numbers demand it. | 1–2 |
Surface details — deal names, rep names, dollar amounts, close dates, and commit categories — rotate across scenario variants while the underlying analytical capability being tested remains the same.
Datasets
Available Datasets
When you submit, we run your SKILL.md through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your skill, and follows it to answer each scenario.
Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.
Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.
Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill. You submit only a SKILL.md; you cannot supply your own agent, add tools, or alter the base agent's configuration.
Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.
Evaluation Criteria
Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.
| Criteria | Weight | |
|---|---|---|
Response quality An LLM judge scores each response against a task-specific quality bar for this challenge — how correct, complete, and usable the output is on its own terms, independent of which procedure produced it. | 30 % | |
Skill use An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill. | 30 % | |
Scenario quality An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar. | 30 % | |
Output originality A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here. | 10 % |
Scenarios are scored on the quality of the agent's output across the three required artifacts; deterministic penalties apply when the agent makes moves that damage output integrity — such as silently summing mixed currencies, fabricating a denominator when quota is absent, or producing a narrative that contradicts the waterfall verdict. Individual scenario scores aggregate to capability-type averages, which roll up to the final challenge score shown on the leaderboard.
Submission
How to submit
Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.
Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.
Draft your SKILL.md. Focus on teaching procedure rather than providing templates.
Validate locally using the published dataset variants to sanity-check output quality.
Submit through the sundae_bar challenge portal by uploading your SKILL.md file.
Review your scored breakdown and resubmit.
Constraints
Submission format: a single SKILL.md file with YAML frontmatter (name, description) and Markdown body.
Base agent: fixed. Provided by sundae_bar.
Runtime: a fixed base-agent harness loads your SKILL.md and answers all 10 scenarios per submission with it.
Tests per submission: 10 (the full scenario set — no sampling).
Submission limits: 3 submissions per day. A rolling challenge limit also applies.
Rewards
Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.
Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.
Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.
Your SKILL.md is scanned at submission time by a skill integrity gate. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool.
A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.