Sundae Bar Logo
Past Challenge
QBR Churn-Risk Briefing Skill
SBC27

QBR Churn-Risk Briefing Skill

Overview

This skill teaches the agent to read a customer account packet and produce a concise internal churn-risk brief that a VP of Customer Success can act on in under three minutes.

Given a structured seed-state account packet — containing some combination of ARR, contract timeline, health-score history, usage telemetry, NPS, support ticket history, stakeholder contacts, and recent wins — the agent produces a seven-section internal QBR brief that names the account's risk tier, identifies the single primary churn signal, traces every claim to packet data, surfaces missing information explicitly, and commits to one actionable next step with a named owner and observable success criterion.

The submission is a single SKILL.md file. Strong skills teach the agent a generalizable reasoning procedure: how to weigh compounding signals, recognise when silence is itself a signal, commit to a primary churn driver, and structure output for a time-pressed executive audience. Weak skills rely on templates or canned language that collapse the moment account details — ARR, stakeholder names, usage patterns, data availability — change.

A skill built to memorise specific scenarios will not hold up as challenges evolve.

You submit a SKILL.md file. A fixed base agent loads it and answers all 10 scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.

We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.

Requirements (10 Tests)

Your skill is tested across 5 capability types, with 10 scenarios in total. Every submission is evaluated against all 10 scenarios — there is no sampling, so consistent quality across every capability type counts.

CategoryOverviewTest cases
Type A — Contradictory Signal ArbitrationTests whether the agent can classify an account's risk tier when the signals in the packet point in different directions — for example, stable usage alongside a deteriorating health score, or a high NPS paired with a looming stakeholder departure. The skill must teach the agent to weigh the full signal set, explicitly override any single-metric default when contradictory evidence is present, and state which signal combination drove the final tier assignment.1–3
Type B — Primary Churn Signal IdentificationTests whether the agent identifies and commits to exactly one dominant churn driver — usage decline, stakeholder change, support spike, billing friction, competitive signal, or silence — rather than listing all plausible drivers at equal weight. The skill must teach the agent to name the signal type, anchor it to a specific metric or verbatim detail from the packet, and carry that commitment coherently through the brief's risk and action sections.1–3
Type C — Scope Boundary EnforcementTests whether the agent correctly declines requests that fall outside the internal QBR churn-risk brief mandate — such as drafting a customer-facing presentation, writing renewal outreach emails, generating synthetic churn scores, or handling onboarding or staffing decisions. The skill must teach the agent to identify the out-of-scope element clearly and explain why it falls outside scope, while still completing any in-scope portion of the request.1–3
Type D — Silence as Signal RecognitionTests whether the agent recognises when the absence of any engagement signal — no logins, no support tickets, no NPS response, no stakeholder outreach — is itself the primary churn indicator rather than a data gap to be overlooked. The skill must teach the agent to classify silence as an active signal, surface it as the primary churn driver, and recommend a discovery-oriented next action rather than a remediation one.1–3
Type E — Thin Data HandlingTests whether the agent handles packets with sparse or missing fields — absent telemetry, no stakeholder contacts, no NPS, a single health-score reading with no prior baseline — without fabricating values or defaulting to generic language. The skill must teach the agent to name each missing data point explicitly, explain its materiality to the risk assessment, and recommend a concrete closure path.1–3

Surface details — account names, ARR figures, contract dates, health scores, stakeholder names, and which fields are present or absent — rotate across variants, while the underlying capability being tested remains the same.

Datasets

Available Datasets

When you submit, we run your SKILL.md through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your skill, and follows it to answer each scenario.

Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.

Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.

Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill. You submit only a SKILL.md; you cannot supply your own agent, add tools, or alter the base agent's configuration.

Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.

Evaluation Criteria

Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.

CriteriaWeight

Response quality

An LLM judge scores each response against a task-specific quality bar for this challenge — how correct, complete, and usable the output is on its own terms, independent of which procedure produced it.

30 %

Skill use

An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill.

30 %

Scenario quality

An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar.

30 %

Output originality

A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here.

10 %

Scenarios are scored on the quality of the agent's brief across all relevant capability dimensions; deterministic penalties apply for moves that directly damage output quality, such as fabricating data, omitting required brief sections, or refusing in-scope work. Individual scenario scores roll up to capability-type averages, which combine into the final challenge score displayed on the leaderboard.

Submission

How to submit

1
Login or create a sundaebar.ai account

Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.

2
Review the requirements

Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.

3
Build your skill

Draft your SKILL.md. Focus on teaching procedure rather than providing templates.

4
Test locally

Validate locally using the published dataset variants to sanity-check output quality.

5
Submit via the platform

Submit through the sundae_bar challenge portal by uploading your SKILL.md file.

6
Iterate

Review your scored breakdown and resubmit.

Constraints

Submission format: a single SKILL.md file with YAML frontmatter (name, description) and Markdown body.

Base agent: fixed. Provided by sundae_bar.

Runtime: a fixed base-agent harness loads your SKILL.md and answers all 10 scenarios per submission with it.

Tests per submission: 10 (the full scenario set — no sampling).

Submission limits: 3 submissions per day. A rolling challenge limit also applies.

Rewards

Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.

Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.

Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.

Your SKILL.md is scanned at submission time by a skill integrity gate. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool.

A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.