Customer Success Intervention Planning Skill
Overview
This skill teaches the agent to assess a confirmed at-risk account, diagnose why it is at risk, and produce a structured, week-by-week save plan that a Customer Success team can execute immediately.
Given a structured account packet — which may include ARR, renewal and opt-out dates, usage telemetry, stakeholder changes, support history, and competitive signals — the agent produces a single seven-section document: an account snapshot with save-window math shown, a justified root-cause classification with the runner-up explicitly rejected, a phased weekly action table, a binary midpoint go/no-go checkpoint, an ARR-calibrated escalation ladder, account-specific success criteria, and a customer-facing executive-call preparation brief. When input signals are too thin to classify root cause, the agent instead returns a structured request naming the exact missing signals. When the request falls outside the skill boundary, the agent declines and redirects.
The submission is a single SKILL.md file. Strong skills teach the agent a reasoning procedure it can apply to accounts it has never seen before — how to read the signal pattern, compute the real deadline, pick the right opening move, and know when to escalate. Weak skills provide templates or memorised answers and break the moment account details change.
A skill built to memorise specific scenarios will not hold up as challenges evolve.
You submit a SKILL.md file. A fixed base agent loads it and answers 8 sampled scenarios with it, then returns a scored breakdown. The submission that holds the top of the leaderboard accrues on-chain emissions for as long as it stays first.
We are looking for skills that demonstrate good design and help the agent perform like a capable practitioner in this domain.
Requirements (8 Tests)
Your skill is tested across four capability types. Every submission sees 2 randomly sampled variants per type, for 8 total scenarios per evaluation.
| Category | Overview | Test cases |
|---|---|---|
| Type A — Closed Opt-Out Window Detection & Mid-Term Pivot | This type tests whether the agent recognises that the auto-renew opt-out deadline has already passed and reframes the plan as mid-term churn prevention rather than a near-term save. The skill must teach the agent to check deadline arithmetic before structuring any plan, and to shift its output — including framing, escalation triggers, and success criteria — when the opt-out window is no longer actionable. | 2 |
| Type B — Root Cause Classification with Decoy Signals | This type tests whether the agent can classify an at-risk account's root cause into exactly one of four types — product gap, stakeholder change, competitive displacement, or value unrealized — when the input deliberately includes signals that point toward a plausible but incorrect runner-up. The skill must teach the agent to apply the acute-change logic (which signals changed sharply in the last 60–90 days?) rather than counting total signals, and to explicitly reject the runner-up with a one-line justification explaining why acute-change reasoning overrides signal volume. | 2 |
| Type C — Root Cause Classification with Save-Window Math | This type tests two interlocked capabilities: correctly computing the true save window from the opt-out or non-renewal notice deadline (not the renewal date) with arithmetic shown, and classifying root cause using acute-change logic. The skill must teach the agent to perform both steps in order — establishing the real deadline first, then diagnosing root cause — so that the weekly plan is scoped to the actual time available. | 2 |
| Type D — Signal Insufficiency Refusal | This type tests whether the agent recognises when the provided input is too sparse to distinguish between root cause types and responds with a structured request for the specific missing signals rather than producing a plan. The skill must teach the agent to identify the minimum signal set required for classification and to name exactly what is absent, rather than defaulting to a generic save plan or an arbitrary root-cause guess. | 2 |
Surface details — account names, ARR figures, renewal dates, notice periods, stakeholder names, and telemetry specifics — rotate across variants, while the underlying capability being tested remains constant.
Datasets
Available Datasets
When you submit, we run your SKILL.md through a standardised base agent provided by sundae_bar. The base agent is the same for every submission and is not supplied by you. It receives the challenge's test scenarios one at a time, calls the load_skill tool to read your skill, and follows it to answer each scenario.
Same base agent, same tests, only the submitted skill changes. Scores therefore reflect the quality of your skill, not agent configuration: the agent's answers are only as good as the procedure your skill teaches.
Graders then score each answer for output quality, correct skill use, and gaming behavior, and the per-scenario scores are averaged into your challenge score.
Important: the base-agent harness is fixed at evaluation time and its available tools are load_skill. You submit only a SKILL.md; you cannot supply your own agent, add tools, or alter the base agent's configuration.
Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.
Evaluation Criteria
Each scenario is graded by an LLM judge. Final scores combine four graders and a set of deterministic penalties.
| Criteria | Weight | |
|---|---|---|
Response quality An LLM judge scores each response against a task-specific quality bar for this challenge — how correct, complete, and usable the output is on its own terms, independent of which procedure produced it. | 30 % | |
Skill use An LLM judge evaluates whether the agent's output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill. | 30 % | |
Scenario quality An LLM judge evaluates how well each output meets the per-type quality bar, independent of whether your skill was followed. A reference good output anchors the judge's sense of the quality bar. | 30 % | |
Output originality A check that your skill drives genuine per-scenario reasoning rather than emitting near-identical, templated responses. Skills that reuse one canned output across scenarios lose credit here. | 10 % |
Scenarios are scored individually, and deterministic penalties apply when the agent's output damages quality in ways that would make the plan unusable or misleading — for example, silently ignoring a closed opt-out window, inventing information not present in the input, or producing a plan when the correct response is a signal-insufficiency refusal. Individual scenario scores aggregate to capability-type averages, which then roll up to the final challenge score shown on the leaderboard.
Submission
How to submit
Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge.
Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.
Draft your SKILL.md. Focus on teaching procedure rather than providing templates.
Validate locally using the published dataset variants to sanity-check output quality.
Submit through the sundae_bar challenge portal by uploading your SKILL.md file.
Review your scored breakdown and resubmit.
Constraints
Submission format: a single SKILL.md file with YAML frontmatter (name, description) and Markdown body.
Base agent: fixed. Provided by sundae_bar.
Runtime: a fixed base-agent harness loads your SKILL.md and answers 8 sampled scenarios per submission with it.
Tests per submission: 8 (2 random variants per capability type, seeded by task ID for reproducibility).
Submission limits: 3 submissions per day. A rolling challenge limit also applies.
Rewards
Emissions accrue on-chain to whichever submission currently holds the top of the leaderboard. There is no fixed per-submission payout: the current leader earns for as long as it stays first.
Time at the top takes all. A strong skill submitted early starts accruing immediately and keeps earning until another submission displaces it. Once you are displaced, accrual stops until you reclaim the top.
Rankings shift as others iterate. Get to the top early, then keep refining your skill to defend the position.
Your SKILL.md is scanned at submission time by a skill integrity gate. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our evaluation pool.
A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible: a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.