Sundae Bar Logo
Past Challenge
Generalist Agent - Email Drafting Skills
SBC9

Generalist Agent - Email Drafting Skills

Overview

SBC9 is the first skills-based challenge on SN121. Instead of submitting an agent file, you submit a single SKILL.md that teaches a standardised base agent how to draft customer-facing emails. We pair your skill with our canonical base agent and evaluate how well it produces quality emails across a range of customer situations.

The submission is a single SKILL.md file. The base agent is fixed and provided by sundae_bar. Your skill is the only variable.

The core objective: write a skill that teaches the agent how to draft customer emails that a real support or success team would send.

Good skills teach procedure. They help the agent recognise the situation first, then shape a response that acknowledges impact, takes clear action, and holds the relationship. Weak skills rely on templates or vague tips and break the moment scenario details change.

This challenge is also the first step toward a broader skills marketplace. The evaluation framework built here will rate skills across frameworks including Letta, Claude, Codex and OpenClaw. Your rating on SBC9 is a direct signal of skill quality.

A skill built to memorise specific scenarios will not hold up as challenges evolve.

The challenge will run for 48 hours from April 22 at 10AM BMT to April 24 at 10AM BMT

You submit a SKILL.md file to the challenge. The validator attaches it to our canonical base agent, runs it against 8 sampled scenarios, and returns a scored breakdown. Top submissions earn Alpha emissions daily and on displacement.

3 TAO per day is distributed to top performers, split daily and on displacement.

We are looking for skills that demonstrate good design and help the base agent behave like a capable customer email drafter.

Good skills do a few things well:

  • Recognise the situation. Before drafting, the agent should identify whether it is repairing a relationship, granting a request, declining gracefully, or correcting a misunderstanding. Each has a different shape.
  • Teach procedure, not templates. Procedural guidance ("acknowledge the specific quantified impact before offering action") generalises across variants. Templates break the moment surface details change.
  • Name the failure modes. The strongest skills warn the agent explicitly against over-apologising, blame-shifting, policy jargon, and condescension.
  • Scaffold without dictating. Give the agent structural guidance on what sections belong in the email, without prescribing exact wording.

The skill that wins is the one that would hold up if you handed the base agent to a real customer success team and asked them to ship.

Requirements (8 Tests)

Your skill is tested across four capability types. Each has 5 variants in the pool. Every submission sees a randomly sampled 2 variants per type, for 8 total scenarios per evaluation.

CategoryOverviewTest cases
Type A — Customer impact and retention stakesAn angry or disappointed customer has cited a specific quantified harm and signalled that their continued business is at risk. Tests whether the skill teaches the agent to acknowledge specific impact, take accountability without blame-shifting, offer a concrete in-policy gesture, and set clear expectations.2
Type B — In-policy refund or goodwill requestA reasonable customer has asked for something we can grant. Tests whether the skill teaches the agent to deliver a clear yes, confirm specifics, set next-step expectations, and avoid the trap of over-apologising or asking for a survey mid-resolution.2
Type C — Out-of-policy requestA customer has asked for something we cannot grant. Tests whether the skill teaches the agent to say no early and kindly, explain without hiding behind policy jargon, offer alternatives within our power, and leave the door open without promising a reversal.2
Type D — Confused customerA customer has misunderstood the product, their account, or their own request. Tests whether the skill teaches the agent to correct explicitly but respectfully, teach in the customer's frame of reference, and confirm shared understanding.

Surface details (customer names, industries, dollar amounts, stakes language) rotate between variants. The underlying capability being tested stays the same.

Datasets

Available Datasets

Your skill attaches to a standardised base agent provided by sundae_bar. The base agent is the same for every submission. It is lean by design: generic memory blocks, standard Letta tools, no customer-specific knowledge. Its performance on the evaluation scenarios is driven almost entirely by the skill you give it.

You do not submit or modify the base agent. Its behaviour is fixed across all submissions so that scoring reflects skill quality, not agent configuration.

Tools available to the base agent: the load_skill tool (which loads your SKILL.md content when the agent decides to use it), plus standard Letta built-ins including memory and conversation search.

Important: the validator attaches your skill to the base agent at evaluation time. You cannot add tools, modify memory blocks, or alter the base agent configuration.

Tip: write your skill as though you are onboarding a capable new hire who needs a procedure, not scripting a chatbot.

Evaluation Criteria

Each scenario is graded by an LLM judge. Final scores combine two graders and a set of deterministic penalties.

CriteriaWeight

Skill use

An LLM judge evaluates whether the agent's email output actually followed the procedures your skill teaches, rather than freelancing or pretending to use the skill.

50 %

Scenario quality

An LLM judge evaluates how well each email meets the per-type rubric, independent of whether your skill was followed. A reference "good" email anchors the judge's sense of the quality bar.

50 %

Prohibited-move penalties Each scenario type has a list of moves that damage trust in a customer email. Examples include "we're sorry you feel that way," "this will never happen again," and substituting policy jargon for plain explanation. Each triggered pattern deducts 0.1 from the scenario quality score. Your final challenge score is the weighted average of all 8 scenarios, after penalties. Per-type rubric criteria examples (Type A): specific impact acknowledged, accountability without blame shifting, concrete action taken, retention gesture within realistic policy, clear next-step expectations, empathetic-but-not-defensive tone, personal human voice. Other types have different criteria matched to their communication goal. Full rubrics are published in rubric_sbc9.txt. Scores roll up from scenario scores to capability type averages to your final challenge score, which is published to the leaderboard.

Submission

How to submit

1
Login or create a sundaebar.ai account

Create an account and add your Bittensor wallet ID to your profile to earn alpha rewards in the challenge

2
Review the requirements

Review the reference skills (good, mediocre, gaming) to understand the quality bar and calibration points.

3
Build your Skill

Draft your SKILL.md. Focus on teaching procedure rather than providing templates.

4
Test locally

Validate locally using the published dataset variants to sanity-check output quality.

5
Submit via the platform

Submit through the sundae_bar challenge portal by uploading your SKILL.md file.

6
Iterate

Iterate. Review your scored breakdown and resubmit.

Constraints

Submission format: a single SKILL.md file with YAML frontmatter (name, description) and Markdown body.

Base agent: fixed. Provided by sundae_bar.

Runtime: the validator attaches your skill to the base agent and runs 8 sampled scenarios per submission.

Tests per submission: 8 (2 random variants per scenario type, seeded by task ID for reproducibility).

Submission limits: 3 submissions per day. A rolling challenge limit also applies.

Rewards

3 TAO per day is distributed across top-performing submissions, split daily and on displacement.

Earlier high-performing submissions earn rewards for longer while holding their position on the leaderboard. Timing matters. A strong skill submitted early accrues rewards while newer submissions work to displace it.

As others iterate, rankings shift. Keep refining your skill to stay in reward positions.

Your SKILL.md is scanned at submission time by a skill integrity gate. The gate checks that your skill teaches procedure rather than encoding scenario-specific content from our variant pool.

A skill that fails the integrity gate scores 0 for the submission. The gate runs deterministically and reports its finding as part of your results. This is how the evaluation framework stays credible for the marketplace. Skill ratings have to mean something — a high score should reflect a skill that genuinely teaches procedure, not one that memorised our test set.