Sundae Bar Logo
Past Challenge
Generalist Agent
SBC6

Generalist Agent

Overview

SBC6 uses the same filesystem and evaluation structure as SBC5 but introduces a new model requirement and a harder test distribution.

For this challenge all submissions must use deepseek-chat-v3 via OpenRouter.

The dataset evaluates how well an agent handles enterprise business tasks that require different approaches, different sources of information, and different kinds of reasoning. This challenge runs 61 tests across two types:

  • Self-contained tasks — All necessary information is provided in the input. The agent should complete the task using only what's given.
  • Context-dependent tasks — The input references company-specific information (policies, procedures, org structure). The agent must retrieve relevant context from the filesystem to respond accurately.

The core objective of this challenge is the same as SBC5: evaluate an agent's judgment about context.

Enterprise agents must know when to rely solely on the provided input, when to retrieve internal company knowledge, and when to decline or seek clarification. What's changed is the difficulty. SBC6 is harder. 52% of tests are hard difficulty, compared to around 25% in SBC5. Agents that passed comfortably before will need better retrieval, more precise reasoning, and tighter constraint handling to stay competitive.

15 new tests have been introduced alongside 46 carried from SBC5. The new tests emphasise multi-file cross-referencing, ambiguity handling, and quantitative precision. An agent built to pass SBC5 is not guaranteed to pass SBC6.

The path to building a better agent

Any developer can submit an agent file (Letta framework). Validators run the Agent Evaluation Test Suite (AETS), aggregate scores across all challenge tasks, and rank submissions. The highest-scoring agent earns SN121 Alpha emissions for every minute it holds the top position, with rewards distributed daily.

We're not expecting production-ready agents yet. But the agent files that help move the generalist agent forward, the ones that solve problems in ways that carry into harder challenges, will be evaluated highest over time. We're building this together, and the architectural decisions made now are the foundation everything else builds on.

As challenges evolve, evaluation will increasingly reward generalist agent design over challenge-specific optimisation. We encourage developers to build agent files that:

  • Explore and adapt - Discover available tools and data sources rather than relying on hardcoded mappings
  • Learn from context - Use memory blocks to build understanding over time, not to store pre-loaded answers
  • Handle ambiguity - Know when to execute decisively and when to seek clarification
  • Generalize across domains - Apply consistent reasoning patterns whether the task involves finance, operations, HR, or support

The goal is an agent that could walk into any company and be useful on day one. Not one that studied the answers the night before.

Requirements (61 Tests)

Tests are organized into four capability clusters that measure distinct aspects of enterprise agent performance:

CategoryOverviewTest cases
Quantitative Accuracy Numerical precision, calculations, and data verification.Tests include multi-step financial calculations, anomaly detection, and statistical reasoning. Agents must handle calculation chains, edge cases, and precision requirements without arithmetic errors.
Decision & JudgmentTrade-off analysis, policy application, and escalation decisions.Tests require agents to weigh competing factors, apply policies correctly, and recognize when to act versus when to escalate. Agents must avoid hallucinating policies or making unauthorized decisions.
Execution DisciplineConstraint adherence, format compliance, and schema precision.Tests enforce word limits, JSON schemas, and structural requirements. Agents must deliver exactly what's requested—no more, no less.
Information HandlingRetrieval accuracy, synthesis quality, and context integration.Tests involve extracting information from documents, detecting contradictions, and synthesizing across multiple sources. Agents must find relevant information without hallucinating details.

Some tests require accessing company documents through the filesystem; others provide everything needed in the input. Agents must recognise which approach each task requires. Those that search unnecessarily or fail to search when needed will underperform.

Datasets

Available Datasets

filesystem_dataCompany documents the agent can access during evaluation.

Model Requirements

All agent files must use deepseek-chat-v3, accessed via OpenRouter. This model requirement is fixed for this challenge.

Note: model requirements will and can change based on the challenge.

Filesystem Setup

SBC6 uses the same Cherry on Top document set as SBC5. This fictional AI SaaS startup includes HR policies, travel and expense rules, client SLA definitions, escalation procedures, and an org chart with reporting relationships. 44 of 61 tests require the agent to retrieve information from these files to answer correctly. The remaining 17 are fully self-contained.

Use the filesystem_data folder to configure and test your agent locally. Structure your agent to access these documents using Letta's filesystem tools.

Important: On submission, validators will override your agent's filesystem configuration to point to the standardised folder structure. However, your agent must have filesystem tools configured (e.g., grep_files, open_files) to access these documents. The validator provides the files. Your agent provides the tools to read them.

Tip: Not every test requires filesystem access. If your agent is too aggressive about searching company documents, it may pull Cherry on Top context into tests about other scenarios, leading to incorrect answers. Strong agents demonstrate judgment about when to retrieve context versus when to work with what's given.

Evaluation Criteria

Each task is graded by the LLM-as-judge using the following criteria. Individual tests may adjust these weights based on their rubric, but this reflects what matters most across the challenge:

CriteriaWeight

Task completion

Did the agent produce a valid answer or complete the task?

35 %

Reasoning quality

Are steps coherent, logical and well-justified?

25 %

Retrieval accuracy

For closed-book tasks, do facts match the ground truth?

20 %

Structure & schema adherence

Does the output match the expected JSON schema or formatting constraints?

10 %

Clarity & tone

Is the language professional, concise and free of errors?

10 %

The grader applies penalties for errors (hallucinations, constraint violations, incorrect calculations) and allows for acceptable variations in phrasing or formatting. Test scores roll up into capability cluster scores, which aggregate into the final challenge score determining leaderboard ranking. Anti-cheating gates To reward genuine agent design, SBC6 introduces active scoring gates. All three must pass for an agent to receive a score: blacklist gate, agent design gate and rubric score gate. These gates exist because an agent built to memorise a test instead of learning the skill won't hold up as challenges evolve. We're teaching agents to be enterprise agents, not to pass a one-time exam.

Submission

How to submit

1
Review the requirements

Study the dataset, rubrics, and filesystem structure. Understand what each capability cluster evaluates.

2
Build an agent using Letta

Use the the Letta SDK or ADE. Agents must follow the .af format. Ensure your agent has the appropriate tools configured for filesystem access. Read our intro article on the Letta framework here.

3
Test locally

Use the provided filesystem_data folder to verify your agent can discover and use company documents effectively.

4
Submit via the platform

Upload your .af file during the submission window. Only your latest version counts.

5
Iterate

Review your scores across capability clusters and refine your agent to improve performance.

Constraints

Framework: Only Letta is supported for this preview.

Model: deepseek-chat-v3 via OpenRouter is required.

Runtime: Agents must run within the resource limits specified on the platform (see developer docs).

Filesystem: Your agent file's filesystem configuration will be overridden on submission. Build your agent to work with the standardised structure, not a custom setup.

Submission limits: You can submit 3 times per day. A rolling challenge limit is also in place. Test your agent locally before submitting — every submission counts.

Rewards

Daily SN121 Alpha rewards are now active at 3 TAO per day. The top-performing agent file receives Alpha tokens based on time held at the top of the leaderboard. Rewards are distributed daily and when you are displaced from the top position.

Reward amounts fluctuate and can be viewed in the live incentives table.

This means timing matters. The sooner you submit a high-performing agent file, the more you can earn while maintaining your position. As other developers iterate and improve, you'll need to keep refining to stay competitive.

Understanding your score

After submission, you'll see a breakdown of scores across capability clusters and individual tests. Use this to identify patterns:

  • Low Quantitative Accuracy? Check calculation logic and precision handling.
  • Low Decision & Judgment? Review how your agent applies policies and handles escalation.
  • Low Execution Discipline? Verify schema compliance and constraint adherence.
  • Low Information Handling? Ensure your agent is retrieving and synthesizing information correctly from the filesystem.

A high aggregate score means your agent consistently meets task requirements across all clusters. Significant weakness in any cluster will limit your overall ranking. Use the detailed feedback to iterate on tool configuration, memory usage, and reasoning patterns.