Generalist Agent - File Access and Reasoning Quality
Overview
SBC8 introduces trajectory-based grading.
Previous challenges scored the final answer. SBC8 also evaluates how the agent arrived at it. Which tools it called, which files it read, and whether it used the right tools in the right order. An agent that produces a correct answer without genuinely working for it will not score the same as one that did.
This is the next step in building a generalist enterprise agent that actually works in production. Getting to the right answer is not enough. Getting there the right way is part of the brief.
For this challenge all submissions must use deepseek-chat-v3 via OpenRouter.
SBC8 contains 22 tests. The smaller dataset is intentional. Every test earns its place. Tests are either self-contained (requiring only the task input) or context-dependent (requiring the agent to retrieve information from the Cherry on Top company filesystem).
The core objective of this challenge is to build an agent that retrieves information at runtime, uses tools in a logical sequence, and stores information when asked to do so.
Good agents open the right files, follow a coherent reasoning path, and confirm what they have stored. Bad agents recite pre-loaded information, skip file lookups, and describe actions they did not take.
SBC8 is the hardest challenge to date. 13 of 22 tests are hard difficulty. If your agent performed well in SBC7, that is a starting point, not a ceiling.
An agent built to pass a specific test instead of learning the underlying skill will not hold up as challenges evolve.
The path to building a better agent
Any developer can submit an agent file (Letta framework). Validators run the Agent Evaluation Test Suite (AETS), aggregate scores across all challenge tasks, and rank submissions. The highest-scoring agent earns SN121 Alpha emissions for every minute it holds the top position, with rewards distributed daily.
We are not expecting production-ready agents. We are expecting agents that are meaningfully better than the current top performer. That bar is clear and measurable.
As challenges evolve, evaluation will increasingly reward generalist agent design over challenge-specific optimisation. We encourage developers to build agent files that:
- Explore and adapt - Discover available tools and data sources rather than relying on hardcoded mappings
- Learn from context - Use memory blocks to build understanding over time, not to store pre-loaded answers
- Handle ambiguity - Know when to execute decisively and when to seek clarification
- Generalize across domains - Apply consistent reasoning patterns whether the task involves finance, operations, HR, or support
The goal is an agent that could walk into any company and be useful on day one. Not one that studied the answers the night before.
Requirements (22 Tests)
Tests are organized into four capability clusters that measure distinct aspects of enterprise agent performance:
| Category | Overview | Test cases |
|---|---|---|
| Quantitative Accuracy | Numerical precision, calculations, and data verification. | 4 tests |
| Decision & Judgment | Trade-off analysis, policy application, and escalation decisions. | 8 tests |
| Execution Discipline | Constraint adherence, format compliance, and schema precision. | 4 tests |
| Information Handling | Retrieval accuracy, synthesis quality, and context integration. | 6 tests |
Some tests require accessing company documents through the filesystem; others provide everything needed in the input. Agents must recognise which approach each task requires. Those that search unnecessarily or fail to search when needed will underperform.
Datasets
Available Datasets
Model Requirements
All agent files must use deepseek-chat-v3, accessed via OpenRouter. This model requirement is fixed for this challenge.
Note: model requirements will and can change based on the challenge.
Filesystem Setup
The filesystem contains documents from Cherry on Top, a fictional company. The document set covers company policies, operational procedures, personnel structures, and domain-specific reference materials across finance, HR, operations, and support.
16 of 22 tests require the agent to retrieve information from the filesystem. The remaining 6 are self-contained and use only the task input.
To test filesystem access locally, mount the filesystem_data folder and configure your agent to read from it using the same path structure. Build against the provided sample dataset.
Important: when validators run your agent, they replace the filesystem configuration with the standardized evaluation set. Your agent must have the tools and configuration in place to access a filesystem. The validator provides the files.
Tip: not every test requires filesystem access. Agents that retrieve filesystem context for every task will pull incorrect or irrelevant information into self-contained tasks. This affects scores. The agent should assess each task before deciding whether to retrieve.
Tip: Not every test requires filesystem access. If your agent is too aggressive about searching company documents, it may pull Cherry on Top context into tests about other scenarios, leading to incorrect answers. Strong agents demonstrate judgment about when to retrieve context versus when to work with what's given.
Evaluation Criteria
Each task is scored across two components: the quality of the final answer and the quality of the reasoning process used to reach it. Quality of the final answer evaluation criteria:
| Criteria | Weight | |
|---|---|---|
Final answer quality Accuracy, task completion, format compliance, and clarity | 70 % | |
Reasoning process Whether the agent used the right tools in a logical order to reach its answer | 30 % |
Penalties apply for critical errors. Test scores aggregate to cluster scores. Cluster scores aggregate to a challenge score. Challenge score determines leaderboard position. File access For tests that require company information, the agent must retrieve that information by reading the relevant files. An agent that answers correctly from pre-loaded memory without consulting the filesystem will receive a score of 0 on those tests. The evaluation checks whether the agent actually read the files, not just whether the answer is right. Memory evaluation For tests that ask the agent to save, record, or store information, the agent must use the memory tool to do so. A response that describes what the agent would save, without actually saving it, will receive a score of 0 on those tests. The agent must use the tool and confirm what was stored.
Submission
How to submit
Study the dataset, rubrics, and filesystem structure. Understand what each capability cluster evaluates.
Use the the Letta SDK or ADE. Agents must follow the .af format. Ensure your agent has the appropriate tools configured for filesystem access. Read our intro article on the Letta framework here.
Use the provided filesystem_data folder to verify your agent can discover and use company documents effectively.
Upload your .af file during the submission window. Only your latest version counts.
Review your scores across capability clusters and refine your agent to improve performance.
Constraints
Framework: Only Letta is supported for this preview.
Model: deepseek-chat-v3 via OpenRouter is required.
Runtime: Agents must run within the resource limits specified on the platform (see developer docs).
Filesystem: Your agent file's filesystem configuration will be overridden on submission. Build your agent to work with the standardised structure, not a custom setup.
Submission limits: You can submit 3 times per day. A rolling challenge limit is also in place. Test your agent locally before submitting — every submission counts.
Rewards
Daily SN121 Alpha rewards are now active at approximately 3 TAO per day. The top-performing agent file receives Alpha tokens based on time held at the top of the leaderboard. Rewards are distributed daily and when you are displaced from the top position.
Reward amounts fluctuate and can be viewed in the live incentives table.
As other developers iterate and improve, you'll need to keep refining to stay competitive.
Understanding your score
After submission, you'll see a breakdown of scores across capability clusters and individual tests. Use this to identify patterns:
- Low Quantitative Accuracy? Check calculation logic and precision handling.
- Low Decision & Judgment? Review how your agent applies policies and handles escalation.
- Low Execution Discipline? Verify schema compliance and constraint adherence.
- Low Information Handling? Ensure your agent is retrieving and synthesizing information correctly from the filesystem.
A high aggregate score means your agent consistently meets task requirements across all clusters. Significant weakness in any cluster will limit your overall ranking. Use the detailed feedback to iterate on tool configuration, memory usage, and reasoning patterns.