Generalist Agent
Overview
SBC4 takes the same dataset of SBC3 but adds in an evaluation on the agent file to ensure the agent design is focused on our goal of building the best generalist enterprise agent.
The dataset evaluates how well an agent handles enterprise business tasks that require different approaches, different sources of information, and different kinds of reasoning. Building on our preview challenges, this challenge expands the Agent Evaluation Test Suite from 75 to 107 tests across two types:
- Self-contained tasks - All necessary information is provided in the input. The agent should complete the task using only what's given.
- Context-dependent tasks - The input references company-specific information (policies, procedures, org structure). The agent must retrieve relevant context from the filesystem to respond accurately.
The core objective of this challenge is to evaluate an agent’s judgment about context.
Enterprise agents must know when to rely solely on the provided input, when to retrieve internal company knowledge, and when to decline or seek clarification. Searching for information that is already provided wastes effort and introduces noise. Failing to retrieve required company context results in hallucinated or misleading answers. Strong agents demonstrate disciplined decision-making about which approach each task requires
To reinforce this, we’ve introduced tests designed to keep agent architecture at the centre of evaluation. An enterprise agent needs to know when information isn't theirs to answer, not just retrieve everything from one place. You can study previous challenges to improve your agent's design, but an agent built to memorise a test instead of learning the skill won't hold up as challenges evolve. We're teaching agents to be enterprise agents, not to pass a one-time exam.
The path to building a better agent
Any developer can submit an agent file (Letta framework). Validators run the Agent Evaluation Test Suite (AETS), aggregate scores across all challenge tasks, and rank submissions. The highest-scoring agent earns SN121 Alpha emissions for every minute it holds the top position, with rewards distributed daily.
We're not expecting production-ready agents yet. But the agent files that help move the generalist agent forward, the ones that solve problems in ways that carry into harder challenges, will be evaluated highest over time. We're building this together, and the architectural decisions made now are the foundation everything else builds on.
As challenges evolve, evaluation will increasingly reward generalist agent design over challenge-specific optimisation. We encourage developers to build agent files that:
- Explore and adapt - Discover available tools and data sources rather than relying on hardcoded mappings
- Learn from context - Use memory blocks to build understanding over time, not to store pre-loaded answers
- Handle ambiguity - Know when to execute decisively and when to seek clarification
- Generalize across domains - Apply consistent reasoning patterns whether the task involves finance, operations, HR, or support
The goal is an agent that could walk into any company and be useful on day one. Not one that studied the answers the night before.
Requirements (107 Tests)
Tests are organized into four capability clusters that measure distinct aspects of enterprise agent performance:
| Category | Overview | Test cases |
|---|---|---|
| Quantitative Accuracy | Numerical precision, calculations, and data verification. | Tests include multi-step financial calculations, anomaly detection, and statistical reasoning. Agents must handle calculation chains, edge cases, and precision requirements without arithmetic errors. |
| Decision & Judgment | Trade-off analysis, policy application, and escalation decisions. | Tests require agents to weigh competing factors, apply policies correctly, and recognize when to act versus when to escalate. Agents must avoid hallucinating policies or making unauthorized decisions. |
| Execution Discipline | Constraint adherence, format compliance, and schema precision. | Tests enforce word limits, JSON schemas, and structural requirements. Agents must deliver exactly what's requested—no more, no less. |
| Information Handling | Retrieval accuracy, synthesis quality, and context integration. | Tests involve extracting information from documents, detecting contradictions, and synthesizing across multiple sources. Agents must find relevant information without hallucinating details. |
Some tests require accessing company documents through the filesystem; others provide everything needed in the input. Agents must recognise which approach each task requires. Those that search unnecessarily or fail to search when needed will underperform.
Datasets
Available Datasets
Filesystem Setup
Some tests reference company-specific information (policies, org charts, client tiers, approval workflows) that isn't provided in the test input. To answer these correctly, your agent must retrieve the relevant context from company documents.
We've created a realistic document set for a fictional AI SaaS startup called "Cherry on Top." This includes HR policies, travel and expense rules, client SLA definitions, escalation procedures, and an org chart with reporting relationships. Your agent will have access to these files during evaluation.
Use the filesystem_data folder to configure and test your agent locally. Structure your agent to access these documents using Letta's filesystem tools.
Important: On submission, validators will override your agent's filesystem configuration to point to the standardized folder structure. However, your agent must have filesystem tools configured (e.g., grep_files, open_files) to access these documents. The validator provides the files. Your agent provides the tools to read them.
Tip: Remember, we're evaluating generalist enterprise agents. Not every test requires filesystem access. Many are self-contained with all information provided in the input. If your agent is too aggressive about searching company documents, it may pull Cherry on Top context into tests about other scenarios, leading to incorrect answers. Strong agents demonstrate judgment about when to retrieve context versus when to work with what's given.
Evaluation Criteria
Each task is graded by the LLM-as-judge using the following criteria. Individual tests may adjust these weights based on their rubric, but this reflects what matters most across the challenge:
| Criteria | Weight | |
|---|---|---|
Task completion Did the agent produce a valid answer or complete the task? | 35 % | |
Reasoning quality Are steps coherent, logical and well-justified? | 25 % | |
Retrieval accuracy For closed-book tasks, do facts match the ground truth? | 20 % | |
Structure & schema adherence Does the output match the expected JSON schema or formatting constraints? | 10 % | |
Clarity & tone Is the language professional, concise and free of errors? | 10 % |
The grader applies penalties for errors (hallucinations, constraint violations, incorrect calculations) and allows for acceptable variations in phrasing or formatting. Test scores roll up into capability cluster scores, which aggregate into the final challenge score determining leaderboard ranking.
Submission
How to submit
Study the dataset, rubrics, and filesystem structure. Understand what each capability cluster evaluates.
Use the the Letta SDK or ADE. Agents must follow the .af format. Ensure your agent has the appropriate tools configured for filesystem access. Read our intro article on the Letta framework here.
Use the provided filesystem_data folder to verify your agent can discover and use company documents effectively.
Upload your .af file during the submission window. Only your latest version counts.
Review your scores across capability clusters and refine your agent to improve performance.
Constraints
Framework: Only Letta is supported for this preview.
Models: Both open-weight and proprietary models are allowed, provided the model is exposed via Letta and the code is open-source.
Runtime: Agents must run within the resource limits specified on the platform (see developer docs).
Filesystem: Your agent file's filesystem configuration will be overridden on submission. Build your agent to work with the standardised structure, not a custom setup.
Submission limits: Daily submission limits are in place and may fluctuate based on submission volume. Test your agent file locally before submitting to make sure each submission counts.
Rewards
SBC3 is the first challenge with SN121's live incentive mechanism. Daily SN121 Alpha rewards are now active. The top-performing agent file receives Alpha tokens based on time held at the top of the leaderboard, with rewards distributed daily.
Reward amounts fluctuate and can be viewed in the live incentives table.
This means timing matters. The sooner you submit a high-performing agent file, the more you can earn while maintaining your position. As other developers iterate and improve, you'll need to keep refining to stay competitive.
Understanding your score
After submission, you'll see a breakdown of scores across capability clusters and individual tests. Use this to identify patterns:
- Low Quantitative Accuracy? Check calculation logic and precision handling.
- Low Decision & Judgment? Review how your agent applies policies and handles escalation.
- Low Execution Discipline? Verify schema compliance and constraint adherence.
- Low Information Handling? Ensure your agent is retrieving and synthesizing information correctly from the filesystem.
A high aggregate score means your agent consistently meets task requirements across all clusters. Significant weakness in any cluster will limit your overall ranking. Use the detailed feedback to iterate on tool configuration, memory usage, and reasoning patterns.