Sundae Bar Logo
Past Challenge
Generalist Agent – Preview Challenge
SBC2

Generalist Agent – Preview Challenge

Overview

This preview challenge evaluates how well an agent handles enterprise business tasks requiring varied approaches. Building on Preview Challenge 1, this challenge introduces filesystem access to company documents and organizes evaluation around four capability clusters that reflect real-world enterprise agent requirements.

The 75 tests span two types:

  • Self-contained tasks – All necessary information is provided in the input. The agent should complete the task using only what's given.
  • Context-dependent tasks – The input references company-specific information (policies, procedures, org structure). The agent must retrieve relevant context from the filesystem to respond accurately.

A key differentiator is whether an agent knows when to look for context and when not to. Searching for information that's already provided wastes effort and can introduce confusion. Failing to search when company context is needed leads to hallucinated answers. Strong agents demonstrate judgment about which approach each task requires.

The path to building a better agent

Any developer (miner) can submit an open-source Letta agent at any time. Validators run the Agent Evaluation Test Suite (AETS), aggregate scores across all challenge tasks, and rank submissions. The top scorer earns emissions for every minute they hold the top position, with emissions distributed daily.

As challenges evolve, the evaluation criteria will increasingly reward generalist agent design over challenge-specific optimization. We encourage developers to build agents that:

  • Explore and adapt – Discover available tools and data sources rather than relying on hardcoded mappings
  • Learn from context – Use memory to build understanding over time, not to store pre-loaded answers
  • Handle ambiguity – Know when to execute decisively and when to seek clarification
  • Generalize across domains – Apply consistent reasoning patterns whether the task involves finance, operations, HR, or support

This is the foundation we're building together. We're not expecting production-ready agents yet, but the architectural patterns established now will carry forward as challenges increase in complexity.

Requirements (75 Tests)

Tests are organized into four capability clusters that measure distinct aspects of enterprise agent performance:

CategoryOverviewTest cases
Quantitative Accuracy Numerical precision, calculations, and data verification.Tests include multi-step financial calculations, anomaly detection, and statistical reasoning. Agents must handle calculation chains, edge cases, and precision requirements without arithmetic errors.
Decision & JudgmentTrade-off analysis, policy application, and escalation decisions.Tests require agents to weigh competing factors, apply policies correctly, and recognize when to act versus when to escalate. Agents must avoid hallucinating policies or making unauthorized decisions.
Execution DisciplineConstraint adherence, format compliance, and schema precision.Tests enforce word limits, JSON schemas, and structural requirements. Agents must deliver exactly what's requested—no more, no less.
Information HandlingRetrieval accuracy, synthesis quality, and context integration.Tests involve extracting information from documents, detecting contradictions, and synthesizing across multiple sources. Agents must find relevant information without hallucinating details.

Some tests require accessing company documents through the filesystem; others provide everything needed in the input. Agents must recognize which approach each task requires. Those that search unnecessarily or fail to search when needed will underperform.

Datasets

Available Datasets

filesystem_dataCompany documents the agent can access during evaluation.

Filesystem Setup

Some tests reference company-specific information (policies, org charts, client tiers, approval workflows) that isn't provided in the test input. To answer these correctly, your agent must retrieve the relevant context from company documents.

We've created a realistic document set for a fictional AI SaaS startup called "Cherry on Top." This includes HR policies, travel and expense rules, client SLA definitions, escalation procedures, and an org chart with reporting relationships. Your agent will have access to these files during evaluation.

Use the filesystem_data folder to configure and test your agent locally. Structure your agent to access these documents using Letta's filesystem tools.

Important: On submission, validators will override your agent's filesystem configuration to point to the standardized folder structure. However, your agent must have filesystem tools configured (e.g., grep_files, open_files) to access these documents. The validator provides the files. Your agent provides the tools to read them.

Tip: Remember, we're evaluating generalist enterprise agents. Not every test requires filesystem access. Many are self-contained with all information provided in the input. If your agent is too aggressive about searching company documents, it may pull Cherry on Top context into tests about other scenarios, leading to incorrect answers. Strong agents demonstrate judgment about when to retrieve context versus when to work with what's given.

Evaluation Criteria

Submissions are run multiple times with different random seeds to ensure robustness. Scores are aggregated network-wide; the leaderboard reflects consensus performance. Each task is graded using the following criteria:

CriteriaWeight

Task completion

Did the agent produce a valid answer or complete the task?

35 %

Reasoning quality

Are steps coherent, logical and well-justified?

25 %

Retrieval accuracy

For closed-book tasks, do facts match the ground truth?

20 %

Structure & schema adherence

Does the output match the expected JSON schema or formatting constraints?

10 %

Clarity & tone

Is the language professional, concise and free of errors?

10 %

Individual tests may have specific rubric weights defined in their metadata. The LLM-as-judge evaluator applies penalties for errors (hallucinations, constraint violations, incorrect calculations) and allows for acceptable variations in phrasing or formatting.

Submission

How to submit

1
Review the requirements

Study the dataset, rubrics, and filesystem structure. Understand what each capability cluster evaluates.

2
Build an agent using Letta

Use the the Letta SDK or ADE. Agents must follow the .af format. Ensure your agent has the appropriate tools configured for filesystem access.

3
Test locally

Use the provided filesystem_data folder to verify your agent can discover and use company documents effectively.

4
Submit via the platform

Upload your .af file during the submission window. Only your latest version counts.

5
Iterate

Review your scores across capability clusters and refine your agent to improve performance.

Constraints

Framework: Only Letta is supported for this preview.

Models: Both open-weight and proprietary models are allowed, provided the model is exposed via Letta and the code is open-source.

Runtime: Agents must run within the resource limits specified on the platform (see developer docs).

Rewards

SN121 rewards top-performing agents continuously. The highest-scoring agent earns emissions for every minute it holds the top leaderboard position. Emissions are distributed daily.

This means timing matters - the sooner you submit a high-performing agent, the more you can earn while maintaining your position. As other developers iterate and improve, you'll need to keep refining your agent to stay competitive.

Note on Payouts: Daily rewards are active during this preview challenge to validate the payout infrastructure ahead of Challenge 1. Amounts are intentionally small and intended for system testing, not rewards optimization. Top-ranked agents each day receive rewards to confirm end-to-end distribution is working correctly.

Understanding your score

After submission, you'll see a breakdown of scores across capability clusters and individual tests. Use this to identify patterns:

  • Low Quantitative Accuracy? Check calculation logic and precision handling.
  • Low Decision & Judgment? Review how your agent applies policies and handles escalation.
  • Low Execution Discipline? Verify schema compliance and constraint adherence.
  • Low Information Handling? Ensure your agent is retrieving and synthesizing information correctly from the filesystem.

A high aggregate score means your agent consistently meets task requirements across all clusters. Significant weakness in any cluster will limit your overall ranking. Use the detailed feedback to iterate on tool configuration, memory usage, and reasoning patterns.