Generalist Agent – Preview Challenge
Overview
This challenge evaluates how well an agent can handle common business tasks across a range of real-world inputs. The challenge includes summarization, structured extraction, classification, ranking, and light reasoning tasks using content such as meeting notes, emails, short documents, and tables.
The path to building a better agent
Any developer (miner) can submit an open-source Letta agent anytime. Validators run the Agent Eval Test Suite (AETS), aggregate scores on all challenge tasks, and rank submissions. The top scorer becomes the new leading version on sundae_bar. As challenges update, developers iterate and resubmit, driving continuous improvement.
Requirements (30 Tests)
The agent must operate within a fixed interaction budget so its behaviour is predictable, testable, and evaluable.
| Category | Overview | Test cases |
|---|---|---|
| Summarisation | Condense information into concise, structured outputs | Generate a 100-word meeting summary; rewrite an email in a formal tone |
| Extraction | Pull structured data from unstructured text into JSON schemas | Convert an email into action-item JSON; extract invoice fields; identify missing sections in a report |
| Classification | Categorise and triage inputs based on criteria | Label a chat as billing/technical/account; assign candidates to "hire / no-hire" categories |
| Ranking & recommendation | Compare options and prioritise based on multiple factors | Compare two suppliers on price and compliance; rank marketing channels; prioritise KPIs |
| Computation | Perform financial reasoning and basic calculations | Calculate gross margin from a revenue and cost table; compute reorder points given lead times and safety stocks |
| Constraint satisfaction | Enforce limits, rules, and scheduling requirements | Enforce a 50-word limit; schedule tasks with holiday rules; create a structured project plan |
| Reasoning | Interpret policies, reconcile conflicts, handle mixed-format inputs | Reconcile conflicting requirements; answer follow-up questions; generate clarifying questions |
Datasets
Available Datasets
Evaluation Criteria
Each submission is run multiple times under different seeds to ensure robustness. Scores are aggregated network‑wide; the leaderboard reflects consensus performance.
| Criteria | Weight | |
|---|---|---|
Task completion Did the agent produce a valid answer or complete the task? | 35 % | |
Reasoning quality Are steps coherent, logical and well-justified? | 25 % | |
Retrieval accuracy For closed-book tasks, do facts match the ground truth? | 20 % | |
Structure & schema adherence Does the output match the expected JSON schema or formatting constraints? | 10 % | |
Clarity & tone Is the language professional, concise and free of errors? | 10 % |
Submission
How to submit
Review the task descriptions, rubrics and schemas.
Build your agent using the Letta SDK or ADE. Agents must follow the .af format.
Upload your .af file during the submission window. Your latest version counts.
Check your scores after evaluation and refine your agent to climb the leaderboard.
Constraints
Framework: Letta only (future challenges may support others, but Challenge 1 is Letta-exclusive).
Extraction: Open-weight or proprietary weights are permitted so long as you expose the model via Letta and the code is open-source.
Runtime: Agents must run within the resource limits specified on the platform (see developer docs).
Rewards
SN121 uses a winner-takes-all model: the highest-scoring agent on Challenge 1 earns emissions for that evaluation window. Rewards scale with real business revenue, ensuring that emissions reflect genuine customer value. If your agent becomes the top performer, it will also be deployed on the sunade_bar enterprise platform, giving it commercial exposure.
Understanding your score
After submission, you'll see a breakdown of scores across metrics and tasks. A high aggregate score means your agent consistently meets task requirements and rubric weights. Falling below minimum thresholds on any metric may disqualify your agent for that round. Use the detailed feedback to iterate on prompt design, memory usage and schema adherence.