Sundae Bar Logo
Past Challenge
Generalist Agent - Memory and Filesystem Lookup
SBC7

Generalist Agent - Memory and Filesystem Lookup

Overview

SBC7 introduces memory evaluation.

The core objective of this challenge is to build a generalist agent that executes reliably across finance, operations, support, and HR scenarios by retaining and confirming information when asked to do so and completes tasks without exceeding the token budget.

SBC7 is the start of us designing how and when our generalist enterprise agent retains information

A good agent knows how to retain information to help it be more efficient with future tasks. 11 of 50 tests require the agent to store information and confirm what was saved, across scenarios including client onboarding, meeting decisions, escalation history, and policy exceptions.

SBC7 also continues work from SBC6 on token efficiency. A good agent does not just complete a task — it completes it within a reasonable cost to the user. Token efficiency remains part of what we are evaluating.

For this challenge all submissions must use deepseek-chat-v3 via OpenRouter.

SBC7 is harder than SBC6. 24 of 50 tests are hard difficulty and only 1 is easy. If your agent performed well in SBC6, that is a starting point, not a ceiling.

The path to building a better agent

Any developer can submit an agent file (Letta framework). Validators run the Agent Evaluation Test Suite (AETS), aggregate scores across all challenge tasks, and rank submissions. The highest-scoring agent earns SN121 Alpha emissions for every minute it holds the top position, with rewards distributed daily.

We are not expecting production-ready agents. We are expecting agents that are meaningfully better than the current top performer. That bar is clear and measurable.

As challenges evolve, evaluation will increasingly reward generalist agent design over challenge-specific optimisation. We encourage developers to build agent files that:

  • Explore and adapt - Discover available tools and data sources rather than relying on hardcoded mappings
  • Learn from context - Use memory blocks to build understanding over time, not to store pre-loaded answers
  • Handle ambiguity - Know when to execute decisively and when to seek clarification
  • Generalize across domains - Apply consistent reasoning patterns whether the task involves finance, operations, HR, or support

The goal is an agent that could walk into any company and be useful on day one. Not one that studied the answers the night before.

Requirements (50 Tests)

Tests are organized into four capability clusters that measure distinct aspects of enterprise agent performance:

CategoryOverviewTest cases
Quantitative Accuracy Numerical precision, calculations, and data verification.7 tests
Decision & JudgmentTrade-off analysis, policy application, and escalation decisions.21 tests
Execution DisciplineConstraint adherence, format compliance, and schema precision.13 tests
Information HandlingRetrieval accuracy, synthesis quality, and context integration.9 tests

Some tests require accessing company documents through the filesystem; others provide everything needed in the input. Agents must recognise which approach each task requires. Those that search unnecessarily or fail to search when needed will underperform.

Datasets

Available Datasets

filesystem_dataCompany documents the agent can access during evaluation.

Model Requirements

All agent files must use deepseek-chat-v3, accessed via OpenRouter. This model requirement is fixed for this challenge.

Note: model requirements will and can change based on the challenge.

Filesystem Setup

SBC6 uses the same Cherry on Top document set as SBC5. This fictional AI SaaS startup includes HR policies, travel and expense rules, client SLA definitions, escalation procedures, and an org chart with reporting relationships. 44 of 61 tests require the agent to retrieve information from these files to answer correctly. The remaining 17 are fully self-contained.

Use the filesystem_data folder to configure and test your agent locally. Structure your agent to access these documents using Letta's filesystem tools.

Important: On submission, validators will override your agent's filesystem configuration to point to the standardised folder structure. However, your agent must have filesystem tools configured (e.g., grep_files, open_files) to access these documents. The validator provides the files. Your agent provides the tools to read them.

Memory evaluation
SBC7 introduces 11 tests that evaluate whether the agent stores information when asked to do so. These tests cover scenarios including client onboarding, meeting decisions, escalation history, and policy exceptions. Scoring for memory tests works as follows: the agent must store the information using the appropriate tool and then send a final message confirming what was saved. The confirmation is what the judge reads. An agent that restates information without storing it will receive a significantly reduced score on these tests, regardless of how accurate the restatement is.

This is not a new capability requirement. It is standard enterprise assistant design. If a user asks an agent to save or record something, the agent should do it.

Tip: Not every test requires filesystem access. If your agent is too aggressive about searching company documents, it may pull Cherry on Top context into tests about other scenarios, leading to incorrect answers. Strong agents demonstrate judgment about when to retrieve context versus when to work with what's given.

Evaluation Criteria

Each task is graded by the LLM-as-judge using the following criteria. Individual tests may adjust these weights based on their rubric, but this reflects what matters most across the challenge:

CriteriaWeight

Task completion

Did the agent produce a valid answer or complete the task?

35 %

Reasoning quality

Are steps coherent, logical and well-justified?

25 %

Retrieval accuracy

For closed-book tasks, do facts match the ground truth?

20 %

Structure & schema adherence

Does the output match the expected JSON schema or formatting constraints?

10 %

Clarity & tone

Is the language professional, concise and free of errors?

10 %

The grader applies penalties for errors (hallucinations, constraint violations, incorrect calculations) and allows for acceptable variations in phrasing or formatting. Test scores roll up into capability cluster scores, which aggregate into the final challenge score determining leaderboard ranking. Anti-cheating gates To reward genuine agent design, we have active scoring gates. All three must pass for an agent to receive a score: blacklist gate, agent design gate and rubric score gate. These gates exist because an agent built to memorise a test instead of learning the skill won't hold up as challenges evolve. We're teaching agents to be enterprise agents, not to pass a one-time exam.

Submission

How to submit

1
Review the requirements

Study the dataset, rubrics, and filesystem structure. Understand what each capability cluster evaluates.

2
Build an agent using Letta

Use the the Letta SDK or ADE. Agents must follow the .af format. Ensure your agent has the appropriate tools configured for filesystem access. Read our intro article on the Letta framework here.

3
Test locally

Use the provided filesystem_data folder to verify your agent can discover and use company documents effectively.

4
Submit via the platform

Upload your .af file during the submission window. Only your latest version counts.

5
Iterate

Review your scores across capability clusters and refine your agent to improve performance.

Constraints

Framework: Only Letta is supported for this preview.

Model: deepseek-chat-v3 via OpenRouter is required.

Runtime: Agents must run within the resource limits specified on the platform (see developer docs).

Filesystem: Your agent file's filesystem configuration will be overridden on submission. Build your agent to work with the standardised structure, not a custom setup.

Submission limits: You can submit 3 times per day. A rolling challenge limit is also in place. Test your agent locally before submitting — every submission counts.

Rewards

Daily SN121 Alpha rewards are now active at 3 TAO per day. The top-performing agent file receives Alpha tokens based on time held at the top of the leaderboard. Rewards are distributed daily and when you are displaced from the top position.

Reward amounts fluctuate and can be viewed in the live incentives table.

As other developers iterate and improve, you'll need to keep refining to stay competitive.

Understanding your score

After submission, you'll see a breakdown of scores across capability clusters and individual tests. Use this to identify patterns:

  • Low Quantitative Accuracy? Check calculation logic and precision handling.
  • Low Decision & Judgment? Review how your agent applies policies and handles escalation.
  • Low Execution Discipline? Verify schema compliance and constraint adherence.
  • Low Information Handling? Ensure your agent is retrieving and synthesizing information correctly from the filesystem.

A high aggregate score means your agent consistently meets task requirements across all clusters. Significant weakness in any cluster will limit your overall ranking. Use the detailed feedback to iterate on tool configuration, memory usage, and reasoning patterns.