Baseline Agent¶

The CHIMERA-agent baseline repository is available here: https://github.com/DIAGNijmegen/chimera-agent-baseline

The baseline repository provides a reference implementation for building a CHIMERA-agent submission. It demonstrates how an agent can read structured patient-level input, call clinical tools served through MCP, retrieve guideline information using RAG, and produce a structured per-case decision through a terminal form-filling node.

The baseline uses a LangGraph ReAct loop and is intended to help participants understand the expected agent workflow, input/output format, reasoning structure, and Docker submission process. It supports the CHIMERA-agent task structure and writes one prediction JSON file per case to the required output directory.

Each case is provided as a structured input folder containing files such as prompt.json, clinical.json, and, where available, features.json. The prompt.json file contains the structured case information visible to the agent, while clinical.json contains additional EHR-style information that can be accessed through MCP tool calls. Optional features.json files may contain frozen foundation-model embeddings from available modalities.

The final output must follow the official structured JSON schema. The baseline defines task-specific output contracts for Task 1, Task 2, and Task 3. Outputs that do not validate against the required schema will be rejected.

Participants may use the baseline as provided, modify it, or replace it with their own agentic pipeline, provided that the final Docker container follows the official challenge rules, output schema, and offline execution requirements.

Offline Execution Requirement¶

All submissions will be executed on the Grand Challenge platform in an offline environment with no internet access.

Participants must not rely on external APIs, cloud-hosted LLMs, web services, online retrieval systems, remote databases, or internet-dependent tools during inference. This includes, but is not limited to, OpenAI, Anthropic, Gemini, hosted Hugging Face inference endpoints, online vector databases, web search APIs, and remote clinical tools.

If participants use an LLM, RAG system, MCP tool, pretrained model, guideline document, retrieval index, or embedding model, all required resources must be included locally inside the submitted Docker container before submission.

Submissions that require internet access or external API calls during execution will fail on Grand Challenge and may be disqualified.

Evaluation Pipeline¶

The official evaluation pipeline is available here: https://github.com/DIAGNijmegen/CHIMERA-agent/tree/main/evaluation

The evaluation repository provides the official local evaluation framework for CHIMERA-agent submissions. It is designed as a Docker-based pipeline that compares agent outputs against expert ground-truth responses using a combination of deterministic scoring and, where enabled, local LLM-based rationale assessment.

The current evaluation implementation includes a per-case gate check, schema validation, decision correctness, confidence scoring, variable-weight agreement, tool-use scoring, section-grounding assessment, and rationale alignment scoring. Cases that fail the required decision gate or schema validation receive a score of zero.

The evaluator produces both per-case and dataset-level outputs, including per-case results, aggregate metrics, and an evaluation summary. These outputs are intended to help participants test their submissions locally before submitting to the Grand Challenge platform.

The evaluation pipeline is also designed for offline execution. For local rationale judging, the repository uses a locally served judge model. Participants should ensure that any required judge model weights are downloaded or mounted locally before running in offline mode. Runtime model downloads or internet access should not be assumed.

Participants should use the official CHIMERA-agent evaluation pipeline for local testing and reporting of results. Custom, modified, or reimplemented evaluation metrics are not permitted for reporting challenge performance unless explicitly approved by the organizers.