SI
Sentinel Integrations
← Back to Research Index

SIA (hexo-ai/sia) — Technical Analysis & Suggested Improvements

Repo analyzed: https://github.com/hexo-ai/sia @ commit on main (cloned to `./workspace/

Paper claim: "SIA: Self-Improving AI with Harness & Weight Updates" (Hebbar et al., 2026, arXiv:2605.27276). Headline numbers: +56.6% LawBench, −91.9% runtime on TriMul Triton kernel, +502% on scRNA-seq denoising; framework labelled SIA-W+H (W = weights, H = harness).

Package: sia-agent==0.2.1, Python ≥3.11, MIT.

1. What the repository actually is

A ~1.7 KLOC orchestration shell around two third-party agent SDKs (Anthropic Claude Agent SDK or OpenHands) that runs a 3-role generational loop:

| Role | Implementation |

|---|---|

| Meta-agent | One prompt in sia/orchestrator.py (META_AGENT_PROMPT, ~65 lines) sent via run_agent(...) to write gen_1/target_agent.py. |

| Target agent | The model-authored Python file. Invoked as a subprocess: python target_agent.py --dataset_dir … --working_dir … inside a per-run venv. |

| Feedback agent | Second prompt (FEEDBACK_AGENT_PROMPT) that consumes execution logs + (optionally) results.json and writes gen_{n+1}/{improvement.md, target_agent.py}. |

Surrounding scaffolding:

2. Confirmation: the "W" in SIA-W+H is **not** in this repo

Exhaustive grep across .py / .md / .toml / .yml for: lora, peft, finetune, fine[-_]tune, sft, grpo, dpo, reward, trainer, huggingface, hf_hub, safetensors, unsloth, axolotl, deepspeed, vllm, modal, torch, transformers, gradient, optimizer, triton, cuda, reinforce. Hits:

Also missing:

Conclusion: The published code implements only the H (harness) half of SIA-W+H. The W half — the actual self-improvement of model weights — is absent. The README and citation nonetheless market the repo as the "official implementation" of the W+H paper. (Side note: the arXiv ID 2605.27276 is malformed for the 2026 year — arXiv IDs are YYMM.NNNNN, so 2026 papers would be 2601–2612.*. This may be a placeholder.)

3. Architectural observations on what *is* implemented

3.1 Correctness / robustness risks

1. subprocess.run(command, shell=True, executable="/bin/bash") with f-string interpolation of python_exec, target_agent_path, ABS_DATASET_DIRECTORY, current_gen_directory, and stdout_log_file (orchestrator.py:660–668, also l.220). Any path containing a space, quote, $, or ; breaks or shell-injects. Same in run_evaluation.

2. venv.create(... with_pip=True) + 9 hard-coded packages on every run (l.398–417). No version pins, no caching, no offline fallback. A cold run on gpqa re-installs pandas/sklearn (slow, network-dependent), and target_agent.py is free to pip install more at runtime without tracking.

3. RUN_DIRECTORY hard-failure if it exists (l.383–386). No --resume, no --overwrite. A crashed run requires manual rm -rf and loses partial state.

4. max_turns="20" passed as a string to ClaudeAgentOptions (l.607, l.897, util.py:24). The SDK expects int; this either silently coerces or breaks on stricter versions.

5. asyncio.run called from inside add_generation → _generate_llm_summary (context_manager.py:157) while the outer orchestrator also calls asyncio.run. Nested asyncio.run is fine here because the outer one has returned, but the pattern is fragile and creates a fresh event loop per generation summary. If anyone ever calls add_generation from inside an async context it will raise RuntimeError: asyncio.run() cannot be called from a running event loop.

6. No timeout on the target-agent subprocess. A runaway agent burns API credits indefinitely (only the in-agent bash tool has timeout=30, set in the reference template that the meta-agent is free to ignore).

7. load_agent_execution reports a "successful trajectory" as isinstance(t, list) and "failed" as isinstance(t, dict) and t.get("error") (orchestrator.py:763–764). Schema-inferred via duck typing; a target agent that legitimately wraps trajectories in a dict will be miscounted as failed.

8. Stdout-metric scraping uses fragile regex (r"accuracy[:\s=]+(\d+\.?\d*)") and silently overwrites the canonical metric whenever results.json is absent (context_manager.py:380–408). Real metrics like F1, MSE_norm, latency_ms are not captured unless a task author specifically writes them to results.json.

9. Best-generation selection in finalize() is hard-coded to metrics["accuracy"] (context_manager.py:266). Tasks like the TriMul kernel (lower-is-better latency) or denoising (lower MSE) cannot be ranked.

10. No sandboxing. permission_mode="bypassPermissions" on Claude SDK (util.py:36) and unconstrained TerminalTool on OpenHands. The model-authored target_agent.py runs with the orchestrator's user privileges and full network. The "agent can only read dataset / only write working_dir" rule is a prompt-level request, not an enforced jail.

11. Cross-process race in MultiTrajectoryLogger (in the reference template): writes execution_q{idx}.json sequentially, but the loop in _shared/reference_target_agent.py does not include checkpointing/resume — a crash on q150/198 loses all prior summary-level state.

12. context.md is the only durable run state but is written with open(..., "a") without fsync and without a structured (JSON/JSONL) mirror, so machine-readable post-hoc analysis requires re-parsing Markdown.

3.2 Design ceilings

13. The feedback agent only sees one prior generation's source + the first 3 of N trajectories truncated to 1000 chars each (orchestrator.py:767–774). For tasks like GPQA-Diamond (198 Qs) or scRNA-seq, the signal-to-noise ratio of "look at 3 samples to design improvements" is very poor. There is no per-trajectory failure clustering.

14. There is no multi-arm exploration: each generation produces exactly one child agent. Real self-improvement frameworks (AlphaEvolve, Voyager, ADAS) maintain a population with selection. The current loop is greedy and monotone in name only — there is no rollback if gen N+1 regresses.

15. No metric-aware selection. gen_n+1 is always derived from gen_n, never from the best historical generation, so a bad mutation poisons the chain.

16. The meta and feedback agents share the same meta_model (l.896) — there is no separation of capabilities (e.g., a cheap clusterer for log analysis + an expensive reasoner for rewrites).

17. The reference target agent uses a bare while-loop with max_tokens=4096 and no token/cost accounting (_shared/reference_target_agent.py:182–220). Long agent runs silently exceed budgets.

18. No task-level evaluation harness: each task ships its own data/public/evaluate.py (when present), invoked via shell. There is no standard BaseEvaluator ABC, no schema for results.json, no required fields, no leaderboard merging across runs.

19. prepare_mlebench_dataset.py uses the deprecated google.generativeai package (l.21) — Google has moved to google-genai. Will break on newer envs.

4. Documentation vs reality gap

5. Suggested Improvements (numbered, actionable)

The list is split into three tiers: (A) faithfulness to the paper (add the missing "W"), (B) correctness/robustness of the existing harness, (C) research extensions. Each item names files/functions to touch and concrete acceptance criteria.

A. Add the missing weight-update path (SIA-W)

1. Introduce a sia/weights/ subpackage with the contract class WeightUpdater(ABC): def update(run_dir, gen_num, trajectories, metrics) -> ModelHandle. Concrete subclasses: LoRASupervisedUpdater, DPOUpdater, RejectionSamplingSFTUpdater, NoopUpdater (default). Wire it into orchestrator.main between run_evaluation and context_mgr.add_generation, so each generation can optionally produce a new target_model_v{n} checkpoint that the next target_agent.py is told to use via --task_model.

2. Trajectory → training-data converter (sia/weights/dataset_builder.py). Given agent_execution/execution_q*.json plus the per-question correctness from results.json, emit:

- sft.jsonl: {messages: [...], reward: float} — keep only trajectories above a reward threshold (rejection sampling / "Self-Taught Reasoner"-style).

- dpo.jsonl: {prompt, chosen, rejected} pairs from same-question high/low-reward trajectories.

- kto.jsonl for the binary-reward case. Validate with pydantic schemas in sia/weights/schemas.py.

3. LoRA training script (sia/weights/train_lora.py) using transformers + peft + trl.SFTTrainer / DPOTrainer. Add an optional [weights] extra in pyproject.toml: transformers>=4.45, peft>=0.13, trl>=0.11, bitsandbytes>=0.43, accelerate>=1.0, datasets>=3.0. CLI: python -m sia.weights.train_lora --gen-dir runs/run_1/gen_3 --base-model meta-llama/Llama-3.1-8B-Instruct --method dpo. Output: runs/run_1/gen_3/adapter/ + adapter_config.json.

4. Modal integration (sia/weights/modal_runner.py) so training does not require a local GPU. Define a modal.App("sia-weight-update") with a @app.function(gpu="A100-40GB", image=modal.Image.debian_slim().pip_install(...)) that mounts runs/ as a modal.Volume and calls train_lora.main(). Add --training-backend {local,modal,slurm} flag to the orchestrator. Acceptance: a single command on a CPU laptop produces an adapter in <30 min for a 7B model on 1k DPO pairs.

5. Adapter-aware run_agent. Extend sia/util.py so the target agent can be served (a) by an OpenAI-compatible vLLM endpoint with --lora-modules name=path or (b) by a HuggingFace pipeline. Add sia/weights/serve_vllm.py that spins a vLLM server pointing at runs/run_X/gen_N/adapter/ and exports OPENAI_BASE_URL for the next generation's target_agent.py.

6. Faithfulness verification harness: a scripts/reproduce_paper.py that runs --task lawbench --backend weights --max_gen 5 and asserts a measurable gain on the held-out split. Even partial reproduction is valuable; if it cannot get within X% of the paper numbers, document that explicitly in EVALUATION_GUIDE.md instead of leaving the W+H claim unsubstantiated.

7. Add the paper's missing tasks: at minimum a stub sia/tasks/trimul-triton/ (Triton kernel correctness + H100 latency target via triton.testing.do_bench) and sia/tasks/scrna-denoising/ (anndata loader + MSE_norm evaluator). Both should have a data/public/evaluate.py and a results.json schema with latency_ms and mse_norm respectively so the metric-direction handling in (B14) can be exercised.

B. Correctness / robustness fixes to the existing harness

8. Eliminate shell=True in orchestrator.run_evaluation and the target-agent invocation. Replace the ... | tee pipeline with subprocess.Popen(args=[python, "-u", target_agent_path, "--dataset_dir", ABS_DATASET_DIRECTORY, "--working_dir", current_gen_directory], stdout=PIPE, stderr=STDOUT) plus a Python tee loop writing to stdout_log_file. Removes injection + quoting bugs.

9. Add a target-agent wall-clock timeout (--target_timeout, default 30 min) enforced via subprocess.Popen.communicate(timeout=…) + SIGTERM-then-SIGKILL escalation. Record the timeout outcome in context.md as a first-class failure mode.

10. Convert max_turns to int at the call sites (util.py:24, orchestrator.py:607, 897) and update the type hint on run_agent.

11. Replace nested asyncio.run in ContextManager._generate_llm_summary with asyncio.get_event_loop().run_until_complete only when no loop is running, otherwise schedule via loop.create_task from an async-aware variant await context_mgr.add_generation_async(...). Make add_generation synchronous and defer the LLM summarization to a finalize_async() batch step — this also cuts per-generation latency.

12. Add --resume and --force flags. If runs/run_X/ exists and --resume is set, detect the highest completed gen_N (presence of agent_execution{.json,/} and either results.json or a recorded failure) and restart from there. Persist orchestrator state to runs/run_X/state.json after every phase transition.

13. Pin and cache the bootstrap venv. Move the 9-package list to sia/runtime_requirements.txt with pinned versions. If uv is present, use uv pip install --offline --cache-dir ~/.cache/sia. Add a --skip-venv flag for environments where the orchestrator is itself containerised.

14. Generalize "best generation" selection. Add a metric_spec to each task's data/public/task.yaml: {primary: "mse_norm", direction: "min"}. Replace the hard-coded accuracy lookup in context_manager.finalize with metric-direction-aware selection, and use it to (a) report the winner and (b) implement elitist selection in (C18).

15. Structured run log. Mirror context.md with context.jsonl (one record per generation) so downstream analysis tools don't have to regex Markdown. Move regex stdout scraping behind a --enable-stdout-fallback-metrics flag; default off.

16. Sandbox the target agent. Provide --sandbox {none,firejail,docker}; default firejail when available with --read-only=$DATASET_DIR --bind=$WORKING_DIR --net=none --net=API_HOST_ALLOWLIST. Document that the prompt-only isolation is advisory, not enforced.

17. Expand the feedback-agent context window. Replace "first 3 trajectories truncated to 1000 chars" with: (a) cluster failed trajectories by error type via cheap LLM or by traceback fingerprint; (b) sample 1 representative per cluster + N hardest; (c) include aggregated per-question correctness matrix and confusion patterns. Add a --feedback_context_strategy {first_n,clustered,worst_n} flag.

C. Research / framework extensions

18. Population-based search. Maintain K parallel agent lineages per run; at each generation, mutate from the best-so-far (elitism) plus 1–2 from explorers. Add --population_size, --selection {greedy,elitist,tournament}. This is the standard ADAS / FunSearch shape and immediately addresses the no-rollback problem.

19. Separate meta and feedback models. Add --feedback_model distinct from --meta_model. The feedback role benefits from a strong reasoner (Sonnet, GPT-5, Gemini Pro); the initial scaffold can be drafted by Haiku/Flash.

20. Cost & token accounting. Wrap every run_agent call in a CostTracker that records {model, input_tokens, output_tokens, usd} to runs/run_X/cost.jsonl. Surface totals in context.md. Add --max_usd hard cap.

21. Verifier/reward-model hook. Generic class Verifier(ABC): def score(trajectory, ground_truth) -> float. Built-in implementations: ExactMatchVerifier, LLMJudgeVerifier, UnitTestVerifier, LatencyVerifier. This becomes the upstream signal both for (A2) DPO pair construction and (C18) selection.

22. Determinism / reproducibility. Record git rev-parse HEAD, all package versions, model identifiers, and a seeded random.Random per run into runs/run_X/manifest.json. Today nothing about an outcome is reproducible.

23. CI: smoke test the loop end-to-end. Add a tests/test_e2e_dryrun.py that mocks run_agent to return a stub target_agent.py printing accuracy: 50.0 and runs orchestrator.main(["--task", "gpqa", "--max_gen", "2", "--run_id", "9999"]). Currently CI only tests helpers.

24. MLE-Bench leaderboard scorer. Add sia/eval/mlebench_score.py that maps a generated submission to the canonical MLE-Bench medal computation, so the "ranks #1" claim in the README can be replayed locally.

25. Modernize the Gemini SDK in prepare_mlebench_dataset.py — google.generativeai is deprecated; migrate to google-genai (the package already declared as an orchestrator runtime dep).

26. Type-check and lint the package, not just lint. Add mypy --strict to CI (currently only ruff runs). The codebase is small enough to be fully typed.

27. Public Python API. Expose from sia import Orchestrator, RunConfig so SIA can be embedded in notebooks/pipelines, not only invoked as a CLI. The current sia/__init__.py is 3 lines and exports nothing usable.

6. TL;DR