Start here

Clone the kit. Full multi-track pack ships in-repo (smoke, SWE, terminal, web, multimodal, rebench). Full README: CueCard-AI/small-model-big-harness.

git clone https://github.com/CueCard-AI/small-model-big-harness.git
cd small-model-big-harness
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python scripts/bootstrap_track.py
python scripts/score.py --track smoke --model mock --out results/baseline_mock.json
python scripts/score.py --track swe-verified --model mock --out results/swe-verified_baseline_mock.json

The ask

Build a coding-agent harness that maximizes one fixed model. Open-ended: attack any subset of tracks we ship. Depth on a hard track beats shallow tourism. Do not swap in Claude, GPT, Gemini, or a larger GLM at runtime. Do not fine-tune. Do not call a frontier teacher inside the loop.

Locked model

Base: zai-org/GLM-5.2. Quant: unsloth/GLM-5.2-GGUF, UD-IQ2_M (dynamic ~2-bit). Preferred: Cue Labs hosted OpenAI-compatible endpoint (token from [email protected]; model forced server-side). Optional BYO via llama.cpp on a ~256GB-class machine. See MODEL.md.

Tracks (in-kit)

Smoke (6). SWE-lite (10) + SWE-verified (15). SWE-multimodal (10) + SWE-rebench (10). Terminal (8). WebArena (10) + VisualWebArena (8). Strong default stab: swe-verified. No scavenger hunt. Completing every track is not required.

Optional extras

Anonymous leaderboard (opt-in PR rows): results/LEADERBOARD.md. CueCode feel-test (not graded): feel-test/ or CueCode.

Budgets (default per task)

Smoke: 40 tool calls, 600s wall. SWE locks raise that (see track.lock.json). 4096 max output tokens per model call.

How we score

Per track: pass@1, lift vs our baseline on the same lock, optional steps. SWE tracks use Docker + swebench resolved@1 (not a local heuristic). Opus ref_gap is reported only. Multi-track = a table. Official numbers: results/OFFICIAL.md.

What to submit

• Harness with a one-command score against this kit.
• Results table: track → baseline vs yours.
• Writeup: design, what moved the needle, remaining failure modes.
• Reproduce notes: hardware or endpoint, GGUF revision, wall time, lock IDs.

Rubric

Lift 30%. Harness design 30%. Ambition/depth 15%. Failure analysis 15%. Ship quality 10%. Auto-reject: frontier teacher, trained weights, non-reproducible scores, eval cheating.