Picking a Codex model by gut feel? Turn your own past sessions into a small eval set first

#What changed

Ibragim (@ibragim_bad on X), who builds coding-agent evals and the SWE-rebench leaderboard at Nebius, published a step-by-step guide on October 16 for turning your own agent history into a private benchmark. He exported three months of sessions from Claude Code, Codex CLI, and Pi, had an agent list candidate tasks, kept the ones with an unambiguous automatic check, and ended up with 10 tasks run three times each in Harbor, an open-source runner for agent evals.

His headline result: on those 10 tasks, Pi running GPT-6.1 Sol solved everything for about $0.070 per task versus about $0.138 for Codex on the same model. He also released an agent skill, evals-from-sessions, that automates most of the steps and installs into Codex as a plugin.

The useful part is the method, not the leaderboard. The sessions where the agent got something wrong and you corrected it make the best test cases, because your correction tells you exactly what the check has to catch.

#Who is affected

Anyone who uses Codex CLI regularly and is choosing between models, reasoning levels, plugins, or harnesses. Codex already keeps a JSONL transcript of each session under ~/.codex/sessions/YYYY/MM/DD/ (the rollout recorder in the open-source Codex code writes them there), so most heavy users already have the raw material.

#What to do now

• Copy your ~/.codex/sessions folder somewhere else before you start, and scan the copy for API keys and other secrets (the guide uses Gitleaks) before sharing it with any tool or teammate.
• Have an agent skim the copy and list candidate tasks: what you asked for, what it got wrong, how you fixed it, and whether the result can be checked automatically. Keep only tasks that look like your normal work and have a clear pass or fail.
• Test each check before trusting it: doing nothing should fail, your reference solution should pass every time, an almost-right solution with one bug should fail, and a different correct solution should pass.
• Block easy cheating, such as leaving future commits or the reference diff in the sandbox, then run each setup several times and compare cost and time only among setups that pass.
• Do the extraction in a separate session or on the copy. A Codex GitHub issue (#27131) describes a session that read its own transcript files and ballooned its token use.

#What is not confirmed

The roughly 2x cost difference comes from one person's 10 tasks and three runs each. It is a field report and DotsFeed has not reproduced it. It says which setup suits his work, not which model or agent is best overall, and your numbers will differ. We confirmed that the evals-from-sessions repository and its Codex plugin install instructions exist and that Codex stores session transcripts at the path above, but we have not run the skill ourselves.

#Sources

• @ibragim_bad on X: https://x.com/ibragim_bad/status/2111102467924496721
• evals-from-sessions skill: https://github.com/ibragim-bad/evals-from-sessions
• Codex rollout recorder (session path): https://github.com/openai/codex/blob/main/codex-rs/rollout/src/recorder.rs
• Codex issue #27131 (self-ingesting session logs): https://github.com/openai/codex/issues/27131