Benchmarks/EFE-98

EFE-98

SimulationCode checkReleased Nov 2026Index weight 0.72
PlaygroundMaintainer ↗

A 98-test short form of Embodied First Exam's standard mode, from its 18 cheapest runnable suites, frozen before any model was run. Coding agents only.

As of November 20, 2026, GPT-6.2 Astra leads with 76.5, followed by Claude Fable 5.2 (50.0) and GPT-6 Astra (38.5). Official leaderboard ↗

Leaderboard

Tasks
98
Attempts / task
1
Median cost / task
$2.86
Median time / task
33 min 49 s

Share of tests passed (%), one attempt per test. GPT-6.2 Astra was run by Playground on all 98 tests. The other scores are computed from Embodied First Exam's published per-test results on the 96 tests that could be matched. Scores of models with a full exam score count only as opponents; since November 20 that includes GPT-6.2 Astra, and Qwen3.8 Max, which the exam no longer lists, counts here.

Run it yourself

EFE-98 runs on Embodied First Exam's own runner and pass or fail checks, unchanged. You need the exam at the version below and the list of 98 tests.

Exam version
ebd10b6
Mode
Standard, 1 attempt
GPT-6.2 Astra cost
$499
GPT-6.2 Astra time
7 h 54 min
  1. Set up the exam

    Clone Embodied First Exam at the version we froze the test list against, then follow its README to install the runner and build the simulator images.

    git clone https://github.com/mll-lab-nu/embodied-first-exam
    cd embodied-first-exam
    git checkout ebd10b62b0f4d5c299c3249555d2746e9eb51880
  2. Get the test list

    One test name per line, 98 in all, across 18 of the exam's suites.

    curl -O https://withplayground.ai/leaderboard/efe-98/tests.txt
    Download tests.txt ↓
  3. Run each test once

    Standard mode, one attempt per test, with the exam's default time budgets, tools and network rules. These are the settings of our GPT-6.2 Astra run; set AGENT, MODEL and EFFORT for the entry you are testing.

    while read -r test; do
      AGENT=codex MODEL=openai/gpt-6.2-astra EFFORT=high MODES=standard \
      JOBS="$PWD/efe98-jobs" RESULTS="$PWD/efe98-results" \
        bash scripts/run_agent_batch.sh efe98 "$test"
    done < tests.txt
  4. Score it

    The score is tests passed divided by 98. A test stopped before grading by an infrastructure failure, an API error or a usage limit is discarded and rerun once. A test that was graded keeps its result.

What the machine needs.

  • A machine that runs Docker with NVIDIA GPUs. We use four RTX A6000 cards on Ubuntu 22.04.
  • Four suites use Isaac Sim, which needs NVIDIA's graphics libraries (libnvidia-gl) at the same version as the installed driver.
  • The runner's scripts need Python 3.11 or newer.
  • To run tests side by side on several GPUs, apply our nine-line runner patch and set GPU_DEVICE for each lane. As shipped, every test asks Docker for the first GPU. gpu-pin.patch ↓

Get listed. To have a result listed here, send the run's jobs folder, which holds each test's result and agent log, to contact@withplayground.ai.

Tasks

98 tasks. Hover a task to play its clip (98 of 98).

98 tasks · 98 with clips

Methodology

Scoring
Code check
Time cap / attempt
1 h 00 min
Control rate
—
Models in index
2 of 11

Scoring. Embodied First Exam's own simulator success function after the agent's last step, pass/fail.

Who runs it. Playground runs each new entry with the exam's own runner and verifier.

Robots. Simulated robots across 18 of Embodied First Exam's suites.

Note. Counts only for models with no full Embodied First Exam score. The exam's ten published agents are listed, from their published per-test results, as opponents only.

Time per attempt. Published or measured for 2 of 2 models.

Cost per attempt. Published for 2 of 2 models.

Cost and time for the other entries. The exam's ten published agents have no result of their own here. Their cost and time per test are the exam's published per-suite averages (its own cost estimate and agent time, over every graded trial in all modes), weighted by EFE-98's tests per suite, so they describe this test mix rather than a run of it. GPT-6.2 Astra's cost is its logged tokens at API list price, a different method from the exam's estimate.