# Model comparison: data tables

Generated by `make-figures.mjs` from the study's `results.json`, model folders and `plan.json`. Do not edit by hand.

## Setup

| Item | Value |
|---|---|
| Claude Code (wireless-101) | 2.1.301 (Claude Code) |
| Claude Code (stats-101) | 2.1.301 (Claude Code) |
| Claude Code (stats-101-planned) | 2.1.302 (Claude Code) |
| Ollama | 0.40.5 |
| Machine | Apple M4 Max, 64 GB |
| Plugin | present-in-html 3.1.0 (commit ae03c20) |
| Time limit (wireless-101) | 120 min |
| Time limit (stats-101) | 120 min |
| Time limit (stats-101-planned) | 240 min |
| Cloud budget cap | $20 per run |
| Judge | claude-opus-5-5 (blind) |
| Run wireless-101 | 20261016-000644 |
| Run stats-101 | 20261016-063639-stats-101 |
| Run stats-101-planned | 20261016-214018-stats-101-planned |

# Main comparison (Wireless 101 and STATS 101, no plan)

These tables cover the two main tasks only; Figs 1-5 are drawn from them. The plan experiment has its own tables below.

## Runs

| Task | Model | Status | Wall | API | Cost | Turns | Tool calls | Output tokens |
|---|---|---|---|---|---|---|---|---|
| Wireless 101 | Opus 5.5 (`claude-opus-5-5`) | finished | 11m 44s | 9m 57s | $2.88 | 34 | 32 | 63296 |
| Wireless 101 | Sonnet 5.5 (`claude-sonnet-5-5`) | finished | 5m 27s | 4m 24s | $0.90 | 29 | 27 | 38188 |
| Wireless 101 | Haiku 5.5 (`claude-haiku-5-5`) | finished | 7m 05s | 5m 22s | $0.24 | 37 | 35 | 68384 |
| Wireless 101 | qwen3.8 27B (`qwen3.8:27b-mlx`) | timed out | 120m 00s | – | $0.00 | – | 19 | – |
| STATS 101 | Opus 5.5 (`claude-opus-5-5`) | finished | 11m 14s | 8m 51s | $2.79 | 32 | 31 | 56857 |
| STATS 101 | Sonnet 5.5 (`claude-sonnet-5-5`) | finished | 5m 24s | 3m 52s | $0.70 | 22 | 21 | 29603 |
| STATS 101 | Haiku 5.5 (`claude-haiku-5-5`) | finished | 8m 44s | 6m 48s | $0.32 | 46 | 45 | 90031 |
| STATS 101 | qwen3.8 27B (`qwen3.8:27b-mlx`) | timed out | 120m 00s | – | $0.00 | – | 22 | – |

## Decks (finished runs)

| Task | Model | Slides | Notes | Diagrams | Animated | Placeholders | Words on slides | Words in notes | Quiz | Checker |
|---|---|---|---|---|---|---|---|---|---|---|
| Wireless 101 | Opus 5.5 | 30 | 30/30 | 13 | 19 | 3 | 1616 | 3042 | yes | pass (0 issues) |
| Wireless 101 | Sonnet 5.5 | 30 | 30/30 | 10 | 14 | 2 | 1419 | 2236 | yes | pass (0 issues) |
| Wireless 101 | Haiku 5.5 | 29 | 29/29 | 4 | 14 | 3 | 1466 | 1763 | yes | pass (0 issues) |
| STATS 101 | Opus 5.5 | 30 | 30/30 | 11 | 20 | 2 | 1513 | 2962 | yes | pass (0 issues) |
| STATS 101 | Sonnet 5.5 | 29 | 29/29 | 7 | 14 | 2 | 1392 | 1682 | yes | pass (0 issues) |
| STATS 101 | Haiku 5.5 | 29 | 29/29 | 7 | 10 | 2 | 1606 | 1973 | yes | pass (0 issues) |

## Judge scores (finished decks)

| Task | Model | Accuracy | Coverage | Teaching | Visual | Diagrams | Notes | Requirements | **Overall /10** | Judge cost |
|---|---|---|---|---|---|---|---|---|---|---|
| Wireless 101 | Opus 5.5 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | **9** | $0.71 |
| Wireless 101 | Sonnet 5.5 | 5 | 5 | 4 | 4 | 4 | 5 | 5 | **9** | $0.68 |
| Wireless 101 | Haiku 5.5 | 4 | 4 | 4 | 3 | 3 | 5 | 5 | **7** | $0.71 |
| STATS 101 | Opus 5.5 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | **9** | $0.66 |
| STATS 101 | Sonnet 5.5 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | **9** | $0.61 |
| STATS 101 | Haiku 5.5 | 4 | 4 | 4 | 3 | 4 | 5 | 4 | **7** | $0.74 |

## Judge findings (finished decks)

| Task | Model | Wrong claims (major) | Invented claims (untrue) | UI issues (major) | Major mistakes |
|---|---|---|---|---|---|
| Wireless 101 | Opus 5.5 | 2 (0) | 5 (0) | 4 (0) | 0 |
| Wireless 101 | Sonnet 5.5 | 0 (0) | 3 (0) | 7 (0) | 0 |
| Wireless 101 | Haiku 5.5 | 5 (1) | 4 (1) | 7 (1) | 2 |
| STATS 101 | Opus 5.5 | 1 (0) | 6 (0) | 4 (0) | 0 |
| STATS 101 | Sonnet 5.5 | 0 (0) | 4 (0) | 6 (1) | 1 |
| STATS 101 | Haiku 5.5 | 1 (0) | 3 (0) | 6 (2) | 2 |

## Unfinished runs: partial progress

| Task | Model | Status | Wall | Slides written in output/parts | With notes | Files | PLAN.md written | Template deck.html scored |
|---|---|---|---|---|---|---|---|---|
| Wireless 101 | qwen3.8 27B | timed out | 120m 00s | 0 | 0 | – | yes | 21 slides, checker pass, judge 1/10 |
| STATS 101 | qwen3.8 27B | timed out | 120m 00s | 20 | 17 | part1.html, part2.html | no | 21 slides, checker pass, judge 1/10 |

## Mean criterion score per model (finished decks, main tasks only; Fig. 4)

| Model | n | accuracy | coverage | teaching | visual | diagrams | notes | requirements | Overall |
|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | 2 | 5.0 | 5.0 | 5.0 | 4.0 | 5.0 | 5.0 | 5.0 | 9.0 |
| Sonnet 5.5 | 2 | 5.0 | 5.0 | 4.5 | 4.0 | 4.5 | 5.0 | 5.0 | 9.0 |
| Haiku 5.5 | 2 | 4.0 | 4.0 | 4.0 | 3.0 | 3.5 | 5.0 | 4.5 | 7.0 |

# Plan experiment (stats-101-planned): local models building from a plan written by Opus

A separate experiment to improve local models, not compared with the runs above. Three local models built STATS 101 from the same Opus plan, with a 240-minute limit (Fig. 6).

## Shared plan (plan experiment)

| Planner | Time | Cost | Turns | Output tokens | Size |
|---|---|---|---|---|---|
| claude-opus-5-5 | 23m 20s (1400 s) | $5.05 ($5.046) | 51 | 101973 | 94 KB |

Note: after planning: D set to $PWD (was the planner temp folder) and the deck renamed deck.html, to match the build prompt.

## Runs (plan experiment)

| Task | Model | Status | Wall | API | Cost | Turns | Tool calls | Output tokens |
|---|---|---|---|---|---|---|---|---|
| STATS 101 + plan | qwen3.8 27B (`qwen3.8:27b-mlx`) | API error | 61m 03s | 61m 01s | $0.00 | 13 | 9 | 11099 |
| STATS 101 + plan | qwen3.6 35B-A3B (`qwen3.6:35b-a3b`) | finished | 20m 07s | 19m 37s | $0.00 | 31 | 30 | 70144 |
| STATS 101 + plan | gemma4 26B (`gemma4:26b-mlx`) | stopped early | 72m 34s | 72m 33s | $0.00 | 18 | 17 | 54053 |

## Decks (finished runs) (plan experiment)

| Task | Model | Slides | Notes | Diagrams | Animated | Placeholders | Words on slides | Words in notes | Quiz | Checker |
|---|---|---|---|---|---|---|---|---|---|---|
| STATS 101 + plan | qwen3.6 35B-A3B | 30 | 30/30 | 10 | 20 | 2 | 1458 | 2830 | yes | pass (0 issues) |

## Judge scores (finished decks) (plan experiment)

| Task | Model | Accuracy | Coverage | Teaching | Visual | Diagrams | Notes | Requirements | **Overall /10** | Judge cost |
|---|---|---|---|---|---|---|---|---|---|---|
| STATS 101 + plan | qwen3.6 35B-A3B | 5 | 5 | 5 | 4 | 4 | 4 | 5 | **9** | $0.70 |

## Judge findings (finished decks) (plan experiment)

| Task | Model | Wrong claims (major) | Invented claims (untrue) | UI issues (major) | Major mistakes |
|---|---|---|---|---|---|
| STATS 101 + plan | qwen3.6 35B-A3B | 0 (0) | 4 (0) | 5 (0) | 1 |

## Unfinished runs: partial progress (plan experiment)

| Task | Model | Status | Wall | Slides written in output/parts | With notes | Files | PLAN.md written | Template deck.html scored |
|---|---|---|---|---|---|---|---|---|
| STATS 101 + plan | qwen3.8 27B | API error | 61m 03s | 10 | 10 | 01-opening.html | given | not judged |
| STATS 101 + plan | gemma4 26B | stopped early | 72m 34s | 10 | 10 | 01-opening.html | given | not judged |

Times are wall-clock. Costs are the CLI's API list-price figure (local runs $0 per token); the judge's cost is separate and not included in the model's cost. One run per model per task.