benchmarks.arnao.ai
AI Model Benchmarks
& Mission Control
Live tracking of frontier AI models — arena rankings, cost benchmarks, and internal fleet performance data.
Mission Control
🚀 Latest Releases
- Sakana Fugu & Fugu Ultra — Jun 22, 2026Multi-agent orchestration model
- GLM 5.2 — Jun 16, 2026ZhipuAI’s latest model
- Claude Opus 4.8 — Jun 2026Anthropic’s new flagship
🏆 Arena Top 3
- Claude Fable 5ELO 1508
- Claude Opus 4.6 (Thinking)ELO 1504
- Claude Opus 4.7 (Thinking)ELO 1502
⚡ Fleet Winner
🏆 DeepSeek V4 Flash
Best cost/performance ratio in internal fleet tests.
Avg Time: 2.9s • Cost: $0.00010 / 4 tasks
LMSYS Chatbot Arena — Top 20
| # | Model | Provider | ELO Score |
|---|
Master Model Catalog
| Model | Provider | Class | Arena ELO | In $/1M | Out $/1M | Context | Release | Status |
|---|
Routing Stack Benchmark — Jul 7, 2026
| Combo | Planning / Execution | Refactor (A) | Classify (B) | Cost/Run | Latency |
|---|---|---|---|---|---|
| Sonnet Duo | sonnet-4-6 / sonnet-4-6 | 100/100 | 77/100 | $0.084 | 38s + 20s |
| Ollama Stack 🏆 | qwen3-coder:30b / mistral-small3.2 (local) | 100/100 | 83/100 | $0.000 | 78s + 75s |
| OpenRouter Value | glm-4.6 / gemini-2.5-flash | — (credits out) | 83/100 | $0.0003 | 15s |
| Gemini Duo | gemini-2.5-pro / 2.5-flash | 60/100 | 81/100 | $0.017 | 30s + 16s |
| Fable+Gemma | claude-fable-5 / gemma4 (local) | 50/100 | 71/100 | $0.022 | 20s + 48s |
- HeadlineThe all-local Ollama stack tied frontier quality on both tasks at zero marginal cost. Latency is the only tradeoff (2-4x slower).
- DoctrineCron and batch jobs route local; interactive execution routes to Flash-class; interactive planning routes to Sonnet. Enforced fleet-wide by Ria via routing-rules.json.
- CaveatFable 5 planning scored 50/100 due to probabilistic safety refusals on code-analysis prompts; strong orchestrator, weak pipeline planner. Benchmarks A/B graded fully automatically (pytest, mypy, AST, gold labels).
Image Model Benchmark — Jul 11, 2026
🏆 Winner: grok-imagine-image-quality (xAI) — nailed text, cat, and puddle count at 6.2s / $0.05
Standard Prompt — identical for every model (stresses exact text, counting, spatial relations, reflections, style)
A rain-slicked Tokyo alley at blue hour, seen from a low three-quarter angle. In the foreground, a silver-haired woman in a transparent vinyl raincoat holds a glowing paper lantern shaped like a koi fish; its warm light reflects in exactly three visible puddles. Behind her, a robot barista leans out of a tiny stall serving matcha, with a hand-painted wooden sign that reads exactly 'ARNAO AI LAB' in weathered white letters. A black cat walks along a power line overhead, silhouetted against neon kanji signs. Shot on 35mm film, shallow depth of field focused on the lantern, cinematic teal-and-amber grade, visible rain streaks, steam rising from a manhole.
| Model | Provider | Sign Text | 3 Puddles | Cat on Wire | Aesthetic | Adherence | Latency | Cost |
|---|---|---|---|---|---|---|---|---|
| grok-imagine-image-quality 🏆 | xAI | ✓ | ✓ | ✓ | 9 | 9 | 6.2s | $0.050 |
| gpt-5.4-image-2 | OpenAI (OR) | ✓ | ✓ | ✓ | 9 | 9 | 180.2s | $0.228 |
| gemini-3-pro-image-preview | ✓ | ✗ | ✓ | 9 | 9 | 19.5s | $0.134 | |
| gemini-3.1-flash-image | Google (OR) | ✓ | ✗ | ✓ | 9 | 8 | 7.7s | $0.069 |
| grok-imagine-image | xAI | ✓ | ✗ | ✓ | 8 | 8 | 8.0s | $0.020 |
| gemini-3.1-flash-lite-image | Google (OR) | ✓ | ✗ | ✓ | 8 | 7 | 2.6s | $0.034 |
| gpt-image-1 | OpenAI | ✓ | ✗ | ✗ | 8 | 7 | 42.0s | $0.167 |
| gpt-5-image-mini | OpenAI (OR) | ✓ | ✗ | ✗ | 8 | 6 | 46.4s | $0.044 |
| gemini-2.5-flash-image | ✗ | ✗ | ✗ | 8 | 7 | 6.3s | $0.039 | |
| dall-e-3 | OpenAI | FAIL — model retired by OpenAI (“does not exist”) | ||||||
| grok-2-image | xAI | FAIL — retired, replaced by grok-imagine-image* | ||||||
| moonshot / deepseek | — | No image-generation API offered | ||||||
| local (ollama / host) | — | None local — ollama is text-only, no ComfyUI/mflux installed | ||||||
- HeadlinexAI’s grok-imagine-image-quality matched gpt-5.4-image-2’s perfect checklist at 1/30th the latency and 1/4.5th the cost. Gia’s image stack now routes winner → Nano Banana Pro → Nano Banana 2.
- NotableEvery 2026-era model except gemini-2.5-flash-image rendered “ARNAO AI LAB” exactly — text rendering is largely solved. Counting (“exactly three puddles”) remains the hardest test: only 2 of 9 passed.
- Catalog gapOpenRouter carries no Chinese image-output models today (no Qwen-Image, CogView, Seedream, or Hunyuan-Image) — only Google nano-banana and OpenAI GPT-5-image families.
- Harnessscripts/benchmarks/image-bench.py — idempotent, per-model reruns via --model, fail-soft error capture. Raw PNGs + results.json in workspace/benchmarks/images/2026-07-11/.
Internal Fleet Benchmark — Jun 21, 2026
| Model | Avg Time | Cost / 4 Tasks | Pass Rate | Notes |
|---|
Research & Deep Dives
- Sakana FuguAn orchestration model that routes tasks to a team of specialist models. The “satanic fugu” concept in practice.
- AgentOSA unified command center for managing multiple agents with persistent memory and a shared interface across AI models. A project at agentos.arnao.ai is feasible.
- Model MonitoringA lightweight cron polls RSS from Hugging Face, arXiv, and top AI labs every 4 hours to surface new releases automatically.