Notes
- Average Inference Time: 25m 16s (1516.2s)
- GPT-5.5 Pro averaged 21m 23s (1283.3s) for context; so slightly longer inference times
- Total Cost (for 15 builds): $710.82 ($47.39 per build)
- Most expensive model benchmarked to-date; previous was GPT-5.5 Pro at $223.90
- Thanks to all supporters for helping fund the benchmark!
Subjectively speaking, GPT-5.6 Sol seems to create the most detailed builds MineBench has seen thus far, while for the most part doing so with great creative choices. I think, personally, there are only a handful of builds I would argue are not clear improvements over GPT-5.5 Pro (like the astronaut and worldtree). On average, GPT-5.6 Sol also creates the largest JSON files across all of its builds by a significant portion.
That being said, this model was also the most expensive model MineBench has benchmarked to date; the previous most expensive model was GPT-5.5 Pro at $223.90 – so 5.6 Sol totaled to being over 3x as expensive. If you're lucky enough to ignore the cost, then yes, the model created the most detailed generations yet. For example, in its cottage build, it added a scarecrow in the garden, added clothes drying on a rack, etc. Its builds also seemed to have a better sense of scale and proportions overall, like the arcade.
We might benchmark GPT-5.6 Terra if there's enough interest, as that would technically be a closer comparison to GPT-5.5 (as Sol is technically the successor to GPT-5.5 Pro, which would also explain the cost).
TLDR: Model is amazing, doesn't tend to be conservative (good or bad depending on your use case), but it's extremely expensive.
Full release-notes/thoughts on the GitHub release
- If you enjoy these posts please feel free to help fund the benchmark
- Sharing the benchmark and starring the Git repository also helps :)
Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench
Previous Posts:
- Comparing Opus 4.8 and Fable 5
- Comparing Opus 4.7 and Opus 4.8
- Comparing GPT 5.4 and GPT 5.5
- Comparing Kimi K2.5 and Kimi K2.6
- Comparing Opus 4.6 and Opus 4.7
- Comparing GPT 5.4 and GPT 5.4-Pro
- Comparing GPT 5.2 and GPT 5.4
- Comparing GPT 5.2 and GPT 5.3-Codex
- Comparing Opus 4.5 and 4.6, also answered some questions about the benchmark
- Comparing Opus 4.6 and GPT-5.2 Pro
- Comparing Gemini 3.0 and Gemini 3.1
Extra Information (if you're confused):
Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.
So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.
The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.
(Disclaimer: This is a public benchmark I created, so technically self-promotion : )