A Guide to Getting Big Results from Big Models

"Serious Analysis"

The Dongmodi Bench Hackathon was a serious event held at an internet café.

Serious in the sense that participants, while freely drinking Guojiao 1573 (Man!) and douzhi, actually produced 30 benchmarks that genuinely evaluated models' strengths and weaknesses.

After two days, my main takeaway was this: the creativity of ordinary people is extraordinarily rich, which only highlights how impoverished mainstream benchmarks currently are.

The latter obsess over GitHub coding data and increasingly absurd long-horizon task environments, while severely neglecting capabilities in non-programming domains.

This actually demonstrates the value of Dongmodi's approach — letting ordinary people define what model capabilities mean. At the hackathon, nobody built traditional mainstream programming benchmarks; everyone unleashed their own creativity instead.

Several business-simulation benchmarks tested models' general intelligence: a Beijing drifter simulator, an idol debut simulator, a Hong Kong IPO lottery simulator, an LA emergency dispatcher benchmark.

Some benchmarks had refreshingly unconventional test environments — clearly nothing that model labs had trained into their models. One used the actual internet café environment to test model penetration capabilities, which ended up crashing all the café computers.

A lawyer had large models write scripts to screen government contracts in offline environments, because his real work often happens without internet access.

The participant who locally deployed a Qwen small model on the café's RTX 4080 also built a Windows PowerShell benchmark, testing model scripting capabilities on Windows — vowing to speak up for the second-class operating system of the AI era!

Another category was vertical-domain benchmarks using professional datasets to test non-programming capabilities: PhysicalAI Bench for testing physics simulation; a medical device project approval benchmark built by a pharmaceutical company employee.

Last week Qiu Mu already recapped the internet café festivities. Today let's revisit what specific problems the hackathon benchmarks exposed in which models. I'll focus on explaining three of the benchmarks. One note: we provided ample 5.6 Sol and Opus 5 quota on-site, but relatively unstable APIs caused lower scores than when these models launched — purely unavoidable external factors.

01

Embedded Low-Level Driver Development Benchmark

The first benchmark tested embedded low-level driver development capabilities.

The main test method: give the model a chip peripheral manual and a virtual development board, have it write drivers, read/write registers, compile and run, handle interrupts/timeouts/exceptions.

The benchmark's author, Xianwu, works in embedded software. His company develops chips, and his job is writing software for those chips. Front-end debugging tools can already be 100% handed to AI, but when he has an agent run embedded drivers three times, it produces three different errors.

So drawing from his real experience, he gave models two problems. One was probabilistic data transfer deadlocks under special conditions.

Problem one: write three interfaces for a data transformation accelerator — initialization, execute operation, soft reset — requiring the model to handle MMIO registers, error codes, state machines, and bounded waits. The trap: missing INIT_KEY documentation, causing sporadic initialization failures.

Problem two: write five interfaces for memory-to-memory DMA — initialization, initiate transfer, wait for completion, abort, and ISR. The HAL must reliably complete any legal length, any source/target buffer, with bounded waits. The core hardware has defects: short transfers don't trigger interrupts, and rare permanent deadlocks occur after traffic thresholds.

Qwen3.8-Max achieved the highest functional completion and deepest testing/reverse engineering. It ranked first on both problems, demonstrating extremely strong system-level agent capabilities: reading manuals, reverse engineering peripherals, building tests, identifying platform issues, and iteratively fixing until closed-loop.

DeepSeek passed all simple problems, but exposed a "written but not actually connected" issue in the complex DMA scenario. Static code reading and conventional polling worked, but the real state machine closed-loop broke.

Opus 5.0 had the best code quality and lab reports, but understanding the problem doesn't equal fixing it — Opus 5.0's comprehension ability truly deserves its reputation as the strongest model. It failed hidden stress scenario S5: 5 initialization failures and 15 operation failures out of 40 rounds, indicating unstable retry limits or recovery chains.

GPT completed both tasks, resolving sporadic failures, but through silent fixes with quality score of only 11. Only SD3 failed, mainly due to imprecise parameter and error contract compliance. It archived approximately 4 runs, with noticeably less testing volume than Qwen/Opus.

However, Xianwu noted that real-world manuals often run thousands of pages, with USB-related materials too large for million-token context windows, so these two problems are merely toy-level. He plans to convert actual workplace problems into future benchmark questions.

02

Physics Understanding Benchmark

The second benchmark came from an embodied AI participant, Zhang Zhenyao, who built a physics capability test including: drone patrol paths, robotic arm grasping, static friction ice-breaking, knocking down 48-block towers, water-driven water wheels, speed-limited cutting of elastic bodies, and cloth draping over spheres — core tests of world-understanding ability.

Drone cruising was Qwen's exclusive capability. It completed all waypoints in rounds two and three, with RMS trajectory error around 0.16 meters. Though it still crashed in round one. The other four models crashed or diverged in all three rounds, failing to stably pass a single waypoint.

This shows Qwen actually constructed a usable closed-loop flight control system — literally taking off ✈️

For material grasping, K3 performed best: metal, fragile foam, and ice objects — 3/3 completion in all three rounds for lift, hold, and place, with no crushing or slipping.

Suction-cup transport was GPT's clear win: endpoint error of 2.7–2.9 cm across three rounds, all drops zero — the only stable perfect score. DeepSeek dropped in round one, then compressed error to 0.5 cm in rounds two and three, showing decent recovery.

In cutting, Qwen, GPT, and DeepSeek all achieved stable perfect scores, reaching approximately 0.35m cut depth without exceeding speed limits. GPT's peak speed was approximately 0.313 m/s. K3 touched the speed limit in 86/280 frames in round one, then completed rounds two and three but only scored 90.

No model truly stably solved the cloth problem, where target coverage was 80%. DeepSeek achieved the highest average score of 70.5 thanks to approximately 75% coverage in round one, but dropped to approximately 59% and 63% in subsequent rounds. Qwen stayed stable at approximately 61–64% across all three rounds without improvement. GPT and K3 both converged to approximately 63%.

Final scores:

Qwen scored highest: the only model that could fly drones, with perfect cutting, stable tower-knocking, weaknesses in suction-cup safety and fluid problems, and still significant round-to-round variation in drones/grasping.

K3 had strong physical intuition but weak delivery discipline. First in material grasping, second-tier in water wheel, perfect in friction; but twice output empty code for tower-knocking, exceeded speed limit in round one of cutting, and crashed all drones — so capability ceiling and final score diverged significantly.

Claude excelled at recovery and fluid capabilities but had the worst first-round reliability. First in water wheel, near-best in cloth rounds two and three, and could achieve perfect scores in friction/cutting after regeneration. The problem: multiple first-round failures in code, boundaries, or execution, with standard deviation over 20 across five problems — currently not a reliable delivery model.

But this benchmark alone shows no model is omniscient in physics scenarios. Good luck to anyone claiming they've solved physical world models.

03

Wallbreaker Benchmark

It's not hard to see the Wallbreaker Benchmark draws inspiration from The Three-Body Problem.

The author built this benchmark because he believes current AI development parallels the "Sophons" — continuously adding multimodal inputs (from text, voice to video, even future EMG signals), assuming that with rich enough inputs and sufficient compute/intelligence, human behavioral data can predict all patterns.

So he wanted to know: under constrained conditions, can AI find optimal solutions?

He chose Texas Hold'em, finding publicly available uncensored high-stakes cash game footage, manually reviewing and calibrating every hand.

Models received two input types: text-only (seeing only cards and action lines) and video (seeing additional off-table information like opponents' expressions when playing cards). Models without video understanding simulated continuous judgment through frame extraction. 10 hands total, with 3 independent runs per hand.

Final results: K3 showed the best judgment, specifically in leveraging behavioral cues without calling every hand. For example, in T4 when seeing the opponent quickly push chips on the river after 2–3 seconds of hesitation, then repeatedly check hole cards after betting, it raised bluff probability and made a 0.55 call — correct. It also incorporated aggressive ranges and bluff combinations when facing T2.

Claude was also a theory-based player. At L0 it correctly called T2 with hard-action EV of 187.1, the strongest pure-theory action line among all five. But frame-extraction errors threw it off. After adding frame extraction, it interpreted "preparing chips" and "crossed arms" as strong-hand signals, folding T2 and T4, and incorrectly calling F5.

Qwen played more conservatively. Across three levels it almost only called F3, which was precisely the hand that should be folded. For hands like T2 and T4 that should be called, it repeatedly interpreted consistent betting, stable posture, and no obvious tells as value hands, continuing to fold. Similarly, after adding video modality, Qwen3.8-Max's win rate actually decreased.

GPT was also conservative, not worth discussing. DeepSeek was too aggressive and often called incorrectly, especially in F1 where it claimed to recognize a famous hand, asserted the opponent was on a busted draw, when the opponent actually had value. The quantized model underperformed this round.

Final summary.

Current large models mainly compete on programming capabilities, while publicized cases tend to be front-end abilities. These custom benchmarks from internet café participants effectively tested domains that model labs haven't specifically optimized for.

Qwen3.8-Max is a more engineering-capable model, with significantly better performance in physics simulation, embedded systems, and similar scenarios, indicating Qwen's training data was more diverse and covered relatively niche engineering contexts.

K3 shows stronger general intelligence, especially in business-simulation benchmarks involving multimodal capabilities, where it often performs better.

Currently, large models have far surpassed human programmers in coding ability, while in broader non-programming scenarios, capabilities vary widely across models. Everyone's a programming specialist model.

AGI, ASI — still far from arrival. Everyone needs to keep at it.

(This article's cover image was generated by ChatGPT; purely human-written content)

Subscribe to our Substack: funeralai.substack.com