LLMs demonstrate a significant drop in performance when the success criterion shifts from "passes all tests" to "would get approved by the maintainer." The time to reach a 50% success rate decreases from 50 minutes to 8 minutes under the more stringent criterion.

LLM-generated Rust code performs a primary key lookup on 100 rows in 1,815.43 ms, significantly slower than SQLite's 0.09 ms. Although the LLM-generated code compiles and passes tests, it is 20,171 times slower for this basic database operation.

Improving coding performance in 15 language models can be achieved by changing the harness used, rather than the models themselves. The harness affects the efficiency and effectiveness of the models, highlighting its role as a critical factor in AI coding capabilities.