CrucibleBench places language models in a persistent MUD environment where NPCs can remember interactions, accumulate trust, and leave traces of mistakes. The system evaluates AI performance over 50 turns with hidden social objectives, utilizing established technology in innovative ways.

High-quality benchmarks are being designed to assess cognitive abilities of frontier models, focusing on reasoning, acting, and judgment rather than just recall. These benchmarks aim to measure progress toward artificial general intelligence (AGI).

Arguments exist for valuing human creativity over AI-generated output, suggesting that humans excel in certain roles due to their ability to perform tasks with greater nuance and style. Despite similarities in output, human artifacts are often preferred for their subtle artistic qualities that AI may struggle to replicate.

The AI Marketing BS Index provides a scoring system to evaluate marketing claims related to AI, similar to John Baez’s Crackpot Index for physics. It aims to identify and critique exaggerated or misleading marketing language in the AI industry.

A set of ten research-level mathematics questions has been created to evaluate the capabilities of current AI systems in providing correct answers. The answers to these questions are known to the authors but will remain encrypted temporarily.