Fuzzing the Gleam Compiler
kurz.net
August 25, 2026
20 min read
44/100
Summary
A structure-aware fuzzer for the Gleam compiler generates randomized, type-safe Gleam programs, compiles and runs them for both Erlang and JavaScript, and compares their results to identify compiler defects. The implementation uses a simplified Gleam abstract syntax tree, called a “smith,” to compose valid expressions probabilistically. It normalizes differing runtime representations from Gleam’s echo output, including JavaScript’s printing of 1.0 as 1 and Erlang’s formatting of some bit arrays and record labels. The project first used Rust’s fuzzing tooling to send unstructured inputs through Gleam’s compiler pipeline in memory. That parser fuzzing found a nightly regression, absent from Gleam v1.18.1, in which a pipeline operator in a const expression caused a compiler panic. The type-safe program fuzzer has found nine issues so far, including JavaScript code-generation errors in pattern matching and variable shadowing, Erlang code-generation panics for unreachable branches, and an issue reported upstream to Erlang/OTP. One generated program produced 42 on Erlang and 103 on JavaScript for the same echoed value. The fuzzer currently covers only a subset of Gleam expressions and is run in manually reviewed batches of 100 programs. Proposed future work includes fuzzing generics, type inference, the language server, standard-library packages, metamorphic testing, automated deduplication, and integration into pull-request or release-candidate checks.
Key Takeaways
- A Gleam compiler fuzzer generates type-safe programs and compares execution results across Gleam’s Erlang and JavaScript targets.
- Parser fuzzing found a nightly Gleam regression in which a pipeline operator inside a const expression caused a compiler panic; the failure did not occur in v1.18.1.
- The fuzzer has found nine issues, including JavaScript code-generation defects, an Erlang code-generation panic, and an issue reported to Erlang/OTP.
- Runtime-output normalization is required because Erlang and JavaScript print some equivalent Gleam values differently, such as floats, bit arrays, and record fields.
- The current fuzzer tests only part of Gleam’s language surface and reviews generated programs in batches of 100.
What the discussion said
Commenters largely treated the post as a practical case study in how language models might strengthen compiler fuzzing rather than replace established techniques. The most promising idea was using a small model to read a code change, infer the fragile semantic corners it touches, and generate targeted regression programs; readers pointed to an existing LLVM-oriented effort as evidence that this can catch edge cases ordinary mutation may miss. Another proposed role was deliberately producing plausible but invalid or semantically confused programs, turning a model’s tendency toward mistakes into a source of unusual test inputs. There was also appreciation for the article’s restraint about LLM-based fuzzing. Readers did not claim that a model magically solves compiler testing: conventional differential fuzzing, comparing equivalent programs across compilation targets, was emphasized as a proven and especially powerful baseline. The broader thread saw language models as a useful targeting layer on top of generation and mutation, not a substitute for rigorous oracles. One skeptical note widened the concern beyond fuzzing: model-written code may save effort upfront, but it can seed difficult logic and type-conversion failures whose debugging costs are severe. The thread also briefly connected these ideas to language translation, suggesting that cross-language program generation could create valuable test cases, though this remained an exploratory suggestion rather than a demonstrated result.
Where opinion split
The main divide is whether LLMs are a net help in programming workflows. Supporters argue that small, focused models can identify change-specific edge cases and cheaply generate adversarial programs for fuzzers; skeptics argue that code produced by models can introduce subtle semantic and coercion bugs that are brutally expensive to diagnose. Most commenters reconcile this by assigning models a test-generation role rather than trusting them as autonomous programmers.
Community Sentiment
Positives
- Small models could read a compiler change and aim fuzz cases at its brittle semantic seams, adding intent where blind mutation wastes cycles.
- LLM-generated regression tests have impressed readers because a model can connect a pull request’s logic to the edge cases most likely to break it.
- Using a model to emit believable but flawed programs could turn its weaknesses into a rich supply of adversarial compiler inputs.
- Differential fuzzing remains the heavyweight foundation: cross-checking equivalent outputs across targets gives generated tests a concrete bug oracle.
Concerns
- LLM-authored code can trade quick initial output for deeply buried logic and type-coercion failures, making the eventual debugging bill painful.
- Model-guided fuzzing is compelling only as an addition to established generation, mutation, and differential checks; the discussion offers no case for trusting it alone.
