“Next-token predictor” is the wrong mental model for LLMs

Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete. It’s a fine zeroth-order approximation, and it is grounded in something real: transformer-based language models emit tokens autoregressively: while not done: tokens.append(model.sample_next_token(tokens)) This certainly has the shape of something you might call a next-token predictor. During pre-tra...