LLM Classification Is Feature Engineering

Overview

LLMs-as-classifiers, prompts applied to a context and returning a label, suck to work with. This is especially painful because they often perform pretty decently. But let’s consider some of the things we’d want in a classifier and see how an LLM-as-classifier stacks up: - Calibration / Threshold Control - LLM verdicts are often hard labels; you can get token log probabilities but there is no mecha...

Summary from minimallysufficient.com.