← All topics
inference-apis
Latest stories covering inference-apis, with summaries and links to related coverage.
2 stories · 2 reports · page 1 of 1
Connected topics
The session you cannot take with you
opinion
14 min read
Inference APIs allow users to send input and receive output, facilitating conversation archiving and replaying. However, challenges such as prompt cache storage on external GPUs, differing tokenization between models, and intentional sampling unreproducibility complicate session portability.
How an inference provider can prove they're not serving a quantized model
tool
10 min read
Tinfoil verifies the identity of models served through inference APIs, ensuring users receive the exact model weights as published. It addresses concerns about potential modifications, such as quantization or altered context windows, in open-source models.