Models
Learn about language models and explore published model information.
A guide to language models
What happens between a prompt and a response?
Turning text into input
A tokenizer breaks text into units called tokens and maps them to IDs. A token may be a word, part of a word, or a character. The exact split depends on the tokenizer.
Generating a response
An autoregressive language model assigns probabilities to the tokens that could come next. Each selected token becomes part of the context for the following prediction.