Models

Learn about language models and explore published model information.

A guide to language models

What happens between a prompt and a response?

Turning text into input

A tokenizer breaks text into units called tokens and maps them to IDs. A token may be a word, part of a word, or a character. The exact split depends on the tokenizer.

Generating a response

An autoregressive language model assigns probabilities to the tokens that could come next. Each selected token becomes part of the context for the following prediction.