Language model basics
How context shapes meaning
The Chinese word 苹果 can refer to a fruit or a technology company. What helps a reader tell them apart?
What happens between a prompt and a response?
A tokenizer breaks text into units called tokens and maps them to IDs. A token may be a word, part of a word, or a character. The exact split depends on the tokenizer.
An autoregressive language model assigns probabilities to the tokens that could come next. Each selected token becomes part of the context for the following prediction.
The Chinese word 苹果 can refer to a fruit or a technology company. What helps a reader tell them apart?
For a model evaluation or product integration, start with a representative input and the output you expect.
Business enquiries