撒汤智能

Language models and AI research

Inside a language model

What happens between a prompt and a response?

Turning text into input

A tokenizer breaks text into units called tokens and maps them to IDs. A token may be a word, part of a word, or a character. The exact split depends on the tokenizer.

Generating a response

An autoregressive language model assigns probabilities to the tokens that could come next. Each selected token becomes part of the context for the following prediction.

Research notes

Language model basics

How context shapes meaning

The Chinese word 苹果 can refer to a fruit or a technology company. What helps a reader tell them apart?

All research

For business

For a model evaluation or product integration, start with a representative input and the output you expect.

Business enquiries