Research

How context shapes meaning

The Chinese word 苹果 can refer to a fruit or a technology company. What helps a reader tell them apart?

Consider the sentences “I peeled an apple” and “Apple released a new phone.” Chinese writes both uses of apple the same way: 苹果. Peeling points to a fruit. Releasing a phone points to a company.

A Transformer language model processes text as tokens. At each position, attention layers combine information from other positions. The same token can therefore acquire different representations depending on the surrounding sentence.

Those representations influence the next-token prediction. They do not guarantee a correct answer. To explore how a particular model uses context, keep the target word and change one or two clues around it, then compare the responses.