Word tokens, dictionary forms, and definitions

Tokens are not always dictionary words

When Valo Reader processes a story, it runs the text through a tokenizer that splits the content into study units called tokens. A token is not always a word in the dictionary sense. Depending on the language, a token might be:

• A conjugated verb form as it appears in the text
• A grammatical particle attached to a noun
• A phrase treated as a single word
• A punctuation-bound segment in languages without spaces

This means that tapping a word in the reader shows you what the tokenizer found, which may include grammatical information that is not part of the dictionary entry. In languages with rich morphology, a single token may pack several grammatical meanings into one word.

Word details

When you tap or click a token, the word details popover or dialog opens. It typically contains:

• Original form — the token as it appears in the text
• Dictionary form — the base form you would look up in a dictionary
• Readings — phonetic readings where applicable, such as Japanese readings
• Definitions — contextual glosses matched to the dictionary form
• Familiarity control — the current level and a control to adjust it

Not every field is available for every token. Dictionary coverage varies by language and by the specific word. Common words and standard conjugations have the best coverage; rare or highly idiomatic expressions may show less information.

How tokens relate to vocabulary

Each token maps to a study key, often derived from its dictionary form, and the study key carries the familiarity level. This means that seeing a conjugated form of a word you already know can still use the same vocabulary item. The exact mapping depends on the language tokenizer.

However, tokens that are genuinely different words (homographs, different etymologies) are tracked separately, even if they look identical in the text.

Cross-story vocabulary state

Changing a word's familiarity updates your per-language vocabulary state. That state is shared across all stories in the same study language, so a promotion you make while reading one story is immediately reflected everywhere.

For the full model of how familiarity stages work, read Word familiarity levels.