Word tokens, dictionary forms, and definitions

Tokens are not always dictionary words

When Valo Reader processes a story, it runs the text through a tokenizer that splits the content into study units called tokens. A token is not always a word in the dictionary sense. Depending on the language, a token might be:

• A conjugated verb form as it appears in the text
• A grammatical particle attached to a noun
• A phrase treated as a single word
• A punctuation-bound segment in languages without spaces

This means that tapping a word in the reader shows you what the tokenizer found, which may include grammatical information that is not part of the dictionary entry. In agglutinative languages (like Japanese or Turkish), a single token may pack several grammatical meanings into one word.

Word details

When you tap or click a token, the word details popover or dialog opens. It typically contains:

• Original form — the token as it appears in the text
• Dictionary form — the base form you would look up in a dictionary
• Readings — phonetic readings where applicable (e.g., furigana for Japanese, pinyin for Chinese)
• Definitions — contextual glosses matched to the dictionary form
• Familiarity control — the current level and a control to adjust it

Not every field is available for every token. Dictionary coverage varies by language and by the specific word. Common words and standard conjugations have the best coverage; rare or highly idiomatic expressions may show less information.

How tokens relate to vocabulary

Each token maps to a dictionary form, and it is the dictionary form that carries the familiarity level. This means that seeing a conjugated form of a word you already know still counts as encountering the same vocabulary item. Your familiarity with the base word applies across all its inflected forms.

However, tokens that are genuinely different words (homographs, different etymologies) are tracked separately, even if they look identical in the text.

Cross-story vocabulary state

Changing a word's familiarity updates your per-language vocabulary state. That state is shared across all stories in the same study language, so a promotion you make while reading one story is immediately reflected everywhere.

For the full model of how familiarity stages work, read Word familiarity levels.