USE CASES · 0.9

Four things to do
with a word's numbers.

The API reference says what each endpoint returns. This page says what to do with it. Every excerpt below is a real file in examples/, standard library only, and the test suite runs all four of them.

Tune a passage toward a tone

Every word carries eight axis scores, each a percentile rank over this build's vocabulary. Ask for a direction by its label and each candidate replacement is scored on it. The scores come from a static embedding with no sentence around the word, so it cannot tell that sick is praise: every suggestion prints with its definition, for you to veto.

GET /api/v1/word/{word} free, once per distinct word 

python examples/tune_tone.py --file memo.txt \
    --toward positive,concrete --min-gain 0.5

# grim -> sober   (+0.82)
#     lacking brightness or color; clothed in sober garments

examples/tune_tone.py

Price a prompt across models estimate

Which of the three reads your prompt in the fewest tokens, and does any of them shatter your key terms into fragments. Pass a second rendering with --also es=file and it compares two real measurements: the script has no translator and will not pretend to one. Prices are yours to supply, because they are the provider's to publish and they change.

POST /api/v1/tokenize free 

python examples/prompt_cost.py --file prompt.txt --price deepseek=0.27

DeepSeek V3      6 tokens  <- cheapest
Mistral 7B      10 tokens
#   most fragmented: tapestry -> tap+est+ry (mistral, 3 tokens)

examples/prompt_cost.py

Should you write the prompt in English? measured

Naming the same concept costs a different number of tokens in every language, and the gap is not the same for every model. These are the build's own tokenizer splits, aggregated: not an opinion about languages, a measurement of three tokenizers.

Language DeepSeek V3 Mistral 7B Qwen3 8B
English 1.18 1.55 1.21
Spanish 2.61 3.07 2.68
French 2.52 2.94 2.60
Japanese 2.98 4.64 4.26
Chinese 2.19 4.16 3.86

Mean tokens per headword, over the 3,983 concepts that have a headword in every language this build carries. For each concept, every headword the language offers is counted, so nothing depends on which one was picked — except in English, which contributes only the pivot. That is the asymmetry to keep in mind reading the table: English averages 1.0 headwords per concept, Spanish 2.0, French 2.2, Chinese 3.3 and Japanese 7.5, so every column but the first carries spelling variants and archaisms that the English column does not.

Every concept also counts once regardless of how often the word is used. The vocabulary is frequency-ranked and the rank is right there in the database, but this mean does not read it, so aardvark weighs exactly as much as able. That is not free: proper nouns are expensive in Chinese and Japanese, and the shared set has plenty of them.

English is the cheapest here, by 1.8x to 3.5x depending on the language and the model. And the model matters as much as the language: for Chinese, DeepSeek reads the same concept in 1.9x fewer tokens than Mistral (2.19 against 4.16), because their vocabularies are 129,280 and 32,768 entries.

How much of that first number is the languages and how much is the averaging? Some of it is the averaging. Take the cheapest headword per concept instead of the mean of all of them — which is what a writer minimising cost would actually type — and the range narrows to 1.4x–2.7x. English still wins, by less. The second number is the solid one: the Chinese gap between DeepSeek and Mistral survives every way we tried aggregating it, because both models are reading the identical strings and only their vocabularies differ.

This is the cost of naming a concept, not of writing a sentence: languages differ in how many words they need, and nothing here measures that. The full sampling method is on the methodology page.

Two endpoints these examples added

EndpointWhat it returnsAccess
POST /api/v1/tokenize per-model token totals for a prompt, the words each model fragments, and the words it could not find free
GET /api/v1/{lang}/word/{lemma} a foreign headword: its own token split, and the English geometry it pivots to free
curl -X POST https://your-host/api/v1/tokenize \
  -H "Content-Type: application/json" \
  -d '{"text": "delve into the tapestry", "lang": "en"}'

/tokenize sums token counts measured at build time; it does not run a tokenizer. It cannot: requirements-api.txt excludes transformers on purpose, which is what keeps the image a few hundred megabytes, so the container cannot tokenize anything at run time. Every reply says so, and says which way the estimate is wrong. It also refuses Japanese and Chinese outright rather than counting zero, because nothing here can find a word boundary inside 犬が好きです.

What these numbers are not

  • A neighbour is not a synonym. Model neighbours are words that turn up in similar sentences, which is why antonyms are often the closest of all.
  • A personality score is a percentile rank over this build's vocabulary, from an embedding with no context around the word.
  • A token total is a sum, not a tokenizer run, and it under-counts prose: punctuation, casing, leading spaces and out-of-vocabulary words all add tokens it does not see.
  • Prices are not ours to publish. Nothing here ships a cost-per-token; you pass today's figure and the output echoes it back.

Which numbers are measured and which are not is on the methodology page. The endpoints themselves are on the API page, and what a key costs is on pricing.