Tokenization – Innohub

1. Introduction: why AI doesn't really “read” the way humans do

When a person reads the sentence “A Finnish entrepreneur is building an international growth company”, they see the words, the meanings and perhaps even picture the situation: the founder, the team, the funding round, the uncertainty and the ambition.

AI sees something completely different. It doesn't start from words. It splits the text into smaller pieces called tokens. A token can be a whole word, part of a word, a punctuation mark or sometimes a string of characters that doesn't look very meaningful to the human eye.

OpenAI's guidance explains that language models process text in the form of tokens, and the number of tokens affects both whether the text fits within what the model can process and how much an API call costs.^[1]

This sounds like a technical detail. It isn't.

Tokenization is one of those invisible mechanisms that determine why AI sometimes forgets the beginning of a conversation, why a long contract won't fit for analysis, why Finnish-language text can use up space surprisingly fast and why using AI costs more than it first appears in the spreadsheet.

If a company uses AI in customer service, legal work, sales, coding, marketing or analyzing internal information, tokens aren't a nerdy side note. They are a resource. A bit like time, money or server capacity.

2. What is tokenization?

Tokenization means splitting text into smaller units that a language model can process. Put simply:

humans see words in a text
computers see characters
a language model processes text as tokens

A token is not the same as a word.

For example, the English word running may be a single token in some tokenizers, but a rarer or longer word can be split into several parts. In OpenAI's tiktoken example, a string of words and characters can be split into pieces such as "t", "ik", "token", " is", " great" and "!".^[1]

A jigsaw puzzle is a good analogy. For a human, a word is a finished picture. For AI, text is a box of small pieces from which it builds the next piece using probabilities. So it doesn't “understand” text the way a human does; instead, it processes the structures of character strings and the relationships between them.

This doesn't make AI useless. Quite the opposite. It is precisely this mechanism that enables a model to handle many languages, specialist terms, code and new words without every possible word having to be taught to it separately. But it also brings limits.

3. Why do tokens matter right now?

Tokens matter for three reasons: capacity, cost and quality.

The first is capacity. Every language model has a context window, meaning the number of tokens it can take into account in a single conversation or task. Claude's documentation describes the context window as a space where the earlier messages in a conversation, the user's instructions and the model's responses accumulate.^[2]

The second is cost. In many AI services, pricing is based on the number of tokens: input tokens and output tokens are priced separately. OpenAI's pricing page shows model-specific prices per million tokens, and OpenAI's guidance emphasizes that token counts appear in the usage metadata and are tied to billing.^[3]

The third is quality. The more text you push into a model, the more it has to separate what matters from what doesn't. A long context doesn't automatically mean a better answer. It only means the model has more material to work with.

This is an interesting point from a leadership perspective. In companies, AI is often talked about like a magic box: “Let's put all the documents in there and ask.” In reality, a better question is: what information should the model see for this particular task?

4. Tokens, the context window and memory

The context window is AI's desk. If the desk is small, it only holds a sticky note, a coffee cup and one contract. If the desk is huge, you can spread out meeting minutes, customer data, code, reports and strategy documents. But even if the desk is large, a person's attention can get lost in the papers too. The same applies to AI.

Current models have developed rapidly in handling long context. Google says that Gemini models process text as tokens and gives a rough rule of thumb: one token corresponds to about four characters, and 100 tokens to about 60–80 English words.^[4] Google has also marketed the long-context capability of Gemini Pro to process up to a million tokens, which can mean, for example, large volumes of documents or entire codebases.^[5]

Anthropic, for its part, describes token counting as a way to manage costs, limits and model routing, that is, which model a task should be sent to.^[6]

One important observation is this: context is not the same as persistent memory.

Context is what the model sees right now. Memory is a system that can retain information across conversations. Many everyday users confuse the two. That leads to disappointment: “Why didn't the AI remember this?” Often the answer is: because it was no longer in the context, or the service doesn't keep it as memory.

5. Finnish as a stress test for tokenization

Finnish is an interesting challenge for tokenization. In English, many words are short and separate. In Finnish, meaning is packed into long inflected forms and compound words.

Consider the word:

kasvuyritysrahoituskierroksellaan

A person recognizes its parts: kasvu (growth), yritys (company), rahoitus (funding), kierros (round), plus a possessive and case ending. A tokenizer, however, may split the word into many pieces. That's why Finnish-language text can use up the context window faster than the user expects.

A practical tip: in Finnish, a single word can often be 2–3 tokens, sometimes more. This isn't a law of nature; it depends on the model and the tokenizer. That's why it's always worth measuring the exact count with the tokenizer tool of the model you're using. OpenAI offers a tokenizer tool, which lets you see how text is split up and how many tokens it produces.^[7]

Why does this matter? Because in a Finnish-speaking organization, long documents, official texts, contracts, research and strategy materials can quickly balloon in token count. The same 20-page body of material isn't equally “light” for the model in every language.

This doesn't mean Finnish should be avoided. It means you should be aware of it when working in Finnish.

6. How does tokenization affect costs?

The cost of AI doesn't come only from how many users use the service. It comes from how many tokens the service processes.

A simple formula:

total cost = input tokens + output tokens + any internal reasoning tokens + the number of repeated calls

OpenAI says that advanced models can also use internal reasoning tokens — tokens the model uses before producing its final answer. These can appear in the usage metadata and be tied to billing.^[8]

This is a big deal for a company.

Imagine a customer service bot that includes the following with every answer:

the company's long service description
the entire customer history
internal guidelines
return terms
the previous conversation
the user's new question

If this is done thousands of times a day, even a small amount of extra text turns into euros.

This is where tokenization turns from a technical detail into a business question. A well-designed AI system doesn't send the model everything possible. It sends only what is relevant to the task.

7. What does this mean for business owners and executives?

For a business owner, tokenization is above all a change in mindset.

You shouldn't lead AI adoption just by asking: “What are all the things we could automate?” A better question is: “What information does the model need to succeed, and what information does it not need?”

This resembles good leadership. A good leader doesn't drown the team in every possible piece of background material. They provide the goal, the context, the limits of decision-making and the right sources. The same applies to AI.

In a company, tokens show up in at least four places:

In costs. Long prompts and extensive documents cost more.
In speed. A long context can slow down the response.
In quality. Too much background information can blur the model's focus.
In security. The more information is sent, the more carefully you need to consider what may be sent.

So when adopting AI, it isn't enough to choose “the best model.” You have to design the flow of information.

8. The perspective of startups and growth companies

For a startup, tokenization is especially important, because in a small company every cost and every product choice is felt quickly.

If you are building a SaaS service, meaning a cloud-based software product that customers use on a subscription, tokens can be part of your unit economics. Unit economics means how much it costs to serve one customer and how much margin they leave.

This is easy to forget in the MVP phase. An MVP is the first working version of a product, used to test whether customers actually want to use the solution.

At first, it may seem sensible to send the model everything:

the entire user profile
all documents
a long system instruction
the entire conversation history
a complete database extract

It works in a demo. But when you have 10,000 users, the token bill starts to add up.

A growth company should ask early on:

Can we compress the context?
Can we retrieve only the relevant pieces of documents?
Can we use a cheaper model for easy tasks?
Can we cache repeated inputs?
Can we limit the length of responses?

Tokens are a product's invisible infrastructure. They don't show up in the user interface, but they can determine whether a service is profitable.

9. Practical examples: bad, better and excellent prompts

Poor example

Here are 80 pages of material. Make a good summary of this.

This may work if the model can process the material. But it is expensive, imprecise and prone to burying the key points.

Better example

Read this 80-page document. Summarize it with a particular focus on customer risks, cost impacts and decision points important to the management team. Don't recap everything.

Here the AI gets a focus. It knows what to look for.

Excellent example

Act as an operational advisor to a growth company. Analyze the material below only through these questions:

1) Which points affect cash flow over the next 6 months?
2) Which risks require a decision from the management team?
3) Which matters are just background information?

Finally, write a one-page decision memo.

This is better because it saves tokens and guides the model's thinking. It doesn't ask for everything. It asks for what matters.

10. Risks and misconceptions

There are many misconceptions about tokenization.

The first is the idea that a bigger context solves everything. It doesn't. A long context helps, but it doesn't remove the need to select, scope and structure information.

The second is the assumption that tokens correspond to words. They don't. This is especially important in Finnish, where long words and inflected forms can be split into many tokens.

The third is the belief that the model remembers everything from the conversation. In reality, the conversation's content accumulates in the context, and when the limit is reached, the system has to summarize, remove or otherwise manage older information. Anthropic describes how the context grows as the conversation progresses, with earlier messages and responses kept in the context window.^[2]

The fourth risk relates to security. When an organization tries to “improve” an AI's answer by adding every possible piece of background information, it may at the same time increase the unnecessary processing of personal data, trade secrets or contract information.

Responsible use of AI isn't just about the model answering well. It's also about giving the model only the information it really needs.

11. The ethical and responsible perspective

Tokenization sounds like a neutral technology. But its effects aren't neutral.

If one language splits into more tokens on average than another, processing it can be more expensive or less efficient. This can influence which languages AI services are developed, tested and optimized for.

For smaller language communities such as Finnish, this is an important question. If a company builds an AI service based only on English-language data and cost logic, Finnish-language use may bring surprises. Not because Finnish is a problem, but because the system was designed carelessly.

A responsible organization tests:

how the model performs in Finnish
how token counts differ across languages
how costs change
how quality holds up
how limitations are communicated to the user

Equality in AI doesn't come from lofty speeches. It also comes from everyday technical choices like these.

12. Tokenization as part of a larger AI transformation

Tokenization reveals something essential about the AI transformation.

We tend to humanize AI. We say it reads, understands, ponders and remembers. These words are useful in everyday language, but they can blur reality.

AI isn't a person sitting at a desk reading a report. It's a system that processes tokens, calculates probabilities and produces the next tokens based on the preceding context. Even so, the result can feel intelligent.

Here lies AI's fascinating paradox: the mechanism is mathematical, but the experience is social. The user feels they are having a conversation. The model processes tokens.

For companies, this means AI has to be managed on two levels:

the level of human experience
the level of the technical system

A good AI strategy understands both.

13. The future: where is tokenization heading?

Tokenization won't disappear in the coming years. But its significance may change.

1. Context windows are growing

A million-token context is no longer a futuristic idea but part of the competition among today's large models. Google and Anthropic have both highlighted the possibilities of long context.^[5] But a bigger window doesn't remove the need for good information management.

2. Token counting is becoming automated

More and more systems count tokens in the background and route the task to a suitable model. Anthropic cites cost management, anticipating limits and model routing among the benefits of token counting.^[6]

3. Context compression is improving

In long conversations, systems are learning to summarize earlier information better. This isn't the same as perfect memory, but it can make the user experience smoother.

4. Multilingual support is improving

Tokenizers and models are likely to get better for languages like Finnish as well. Even so, smaller languages need deliberate testing.

5. Hype gives way to operational expertise

In the early days, many asked: “What are all the things AI can do?” The next phase is more practical: “How do we do this with quality, safely and profitably?” Tokens are central to this conversation.

14. Conclusions

Tokenization is a basic mechanism behind everyday AI use.

It explains why AI doesn't process text the way humans do. It explains why long documents cost money. It explains why Finnish-language text can use up the context window surprisingly fast. It also explains why a good prompt isn't just a nicely worded question but a resource-wise instruction.

The most important lesson is this:

Using AI effectively doesn't mean giving the model as much information as possible. It means giving the model the right information in the right format.

For a business owner, this is a competitive advantage. For an executive, it's a matter of cost and quality. For a professional, it's a new basic skill of the job. For a student, it's a way to understand why AI's answers sometimes succeed and sometimes fall apart.

Tokenization is invisible, but its effects are visible everywhere.

15. Practical frameworks, checklists and templates

A general operating model: token-wise use of AI

Define the task precisely.
Select only the material relevant to the task.
Remove unnecessary repetition.
Give the model a role and a goal.
Specify the response format.
Test the token count.
Monitor costs in use.
Improve the prompt based on data.

Checklist before sending long material to a model

Ask:

Does the model need the entire document?
Would a summary be enough?
Can I split the material into parts?
Does it contain personal data or trade secrets?
Is the task analysis, summarization, a decision memo or ideation?
How long does the answer really need to be?
Is the Finnish-language material surprisingly heavy in token count?

A set of questions for the management team

In which processes does AI process the most text?
Which use cases cause the largest token costs?
Do we have guidelines on what data may be given to the model?
Do we measure AI costs per user, per customer or per task?
Has Finnish-language use been tested separately?
Could some tasks be handled with a cheaper or smaller model?

Beginner template

Use this prompt framework:

Act as [role].
Your task is [goal].
Use only the material below.
Focus especially on these points: [1–3 items].
Don't recap everything.
Format the answer as a [list / memo / table / summary].
Maximum length [x] words.

An advanced user's approach

Analyze the material in three steps:

1) Extract only the points relevant to the task.
2) Separate facts, interpretations and uncertainties.
3) Finally, make a decision proposal.

If the material isn't enough for a conclusion, say so directly. Don't fill gaps with guesses.

A template for teams

Task: What do we want the AI to do?
Material: What information does it need?
Scope: What information does it not need?
Risks: Does the material contain sensitive information?
Tokens: Has the material been tested with a tokenizer tool?
Quality: How do we know the answer is good?
Cost: Roughly how much does a single use cost?
Improvement: How can the prompt or the material retrieval be made lighter?

Tokenization isn't a boring technical footnote of AI. It's the interface between human thinking and machine computation.

Once you understand it, AI becomes less mysterious — and much more useful.

Tokenization: AI's invisible language that sets the limits of its understanding — and its costs