Multimodality – Innohub

1. Introduction: AI has gained new senses

AI no longer just writes answers on a screen. It looks at images, listens to audio, interprets video and combines different data sources in a single line of reasoning. This is a big shift.

Until recently, AI was above all a fast text worker: it helped you summarize, brainstorm, translate and write. Now it is becoming more like a coworker you can show a receipt, a draft contract, a photo from the production line, a recording of a customer service call or a video of a sales pitch.

That is exactly what multimodality means: AI's ability to handle several different “forms,” or modalities, such as text, images, audio and video.

OpenAI described its GPT-4o model in 2024 as a step toward more natural human-computer interaction: the model can take in text, audio, images and video and produce text, audio and images.^[1]

For businesses, this is not just a technical upgrade. It changes what can be automated, how information is retrieved and what kinds of products can be built.

2. What does multimodality mean?

Multimodality refers to AI that understands and processes several different types of data. Put simply:

text: reports, emails, contracts, chats
image: photos, charts, receipts, drawings, user interfaces
audio: speech, meetings, interviews, customer service calls
video: motion, sequences of events, presentations, surveillance footage, training videos

A traditional language model was like a person who learns about the world only by reading. A multimodal model also gains abilities akin to seeing and hearing.

This doesn't mean AI “understands” the world the way a human does. It doesn't feel, experience or perceive biologically. But it can computationally recognize connections between different types of data: what an image shows, what is said in speech, how a chart relates to the figures in a report and what happens in a video.

OpenAI's developer documentation defines multimodality as a model's ability to understand and generate content across different input types, such as text, images, audio and video.^[2]

3. Why does this matter right now?

Multimodality has become important because much of the information in working life doesn't come in tidy text form.

Leadership team decisions rely on slide decks, financial figures, conversation and the mood in the room. In industry, information lives in images of machinery, gauges, sensors and maintenance videos. In healthcare, it lives in patient records, medical imaging and conversations. In sales, signals come from customer calls, quotes, online behavior and presentation materials.

Bringing all of this together used to be slow. A person looked at the image, read the report, listened to the meeting and made an interpretation. Now AI can help with that work.

According to Stanford HAI's 2025 AI Index Report, AI's impact on society, the economy and governance is intensifying.^[3] Multimodality is one of the most visible directions of this development: AI no longer stays inside its own text box but is starting to touch almost all digital work.

4. The technology, made understandable

Multimodal AI rests on the idea of a shared representation. When a model processes text, images or audio, it converts them into mathematical representations. You can think of these as coordinates on a map of meaning.

If an image shows a dog, a text contains the word “dog” and an audio file contains a bark, the model learns that these can relate to the same thing.

Three things make this possible:

First, computing power has grown. Training large models takes an enormous amount of computation.

Second, there is more data. The web holds unprecedented amounts of text, images, video, audio and documents.

Third, user interfaces have improved. Multimodality isn't useful if it can only be used in a research lab. Its value emerges when users can add an image to a conversation, talk to the model or ask AI to interpret a document.

OpenAI's GPT-4o System Card states that the model was trained end-to-end across text, vision and audio, so that all inputs are processed by the same neural network.^[4]

5. The individual's perspective: AI as an everyday interpreter

For individuals, multimodality makes AI more tangible. You don't need to craft the perfect text question — you can just show it.

Practical examples:

Take a photo of a receipt and ask AI to extract the total, date and VAT.
Send a picture of a chart and ask for an explanation: “What does this mean in plain English?”
Show a photo of a plant leaf and ask what might be wrong with it.
Upload a meeting recording and ask for the decisions, open questions and next steps.
Show a photo of your wardrobe and ask for outfit ideas for a specific occasion.

In an article by Elina Lappalainen in Helsingin Sanomat, AI is described as an everyday helper that can assist with things like skincare, workouts, cooking and gardening — but whose advice you also need to know how to question.

That sums up multimodality's value proposition well: it can make AI more approachable even for people who don't think of themselves as tech-savvy. But it also carries a warning. When AI answers confidently based on an image, it can sound like an expert even when it is guessing.

6. The perspective of businesses and organizations

For businesses, multimodality opens up an entirely new layer of efficiency. The biggest gain doesn't necessarily come from AI “writing better.” It comes from AI being able to process scattered information.

In many organizations, information is spread across:

PDF reports
customer recordings
Teams meetings
CRM systems
production photos
emails
support tickets
sales presentations
training videos

Multimodal AI can help bring all of this together.

For example, a customer service company can analyze call recordings, chat conversations and photos attached to complaints. A manufacturer can combine maintenance photos, sensor data and technicians' written notes. A consulting firm can summarize a workshop video, read photos of flip charts and turn them into a project plan.

This is where leadership makes the difference. In a Startup-ministeriö conversation, Supercell's Ilkka Paananen emphasizes that decisions in an organization should be made as close to the product and the end customer as possible. The same idea applies to multimodal AI: the best use cases don't emerge in a closed IT department workshop, but where everyday friction actually shows.

7. The opportunity for startups

Multimodality is especially interesting for startups because it enables products that used to be hard to build. A good startup usually looks for a painful problem that existing solutions handle clumsily. Multimodality widens the range of problems you can solve.

Possible startup ideas:

AI that analyzes construction site photos and spots safety hazards
a tool that automatically turns sales calls into CRM updates
an app that reads receipts, contracts and invoices for a small business owner's bookkeeping
a service that automatically turns training videos into learning materials
a healthcare assistant that combines a patient's described symptoms, images and documents for a doctor to review

There is an important distinction, though: technology alone is not enough. A startup has to understand the workflow. Who uses the product? In what situation? Which decision does the AI's answer influence? What happens if the answer is wrong?

Multimodal AI can be the technical engine. Business value is created only when it is packaged into a reliable, easy-to-use and responsible product.

8. Practical applications

Finance and accounting

Receipts, invoices, contracts and payment data are bread and butter for multimodality. AI can extract the details from a receipt, suggest an expense category and flag an inconsistency: for example, a restaurant bill booked as a travel expense even though the date doesn't match the travel report.

Sales

A sales call contains much more than words. There are questions, hesitation, objections, buying signals and moments of silence. A multimodal system can combine the call, the presentation materials and CRM data. It can suggest the next message, pick up on the customer's concerns and remind you to follow through on what you promised.

Marketing

A campaign's success isn't just about text. It involves images, video, audio, the brand and the target audience's reactions. AI can analyze ad images, compare them against brand guidelines and suggest versions for different channels.

Education

A student can show a photo of an assignment, explain out loud where they are stuck and ask for guidance. A teacher can turn a lecture recording into a summary, an assignment pack and review questions.

Manufacturing

In maintenance, a worker can take a photo of a faulty part and ask AI to compare it with the service manual. This doesn't replace the professional. It can speed up troubleshooting and reduce errors.

Media and communications

A journalist, communications professional or analyst can feed an interview recording, images, reports and background materials into the system. AI helps find what matters, but a human makes the journalistic, ethical and contextual judgment calls.

9. Examples: receipts, charts, meetings and video

Example 1: a receipt

A user sends a photo of a receipt and asks:

Extract the date, total, VAT and payment method, and suggest an expense category. Also let me know if anything is unclear.

Good AI doesn't just give an answer. It points out what it is unsure about. A better answer would be:

The total appears to be €48.90, but the VAT line is hard to read because the image is blurry. Double-check the VAT before entering it into the books.

Example 2: a chart

An executive uploads an image of a sales chart. The question:

What from this chart is worth highlighting to the board?

AI can spot a trend, an anomaly and an unclear axis label. It can also suggest questions: did the growth come from new customers or from price increases?

Example 3: a meeting

A team uploads an audio recording or a video of a workshop. AI produces:

decisions
open questions
owners
timelines
risks
points of disagreement

This can save time. But decisions shouldn't be accepted blindly, because the model may misinterpret the conversation.

Example 4: a video

A training video can be turned into a text summary, learning questions, a process description, onboarding material or short highlights suited for social media clips. This is where multimodality meets content production. A single source can serve many purposes.

10. Poor, better and excellent use

Poor example

Look at this image and tell me the truth.

This is a poor request because it gives the model too much latitude. It doesn't specify what should be assessed, with what precision or what the answer will be used for.

Better example

Analyze this chart. Give me the three most important observations, two possible interpretations and one thing that should be checked against the original data.

This steers the model to separate observations from interpretations.

Excellent example

You are a CFO's analytical assistant. Look at this sales chart and compare it with the monthly report below. Write a 150-word summary for the board. Separate facts, assumptions and follow-up questions. Don't make up figures that aren't shown in the image or the text.

This is far better. It includes a role, a task, a format, constraints and caution.

11. Risks and pitfalls

The biggest risk of multimodality is the same as its biggest strength: it feels natural. When AI looks at an image and answers convincingly, users easily assume it saw things the way a human would. It didn't. It produced a statistical interpretation.

The key risks:

1. Hallucinations

AI can make up things that aren't in the image, audio or video. Misidentifying a plant, a product, a person or a document, for example, can lead to bad decisions.

2. Overreliance

People may outsource their judgment to the model. This is especially dangerous in high-stakes situations such as health, law, finance, safety and recruiting.

3. Privacy

Images, audio and video often contain personal data. Faces, voices, locations, workplace premises, documents and customer information can be exposed without anyone noticing. An OpenAI report on privacy notes that in multimodal settings, personal data can appear in text, images, audio and video, which makes it harder to identify and protect.^[5]

4. Copyright and use of materials

Processing images, video and audio involves rights. Are you allowed to use the material? Can you feed it into an external service? Can you create new versions of it?

5. Bias

Models may perform differently for different groups of people, languages, accents, camera angles and cultural contexts.

6. Security

Multimodality can also make scams more convincing. Fake voices, images and videos can make it harder to judge what to trust. NIST's AI Risk Management Framework gives organizations a framework for managing AI risks from the perspective of individuals, organizations and society.^[6]

12. Ethics, privacy and regulation

Multimodal AI directly touches people's everyday lives. That's why responsibility isn't an add-on bolted on afterward. Companies should ask at least the following:

What data are we feeding the model?
Does it contain personal data?
Have customers or employees been informed of, and consented to, the processing?
Which decisions does the AI's output influence?
Who reviews critical answers?
How are errors documented?
Can users challenge or correct the AI's conclusion?

The EU AI Act is the first comprehensive legal framework on AI, aiming to address the risks of AI and position Europe in AI development.^[7]

In the EU, obligations for general-purpose AI models are being phased in, and in 2025 the Commission published the General-Purpose AI Code of Practice to support compliance with the AI Act.^[8]

This is an important message for businesses: AI experiments can no longer be just enthusiastic tinkering. You need ground rules.

13. What does this say about the bigger shift?

Multimodality signals a major change of direction in working life. Digitalization used to mean turning paper into PDFs and moving processes into a system. Now AI is starting to interpret the messy, incomplete and varied information that work actually consists of.

This can change expert work in three ways.

First, the threshold for analysis drops. You no longer have to convert everything into text or a spreadsheet first.

Second, documentation becomes more automated. Meetings, images and processes can be turned into structured information.

Third, interaction becomes more natural. You can talk to AI, show it things and ask it questions.

But the change doesn't eliminate people. It changes their role. The expert of the future isn't the one who has memorized the most facts. It's the one who can ask good questions, evaluate answers, recognize uncertainty and connect AI-generated analysis to the real situation.

14. The future: hype vs. realistic development

There is a lot of hype around multimodality. Some of it is justified, some isn't. Realistically, the following trends are likely.

1. AI becomes the user interface

Many applications will gain conversational, image and voice interfaces. Instead of clicking through everything themselves, users will ask the system to do it.

2. Documents come alive

A report is no longer just a PDF. It can include text, charts, source data, a spoken explanation and the ability to query it with AI.

3. The value of meetings is measured better

AI can identify what was decided, what was left open and who promised to do what. This can reduce hidden waste in organizations.

4. Video analysis becomes mainstream

Training, safety, maintenance, sports, media and customer experience all benefit from video interpretation.

5. Trust becomes the most important competitive advantage

As AI handles increasingly sensitive material, users will choose solutions they can trust.

What won't disappear? Human judgment won't disappear. Accountability won't disappear. Context won't disappear. And perhaps most importantly: the need to understand the customer won't disappear. AI can look at an image, but it doesn't automatically know what matters in a business.

15. Conclusions

Multimodality makes AI more practical. It brings AI closer to real work: receipts, images, conversations, videos, reports, customer situations and decisions.

Its promise is big: less manual interpretation, faster analysis, better documentation and more natural interaction with technology. But that is exactly why you need to be careful with it. Multimodal AI is not a magic eye. It is a powerful tool for analysis and interpretation that can be wrong in a very convincing way.

The best way to use it is not to ask: “Can AI do this for me?”

A better question is: “How can AI help me see, hear and understand this situation better — while keeping accountability where it belongs?”

That is the essence of multimodality. It doesn't make people unnecessary. It can make them more accurate, faster and better informed.

16. Practical frameworks and templates

General operating model

Pick one concrete use case.
Define what data will be used: text, images, audio, video.
Decide what the AI should produce.
Define who will review the output.
Test with a small dataset.
Log errors and uncertainties.
Decide whether the use can be expanded.

Checklist before sending an image to AI

Does the image contain personal data?
Does it show a customer, an employee or confidential information?
Are you allowed to use the image?
Which decision will the analysis influence?
Is an expert review needed?
Is the image sharp enough?
Am I asking the AI to state its uncertainties?

A good prompt for image analysis

Analyze this image. First, tell me what you can see with certainty. Then list possible interpretations. Separate facts from estimates. Don't make up missing information. Finally, mention what I should check myself.

A good prompt for a receipt

From this receipt, extract the date, total, VAT, payment method, merchant and a possible expense category. If anything is unclear, mark it as uncertain instead of guessing.

A good prompt for a meeting recording

Summarize this meeting. Separate decisions, open questions, agreed tasks, owners and deadlines. Finally, highlight three things that could cause misunderstandings.

Decision-making model

Use an AI answer in decision-making only if these conditions are met:

the source material is of sufficient quality
the AI's uncertainties have been identified
critical points have been checked
a human makes the final decision
the consequences of the decision are understood

Beginner template

Start with low-risk everyday tasks: explaining a chart, extracting receipt details, suggesting improvements to a presentation slide, describing what's in an image, drafting meeting notes.

An advanced user's approach

Build a workflow:

Gather the material: image, text, audio or video.
Ask the AI for an initial analysis.
Ask it to list its uncertainties.
Compare the output with the original material.
Make corrections.
Save the lessons learned for next time.

Team template

Ask together:

Where in our work do we repeatedly look at, listen to or interpret the same thing?
Where are mistakes expensive?
Where could AI speed up the groundwork while a human makes the decision?
What data must we never feed into an external AI service?
What would be a safe first pilot?

Multimodality isn't just a new AI feature. It is a step toward a world where technology understands our work in a form closer to how it really is.

Not just as text. But as images, sound, movement, conversations and situations.