Multimodal LLMs Explained: How AI Models Read Images, Audio, Video and PDFs
· 16 min read
A buyer takes a screenshot of your pricing page, drops it into ChatGPT, and asks, "Is this cheaper than what we pay now for 40 seats?" A sales engineer uploads a 60-page security questionnaire as a PDF and asks Claude to find every question about data residency. A product manager records a five-minute voice memo after a customer call and asks Gemini to turn it into a prioritized list of feature requests. None of these people typed a long prompt. They handed the model a picture, a document, or a recording, and expected it to understand.
That is what multimodal large language models do. They accept more than text, most commonly images, scanned documents, audio, and video, and reason about that input in the same conversation as your written question. In a few years this went from research demo to a default feature in the AI tools your buyers, customers, and employees already use.
This guide explains multimodal LLMs in plain English for U.S. B2B SaaS teams: what "multimodal" actually means, how models turn pixels and sound into something a language model can process, what these models are good and bad at, what they cost, the new security and privacy risks they introduce, and what they mean for how AI engines read your website, screenshots, PDFs, and videos. If you are new to the underlying technology, start with our practical guide to large language models, then come back.
Published November 16, 2026. This is an educational explainer. Model names, features, limits, and prices change frequently, and providers do not disclose every detail of how their production models are built. Where this guide describes a specific product, it relies on that vendor's public documentation or published research; where it describes multimodal models in general, it describes common approaches in the research literature rather than any one company's private methods.
What "Multimodal" Means
A modality is a type of input or output: text, images, audio, video, and so on. A text-only model takes text in and sends text out. A multimodal model handles at least one other modality.
In practice, the term covers several different capabilities that are worth separating, because products often support some and not others:
- Image understanding. The model can read a photo, chart, screenshot, diagram, or scanned page and answer questions about it. This is the most widely available multimodal capability.
- Document understanding. The model can take a PDF and use both its text and its visual layout, including tables, charts, and figures. Some tools only extract the text from a PDF; others also look at each page as an image.
- Audio understanding. The model can take recorded speech or other sound and transcribe, summarize, or answer questions about it.
- Video understanding. The model can take a video, usually by sampling frames and the audio track, and describe or answer questions about what happens in it.
- Multimodal output. The model can produce something other than text, such as speech or images. Generating images is technically a different problem from understanding them, and it is often handled by separate or specialized components.
When a vendor says a model is "multimodal," check which of these it means. A model that reads images well may not accept audio at all, and a chat app may add features through separate tools that the underlying model does not have on its own.
A Short History: From Image Captions to "Omni" Models
Computers that describe pictures are not new. What changed is that image and audio understanding were joined to large language models, so the same system that writes and reasons in text can also look and listen.
A few milestones explain how we got here:
- Vision Transformers (2020). Google researchers showed in An Image Is Worth 16x16 Words that the transformer architecture behind LLMs could also process images, by cutting a picture into small square patches and treating each patch roughly like a word. This made it natural to connect vision and language models later.
- CLIP (2021). OpenAI's CLIP was trained on about 400 million image-and-caption pairs collected from the internet, learning to place images and their matching descriptions close together in a shared space. It could recognize many kinds of images without being trained on each category specifically, and CLIP-style image encoders became a common building block.
- Flamingo (2022). DeepMind's Flamingo connected a pretrained image encoder to a pretrained language model and showed that the combined system could learn new visual tasks from just a few examples in the prompt.
- Whisper (2022). OpenAI's Whisper speech-recognition model was trained on about 680,000 hours of multilingual audio from the web and made robust transcription widely accessible.
- LLaVA (2023). The open research project LLaVA showed a relatively simple recipe: connect an image encoder to an open language model with a small connecting layer, then fine-tune on image-based instructions and conversations. Many open multimodal models followed similar designs.
- GPT-4 with vision and Gemini (2023). OpenAI released image input for GPT-4, documented in its GPT-4V system card, and Google introduced Gemini, which its technical report described as trained to be multimodal from the start across images, audio, video, and text.
- "Omni" models (2024). OpenAI's GPT-4o announcement described a single model handling text, audio, and images, and reported it could respond to audio in as little as 232 milliseconds, with an average of 320 milliseconds, which is close to human conversational response time. Google's Gemini 1.5 report showed very long multimodal inputs, including hours of video and audio, in a single prompt.
Since then, image input has become standard across major assistants, and providers document their capabilities in guides such as OpenAI's images and vision guide, Anthropic's vision documentation, and Google's image understanding guide.
How a Language Model "Sees" an Image
Language models work with tokens, small chunks of text converted into numbers. (Our guide to tokens and context windows explains this in detail.) An image is not text, so it has to be converted into something the model can process alongside your words. The details differ by model, but the common research recipe looks like this:
- Split the image into patches. The picture is resized and cut into a grid of small squares.
- Encode the patches. A vision encoder, often a transformer trained in the CLIP style, turns each patch into a list of numbers that represents what it contains: an edge, a letter, part of a face, a bar in a chart.
- Translate into the language model's space. A connecting layer maps those numbers into the same kind of representation the language model uses for word tokens. The result is a sequence of image tokens.
- Process everything together. The image tokens are placed into the context window next to your text tokens, and the language model attends to both when generating its answer.
Some models are built this way, with separately trained parts joined later. Others are trained on mixed text, images, and audio from early on, which providers describe as "natively" multimodal. Either way, the important practical point is the same: the image becomes tokens, and tokens take up space and cost money.
This also explains several familiar quirks. Models resize large images, so tiny text in a dense screenshot may be lost. Fine spatial detail, such as exactly which cell in a large table a number sits in, is harder than general description. And because the model is generating likely text about what it sees rather than measuring the image, it can describe things that are not there.
How Models Handle Audio and Video
Audio is usually converted into a visual-like representation of sound over time, called a spectrogram, and encoded into tokens in a similar way. Some systems first transcribe speech to text and pass the transcript to a text model. Others process the audio directly, which lets them pick up information a transcript loses, such as tone, hesitation, multiple speakers, or background sound. Voice assistants that respond quickly and can be interrupted generally rely on the direct approach.
Video is typically handled by sampling frames at intervals, encoding each frame like an image, and combining those with the audio track. For example, Google's documentation describes Gemini sampling video at about one frame per second by default. That matters: a model watching at one frame per second can miss a quick on-screen change, such as a menu that flashes open in a product demo. Video is also token-heavy, since every sampled frame becomes many tokens.
What Multimodal Inputs Cost
Images, audio, and video are billed in tokens like text, but the conversion rules vary by provider and change over time. At the time of writing, public documentation gives a sense of scale:
- Anthropic documents an approximate image token cost of the image's width times height in pixels divided by 750, so a 1,000 by 1,000 pixel image is roughly 1,300 tokens. See its vision documentation.
- OpenAI calculates image tokens from the image's size and a "detail" setting, where low detail uses a small fixed amount and high detail scales with the number of image tiles. See its images and vision guide.
- Google documents fixed token amounts for small images, tiling for larger ones, and per-second rates for audio and video in its token counting guide.
The practical lessons are simple. A single screenshot can cost as much as a page or two of text. A long video can fill a large share of a context window. Sending images at higher resolution than the task needs wastes money, and sending them at lower resolution than the task needs, such as when small print matters, wastes the whole request. Test a few sizes on your real content and pick the smallest that keeps accuracy.
What Multimodal Models Are Good At
Used on the right tasks, multimodal models save real time. Common, well-suited business uses include:
- Reading screenshots and interfaces. Explaining an error message, summarizing a dashboard, or turning a whiteboard photo into notes.
- Extracting information from documents. Pulling fields from invoices, receipts, contracts, and forms, especially when the layout carries meaning. Providers document dedicated PDF features, such as Anthropic's PDF support and Google's document processing guide.
- Interpreting charts and diagrams. Describing trends, comparing bars, or explaining an architecture diagram in plain language.
- Transcribing and summarizing calls. Meeting notes, customer interviews, and support calls, with speaker changes and action items.
- Reviewing video. Summarizing recorded demos or training sessions and finding the moment a topic is discussed.
- Accessibility. Drafting alt text and image descriptions, which humans should then review.
Public benchmarks track progress on these skills. MMMU, for example, tests college-level questions across subjects that require understanding images such as charts, diagrams, and chemical structures. As our guide to evaluating LLMs explains, benchmark scores are a starting point, not proof the model will work on your documents.
Where Multimodal Models Fail
Multimodal models inherit every weakness of text models, described in our guide to LLM hallucinations, and add some of their own.
They describe things that are not there
Researchers call this object hallucination: a model confidently mentions an object, label, or number that does not appear in the image. A 2023 study, Evaluating Object Hallucination in Large Vision-Language Models, found the problem was widespread and that models were more likely to claim objects that frequently appear together with what really is in the picture. In business terms: a model "reading" a blurry invoice may supply a plausible total rather than the real one.
They struggle with exact reading and counting
Small print, dense tables, handwriting, rotated text, and counting many similar items remain common failure points. A model may summarize a chart's overall trend correctly while misreading the exact value on one bar.
They lose detail through resizing and sampling
Because images are resized and video is sampled, small or brief details can disappear before the model ever processes them. If the answer depends on a footnote in a screenshot or a two-second moment in a video, assume it may be missed and check.
They can be steered by text inside images
A multimodal model reads text that appears in an image and can treat it as instructions. That creates a new route for prompt injection, covered below.
They are less consistent than they look
Ask the same question about the same image twice and you may get different details. For any extraction that feeds a business decision, such as contract terms, invoice amounts, or compliance answers, validate the output against the source and keep a human in the loop.
Security and Privacy Risks
Our guide to LLM security and privacy covers the general risks. Multimodal input adds three specific ones.
Hidden instructions in images and audio. If an assistant reads an image or recording from an untrusted source, such as an uploaded file, an email attachment, or a web page it browses, that content can contain instructions aimed at the model. Text can be printed in an image in tiny or low-contrast type that people overlook, and researchers have demonstrated instructions hidden in images and sounds in ways that are hard for humans to notice (Bagdasaryan et al., 2023). Treat every image, PDF, and recording from outside your organization as untrusted input, and do not give a model that processes such content broad authority to send messages, change records, or call tools without approval. Our guide to LLM agents and tool use explains why that matters.
Sensitive information you did not notice. Screenshots often capture more than intended: open browser tabs, customer names, internal dashboards, email addresses, or a password manager in the corner. Audio recordings capture everyone in the room. Before uploading, crop and redact, and check your provider's data retention and training terms for the specific plan you use.
Biometric and recording laws. Images and recordings of people can raise legal issues that text does not, such as consent requirements for recording calls in some U.S. states and biometric privacy laws like Illinois' Biometric Information Privacy Act. This guide is not legal advice; involve counsel before building workflows that analyze faces, voices, or recorded conversations of customers or employees.
How to Use Multimodal Models Well
A few practical habits make results much more reliable:
- Say what to look at. "What is the annual price for the Business plan in this screenshot?" works better than "What does this say?" Our guide to prompt engineering covers this in depth.
- Give the model text when you have it. If you have the original document, spreadsheet, or transcript, send that instead of a screenshot of it. Text is cheaper and usually more accurate than reading pixels.
- Use sharp, cropped images. Crop to the relevant area, keep text legible, and avoid photos taken at an angle.
- Ask for evidence. Ask the model to quote the exact text it read or name the page and section for each extracted value, so a reviewer can check it quickly.
- Split long documents. For very long PDFs or videos, ask about one section or time range at a time, or retrieve the relevant pages first. Our guide to retrieval-augmented generation explains how.
- Test on your own files. Build a small set of 30 to 50 real images, documents, or recordings with known correct answers, and measure accuracy, cost, and speed before you roll a workflow out.
What Multimodal AI Means for Your Brand's Visibility
For B2B SaaS teams, the most important question is not how to use multimodal models internally. It is how AI engines read the content you publish, and how buyers use these models while researching software.
Buyers bring your visuals to AI
Buyers increasingly paste screenshots of pricing pages, comparison tables, and product interfaces into AI assistants and ask for an evaluation. Google has also added image-based questions to its AI search experiences through Google Lens. That means the AI may be reading your visuals directly, with all the resizing and misreading risks described above. Clean, legible pricing and comparison tables, with plan names, prices, billing terms, and dates clearly printed, are less likely to be misread than cramped graphics.
Crawlers mostly rely on your text
Being able to read images does not mean every AI system reads every image on your site. Search and AI crawlers fetch your pages, and providers do not fully document how much image, video, or PDF content their answer systems process versus the surrounding text. The safe assumption is that the facts that matter should exist as plain, crawlable text, not only inside images, slides, or videos. Our guide to structuring a website for LLM crawlers covers the basics.
Common gaps we see on SaaS sites:
- Pricing or plan limits shown only in an image or an interactive widget that renders nothing readable in the page source.
- Customer logos as images with no text naming the customer or what they use the product for. Our article on customer proof in AI answers explains why named, specific proof matters.
- Product demos and webinars with no transcript, so the answers spoken in the video exist nowhere a text-based system can read them.
- Key facts locked in PDFs, such as security overviews or data sheets, with no HTML page summarizing the same facts.
- Screenshots in help docs with no written steps, so the instructions are only in the pictures. See our guide to help center content and AI visibility.
Text equivalents help people and machines
The fixes are mostly the same things accessibility standards already ask for. Google's image SEO best practices recommend descriptive alt text, descriptive file names, and placing images near relevant text. Its video best practices recommend making videos discoverable with dedicated pages and structured data. The W3C's alt text decision tree helps decide when and how to describe an image.
In practice:
- Write alt text that states what an informative image shows, including any key number or label, and keep decorative images out of the way.
- Publish a transcript or a written summary next to every important video.
- Put the same facts that appear in charts, screenshots, and PDFs in nearby HTML text, with dates.
- Keep the facts consistent everywhere. If your pricing screenshot, pricing page, and sales PDF disagree, AI systems may repeat any of them. Our article on why AI engines get SaaS pricing wrong shows how this happens.
None of this guarantees that an AI engine will mention or recommend your brand. It removes avoidable reasons for it to misread or overlook you. The only way to know what AI engines actually say is to observe them: run defined buyer prompts on named engines, record the date and the exact answer, and repeat. Our guide to tracking LLM recommendations explains how.
A 30-Day Plan for SaaS Teams
Week 1: Inventory your visual content. List the pages where important facts appear mainly in images, video, or PDFs: pricing, plan limits, integrations, security, customer results, and product demos. Note which have text equivalents and which do not.
Week 2: Add text equivalents. Write alt text for informative images, add transcripts or summaries to your top demo and webinar videos, and create HTML summaries for key PDFs. Add dates to pricing and plan information.
Week 3: Test how models read your visuals. Take screenshots of your pricing page, a comparison table, and a key product screen. Ask two or three AI assistants specific questions about each, such as "What does the Business plan cost per user per year?" Save the answers and note any misreadings. Fix the layouts that cause them, such as tiny footnotes, low-contrast text, or prices separated from plan names.
Week 4: Set internal guardrails. If your team uses multimodal AI for documents or calls, write down which data may be uploaded, which plans and retention settings are approved, where a human must verify extracted values, and which tools a model may never trigger on its own based on uploaded content.
The Bottom Line
Multimodal LLMs let people hand AI a screenshot, a document, a recording, or a video and get useful answers about it. Under the hood, images and sound become tokens, which means they take up context, cost money, and can be misread, especially when detail is small, brief, or blurry. Use them where they save real time, give them text whenever you have it, ask for evidence, and verify anything that feeds a decision.
For brands, multimodal AI raises the stakes on clarity. Buyers bring your visuals into AI conversations, and AI systems that build answers from your site depend on what they can actually read. If you want to see which buyer questions AI engines answer without your brand, and which sources they rely on instead, book a strategy call with LLM Recommend.
Frequently Asked Questions
What is a multimodal LLM?
A multimodal large language model is an AI model that accepts more than text, commonly images, PDFs, audio, or video, and reasons about that input together with a written question. Many can also produce other outputs, such as speech, though image understanding is the most widely available capability.
How does an AI model read an image?
Most models resize the image, cut it into small patches, and use a vision encoder to turn each patch into numbers. Those are converted into image tokens and placed in the context window alongside text tokens, so the language model can process both when it answers.
Are images more expensive than text in AI models?
Often, yes. Images are billed as tokens, and a single screenshot can use as many tokens as a page or two of text, depending on its size, the detail setting, and the provider. Audio and video are billed by length and can be much more token-heavy.
Can multimodal models make mistakes reading images?
Yes. They can describe objects or numbers that are not in the image, misread small or dense text, miss brief details in video, and give different answers to the same question. Ask for quoted evidence and verify any extracted value used for a business decision.
Is it safe to upload screenshots and documents to AI tools?
It depends on your provider's terms and your settings. Screenshots often contain more than intended, and images from untrusted sources can carry hidden instructions. Crop and redact before uploading, use approved business plans with suitable retention terms, and do not let models act on untrusted files without approval.
How does multimodal AI affect my brand's visibility in AI answers?
Buyers paste screenshots of pricing and comparison pages into AI assistants, and AI engines build answers from the content they can read on your site. Publish key facts as plain, dated text alongside images, add transcripts to videos and HTML summaries to PDFs, and keep facts consistent everywhere. That reduces misreading but never guarantees inclusion.