Multimodal AI
A multimodal model works not only with text but with images, audio or video, inside the same mechanism. You can show it a chart, ask about it in words and get a written answer.
How it works
Different data types are brought to a common form: vectors. An image is cut into patches, each becomes a set of numbers, and the model then processes them with the same attention mechanism it uses for words. To it this is one sequence, not two separate worlds.
Which is what lets modalities be linked. The model answers a question about a photograph's contents, reads text from a screenshot, describes a chart, or spots a mismatch between a table and its caption.
Images cost more than text: a single picture takes hundreds or thousands of tokens depending on resolution. Working with documents, that noticeably affects both the bill and how much fits in the context.
Where it is used
- Reading documents and screenshotsPulling data out of an invoice, a table or a screen capture is the most common practical use.
- Checking charts and diagramsThe model spots a discrepancy between figures in the text and what an image shows.
- AccessibilityDescribing images in words and transcribing speech cover tasks that used to need separate systems.
- Fine print remains a problemOn dense tables and low resolution, reading errors are common. Important numbers deserve rechecking.