Multimodal Extraction: Images, Audio & Video

Beyond documents

Image extraction

Beyond documents, Content Understanding processes photographs and images:

Image Type	What’s Extracted
Product labels	Brand, ingredients, nutrition facts, warnings, barcodes
Whiteboards	Handwritten text, diagrams, sketches
Screenshots	UI text, form data, error messages
Signs and posters	Title, body text, contact info
Retail shelves	Product names, prices, positions

GreenLeaf scenario: GreenLeaf photographs product labels on incoming seed packages. Content Understanding extracts the seed variety, planting instructions, expiry date, and lot number — automatically populating their inventory system.

Audio extraction

Content Understanding processes audio recordings to extract structured information:

Audio Source	What’s Extracted
Meetings	Key topics, action items, decisions, speakers
Customer calls	Account info, issue category, sentiment, resolution
Interviews	Questions asked, responses, key quotes
Voicemails	Caller name, callback number, purpose

The process:

1. Speech recognition — transcribes the audio
2. Speaker diarisation — identifies who said what
3. Semantic extraction — pulls out structured fields (topics, actions, entities)

DataFlow Corp scenario: DataFlow records 10,000 customer support calls daily. Content Understanding extracts: customer account number (spoken), issue category, steps the agent took, resolution status, and customer satisfaction (inferred from tone).

Video extraction

Video combines visual AND audio extraction:

Video Source	What’s Extracted
Training videos	Slide text, spoken content, key topics, timestamps
Security footage	Events, movements, anomalies, timestamps
Presentations	Slide content, speaker narrative, Q&A sections
Product demos	Feature descriptions, UI text, spoken explanations

The process:

1. Scene detection — identifies key moments and transitions
2. Slide extraction — captures on-screen text and slides
3. Speech transcription — transcribes spoken content
4. Semantic synthesis — combines visual and audio into structured output

ℹ️ Multimodal extraction = RAG gold mine

Multimodal extraction is incredibly powerful for building RAG (Retrieval-Augmented Generation) systems:

• Extract text from all company documents → searchable
• Transcribe all meeting recordings → searchable
• Extract slides from all training videos → searchable
• Extract data from product images → searchable

Now your AI agent can search across ALL company knowledge — documents, meetings, videos, images — from a single query.

Exam relevance: Understanding how Content Understanding feeds into RAG systems connects information extraction to generative AI and agents.

Comparing extraction across modalities

Content Understanding across four modalities

Feature	Key Technique	Output Example
📄 Documents	OCR + layout analysis + field mapping	Vendor: GreenLeaf, Total: $3,400, Date: 15 May 2026
🖼️ Images	OCR + object recognition + field mapping	Product: Tomato Seeds, Expiry: Dec 2026, Lot: A4521
🎙️ Audio	Speech recognition + diarisation + semantic extraction	Speaker 1: reported billing issue, Action: refund processed
🎬 Video	Scene detection + OCR + speech + semantic synthesis	Slide 3: 'Q2 Revenue: $4.2M', Speaker: 'We exceeded targets by 15%'

🎬 Video walkthrough

Flashcards

Question

What three types of media can Content Understanding extract data from (beyond documents)?

Click or press Enter to reveal answer

Answer

Images (product labels, whiteboards, screenshots), Audio (meetings, calls, voicemails), and Video (training, presentations, demos). Each uses different techniques but produces structured data.

Click to flip back

Question

How does Content Understanding process video?

Click or press Enter to reveal answer

Answer

Four steps: 1) Scene detection (key moments), 2) Slide extraction (on-screen text), 3) Speech transcription (spoken content), 4) Semantic synthesis (combines visual + audio into structured output).

Click to flip back

Question

How does multimodal extraction support RAG systems?

Click or press Enter to reveal answer

Answer

By making all company knowledge searchable — documents, meeting recordings, training videos, product images — a single query can find relevant information across ALL modalities. This powers comprehensive AI agents and chatbots.

Click to flip back

Knowledge Check

Knowledge Check

DataFlow Corp wants to extract action items and decisions from their weekly team meeting recordings. Which Content Understanding capability handles this?

ADocument extraction — meetings produce meeting minutes documentsBImage extraction — to read any slides shared during the meetingCAudio extraction — speech recognition with semantic extraction of action itemsDVideo extraction — only if the meeting was on video

Check Answer

Knowledge Check

MediSpark wants to make their entire training video library searchable. Doctors should be able to type a question and find the exact video moment that answers it. What's the best approach?

AAsk GPT-4o to watch each video and write summariesBUse Content Understanding video extraction to index slide text and spoken content, then use embeddings for searchCUse Azure Speech to transcribe the audio onlyDManually tag each video with keywords

Check Answer

---

Next up: Building an Extraction App — putting Content Understanding into a complete application.