[MS] From Noisy Queries to Precise Frames: Query Decomposition for Media Asset Search - devamazonaws.blogspot.com

Introduction: The Problem

Media and entertainment teams work with massive asset catalogs for films and TV series. Each asset often includes structured metadata such as episode ID, scene ID, and shot ID, along with the visual content itself. Teams want a unified search experience that understands both metadata constraints and natural language intent. The problem appears when users provide both types of information in one query, for example:

Two soldiers facing off across a battlefield in episode 20, scene 5, shot 2

If we embed this raw query directly for semantic retrieval, two issues appear:

1. Metadata tokens dilute visual intent in the embedding representation.
2. Retrieval can return visually similar assets from the wrong episode, scene, or shot.

This gap is common in real workflows. Editors, archivists, and production teams often know part of the context, but they still describe the visual moment in natural language. Query decomposition solves this by splitting one noisy query into two useful parts: structured filters and semantic intent.

The Journey: Our Approach and Solution

In production, this works as a query pre-processing layer between user input and retrieval: the parser first identifies grounded metadata signals such as episode, scene, and shot, then removes those tokens to generate a clean sub-query that captures only the visual intent. The metadata fields are converted into strict filter clauses to narrow the candidate pool, while the sub-query is embedded for semantic ranking within that filtered set. This two-path retrieval flow prevents context drift, reduces false positives from adjacent scenes, and still preserves semantic flexibility when users describe actions, objects, or mood in free-form language.

Approach

Our approach decomposes each user query into:

1. Structured filters: episode, scene, shot (and optionally characters)
2. A clean sub-query: natural language description with metadata removed

We then run hybrid retrieval:

1. Apply filters as pre-conditions in metadata search.
2. Use the sub-query for semantic or vector retrieval.
3. Rank merged candidates with a hybrid strategy.

At a high level, the decomposition pipeline follows three steps:

1. Parse query text with any one of the approaches, i.e. Regex or LLM or fine-tuned Entity extraction.
2. Extract explicit metadata fields.
3. Rewrite the residual visual intent as a sub-query.

This design improves precision through filtering and improves recall through semantic matching.

Data available

For this use case, we evaluated approximately 500 user queries with mixed structure:

1. Queries with full contextual metadata (episode, scene, and shot)
2. Queries with partial contextual metadata
3. Queries with no explicit contextual metadata

Ground-truth template

Domain experts labeled the 500 query set using this template:

Query	Shot-id	Scene-id	Episode-id	Sub-query
User input query	Extracted shot value	Extracted scene value	Extracted episode value	Metadata-free semantic query

Sample Queries

Query	Episode ID	Scene ID	Shot ID	Sub-query
Two astronauts repairing a satellite in episode 4 scene 3 shot 11	4	3	11	Two astronauts repairing a satellite
A chef plating dessert in scene 9 shot 2	N/A	9	2	A chef plating dessert
Crowd celebrating at a city marathon finish line	N/A	N/A	N/A	Crowd celebrating at a city marathon finish line
Detective opening an old letter in episode 2	2	N/A	N/A	Detective opening an old letter
Family watching fireworks by a river at night	N/A	N/A	N/A	Family watching fireworks by a river at night

Experiments & Evaluation

Regex vs LLM vs Fine-Tuned Entity Extraction

Each extraction approach comes with different trade-offs depending on your query patterns, budget, and maintainability needs:

Criterion	Regex	LLM (GPT-4.1-mini / GPT-4.1)	Fine-tuned Entity Extraction
Latency	Zero - no network call	4–5 seconds per query	Near-zero after deployment
Cost	Free	Per-token API cost	One-time training + hosting cost
Sub-query rewriting	Manual / rule-based	Natural and fluent	Manual / rule-based - entity extraction only
Typo & variation tolerance	Low - brittle on irregular input	High - generalises across phrasing	Medium - bounded by training distribution
Extensibility to new fields	Requires new rules per field	Prompt update only	Requires re-training or fine-tuning
Operational complexity	Very low	Low (API)	High - model lifecycle, infra, versioning
Best suited for	Consistent, well-structured input	Mixed or ambiguous input	High-volume production with domain shift

Experiment Setup and Variants

We tested four decomposition modes:

Evaluation

We evaluated decomposition quality and retrieval impact using three metric groups:

1. Filter extraction accuracy: exact match for episode, scene, and shot
2. Sub-query quality: ROUGE-L and BLEU against labeled rewrites
3. Retrieval effectiveness: Recall@K (K=5)

The Destination: Outcomes and Learnings

Decomposition Quality

Method	Episode match	Scene match	Shot match	ROUGE-L	BLEU	Latency
GPT-4.1 (zero-shot)	100%	99.7%	100%	1.00	0.86	5s
GPT-4.1-mini (zero-shot)	100%	99.7%	100%	1.00	0.85	4s
GPT-4.1 (few-shot)	100%	100%	100%	1.00	0.86	5s
GPT-4.1-mini (few-shot)	100%	100%	100%	1.00	0.85	4s
Regex	100%	100%	100%	0.97	0.79	0s

Token Consumption

Variant	Avg input tokens	Avg output tokens
Regex	0	0
zero-shot	700	35
few-shot	1053	35

Retrieval Effectiveness (Recall@K, K=5)

1. Query-only baseline: about 27%
2. Sub-query only: about 31.8% to 32.2%
3. Filters + sub-query: best overall, up to 49.4%

Detailed highlights:

Method	Query only	Sub-query only	Filters + sub-query
Regex	27%	32.2%	49.4%
GPT-4.1-mini (zero-shot)	27%	32.2%	46.2%
GPT-4.1-mini (few-shot)	27%	31.8%	49%
GPT-4.1 (zero-shot)	27%	31.8%	43.6%
GPT-4.1 (few-shot)	27%	31.8%	49%

Key observations

1. Few-shot closed the Scene ID gap seen in zero-shot.
2. Regex matched or exceeded LLM quality for structural extraction.
3. Filters plus semantic sub-query produced the strongest retrieval outcome.

Why and When to Use an LLM

Regex is the best choice when queries are structured and format patterns are stable. LLMs become valuable when you need:

1. Robust sub-query rewriting for semantic search quality
2. Tolerance for typos and natural phrasing variations
3. Extensibility to future fields such as character names or location entities

A practical strategy is to use Regex first, then invoke an LLM fallback (say, GPT-4.1-mini) for edge cases that fail rule coverage.

Edge-Case Comparison: Where LLM Wins and Where Regex Leads

The examples below show why a hybrid strategy works best in production. LLM-based decomposition excelled at preserving natural phrasing and correctly separating metadata from intent in ambiguous patterns, while Regex remained strongest on speed, determinism, and zero-cost extraction for stable formats.

Query	Ideal output	Regex output	LLM output	What this shows
Scene 3 A frontal pose of warriors with a hill background	- scene: 3 - sub-query: A frontal pose of warriors with a hill background	- scene: 3A - sub-query: frontal pose of warriors with a hill background	- scene: 3 - sub-query: A frontal pose of warriors with a hill background	LLM handled boundary ambiguity better by keeping 3 as scene and preserving the complete semantic phrase.
Frontal Profile of Spiderman and Superman in scene 10A	- scene: 10A - sub-query: Frontal Profile of Spiderman and Superman	- scene: 10A - sub-query: Frontal Profile of Spiderman and Superman in	- scene: 10A - sub-query: Frontal Profile of Spiderman and Superman	LLM produced cleaner rewrite quality by removing trailing connector tokens after metadata extraction.
A wide shot of cyclists crossing a bridge in scnee 12	- scene: 12 - sub-query: A wide shot of cyclists crossing a bridge	- scene: N/A - sub-query: A wide shot of cyclists crossing a bridge in scnee 12	- scene: 12 - sub-query: A wide shot of cyclists crossing a bridge	LLM tolerated the typo in scene and still extracted the correct filter, while Regex missed the malformed token pattern.

Regex strengths	LLM strengths
Deterministic behavior for known templates	Better handling of ambiguous token boundaries
Zero latency and zero token cost	More fluent sub-query rewriting
Easier debugging for rule failures	Better resilience to natural phrasing variation

Conclusion

Query decomposition is not an optional optimization for media asset retrieval. It is a foundational step when user queries mix contextual metadata with free-form visual descriptions. Our evaluation shows that separating metadata filters from semantic intent nearly doubles retrieval performance versus raw semantic search alone, moving the Recall@K (K=5) from 27% to around 49%. For this use case, Regex is the preferred default because it delivers perfect structural extraction with zero latency and zero token cost. When coverage needs to expand to more ambiguous input styles, GPT-4.1-mini with prompt v2 provides the best cost-to-quality fallback.

Thanks

A special thanks to Mark D'Souza, Prabal Deb, Preethi Natarajan, Priya Bhimjyani for their valuable input.

The featured image was generated using Copilot. Terms can be found here.

Post Updated on July 9, 2026 at 08:00AM
Thanks for reading
from devamazonaws.blogspot.com