Multimodal hybrid retrieval: combine methods without blurring their roles

1. Understand what each retrieval branch contributes

Structured retrieval uses stored fields such as camera identifiers, source time, object labels and valid numerical values. It is useful for explicit restrictions and recorded facts, but cannot discover an unrecorded visual relationship merely by adding another field filter. Image and text embeddings represent appearance and language in a form that supports similarity search. They can surface semantically related candidates whose exact wording differs from the query.

CLIP demonstrates learning image-text representations from paired supervision; it does not guarantee that every requested attribute or relationship is present. A vector near “person in a red shirt holding a box” might depict a red object near someone, or a matching shirt without a holding action. Hybrid retrieval is valuable because the branches have different strengths, not because combining their scores converts weak evidence into proof.

2. Define the eligible universe before ranking

Apply permission restrictions before retrieval and candidate ranking in every branch. Confirm the date, timezone and approximate window around a clock expression before using source-time filters. Camera restrictions and numerical-value ranges are hard constraints too. A null or invalid value is not zero, and a result outside the confirmed time range remains ineligible even when its image-text similarity is exceptionally high.

Keep these constraints attached through candidate expansion, fusion, reranking and event aggregation. Filtering only after a global top K can lose eligible results that never reached the candidate list, and unauthorized candidates should not participate in ranking at all. If a vector engine cannot directly express the required restrictions, design an authorized eligible partition or constrained search strategy. Merely increasing K is not a permission boundary or a guarantee of completeness.

• Resolve authorized sources and confirmed time context before creating candidates for either retrieval branch.
• Use schema-validated, allowlisted, parameterized read-only operations for metadata access, not direct generated SQL.
• Preserve camera, time, value-validity and access constraints during expansion and reranking.
• Recheck permissions before evidence playback; ranking access does not grant permanent viewing rights.

3. Choose representations and fusion deliberately

A full-frame vector describes the scene, while a crop vector emphasizes a region and may lose surrounding action context. Clip-level representations can capture temporal context, but their usefulness depends on sampling and training. Record which source region or interval each embedding represents, its preprocessing and model version. Vectors from incompatible model spaces should not be compared as if their distances had the same meaning.

FAISS supplies similarity-search methods; index type, approximation and candidate depth affect speed and which neighbors are recovered. Elasticsearch documents reciprocal rank fusion, which combines rankings using reciprocal rank contributions rather than assuming all raw scores share a scale. RRF can reconcile complementary lists, but does not prove relation correctness. Its window sizes, rank constant and candidate coverage still need evaluation for the actual dataset and query distribution.

Neighbor recall compares an approximate index with exact neighbors under the same representation and distance. Event recall compares retrieved events with independently judged relevant events. Recovering every exact neighbor can still miss the correct event when the representation ignores the requested relationship. Diagnose representation, candidate retrieval and relevance separately; increasing candidate depth cannot reconstruct an action that no sampled frame supports.

{
  "eligible_scope": {"camera_ids": ["cam-03"], "time_confirmed": true, "authorized_only": true},
  "candidate_branches": ["structured_fields", "image_text_vectors"],
  "fusion": "reciprocal_rank_fusion",
  "rerank_checks": ["same_person_red_shirt", "same_person_holding_same_box", "temporal_support"],
  "output_unit": "event",
  "playback": "requires_fresh_authorization"
}

4. Rerank with composition and evidence

Reranking should inspect why a candidate answers the complete question. For the translated example “Find a woman in a red shirt holding a box around 13:00”, shirt color must belong to the relevant person and holding must involve that person and the same box. Treat the person category as user-described apparent presentation, never proof of gender identity or sex. Do not infer protected traits or personal identity; uncertain visual interpretations require authorized review.

Separate supported, contradicted and unknown conditions. A crop showing a red shirt cannot resolve an occluded hand, and a box elsewhere in the scene cannot establish holding. Strong semantic similarity must not erase a contradiction or silently satisfy logical AND. A reviewer should be able to inspect relevant regions and intervals, see the evidence source, and understand which condition prevented an exact match. Avoid punitive automation based on these uncertain descriptions.

5. Aggregate frames into coherent event evidence

A video search should usually return events rather than many adjacent frames of the same moment. Group observations using documented temporal and source rules, with track identifiers scoped to a bounded camera or sequence. ByteTrack offers an association approach, not proof of personal identity. Track fragmentation and identity switches can undermine a proposed relationship, so preserve association uncertainty rather than joining disconnected appearances as one person.

Temporal aggregation must preserve compatibility of evidence. A red shirt at one time and a box held much later are not automatically one matching action. Specify permitted gaps, overlap rules and event boundaries, and retain contradictions within the interval. Sampling may miss a short interaction even when scene vectors look appropriate. Show supporting segments and known coverage gaps instead of presenting a single still as proof of an entire action sequence.

6. Track candidate loss, freshness and explanation limits

Measure where relevant events disappear: missing detections, incomplete fields, embedding approximation, shallow candidate lists, fusion windows or reranking mistakes. Late arrivals and model changes can alter the active evidence. Source media, structured records, embeddings and caches need consistent expiry and deletion; otherwise a stale candidate may point to unavailable footage. A no-result response should reflect searchable coverage without revealing inaccessible events or sources.

Queries, OCR, captions and retrieved evidence can contain prompt injection. Treat them as data and keep access decisions and query authority outside model-generated explanations. Cite authorized event evidence and acknowledge unknown relationships rather than synthesizing a confident story. The planned AIFacon Detection context motivates combining stored fields with image or video meaning, but these principles do not establish an implemented retrieval stack, particular performance level or certified result.