Visual search evaluation: measure the answer, not just the model

1. Define relevance at the event level

A search benchmark needs a clear unit of judgment. For video, use a documented event boundary and evidence interval rather than counting every nearby frame as another success. Define how overlapping detections, fragmented tracks and repeated views map to one event. An attractive thumbnail or a high embedding score is not a relevance label. Reviewers need enough authorized footage to assess the requested attributes and their relationship together.

For the translated query “Find a woman in a red shirt holding a box around 13:00”, relevance requires the confirmed time scope, the relevant person’s red shirt and that person holding the same box. The person category is user-described apparent presentation, not proof of gender identity or sex. Do not infer protected traits or identity, and do not use uncertain matches for punitive automation. Document supported, contradicted and unknown conditions, with manual review when evidence is insufficient.

2. Build reference judgments without similar-frame leakage

Create a query set that includes ordinary requests, ambiguous time expressions, negatives, occlusion, relation confusion and genuine no-match cases. Establish date, timezone and approximate windows through user-confirmed interpretations rather than evaluator assumptions. Reference judgments should describe what footage was available and what was not observable. Use a written labeling guide, independent review where appropriate and a process for resolving disagreements; human annotations can also be wrong.

Split development and test by camera and day, keeping adjacent frames and related event segments in the same partition. Where feasible, include held-out sites or conditions without claiming they represent every deployment. Tune thresholds, candidate depth and fusion settings on development data, then freeze the test protocol. Record schema, detector, tracker, embedding and reranker versions so a change in labels or preprocessing is not mistaken for an improvement in retrieval.

• Include a red shirt without a box, a red box beside a different person, and a holding relation hidden by occlusion.
• Include explicit negatives, unknown observations, invalid numerical values and valid zero values as separate cases.
• Include missing date or timezone requests that must pause for confirmation, not silently select a day.
• Keep camera and day partitions distinct, document exclusions and retain event-level rather than frame-duplicated judgments.

3. Measure retrieval quality and whole-pipeline latency

Recall@K is the number of relevant events retrieved in the first K unique event results divided by the number of relevant events in the judged, eligible corpus. Precision@K divides the relevant events in those positions by K, treating missing positions in a shorter list as non-relevant. For K=5, returning only one relevant event gives Precision@K of 1/5=0.2, not 1.0. Precision over returned results is a separate metric and needs a distinct label. If there are no relevant events, recall is undefined; report such queries separately rather than replacing it with a flattering score.

Measure latency from query submission through interpretation, permission checks, retrieval, fusion, reranking and evidence readiness. Report distributions and timeout or failure counts, not only model inference time. A faster answer with reduced candidate coverage may lose recall. Compare structured-only, vector-only and hybrid approaches under the same authorized corpus and confirmed hard constraints, while reporting how unknown relationships and partial matches are counted.

{
  "evaluation_unit": "unique_event",
  "split_keys": ["camera_id", "source_day"],
  "metrics": ["Recall@K", "Precision@K", "end_to_end_latency"],
  "precision_at_k_denominator": "K",
  "missing_positions": "non_relevant",
  "zero_relevant_recall": "undefined_report_separately",
  "states": ["no_match", "restricted_scope", "incomplete_index", "source_unavailable"],
  "versions": ["schema", "detector", "tracker", "embedding", "reranker"]
}

4. Evaluate no-result states and index completeness

A no-result response can mean no eligible match, restricted search scope, incomplete indexing or unavailable source evidence. Test these states separately. The interface should communicate relevant limitations without revealing whether a hidden event exists. Permissions must constrain retrieval and candidate ranking in all branches and be checked again at playback. A permission denial is not a relevance failure that can be solved by relaxing access controls.

Test delayed arrivals, missing segments, deduplication, stale embeddings and reprocessed event revisions. Verify that source capture time is not confused with arrival time and that local bounded track identifiers are not treated as person identity. Check hard time, camera and value constraints at every stage. A semantic result outside the confirmed interval is incorrect even if it looks convincing, while a null or invalid measurement must never pass as zero.

5. Evaluate privacy boundaries and derived-data lifecycle

Map which media, event fields, vectors, query logs and cached results are retained, for what purpose and for how long under the applicable design. Minimize unnecessary collection and avoid turning descriptive search into protected-trait or identity inference. Test whether authorized reviewers see only appropriate evidence. Logs and evaluation exports can expose personal information too, so access control and data minimization must extend beyond the playback screen.

Exercise expiry and deletion across source media, event records, indexes, embeddings and caches. Confirm that deleted evidence cannot reappear through delayed processing, saved candidate lists or stale snippets. Record and investigate cleanup failures rather than treating a successful primary-file deletion as complete. Tests should cover rights changing between search and playback, revoked source access and broken evidence references. These are verification topics, not commitments about a product’s retention policy.

6. Review authority boundaries and communicate limits

Security evaluation should include adversarial query text and prompt injection in OCR, captions or retrieved evidence. Attempts to change permissions, invoke tools or override the interpretation must remain untrusted content. Test schema validation and parameterized, allowlisted read-only query execution instead of direct generated SQL. Verify that generated explanations cannot authorize a source, invent missing evidence or hide an unknown relationship behind confident wording.

Use NIST AI RMF to organize risk questions and responsibilities, not as a claim of certification or universal safety. Publish the evaluation scope, limitations, failure categories and conditions requiring review. Reevaluate when cameras, scene conditions, schema, models or retrieval settings change. AIFacon Detection is planned, with a direction involving detection, indexes and NLP search over stored fields plus image or video meaning; this article reports no company benchmark, service guarantee or validated product outcome.