Web Analytics
FaceIndex.org

Visual search research methods

FaceIndex Labs

How to evaluate visual search with known answers, comparable inputs, and documented errors.

Camera optics and test materials in a photographic laboratory

What a comparison can measure

NIST’s one-to-one verification track compares a probe with a claimed reference. Its one-to-many track searches across a predefined gallery. A public image-search product can also miss a relevant page because that page was never indexed, was blocked, or disappeared. This distinction matters when a product returns no result.

This is a proposed method. FaceIndex reports no benchmark scores on this page.

A reproducible test design

  1. Build a consented reference set

    Use opt-in adult participants or images licensed for this exact purpose. Record the dataset version. Keep material off public search services if the consent or license does not cover their handling.

  2. Separate search conditions

    Measure same-file reuse and known edited copies separately from one-to-many similarity. Include an authorized negative set and distinguish absent sources from retrieval misses.

  3. Compare the same inputs

    Record the date, product/version where known, conditions, number of trials, and how no-result cases were counted. Vary one condition at a time when feasible.

  4. Review errors and disclose incentives

    Count useful correct results and wrong candidates separately. Report denominators and limitations. Identify FaceIndex as a product operator in any comparison it publishes.

Read more

Build a scorecard you can reproduce

Set aside separate groups for known exact copies, authorized edits, and queries with no expected result. Record the expected destination for positive examples before searching. Search interruption, provider error, and a completed search with no result deserve different labels. Changing more than one photo condition at once makes it hard to interpret a miss.

Record the observations

Known copy found
Count a correct source only if the result opens to the expected public page or a documented equivalent. Report the number found over the number of eligible, accessible known copies.
Incorrect candidate
Record misleading candidates separately and identify which comparison produced them. Count negative queries that returned an incorrect lead as well as individual wrong candidates.
Source unavailable
Put a removed or blocked destination in its own category. Preserve the denominator and explain how unavailable sources affect it.

For repeat tests, preserve the approved reference set, the query date, and the decision threshold where the provider exposes one. Comparing two runs from different dates can mix changes to the index with changes to the algorithm.

Worked example: an edited copy falls out of the results

Suppose an exact duplicate appears, but a resized and cropped version does not. Before judging the tool, confirm the expected public destination remains accessible. Repeat the comparison with resize and crop as separate conditions. If only the crop drops out, report that observation for this set and this test version. Small examples help design a larger test. They do not establish a general accuracy claim.

Use a consenting or appropriately licensed test set. Confirm what each product may retain or disclose before including it in the study.

Industry benchmarks can explain the error terms, but their datasets and thresholds are different from a public search service’s coverage. NIST verification evaluation · NIST identification evaluation.

Primary sources