Documenting a dataset for AI evaluation
Where did this evaluation set come from, exactly?
A model team keeps a reviewable record of which files went into an evaluation set and who approved them.
Humiq / Worked example · AI teams · research labs
A fictional scenario that shows how the workspace is meant to be used. People, places and figures are made up.
The question
A team builds an evaluation set of street-sign images for a navigation model. Months later someone will ask where each image came from and whether the team had the right to use it. The answer should not depend on someone's memory.
The research question: is every image in version 3 of the set licensed for evaluation use and free of identifiable people?
Fingerprints instead of uploads
Each image is fingerprinted in the browser. The license text and the photographer's written agreement are added as notes. Images with recognisable faces are left out of the set before they are ever added.
The team stamps the fingerprints on chain in batches of up to thirty-two, so the existence of each file at that date can be shown later without publishing the files.
Review by someone outside the team
A colleague from the legal side reads the exported brief, checks a sample of files against their fingerprints and records a sign-off from her own wallet. Her verdict and the hash of her checklist are on chain; the files are not.
What it does not do
The record shows what the team checked and who reviewed it. It does not grant rights, does not replace a license and does not prove the images are correct labels. It makes the paper trail easy to follow.