Dataset Preparation for AI

Principles and Suitability

Dataset preparation organizes the lifecycle from collection and splitting through labeling to reproducible versions. Unlike annotation, which concentrates on answers for individual examples, preparation governs the collection as a whole. Split data before any step that learns statistics from it, so information reserved for evaluation does not leak into development.

A dataset includes images, answers, and metadata explaining their origins; arranging files is not enough. Training data learns parameters, validation selects methods or thresholds, and test data checks results after those choices. Leakage means information that should be unavailable during development becomes available, inflating scores without representing new operational images. Adjacent, nearly identical frames placed in different sets are one example.

Engineers choose the separation unit to match what will be new in deployment. If the claim concerns new cameras, hold cameras out according to that scope rather than randomly splitting images. A manifest lists files and version information for reconstruction; hashes detect content changes. Preserve these alongside answers and preparation methods. Filenames alone do not prove that a test set remains unchanged after data corrections.

Scroll the diagram horizontally to inspect all details.

Illustrative Industrial Example

Design example: consecutive images of one gauge are grouped by device and recording session, then devices or time periods are held out for testing. Adjacent frames are not randomly distributed across training and test sets. The split should preserve the intended boundary rather than merely produce conveniently sized folders for an experiment.

Suppose a recording session contains many images with almost identical needle positions. Random image-level splitting could expose almost every condition during training. Holding out the whole session tests images with less repeated context, but ask whether the same camera and housing justify the intended conclusion. A time holdout does not establish suitability for a new factory whose lighting and materials are entirely different.

Engineers inspect near duplicates before splitting and preserve device and session groups in the manifest. They calculate scaling only from training data and apply locked values to validation and test sets. Label corrections require a new version and checks that comparisons use identical reference answers. Excluding damaged images requires reasons and counts, not removal of difficult test cases merely to reach a desired score.

Study and Trial Workflow

Evaluation and Metric Definitions

Inspect class proportions and lighting conditions in each set, and verify that devices or near-duplicate images do not cross the declared boundary. Record exclusion counts and reasons. These checks concern the actual separation rule and sample contents, not simply whether train, validation, and test directories have different names or contain different filenames.

Differences between set proportions may be a problem or an intentional test condition; explain why rather than forcing equality without context. Determine whether important groups have enough examples for separate reporting, or disclose that they do not. State uncovered conditions and inspect usage rights and personal information before publication. Correct splitting does not remove restrictions governing the use of the images themselves.

Engineers reconstruct sets from manifests and verify hashes and answers before evaluation. Compare identifiers and image similarity across sets using the declared separation unit, and confirm that statistics-learning steps never read test data. If repeated inspections select models, that set may have become development data. Reserve fresh independent data for final conclusions rather than relying on a filename containing the word test as proof of independence.

Limitations and Failure Cases

Device separation does not guarantee robustness at a new factory when scenes and cameras remain similar. Future information and test statistics must not select models. The boundary supports only the declared question, so conclusions should describe what was genuinely held out rather than imply coverage of every future operating environment.

Grouping reduces memorization of context but also reduces training data, and some groups may contain distinctive gauge types that complicate comparisons. Plan adequate coverage before collection instead of moving images back because results are low. If only one source is available, report that limitation honestly. A feasible split does not prove robustness to sources that have never been examined, even when its internal separation is correct.

Data and answers change during development. Version preservation adds storage and maintenance work but supports retrospective review. Assign approval responsibility for definition changes and keep exclusion history. Predictions from the model under evaluation must not become reference answers without independent checking. Dataset preparation describes a design process, not certification that a collection meets a standard or a claim that a product already handles versioning automatically.

Practical Checklist

• Can the split be reconstructed from its manifest?
• Is validation genuinely separate from testing?
• Do label corrections create a new version?

References and Further Reading

• scikit-learn — Cross-validation
• scikit-learn — Common pitfalls and data leakage