How to Test Computer Vision on Your Site Instead of a Demo Video

Why a clean demo clip misleads you

A vendor demonstration is recorded under conditions that make the model look good: even light, framed subjects, a closed set of classes the model was trained on, and hardware with no concurrent load. Each of those conveniences maps to a distinct failure once the system meets a real site. A daytime warehouse clip will see an entirely different pixel distribution at dusk, under sodium lamps, or when a roller door opens and floods the frame with backlight. A model trained to recognize a fully visible object can silently mislabel the same object half-hidden behind a forklift.

The deeper problem is what is called unknown-class flow. In a demo the operator chooses the objects, so everything in frame belongs to a known class. In production the world is open: a softmax classifier does not say “I do not know” — it spreads probability across the classes it knows and reports the most likely one with whatever confidence the geometry produces. That is silent misclassification, and it is dangerous precisely because the wrong answer looks identical to a confident right answer. The fix is architectural, not a matter of collecting more labels: calibrate confidence thresholds and give out-of-distribution inputs somewhere to go other than the nearest known label, for example a review queue for humans.

The practical lesson is that accuracy measured in a lab is a number about the distribution the demo captured, not about the distribution your site will produce. Treating a benchmark or demo score as a production guarantee is the same error as treating a GPU spec-sheet number as real workload throughput. Before you accept anything, decide which conditions are normal for your operation and test explicitly against those, plus the difficult ones.

• Lighting variability is the most underestimated failure: test dawn, dusk, artificial light and backlight separately.
• Occlusion and identity swaps corrupt every downstream frame in tracking tasks, not just the occluded moment.
• Unknown objects need an explicit “unknown” or low-confidence path, or they will be silently misclassified.
• Edge throughput is a systems problem: measure the whole pipeline under thermal load, not the model on a clean bench.

Capture a reference set from your own cameras

A useful validation set is a small matrix of conditions, not one long unstructured recording. Start by writing down the actual detection task in measurable terms. “Detect people” is not enough; “count people crossing this entrance in each direction while excluding staff” gives the footage a concrete purpose. Record the target class or state, the zone where the rule applies, the shortest event or dwell time that matters, and what the result should trigger.

Then collect normal, difficult, failure and transition samples. Normal samples establish the common workload and baseline behavior. Difficult samples — night, glare, rain, crowding, occlusion, distant targets, motion blur — test the conditions most likely to strip away usable evidence. Failure and ambiguous samples should be kept and labeled rather than deleted, because they reveal whether the camera view, the business rule, the model or the review workflow is the problem. Transition samples around day-to-night changes, scene resets and stream profile changes check recovery.

Capture from real production cameras whenever possible and aim for a majority of your data from genuine site conditions rather than staged setups. Spread collection across at least three cameras and multiple days and shifts; a single-day, single-camera collection produces inflated validation metrics that collapse on the second camera at dusk. Log metadata for every image — camera ID, timestamp, location, shift, weather and any processing applied — because without it you cannot diagnose a regression to a specific source or stratify your test later. Watch for duplicate frames and time correlation: split validation data by day or scene rather than randomly per image, or frames captured a second after the training frames leak into your test set.

• Cover at least three cameras or capture conditions so the test generalizes beyond one device.
• Spend 80% or more of the set on real production footage; use public or synthetic data only to fill rare gaps.
• Every image should carry camera, timestamp and location metadata before you annotate.
• Remove duplicates and split by day or scene, not per-image, to avoid leaked validation.

Measure at the model input, per condition and per class

Judge the footage at the resolution the model actually sees, not at the resolution of a large monitoring window. A target that looks clear in the VMS may lose the distinguishing detail after decode, resize and crop. Record the target's width and height in pixels at the model input across near, middle and far regions of the scene, and also log angle, pose, occlusion frequency, motion blur and compression. There is no universal pixel threshold that suits every task — license plates, helmets, people and occupancy states all demand different evidence — so use the measurements to compare regions and then validate the model on those samples.

On the labeled reference set, compute precision and recall separately for each class and for each condition group, and count false positives and false negatives explicitly. Precision tells you how often an alert is correct; recall tells you how many real events you catch. The two trade against each other, and where you set the operating point depends on which failure is more expensive on your site. On an inspection line a model tuned to catch every defect will flag more good parts; if every false positive stops the line, the tolerable ceiling is much lower than if a flagged part routes to a rework station that clears it in seconds. Report the achievable band under stated conditions, not one headline number.

Also watch confidence calibration. Modern networks are systematically overconfident, so a threshold set on raw outputs may not mean what you think. Where the model is used for counting, access control or safety-adjacent decisions, compare the claimed confidence distribution against the observed correctness on your own footage and route the low-confidence tail to a human review queue. A system that fails visibly — saying “I am uncertain” — is far more trustworthy in operation than one that emits confident wrong answers nobody downstream can distinguish from correct ones.

• Measure target size in pixels at the model input, across near, middle and far scene regions.
• Report precision and recall per class and per condition, plus separate false positive and false negative counts.
• Decide the false-positive ceiling from what an alert costs on your line, not from a vendor's operating point.
• Check whether confidence scores are calibrated on your footage before trusting a threshold.

Run two test tracks: offline model, then live system

Offline footage is ideal for comparing model behavior on repeatable evidence, but it is not a substitute for testing the live system. A full pipeline must decode the actual RTSP stream, handle timestamps and reconnects, run several cameras concurrently, and deliver events and evidence with timing you can trust. Splitting validation into two linked tracks keeps the questions separate. The model track measures detections, misses, false alerts and condition-specific behavior on labeled clips. The system track measures decode stability, end-to-end latency, compute and memory, storage, outputs, and recovery after disconnects.

Throughput is the trap most teams hit. A detector that runs at a comfortable frame rate on an unloaded workstation often collapses on the edge device in the field, where it competes for memory bandwidth, shares the device with decode and preprocessing, and thermally throttles after sustained load. When sustained throughput falls below the camera's frame rate, the pipeline drops frames, and dropped frames break exactly what tracking and action recognition depend on — temporal continuity. The number that matters is sustained whole-pipeline throughput on the target device under realistic load, not the model's peak rate on a clean bench. Measure with the camera attached and the device thermally loaded, and if the model cannot hold the frame rate, right-size it or reduce inference frequency rather than shipping a model that silently drops evidence.

Because exported or transcoded clips differ from the live stream in resolution, frame timing, bitrate and compression artifacts, treat offline model results as a necessary first gate and always confirm them in the live system track before selecting hardware or committing to go-live. Record in a manifest which clip is original and which is exported so every result can be traced back to a source camera and condition when a model version or camera configuration changes.

• Model track: repeatable detection quality on labeled footage, per condition.
• System track: live decode, end-to-end timing, concurrency, storage, outputs, recovery.
• Validate sustained throughput under thermal load with the real camera attached.
• Keep a manifest linking every test result to source camera, clip and stream metadata.

Commission in stages and prove trust before cutover

Treat the move from pilot to production as a staged commissioning rather than a single go-live decision. In the terminology used for industrial vision systems, installation qualification verifies the hardware is installed to specification — camera serials, firmware, cabling and protective enclosures documented. Operational qualification verifies every function works at nominal and boundary conditions: image capture, inference, alarms, and outputs to downstream systems. Performance qualification is the statistical acceptance test in which the system must meet accuracy, false-positive and latency targets on data that represents real operating conditions. Skipping the operational step before the statistical one is a documented source of field failures that are expensive to remediate.

Roughly a week of parallel inspection, and ideally two full weeks across shifts and product variants, is the practical minimum to surface edge cases before you trust the system to make decisions alone. During this parallel run the model observes the real flow while a human still holds the decision, giving you time to tune confidence thresholds and catch alert-fatigue problems. Teams that run long enough see the same patterns: lighting drift from ambient light leaks, silent accuracy decay when the model is never retrained on new variants, and operators who silence alerts by the third month because no refresher training was scheduled.

After cutover, validation does not stop. Monitor the distribution of confidence by class, the number of activations per source and time of day, the share of empty frames, pipeline errors by stage, and latency per step. Keep a modest set of manually labeled production examples and recompute quality on real data periodically, because accuracy that looked stable in acceptance drifts as the environment moves. Re-audit and, where needed, retrain on newly captured conditions rather than assuming the original validation still describes the deployed system.

• IQ/OQ/PQ: verify install, then function at boundaries, then statistical acceptance on real-representative data.
• Run parallel human-plus-system inspection for one to two full weeks before cutover.
• After go-live, monitor confidence distributions, activations by time, and empty frames.
• Schedule retraining and refresher operator training, or trust silently erodes over a quarter.

Budget for the false-positive cost and operator trust

Most industrial vision projects fail on economics, not on raw model quality. Even a small false-positive rate floods operators with nuisance alerts and erodes trust until the system is ignored exactly when it matters. A demo reports the detection rate; production pays for the false positives at real throughput. Before you accept a model, model the cost of a false positive on your operation — does it stop a line, pull an operator away, or clear in seconds at a rework station? — and set the operating point accordingly.

Trust is a live operational asset that you measure indirectly. If operators start dismissing or silencing alerts, that is a signal about alert frequency and usefulness, not about discipline. Pair every go-live with a living standard operating procedure and periodic refresher training, and treat “nobody is complaining” as a lack of instrumentation rather than proof that all is well. The goal of field validation is not a perfect accuracy number but a defensible statement about how the system behaves under the conditions it will actually face, with the gaps named and instrumented before the demo turns into an obligation.