Running a Vision Model on an Edge NPU: What Actually Decides Real-Time Performance

A model that hits its accuracy target on a GPU can still miss its frame rate on a device. On edge hardware, real-time performance is decided as much by the pipeline around the model as by the model itself. These are the decisions that matter most.

Pick the runtime before you finalise the model

Each NPU toolchain, such as Rockchip RKNN, TensorFlow Lite with a hardware delegate, or ESP-DL on microcontrollers, supports a specific set of operators. An operator the NPU does not support falls back to the CPU, and a single fallback in the middle of a network can cost more time than the rest of it. Check operator support against the target toolchain while the architecture is still easy to change.

Quantise with data from the real camera

INT8 quantisation is usually required for good NPU throughput. Calibrate it with frames captured from the actual sensor and optics on the device, not only with a public dataset. Then validate accuracy on the device after conversion, because quantisation error is easy to miss if you only test the float model.

Measure the whole pipeline, not just inference

End-to-end latency includes capture, colour conversion and resizing, inference, post-processing such as non-maximum suppression, and whatever happens with the result. Pre-processing done on the CPU is a common bottleneck. Moving it to the SoC's 2D or ISP hardware often gives back more time than further model optimisation.

• Confirm operator support on the target NPU before training
• Calibrate INT8 quantisation with on-device camera frames
• Re-check accuracy on the device after conversion
• Profile capture, pre-processing, inference and post-processing separately
• Offload resize and colour conversion to hardware where the SoC allows it

If you have a model that works on a workstation and need it running reliably on a device, send a project brief with the target SoC and frame rate you need.