Edge AI Benchmark Lab

A model that works in a notebook still has to run somewhere: a CPU server, a camera, a phone. Teams reach for ONNX and 8-bit quantisation because they are “faster and smaller”, then find the speed-up depends on the model and the processor, and that an 8-bit model can quietly give different answers.

Published benchmarks rarely help: they come from someone else's hardware, time only the network and not the image decoding around it, and seldom say whether the faster model still agrees with the original.

Choosing where and how a model runs is a cost and reliability decision. The useful numbers are the ones from the machine you will deploy on, for the whole request (decode, resize, infer), with the memory each option needs and evidence that its answers did not change. That is what this lab produces, every day, for the server this site runs on. It also picks the models for the Street Scene project on the same evidence.

Two ImageNet classifiers with very different designs, MobileNetV3-Small (built for phones) and ResNet-18 (plain convolutions), are prepared three ways when the server image is built: the PyTorch model as trained, its ONNX export, and an 8-bit version calibrated on real photos. A daily job runs every variant in a fresh, isolated process on the same photos and records latency per phase, throughput, CPU use, memory, file size and how often each variant picks PyTorch's top class.

Why ONNX? PyTorch runs a model op by op as Python asks for it. An ONNX file is the whole graph, so a runtime can fuse operations (a convolution and its batch normalisation become one), plan memory once, and pick kernels for the CPU it finds. The weights and the answers stay the same.

Why 8 bits? Storing weights as 8-bit integers makes the file four times smaller, and CPUs with VNNI or AMX instructions multiply 8-bit numbers several times faster than 32-bit ones. The cost is precision: each tensor gets 256 levels, chosen from the ranges seen on calibration photos, and a layer whose values span a huge range loses the detail in the small ones.

When is a CPU enough? When a photo takes milliseconds and requests arrive one at a time, as here: a GPU would sit idle. Edge hardware changes the picture: NPUs and phone accelerators often run only 8-bit models, so the quantisation step stops being optional, and its accuracy check matters more.

Models

MobileNetV3-Small and ResNet-18 (timm, ImageNet-1k), weights pinned by SHA-256

Variants

PyTorch FP32 (eager) · ONNX Runtime FP32 · ONNX Runtime INT8 (static, QDQ)

PyTorch

2.14 CPU build from conda-forge in its own environment, exact packages locked

Export

torch.onnx.export with torch.export graph capture, dynamic batch, opset 18

Quantisation

ONNX Runtime static quantisation: weights per channel, activations per tensor

Calibration

128 COCO 2017 photos under reuse licences; 200 others for agreement

Runner

A fresh child process per measurement, with CPU, memory and time limits

Interface

FastAPI, a daily Procrastinate job, Next.js; results stored in PostgreSQL

One input for every variant. Preprocessing is a single function (shorter side to 256 pixels with bicubic resampling, centre crop, normalise), checked against timm's own evaluation transform to within 0.00001. It runs inside each variant's process and is timed separately, so the “whole photo” figure is what a real request costs.

What is measured. After warm-up runs, each process classifies the same photos one at a time and times preprocessing, inference and postprocessing separately; the median and 95th percentile are reported. CPU use is CPU time over wall time (200% means both cores busy). Throughput uses batches of eight. Memory is read from the process itself after importing the runtime, after loading the model, and at its peak. Start-up (importing the runtime, loading the model) is reported apart, because a server pays it once and a command-line tool every time.

Agreement, not just speed. At the highest thread count every variant also classifies 200 photos that were not used for calibration, and its probabilities are compared with PyTorch's: the share with the same top class (on every photo, and on those where PyTorch is at least 50% sure), the overlap of the top five, and the largest probability difference. ONNX at 32 bits should match to about one part in a million; 8 bits will not, and the question is by how much.

Finding the layers that cannot be 8-bit. Quantised in every layer, MobileNetV3 gave a different top class from the 32-bit model on every one of the 128 calibration photos. Right after its first layer, one channel reaches about 56 while ten of the sixteen stay below 1; one 8-bit range for the whole tensor has steps of about 0.2, so those ten keep only a few distinct values, and the error compounds from there. So at build time each layer is quantised on its own and the damage measured (KL divergence from the 32-bit probabilities), and the most damaging are kept in 32-bit, a few at a time, until the 8-bit model gives the 32-bit model's answer on at least 99% of the calibration photos the 32-bit model is at least 50% sure of. Photos showing nothing in ImageNet's classes get near-tied answers that any change flips, so they are reported but not used to judge. The published agreement is measured on different photos.

Pick a photo or upload your own: it is classified now, on this server, by the three variants of the model you choose, each in a fresh process. You see each variant's top five classes, how long inference took and whether it agrees with PyTorch.

Portfolio demonstration. Each visitor can run 20 photos an hour. Uploaded photos are decoded, classified and discarded; nothing is stored. The models know the 1,000 ImageNet classes only, so most photos of people or scenes get a guess about an object in them.

Results are read from the running system and are not available right now. They appear after the first daily run of the benchmark on this server.

8-bit conversion ruins a model

Every 8-bit model is checked against the 32-bit one on the calibration photos when it is built. If it gives a different top class on photos the 32-bit model is sure of, each layer is quantised on its own to find the ones that do the damage, and the worst are kept in 32-bit until 99% agree. The report shows the all-8-bit result, every step of the search and the layers kept.

An uploaded file is not what it claims

It is decoded in its own process with no secrets, a CPU cap and no file writes. Only JPEG, PNG and WebP are accepted, images over 25 megapixels are refused before their pixels are decoded, EXIF orientation is applied and all metadata dropped, and the models only ever see the clean re-encoded copy.

A measurement hangs or crashes

Each child process has a wall-clock deadline and is killed with its process group when it passes; the API answers with a plain message. A crashed child takes nothing else down.

One variant colours another's numbers

Every model, variant and thread count runs in a new process: no shared memory, thread pools or caches, so peak memory and start-up are each variant's own.

A different CPU gives different numbers

Every run records the CPU model and the instruction sets that matter for inference (AVX2, AVX-512, VNNI, AMX), the CPUs available, the container quota and the software versions. Runs are only comparable on the same hardware id.

The server is busy

Benchmarks and “Try it” take the server's single CPU lease; a visitor waits at most 20 seconds and is then asked to try again. Each visitor gets 20 runs an hour.

PyTorch is missing or broken

The ONNX variants keep working and the API says which part is unavailable; the image build fails instead of shipping a lab with a variant silently missing.

Everything heavy happens when the server image is built: the weights and photos are downloaded from pinned sources and checked against their SHA-256 hashes, each model is exported in the PyTorch environment, and the 8-bit versions are calibrated, checked and written with the same ONNX Runtime that serves them. The server only loads files.

PyTorch comes from conda-forge's CPU builds, installed from a lock file that lists every package with its hash, into its own directory. The application's Python never imports it; only the benchmark's child processes do. A daily job runs the suite (about two minutes of CPU) under the server's CPU lease, so the results above always come from this machine.

$0. No API calls and no GPU: models, photos and runtimes are open source or openly licensed, and everything runs on the same 2-vCPU server as the rest of the site. The daily run costs about two minutes of CPU; the PyTorch environment adds about 1 GB to the server image.

• One CPU, shared: the server also runs the site, so the 95th percentile includes other work, and absolute numbers only hold for this hardware id.
• ImageNet accuracy is not measured here (its images cannot be redistributed); agreement with PyTorch on other photos stands in for “did quantisation hurt”, and published accuracy is quoted from timm.
• Two classification models. Detection, pose and face and number-plate models for Street Scene Intelligence are the next part of the lab.
• PyTorch runs in eager mode, as most code does; torch.compile, OpenVINO and other runtimes are not compared.
• The 8-bit conversion keeps whole layers in 32-bit; finer fixes (per-channel activation scales, cross-layer equalisation, quantisation-aware training) would recover more speed and are not attempted.

The full source is on GitHub under the MIT licence, with the tests.

• registry.pyThe models, their pinned weights and the three variants
• preprocess.pyThe one preprocessing function, and upload sanitising
• export_torch.pyPyTorch to ONNX, in the PyTorch environment
• quantize.py8-bit conversion and the search for sensitive layers
• child.pyThe isolated process that loads a variant and measures it
• benchmark.pyThe suite, and agreement with PyTorch
• hardware.pyThe hardware fingerprint
• service.pyStored runs and “Try it”

Models: MobileNetV3-Small (Howard et al., 2019) and ResNet-18 (He et al., 2016) with weights from the timm library by Ross Wightman, Apache 2.0, trained on ImageNet-1k. Photos: COCO 2017 validation set (Lin et al., 2014), only photos whose Flickr licence allows reuse; annotations CC BY 4.0. Details on the data page.