Why edge deployment is the hardest part of surgical AI (and what it takes to get right)
By Dr. Musa Abdulkareem ·
A common failure mode of medical AI projects: the model works beautifully on the research GPU and collapses the moment you try to deploy it in a live clinical setting. Here's why, and what it actually takes to run deep learning inside an operating theatre.
The demo-to-deployment chasm
Most AI work in healthcare stops at the research notebook. You have ultrasound frames, annotated masks, a U-Net variant that segments veins and arteries at 0.87 Dice, and a PowerPoint showing beautiful side-by-side overlays. By every academic metric, the model is done.
Then someone asks the question that kills the project: "Can we use this during surgery?"
And suddenly the work that's been done is less than 20% of the work that needs to be done. Because between "the model is accurate" and "a surgeon is using this in a live procedure" sits a wall of engineering problems that most teams never have to think about.
What actually changes when you leave the research cluster
A cloud GPU doesn't care how long inference takes within reason — 200ms, 500ms, whatever. The research workflow measures accuracy, not latency. It's running on a well-cooled A100 with 80 gigs of HBM and effectively unlimited power. None of that is true in an operating theatre.
At the point of care, four things change at once:
1. Latency becomes a hard constraint, not a preference
Surgeons operate in a tight visual-feedback loop with the ultrasound probe. When a surgeon moves the probe, the image on the screen updates within a frame. Any AI overlay you add has to keep up with that — which in practice means delivering a segmentation result within about 50ms, sometimes less. Past 100ms, the overlay starts to "chase" the probe movement, and the surgeon will turn the feature off inside 30 seconds. It becomes a distraction instead of a tool.
This shifts the entire optimisation problem. You're no longer trying to maximise Dice on a held-out test set; you're trying to maximise Dice subject to sub-50ms inference on a device that fits on a medical cart. That usually means a substantially smaller model than your research one, aggressive quantisation, and TensorRT optimisation — all of which cost you accuracy. The work is finding the best model on the Pareto frontier, not the best model in the abstract.
2. The hardware budget is unforgiving
An Nvidia Jetson AGX Orin gives you ~275 TOPS of INT8 compute in a package you can mount on a procedural cart. That's genuinely a lot — but it's shared with image acquisition, preprocessing, the display pipeline, and any other software running on the box. And it's under thermal constraints that a data-centre GPU will never experience. Jetsons throttle. Hard. If your model pushes the device to 85°C during a three-hour procedure, your inference time doubles and you've just lost the surgeon's trust.
Practical implication: you profile under realistic load, not in a two-minute benchmark. Run the model continuously for an hour in a warm room before you call it deployed.
3. Cloud is not an option
Operating theatres have unreliable network connectivity, they can't tolerate the latency of a round-trip to a cloud endpoint, and regulators won't let patient imagery leave the premises without formal data-governance agreements. "Call an API" is not on the table. This means the entire inference stack — model, preprocessing, post-processing, display — runs locally on a device the size of a small set-top box.
4. Reliability expectations are absolute
If the model fails to produce output for a single frame during surgery, it has to fail gracefully — not show a stale result, not throw a visible error on the screen, not crash the application. The surgeon cannot be managing your software's state. Which means error handling, fallback behaviour, and runtime monitoring are not afterthoughts; they're features you design in from the first commit.
What the pipeline actually looks like
Our deployment pipeline for the surgical segmentation system, roughly:
Train in PyTorch on annotated intraoperative data. Standard, though the data curation is genuinely harder than the modelling — intraoperative ultrasound is noisy, variable between patients, and annotation is expensive because you need a clinician to do it. We spent considerably more time on the data pipeline than on the architecture search.
Export to ONNX, then optimise with TensorRT. The TensorRT step is where you actually get real-time inference. It applies INT8 quantisation (with a calibration pass on representative data to keep accuracy loss bounded), layer fusion, and kernel selection specific to the target GPU. A model that runs at 90ms in PyTorch on the Jetson can easily run at 25ms after TensorRT, without meaningful accuracy loss — provided your calibration set is good.
Build preprocessing and post-processing as part of the graph where possible. Speckle reduction, signal normalisation, contrast enhancement — all of these can be part of the TensorRT graph, eliminating host-device copy overhead. You want as much of the per-frame work as possible living on-device.
Integrate with the display pipeline. Your segmentation mask has to composite with the live ultrasound at the display layer, without introducing extra latency. This is usually where projects discover they need to rewrite parts of their inference harness to work with the device's actual video pipeline rather than a PNG-based debugging loop.
Run continuous monitoring. Inference latency distributions, thermal state, memory pressure, and — critically — a bounded watchdog that handles the case where a frame fails to produce output. Fallback to showing the raw ultrasound is better than freezing the overlay.
The accuracy-vs-latency trade-off is a design decision, not a failure
One of the tensions in this kind of work: deploying at the edge means you're almost always running a model that is measurably less accurate than the one you could run in the cloud. Research teams find this uncomfortable, because the headline number on the paper goes down.
But from the clinical perspective, this is the correct trade-off. A model that is 3% less accurate but runs in real time on-site is clinically useful. A model that is 3% more accurate but runs with 2-second latency on a server 200ms round-trip away is clinically unusable. The second model doesn't actually get deployed. So the accuracy comparison is meaningless.
This is a hard thing to internalise, especially for teams coming from academic backgrounds where maximising a benchmark is the job. Once you accept it, you start designing for deployment from day one — and the whole project changes.
What makes this hard, in the end
It's not that edge deployment is technically impossible; the tooling has matured remarkably. TensorRT works. ONNX works. The Jetson is a capable device. Any reasonable ML engineer can get a model running on a Jetson in a few weeks.
What's hard is getting a model running well enough to trust in a live procedure. That's a different problem. It requires engineering discipline that most research teams don't have, clinical partnership that most engineering teams don't have, and a willingness to stop chasing accuracy numbers and start chasing reliability. The production system ends up 3–6× more code than the research model, and none of that extra code is glamorous.
This is why edge medical AI projects take longer than people think, and why most of them never actually deploy. The chasm between "the model works" and "a surgeon uses this during surgery" is real, and it's where the interesting engineering lives.
If you're working on a similar problem and want to compare notes, we'd be happy to talk — reach us via the contact form on the main site.
Related capability: Clinical & precision-health AI