Real-Time Thermal Obstacle Detection

Low-altitude tactical flight leaves almost no margin for surprises. Towers, poles, and masts are among the deadliest collision hazards a crew can face, and they are exactly the objects that degraded visual environments hide best. Visible-spectrum cameras fail in low light, adverse weather, and high-contrast scenes; thermal sensors keep working, but they bring their own problems. Our client needed obstacle detection running on 14-bit imagery from uncooled long-wave infrared sensors (8–12 µm spectral band, 768×1024 native resolution) — and uncooled thermal is unforgiving. Vertical structures often show only subtle thermal gradients against sky or terrain. Dynamic range is narrow, temporal noise drifts with sensor temperature, and an obstacle can appear darker or brighter than its background depending on solar loading.

The operational constraints were just as hard as the sensing ones. The system had to process multiple frames per second at low latency on embedded avionic hardware, which ruled out dense sliding-window deep learning over full frames. Pure deep learning was also a poor fit for another reason: thermal training data is scarce, and models transferred from visible-spectrum imagery struggle with the fundamentally different statistics of thermal scenes. Purely classical methods, meanwhile, lack the representational capacity to handle the full variability of obstacle appearances — the world is full of thin branches, fence posts, and antenna structures that look a lot like the hazards we care about. And because this is flight safety, false alarms carry a real cost: a system that cries wolf gets ignored. We needed a detector that was fast enough for an embedded flight computer, sensitive enough to catch thin pole-like structures in weak thermal contrast, and disciplined enough to stay quiet about transient thermal artifacts.

We built a hybrid pipeline: classical image processing decides where to look, deep learning decides what it's looking at, and temporal logic decides whether to believe it.

The architecture also supports multi-camera configurations: each sensor runs the full pipeline independently, and detections fuse in a shared azimuth/elevation coordinate frame relative to the aircraft, using per-camera polynomial calibration to map pixels to angles. A global obstacle database accumulates evidence across viewpoints, extending effective coverage without coupling the per-camera pipelines.

The design keeps deep learning where it earns its cost and classical processing everywhere else, and the compute profile shows it:

GPU acceleration cuts ResNet inference time by a factor of 10 versus CPU-only execution, making real-time operation possible. Because the classifier runs only every fifth frame — with optical flow bridging the gaps — and only on the ≤50 candidates the classical stage nominates, worst-case compute is bounded and configurable. Total working memory stays under 500 MB including frame buffers, track databases, and network weights: a footprint compatible with embedded avionic systems.

On the false-alarm side, the 0.95 classification threshold combined with the 4-of-8-frame persistence requirement is designed to keep spurious reports at levels acceptable for operational flight systems. The price is latency, and it's a known, bounded one: confirmed detection averages 4 frames from obstacle appearance, with a minimum of 1 frame for high-confidence obstacles — comfortably inside the multi-second timescales of collision-avoidance maneuvers.

Compute also scales sensibly with scene content: sparse scenes yield few candidates and fast frames, while the per-frame candidate cap bounds cost in heavy clutter.

Evaluated on the BN Dataset — thermal video sequences from low-altitude tactical flights over diverse terrain, times of day, and aircraft maneuvers, with ground-truth obstacle annotations — the system detects poles, towers, and masts down to 10×20 pixels, degrading gracefully as range grows and thermal contrast thins. The residual challenges are the honest ones: thin branches, fence posts, and antennas that genuinely resemble obstacles, and brief track gaps during aggressive maneuvers that the temporal filter is built to tolerate.

The single best trick in this system is the multi-gamma exposure bracket. An obstacle in thermal imagery can be hotter or cooler than its background depending on material and solar loading, and training one classifier to handle both dark-on-light and light-on-dark appearances with limited thermal data is a losing game. Instead, we synthesize three exposures from every patch — linear (γ = 1.0), brightened (γ = 0.5) for dark obstacles, darkened (γ = 2.0) for bright ones — and max-pool the classifier scores across them. Whatever the thermal polarity, at least one variant presents the obstacle with favorable contrast. That's exposure invariance without HDR hardware, multiple captures, or a massive training set. Ablations bore this out: single-gamma variants degraded on very dark or very bright obstacles, while the three-gamma bracket matched the performance of exhaustive search over many gamma values at a fraction of the cost.

The second decision that paid off was refusing to make deep learning do everything. Classical Big-Small filtering is a nearly free attention mechanism: it knows what a vertical step edge looks like and hands the network a short list instead of a full frame. The ResNet does the one thing classical filters can't — discriminate real obstacles from look-alike clutter.

Finally, we treated false alarms as an architecture problem, not a threshold-tuning problem. Optical flow plus IOU matching over an 8-frame window gives robust persistence checking without a heavyweight multi-object tracker — simple, inspectable, and cheap enough for the flight computer.