YOLOv8-Based Load Detection & Jetson Edge Deployment

In modern intelligent crane systems, visual AI is what enables an overhead crane to interpret its surroundings and act accordingly. Kelude Heavy Industry has built a complete visual AI technology stack—spanning camera selection, model training, and edge deployment—around the YOLOv8 object detection algorithm and the Jetson Orin edge computing platform. This stack delivers unmanned operation capabilities across multiple scenarios, including load identification, precision positioning, and safety monitoring. This article breaks down the full technical pipeline of crane visual AI from an engineering perspective.

Four Core Application Scenarios for Crane Vision AI

Different crane vision AI use cases impose varying demands on accuracy and real-time performance. Through years of project experience, Kelude has developed a technical solution framework covering four typical scenarios:

Industrial Camera Selection and Lens Parameter Calculation

Camera selection is the foundation of any overhead crane vision system. Kelude has developed a systematic camera selection framework tailored to the specific requirements of different crane operating scenarios. For standard load identification tasks, a 1.3-megapixel industrial camera paired with a depth sensor is recommended. For precision pick-and-place alignment, a 5-megapixel color camera combined with a 3D structured-light camera delivers the required accuracy.

Focal length calculation is a critical step in the selection process, governed by the core formula: f = (sensor_width × working_distance) / FOV. In a typical top-down crane mounting scenario — with a target area width of 1.5 m and a working distance of 8 m — an IMX265 sensor (sensor width 7.1 mm) yields a focal length of approximately 38 mm. In practice, 35 mm or 50 mm lenses are commonly used. Recommended focal lengths vary by mounting position: 25–50 mm for top-down crane mounting (working distance 8–15 m), 12–25 mm for side-mounted cameras, and 6–12 mm for close-range gripping applications.

For high-temperature steel mill environments, thermal imaging cameras are required, housed in dedicated protective covers with active cooling. Kelude's Technical Solution specifies a 640×480 resolution thermal imager capable of continuous, stable operation at ambient temperatures above 95°C — meeting the demanding requirements of steel billet identification and continuous casting zone monitoring.

YOLOv8 Training Pipeline and Data Augmentation for Crane Vision

Training a YOLOv8 model for crane applications requires a dedicated engineering methodology. For data collection, a minimum of 500 samples per class is recommended, covering varied camera angles (80% top-down, 20% side-view), diverse lighting conditions (daylight, nighttime, backlight, strong glare), and different operating states (empty hook, full load, various load types). LabelImg is recommended for PascalVOC format annotation, while CVAT enables collaborative online annotation for larger teams.

Data augmentation strategies must be customized for the top-down crane perspective. Unlike autonomous driving scenarios, crane vision does not require large-angle rotation augmentation. Instead, the focus is on Mosaic mixing augmentation (to simulate cluttered, irregularly stacked scenes) and brightness adjustment (to handle rapid illumination changes in steel mills). Kelude's standard training configuration starts from a YOLOv8n pretrained model, with 200 epochs, 640×640 input image size, an initial learning rate of 0.001, and carefully tuned HSV color-space augmentation parameters.

crane_dataset/ ├── train/images/ # Training Images (*.jpg) ├── train/labels/ # YOLOFormat Annotation (*.txt) ├── val/images/ # Validation Images ├── val/labels/ └── crane_dataset.yaml

The dataset configuration defines six target classes: steel_coil, steel_slab, container, workpiece, crane_hook, and spreader — covering the most common objects encountered in overhead crane lifting operations.

TensorRT Optimization and Jetson Edge Deployment

Edge deployment is the critical bridge that takes crane vision AI from the laboratory to real-world industrial operation. Kelude relies on the NVIDIA Jetson Orin series as its core edge computing platform, with the Jetson Orin NX — delivering 70–100 TOPS of AI compute at just 10–25 W — as the recommended first choice for meeting real-time inference requirements across the vast majority of crane applications.

The model deployment pipeline comprises three technical stages. First, the trained PyTorch model is exported to a TensorRT engine file via model.export(format="engine"), leveraging TensorRT's layer fusion, quantization calibration, and memory optimization to significantly boost inference speed. Next, the TensorRT engine is deployed on the Jetson platform, configured with DeepStream or a custom inference pipeline. Finally, inference results are transmitted in real time to the host dispatching system via the MQTT protocol, completing the closed loop from detection to decision to control.

Inference performance varies significantly across YOLO versions on the Jetson Orin NX. YOLOv8n achieves an inference speed of approximately 8 ms with a model size of just 6.3 MB. The newer YOLOv10n further improves on this — 7 ms and 5.5 MB — delivering near-YOLOv8s accuracy with faster inference, making it Kelude's recommended first-choice model for edge deployment.

Coordinate Transformation and Vision-Guided Positioning Principles

The ultimate goal of visual recognition is to translate pixel data into precise mechanical motion commands, which requires a complete transformation chain from pixel coordinates to world coordinates. Kelude's vision AI system employs the PnP (Perspective-n-Point) pose estimation algorithm, combined with camera intrinsic and extrinsic calibration, to convert the 2D bounding boxes detected by YOLOv8 into 6-DOF pose information (X/Y/Z translation plus pitch/yaw/roll rotation).

In practical engineering, once the camera mounting position is precisely calibrated, detection results are mapped through a coordinate transformation matrix into control commands for the three physical axes: crane bridge travel (X-axis), trolley travel (Y-axis), and hoisting (Z-axis). The end-to-end latency of the complete vision-guided workflow is kept within 150 ms, meeting the real-time control requirements of L4-level unmanned overhead cranes.

Frequently Asked Questions