A Novel Hierarchical Multi-Scale Attention Mechanism for Enhanced YOLOv8 Performance in Unmanned Aerial Vehicle-Based Small Object Detection

Authors

• Mohammed Hasan Mutar, Department of Electrical Engineering, Faculty of Electrical and Computer Engineering, Urmia University, West Azerbaijan
• Morteza Valizadeh, Department of Electrical Engineering, Faculty of Electrical and Computer Engineering, Urmia University, West Azerbaijan
• Alaa Hussein Abdulaal, Department of Electrical Engineering, College of Engineering, Al-Iraqia University, BaghdadFollow
• Mehdi Chehel Amirani, Department of Electrical Engineering, Faculty of Electrical and Computer Engineering, Urmia University, West Azerbaijan

Abstract

Object detection in Unmanned Aerial Vehicle (UAV) images presents significant challenges due to the prevalence of small and densely packed objects, as well as variations in scale, orientation, and lighting conditions. This paper introduces a novel object detection algorithm, Hierarchical Multi-Scale Attention YOLO (HMSA-YOLO), which is an improved version of YOLOv8 designed to address these challenges. The proposed method incorporates a novel Hierarchical Multi-Scale Attention (HMSA) module, a Bidirectional Feature Pyramid Network (BiFPN) for enhanced feature fusion, a modified loss function, and an adaptive anchor optimization technique. The HMSA module effectively captures both channel and spatial dependencies at multiple scales, enabling the network to focus on relevant features for small and densely packed objects. The BiFPN enables more efficient and effective feature fusion across different pyramid levels, while the modified loss function addresses class imbalance and improves localization accuracy. The adaptive anchor optimization technique dynamically adjusts anchor boxes based on attention weights, further enhancing the detection of small objects. Comprehensive experiments on the challenging DOTA and VisDrone datasets demonstrate the superior performance of the proposed method. HMSA-YOLO achieves a remarkable mAP@0.5 of 96.2%, representing a significant 19.4% improvement over the baseline YOLOv8. The proposed method also shows substantial improvements in small object detection (91.2% mAP) and dense object detection (93.7% mAP), while maintaining competitive inference speed. The effectiveness of each component is validated through extensive ablation studies, and the qualitative analysis demonstrates the robustness of the proposed method in various UAV scenarios.

Recommended Citation

Mutar, Mohammed Hasan; Valizadeh, Morteza; Abdulaal, Alaa Hussein; and Amirani, Mehdi Chehel (2026) "A Novel Hierarchical Multi-Scale Attention Mechanism for Enhanced YOLOv8 Performance in Unmanned Aerial Vehicle-Based Small Object Detection," Iraqi Journal for Computer Science and Mathematics: Vol. 7: Iss. 3, Article 1.
DOI: https://doi.org/10.52866/2788-7421.1417
Available at: https://ijcsm.researchcommons.org/ijcsm/vol7/iss3/1