Abstract This study proposes an automated defect detection system for large-scale fudge production, addressing the limitations of manual inspection, which is both labor-intensive and error-prone. Deep learning-based image recognition was employed to classify normal samples and four defect types using multiple object detection models, including SSD and YOLOv4/YOLOv5/YOLOv7/YOLOv8/YOLOv11. Instead of relying on a single model, the system integrates a multi-model strategy combining confidence-weighted voting, rule-based selection for specific defects, and Non-Maximum Suppression (NMS), enabling complementary strengths across models and improving robustness. The selected models were deployed in a real-time inspection system equipped with a flipping mechanism that allows each piece of fudge to be inspected on both sides, thereby expanding the coverage of defects. In evaluations using 1,000 real production samples, first-round accuracies were 47.5% (hole), 56.7% (leak), and 60.9% (white). After applying the flipping mechanism for a second inspection, accuracies increased to 75.8%, 83.6%, and 89.3%, respectively, with hole defects showing the largest improvement (28.3%). Our validation-learned fusion achieves 0.995 mAP@0.5 (on par with the best single model) and 0.944 mAP@0.5:0.95, outperforming YOLOv11 and YOLOv5 by + 0.2 and + 2.4 percentage points, respectively; gains are most evident for White defect and Hole defect. These results indicate that combining multi-model detection with dual-side inspection significantly enhances accuracy and enables the reliable screening of defects in real-time production environments. Similar content being viewed by others Acknowledgements The authors would like to express their sincere appreciation to Hsiehlung Electromechanical Technology Co., Ltd. for their valuable contribution and support to this research under Contract No. 111255. Funding This research was supported by the National Science and Technology Council (NSTC) of Taiwan, R.O.C., under Contract No. NSTC 111–2622-E-029–004. Author information Authors and Affiliations Corresponding author Ethics declarations Competing interests The authors declare no competing interests. Additional information Publisher’s note Springer Nature remains neutral