Abstract Clouds significantly affects the power output of solar energy systems, and it also decrease the life of modules in photovoltaic system and receivers in the concentrating solar power system, resulting in costs of operation and maintenance. In the present study, the MRR-YOLO model was proposed, which was based on deep learning and instance segmentation technique. The MSDA, the RCS-OSA, and the RFAConv modules were used, and their functions to the cloud segmentation were investigated. Results found that the MSDA module helped maintain the model’s lightweight, the RFAConv module had a better feature extraction of the clouds. The instance segmentation method was better than semantic segmentation in clouds detection, especially for clouds of varying shapes. The PB, the RB, and the mAP50B of the MRR-YOLO model were 79.2%, 66%, and 74.7%, respectively. For the segmentation task, the PM, the RM, and the mAP50M were 79.3%, 64.8%, and 73%, respectively. The MRR-YOLO model was validated by the SWIMSEG, the CCSN, the all-sky datasets, and the real cloud images in Zhengzhou city, it had a better detection performance and applicability than other models. A heatmap comparison was also conducted, showing that the model accurately detected all features of the clouds. Similar content being viewed by others Data availability The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. References Abdalla, S. A mathematical model for economic and prognostic studies of solar photovoltaic power: Application to China, the EU, the USA, Japan and India compared to worldwide production. Renew. Energy Focus. 50, 100607 (2024). Borodiņecs, A., Lebedeva, K. & Odineca, T. Evaluation of household electricity consumption in multi-apartment buildings for optimization of rooftop PV systems. Energy Build., 114971. (2024). Rakhmatov, A. et al. Advancements in renewable energy sources (solar and geothermal): A brief review[C]//E3S Web of