UAV remote sensing image extraction of single tree crown of Chinese fir based on Mask R-CNN and transfer learning

Xie Yunhong; Sun Zhao; Ding Zhidan; Luo Mi; Li Yun; Sun Yujun

doi:10.12171/j.1000-1522.20210343

Journal of Beijing Forestry University > 2024 > 46(3): 153-166. > DOI: 10.12171/j.1000-1522.20210343

Xie Yunhong, Sun Zhao, Ding Zhidan, Luo Mi, Li Yun, Sun Yujun. UAV remote sensing image extraction of single tree crown of Chinese fir based on Mask R-CNN and transfer learning[J]. Journal of Beijing Forestry University, 2024, 46(3): 153-166. DOI: 10.12171/j.1000-1522.20210343

Citation:

PDF (5254 KB)

UAV remote sensing image extraction of single tree crown of Chinese fir based on Mask R-CNN and transfer learning

National Forestry and Grassland Administration Key Laboratory of Forest Resources & Environmental Management, Beijing Forestry University, Beijing 100083, China

More Information

Received Date: September 02, 2021
Revised Date: October 19, 2023
Accepted Date: January 02, 2024
Available Online: January 07, 2024

Graphical Abstract

Abstract

Abstract

Objective
The UAV image automatically extracts canopy for precise information.
Method
In this study, single tree crown extraction method from UAV images based on Mask R-CNN and transfer learning was proposed. Firstly, the optimized Mask R-CNN instance segmentation model based on faster R-CNN was selected, and the optimal feature extraction network was chosen between ResNet50 and ResNet101. Secondly, transfer learning was introduced to train with Mask R-CNN together, combined with the guiding role of transfer learning to reduce training time and improve training accuracy.
Result
The results showed that the overall accuracy of the Mask R-CNN model was 93.59%, the user accuracy, F1 score and mean average precision were 65.46%, 76.05% and 0.31, respectively. After adding transfer learning, the user accuracy of Mask R-CNN model was increased by 29.53%, the F1 score was increased by 19.63%, and the average accuracy was increased by 0.21. In the Mask R-CNN model with ResNet50 and ResNet101 as feature extraction networks, the average values of overall accuracy, user accuracy, F1 score and mean average precision of the ResNet50 + Mask R-CNN model were 96.94%, 95.57%, 96.17% and 0.54, respectively. The average values of overall accuracy, user accuracy, F1 score and mean average precision of the ResNet101 + Mask R-CNN model were 96.20%, 94.41%, 95.19% and 0.49, respectively. The R² of ResNet50 + Mask R-CNN model loaded with transfer learning in predicting east-west crown width, north-south crown width, crown area and quadrat canopy density were 0.87, 0.84, 0.93 and 0.83, respectively.
Conclusion
The method based on Mask R-CNN and transfer learning proposed in this study obtains more accurate results of tree crown parameters, which provides a fast and efficient solution for tree resource assessment in UAV images.
- UAV,
- remote sensing image,
- deep learning,
- Mask R-CNN,
- transfer learning,
- crown extraction

FullText(HTML)

References (33)

References

[1]	覃先林, 李增元, 易浩若. 高空间分辨率卫星遥感影像树冠信息提取方法研究[J]. 遥感技术与应用, 2005(2): 228−232. Qin X L, Li Z Y, Yi H R. Extraction method of tree crown using high-resolution satellite image[J]. Remote Sensing Technology and Application, 2005(2): 228−232.
[2]	孙振峰, 张晓丽, 李霓雯. 机载与星载高分遥感影像单木树冠分割方法和适宜性对比[J]. 北京林业大学学报, 2019, 41(11): 66−75. Sun Z F, Zhang X L, Li N W. Comparison of individual tree crown extraction method and suitability of airborne and spaceborne high-resolution remote sensing images[J]. Journal of Beijing Forestry University, 2019, 41(11): 66−75.
[3]	Santos A A, Marcato J J, Araújo M S, et al. Assessment of CNN-based methods for individual tree detection on images captured by RGB cameras attached to UAVs[J]. Sensors, 2019, 19(16): 3595. doi: 10.3390/s19163595
[4]	于东海, 冯仲科. 基于无人机倾斜航空影像的树冠体积测算方法[J]. 农业工程学报, 2019, 35(1): 90−97. doi: 10.11975/j.issn.1002-6819.2019.01.011 Yu D H, Feng Z K. Tree crown volume measurement method based on oblique aerial images of UAV[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2019, 35(1): 90−97. doi: 10.11975/j.issn.1002-6819.2019.01.011
[5]	汪小钦, 王苗苗, 王绍强, 等. 基于可见光波段无人机遥感的植被信息提取[J]. 农业工程学报, 2015, 31(5): 152−159. doi: 10.3969/j.issn.1002-6819.2015.05.022 Wang X Q, Wang M M, Wang S Q, et al. Extraction of vegetation information from visible unmanned aerial vehicle images[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2015, 31(5): 152−159. doi: 10.3969/j.issn.1002-6819.2015.05.022
[6]	Huang H, Li X, Chen C. Individual tree crown detection and delineation from very-high-resolution UAV images based on bias field and marker-controlled watershed segmentation algorithms[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(7): 2253−2262. doi: 10.1109/JSTARS.2018.2830410
[7]	李丹, 张俊杰, 赵梦溪. 基于FCM和分水岭算法的无人机影像中林分因子提取[J]. 林业科学, 2019, 55(5): 180−187. doi: 10.11707/j.1001-7488.20190520 Li D, Zhang J J, Zhao M X. Extraction of stand factors in UAV image based on FCM and watershed algorithm[J]. Scientia Silvae Sinicae, 2019, 55(5): 180−187. doi: 10.11707/j.1001-7488.20190520
[8]	张凝, 冯跃文, 张晓丽, 等. 结合航空影像纹理和光谱特征的单木冠幅提取[J]. 北京林业大学学报, 2015, 37(3): 13−19. Zhang N, Feng Y W, Zhang X L, et al. Extracting individual tree crown by combining spectral and texture features from aerial images[J]. Journal of Beijing Forestry University, 2015, 37(3): 13−19.
[9]	孙钊, 潘磊, 孙玉军. 基于无人机影像的高郁闭度杉木纯林树冠参数提取[J]. 北京林业大学学报, 2020, 42(10): 20−26. Sun Z, Pan L, Sun Y J. Extraction of tree crown parameters from high-density pure Chinese fir plantations based on UAV images[J]. Journal of Beijing Forestry University, 2020, 42(10): 20−26.
[10]	徐永胜. 基于无人机影像的林木冠幅提取与树种识别研究[D]. 哈尔滨: 东北林业大学, 2021. Xu Y S. Research on tree crown extraction and tree species recognition bases on UAV images[D]. Harbin: Northeast Forestry University, 2021.
[11]	Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation[C]//International Conference on Medical image computing and computer-assisted intervention. Munich: Technische Universität München, 2015.
[12]	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Boston: Computer Vision, 2015.
[13]	Li W J, Fu H h, Yu L, et al. Deep learning based oil palm tree detection and counting for high-resolution remote sensing images[J]. Remote Sensing, 2017, 9(1): 22.
[14]	田萱, 王亮, 丁琪. 基于深度学习的图像语义分割方法综述[J]. 软件学报, 2019, 30(2): 440−468. Tian X, Wang L, Ding Q. Review of image semantic segmentation based on deep learning[J]. Journal of Software, 2019, 30(2): 440−468.
[15]	He K, Gkioxari G, Dollár P, et al. Mask R-CNN[C]//Proceedings of the IEEE international conference on computer vision. Venice: Computer Vision, 2017.
[16]	Li Q, Arnab A, Torr P H S. Weakly-and semi-supervised panoptic segmentation[C]//Proceedings of the European conference on computer vision (ECCV). Munich: Computer Vision, 2018.
[17]	高云, 郭继亮, 黎煊, 等. 基于深度学习的群猪图像实例分割方法[J]. 农业机械学报, 2019, 50(4): 179−187. Gao Y, Guo J L, Li X, et al. Instance-level segmentation method for group pig images based on deep learning[J]. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50(4): 179−187.
[18]	Dai J, He K, Sun J. Instance-aware semantic segmentation via multi-task network cascades[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas: Computer Vision, 2016.
[19]	Bai M, Urtasun R. Deep watershed transform for instance segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: Computer Vision, 2017.
[20]	Tan C, Sun F, Kong T, et al. A survey on deep transfer learning[C]//International conference on artificial neural networks. Springer: European Neural Network Society, 2018.
[21]	Gu S, Ding L. A complex-valued vgg network based deep learning algorithm for image recognition[C]//2018 Ninth International Conference on Intelligent Control and Information Processing (ICICIP). Wanzhou: International Conference on Intelligent Control and Information Processing (ICICIP), 2018.
[22]	李博, 江朝晖, 谢军, 等. 基于迁移学习的园艺作物叶部病害识别及应用[J]. 中国农学通报, 2021, 37(7): 138−143. Li B, Jiang C H, Xie J, et al. Leaf disease recognition of horticultural crops based on transfer learning and its application[J]. Chinese Agricultural Science Bulletin, 2021, 37(7): 138−143.
[23]	查志华, 周文静, 吴杰. 基于迁移学习Faster R-CNN模型田间红提葡萄果穗的识别[J]. 石河子大学学报(自然科学版), 2021, 39(1): 26−31. Zha Z H, Zhou W J, Wu J. Identification of red globe grape cluster in grapery with Faster R-CNN model based on transfer learning[J]. Journal of Shihezi University (Natural Science Edition), 2021, 39(1): 26−31.
[24]	Ocer N E, Kaplan G, Erdem F, et al. Tree extraction from multi-scale UAV images using Mask R-CNN with FPN[J]. Remote Sensing Letters, 2020, 11(9): 847−856. doi: 10.1080/2150704X.2020.1784491
[25]	Safonova A, Guirado E, Maglinets Y, et al. Olive tree biovolume from uav multi-resolution image segmentation with mask r-cnn[J]. Sensors, 2021, 21(5): 1617. doi: 10.3390/s21051617
[26]	Hao Z, Lin L, Post C J, et al. Automated tree-crown and height detection in a young forest plantation using mask region-based convolutional neural network (Mask R-CNN)[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 178: 112−123. doi: 10.1016/j.isprsjprs.2021.06.003
[27]	Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. Advances in Neural Information Processing Systems, 2015, 28: 91−99.
[28]	Sun W, Wang R. Fully convolutional networks for semantic segmentation of very high resolution remotely sensed images combined with DSM[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(3): 474−478.
[29]	Sharma N, Jain V, Mishra A. An analysis of convolutional neural networks for image classification[J]. Procedia Computer Science, 2018, 132: 377−384. doi: 10.1016/j.procs.2018.05.198
[30]	Torrey L, Shavlik J. Transfer learning[M]//Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. Hershey: IGI Global, 2010.
[31]	梁泽, 黄姣, 韦飞黎, 等. 基于夜光遥感影像与百度POI数据的中国城市群空间范围识别方法[J]. 地理研究, 2020, 39(1): 92−102. Liang Z, Huang J, Wei F L, et al. Identifying the spatial range of urban agglomerations in China based on night light remote sensing and POI data[J]. Geographical Research, 2020, 39(1): 92−102.
[32]	伍艳莲, 赵力, 姜海燕, 等. 基于改进均值漂移算法的绿色作物图像分割方法[J]. 农业工程学报, 2014, 30(24): 161−167. Wu Y L, Zhao L, Jiang H Y, et al. Image segmentation method for green crops using improved mean shift[J]. Transactions of the Chinese Society of Agricultural Engineering (Transactions of the CSAE), 2014, 30(24): 161−167.
[33]	Özcan A H, Hisar D, Sayar Y, et al. Tree crown detection and delineation in satellite images using probabilistic voting[J]. Remote Sensing Letters, 2017, 8(8): 761−770. doi: 10.1080/2150704X.2017.1322733

[1]	Zhang Haiyan, Yuan Mingshuai, Jiang Qi, Sun Yu, Cui Jian, Ren Lili, Luo Youqing. Deep learning model compression for real-time listening of boring vibration[J]. Journal of Beijing Forestry University, 2021, 43(6): 92-100. DOI: 10.12171/j.1000-1522.20200100
[2]	Sun Zhao, Pan Lei, Sun Yujun. Extraction of tree crown parameters from high-density pure Chinese fir plantations based on UAV images[J]. Journal of Beijing Forestry University, 2020, 42(10): 20-26. DOI: 10.12171/j.1000-1522.20190386
[3]	Sun Zhenfeng, Zhang Xiaoli, Li Niwen. Comparison of individual tree crown extraction method and suitability of airborne and spaceborne high-resolution remote sensing images[J]. Journal of Beijing Forestry University, 2019, 41(11): 66-75. DOI: 10.13332/j.1000-1522.20180446
[4]	Liu Jiazheng, Wang Xuefeng, Wang Tian. Research on image recognition of five bark texture images based on deep learning[J]. Journal of Beijing Forestry University, 2019, 41(4): 146-154. DOI: 10.13332/j.1000-1522.20180242
[5]	FENG Jing-jing, ZHANG Xiao-li, LIU Hui-ling.. Single tree crown extraction based on gray gradient image segmentation[J]. Journal of Beijing Forestry University, 2017, 39(3): 16-23. DOI: 10.13332/j.1000-1522.20160373
[6]	FENG Jing-jing, ZHANG Xiao-li, LIU Hui-ling. Single tree crown extraction based on gray gradient image segmentation[J]. Journal of Beijing Forestry University, 2017, 39(3): 16-23. DOI: 10.13332/j.1000--1522.20160373
[7]	ZHANG Shuai, HUAI Yong-jian.. Leaf image recognition based on layered convolutions neural network deep learning.[J]. Journal of Beijing Forestry University, 2016, 38(9): 108-115. DOI: 10.13332/j.1000-1522.20160035
[8]	ZHANG Ning, FENG Yue-wen, ZHANG Xiao-li, FAN Jiang-chuan. Extracting individual tree crown by combining spectral and texture features from aerial images[J]. Journal of Beijing Forestry University, 2015, 37(3): 13-19. DOI: 10.13332/j.1000-1522.20140309
[9]	ZHANG Zhi-du, XU Cheng-yang, CAI Bao-jun, LI Cui-cui, YUAN SHi-bao, YANG Cheng, PENG Qiang, JIN Gui-xiang. Influence of stock density on crown structure of Prunus davidiana.[J]. Journal of Beijing Forestry University, 2009, 31(6): 187-192.
[10]	LIU Qing-wang, LI Zeng-yuan, CHEN Er-xue, PANG Yong, WU Hong-gan. Extracting height and crown of individual tree using airborne LIDAR data.[J]. Journal of Beijing Forestry University, 2008, 30(6): 83-89.

Cited By

Cited by

Periodical cited type(3)

1.	李林. 生活垃圾绝氧热解发电工艺构建与评估. 工业技术创新. 2020(01): 53-59 .
2.	陈立林，刘琼，杜沄燕. 森林自燃火灾中的凋落物辐射升温特性研究——以岳麓山为例. 中国安全生产科学技术. 2018(02): 174-179 .
3.	李华，李林. 竹类生物质热解技术构建与评估. 工业技术创新. 2017(03): 49-53 .

Other cited types(5)

Get Citation

PDF

XML

Article views (654) PDF downloads (92) Cited by(8)

Turn off MathJax

Article Contents

Abstract

References

UAV remote sensing image extraction of single tree crown of Chinese fir based on Mask R-CNN and transfer learning