Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge

Yuanqiao Wang, Wenbo Gou, Chuanyu Wang, Jiangchuan Fan, Weiliang Wen, Xianju Lu, Xinyu Guo, Chunjiang Zhao

Abstract


In order to address the challenge of non-destructive detection of tomato fruit ripeness in controlled environments,this study proposed a real-time instance segmentation method based on the edge device. This method combined the principlesof phenotype robots and machine vision based on deep learning. A compact and remotely controllable phenotype detectionrobot was employed to acquire precise data on tomato ripeness. The video data were then processed by using an efficientbackbone and the FeatFlowNet structure for feature extraction and analysis of key-frame to non-key-frame mapping from videodata. To enhance the diversity of training datasets and the generalization of the model, an innovative approach was chosen byusing random enhancement techniques. Besides, the PolyLoss optimization technique was applied to further improve theaccuracy of the ripeness multi-class detection tasks. Through validation, the method of this study achieved real-time processingspeeds of 90.1 fps (RTX 3070Ti) and 65.5 fps (RTX 2060 S), with an average detection accuracy of 97% compared tomanually measured results. This is more accurate and efficient than other instance segmentation models according to actualtesting in a greenhouse. Therefore, the results of this research can be deployed in edge devices and provide technical support forunmanned greenhouse monitoring devices or fruit-picking robots in facility environments.
Keywords: instance segmentation, phenotype robot, tomato, greenhouse-based plant phenotyping, ripeness detection
DOI: 10.25165/j.ijabe.20241702.8403.

Citation: Wang Y Q, Gou W B, Wang C Y, Fan J C, Wen W L, Lu X J, et al. Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge. Int J Agric & Biol Eng, 2024; 17(2): 200–210.

Keywords


instance segmentation, phenotype robot, tomato, greenhouse-based plant phenotyping, ripeness detection

Full Text:

PDF

References


Begum N, Hazarika M K. Maturity detection of tomatoes using transfer learning. Measurement: Food, 2022; 7: 100038.

Chen G, Muriki H, Pradalier C, Chen Y, Dellaert F. A hybrid cable-driven robot for non-destructive leafy plant monitoring and mass estimation using structure from motion. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023; pp.11809–11816.

Chen S M, Xiong J T, Jiao J M, Xie Z M, Huo Z W, Hu W X. Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map. Precision Agriculture, 2022; 23: 1515–1531.

Huang Y P, Si W, Chen K J, Sun Y. Assessment of tomato maturity in different layers by spatially resolved spectroscopy. Sensors, 2020; 20: 7229.

Zheng T X, Jiang M Z, Li Y F, Feng M C. Research on tomato detection in natural environment based on RC-YOLOv4. Computers and Electronics in Agriculture, 2022; 198: 107029.

Atefi A, Ge Y, Pitla S, Schnable J. Robotic technologies for high-throughput plant phenotyping: Contemporary reviews and future perspectives. Front Plant Sci, 2021; 12: 611940.

Dong M, Yu H Y, Zhang L, Wu M Z, Sun Z P, Zeng D Q, et al. Measurement method of plant phenotypic parameters based on image deep learning. Wireless Communications and Mobile Computing, 2022; 2022: 7664045.

Kolhar S, Jagtap J. Convolutional neural network based encoder-decoder architectures for semantic segmentation of plants. Ecological Informatics, 2021; 64: 101373.

Rahul M S P, Rajesh M. Image processing based automatic plant disease detection and stem cutting robot. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli: IEEE, 2020; pp.889–894.

Numsong A, Posom J, Chuan-Udom S. Artificial neural network-based repair and maintenance cost estimation model for rice combine harvesters. Int J Agric & Biol Eng, 2023; 16(2): 38–47.

Feng X B, He P J, Zhang H X, Yin W Q, Qian Y, Cao P, et al. Rice seeds identification based on back propagation neural network model. Int J Agric & Biol Eng, 2019; 12(6): 122–128.

Lu W, Zeng M J, Qin H H. Intelligent navigation algorithm of plant phenotype detection robot based on dynamic credibility evaluation. Int J Agric & Biol Eng, 2021; 14(6): 195–206.

Xiao D Y, Liang G, Liu C L, Huang Y X. Phenotype-based robotic screening platform for leafy plant breeding. IFAC-PapersOnLine, 2016; 49(16): 237–241.

Li Y T, He L Y, Jia J M, Chen J N, Lyu J, Wu C Y. High-efficiency tea shoot detection method via a compressed deep learning model. Int J Agric & Biol Eng, 2022; 15(3): 159–166.

Lu Y, Chen X Y, Wu Z X, Yu J Z, Wen L. A novel robotic visual perception framework for underwater operation. Frontiers of Information Technology & Electronic Engineering, 2022; 23(11): 1602–1619.

Wang Y Q, Fan J C, Yu S, Cai S Z, Guo X Y, Zhao C J. Research advance in phenotype detection robots for agriculture and forestry. Int J Agric & Biol Eng, 2023; 16(1): 14–25.

Yu S, Fan J C, Lu X J, Wen W L, Shao S, Guo X Y, et al. Hyperspectral technique combined with deep learning algorithm for prediction of phenotyping traits in lettuce. Frontiers in Plant Science, 2022; 13: 927832.

Hu H M, Kaizu Y, Zhang H D, Xu Y W, Imou K, Li M, et al. Recognition and localization of strawberries from 3D binocular cameras for a strawberry picking robot using coupled YOLO/Mask R-CNN. Int J Agric & Biol Eng, 2022; 15(6): 175–179.

Widiyanto S, Nugroho D P, Daryanto A, Yunus M, Tri D. Monitoring the growth of tomatoes in real time with deep learning-based image segmentation. International Journal of Advanced Computer Science and Applications, 2021; 12(12): 0121247.

Zhang Y, Tian Z H, Ma W Q, Zhang M, Yang L L. Hyperspectral detection of walnut protein contents based on improved whale optimized algorithm. Int J Agric & Biol Eng, 2022; 15(6): 235–241.

Liu W, Zou S S, Xu X L, Gu Q Y, He W Z, Huang J, et al. Development of UAV-based shot seeding device for rice planting. Int J Agric & Biol Eng, 2022; 15(6): 1–7.

Weyler J, Milioto A, Falck T, Behley J, Stachniss C. Joint plant instance detection and leaf count estimation for in-field plant phenotyping. IEEE Robot and Automation Letters, 2021; 6(2): 3599–3606.

Hosoi F, Nakabayashi K, Omasa K. 3-D modeling of tomato canopies using a high-resolution portable scanning Lidar for extracting structural information. Sensors, 2011; 11(2): 2166–2174.

Yang L L, Tian W Z, Zhai W X, Wang X X, Chen Z B, Wen L, et al. Behavior recognition and fuel consumption prediction of tractor sowing operations using smartphone. Int J Agric & Biol Eng, 2022; 15(4): 154–162.

Li H, Issaka Z, Jiang Y, Tang P, Chen C. Overview of emerging technologies in sprinkler irrigation to optimize crop production. Int J Agric & Biol Eng, 2019; 123): 1–9.

Zu L L, Zhao Y P, Liu J Q, Su F, Zhang Y, Liu P Z. Detection and segmentation of mature green tomatoes based on mask R-CNN with automatic image acquisition approach. Sensors, 2021; 21(23): 7842.

Fan J C, Zhang Y, Wen W L, Gu S H, Lu X J, Guo X Y. The future of Internet of Things in agriculture: Plant high-throughput phenotypic platform. Journal of Cleaner Production, 2021; 280: 123651.

Jin Y, Liu J, Xu Z, Yuan S, Li P, Wang J, et al. Development status and trend of agricultural robot technology. Int J Agric & Biol Eng, 2021; 14(4): 1–19.

Xiang L R, Nolan T M, Bao Y, Elmore M, Tuel T, Gai J Y, et al. Robotic assay for drought (RoAD): An automated phenotyping system for brassinosteroid and drought responses. Plant Journal, 2021; 107: 1837–1853.

Yang C Z. Plant leaf recognition by integrating shape and texture features. Pattern Recognition, 2021; 112: 107809.

He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. 2017 IEEE international Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.2980–2988.

Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019; pp.9156–9165.

Liu H, Rivera Soto R A, Xiao F, Jae Lee Y. YolactEdge: Real-time instance segmentation on the edge. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an: IEEE, 2021; pp.9579–9585.

Young S N, Kayacan E, Peschel J M. Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum. Precision Agriculture, 2019; 20: 697–722.

Miao Z H, Yu X Y, Li N, Zhang Z, He C X, Li Z, et al. Efficient tomato harvesting robot based on image processing and deep learning. Precision Agriculture, 2023; 24: 254–287.

Peng H X, Xue C, Shao Y Y, Chen K Y, Liu H N, Xiong J T, et al. Litchi detection in the field using an improved YOLOv3 model. Int J Agric & Biol Eng, 2022; 15(2): 211–220.

Omasa K, Ono E, Ishigami Y, Shimizu Y, Araki Y. Plant functional remote sensing and smart farming applications. Int J Agric & Biol Eng, 2022; 15: 1–6.

Li H H, Wei Y Y, Zhang H M, Chen H, Meng J F. Fine-grained classification of grape leaves via a pyramid residual convolution neural network. Int J Agric & Biol Eng, 2022; 15(2): 197–203.

Yin X, Li W H, Li Z, Yi L L. Recognition of grape leaf diseases using MobileNetV3 and deep transfer learning. Int J Agric & Biol Eng, 2022; 15(3): 184–194.

Yang Z K, Li W Y, Li M, Yang X T. Automatic greenhouse pest recognition based on multiple color space features. Int J Agric & Biol Eng, 2021; 14(2): 188–195.

Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City: IEEE, 2018; pp.4510–4520.

Leng Z Q, Tan M X, Liu C X, Cubuk E D, Shi X J, Cheng S Y, et al. Polyloss: A polynomial expansion perspective of classification loss functions. arXiv Preprint, 2022; arXiv: 2204.12511.




Copyright (c) 2024 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office