Promising real-time fruit and vegetable quality detection technologies applicable to manipulator picking process
Abstract
Key words: fruit and vegetable quality detection; machine vision; spectroscopy; tactile sensors; picking scenario
DOI: 10.25165/j.ijabe.20241702.7678
Citation: Feng J T, Yang Q Y, Tian H, Wang Z P, Tian S J, Xu H R. Promising real-time fruit and vegetable quality detection
technologies applicable to manipulator picking process. Int J Agric & Biol Eng, 2024; 17(2): 14–26.
Keywords
Full Text:
PDFReferences
Zhang Z, Lu R. Automated infield sorting and handling of apples. In: Karkee M, Zhang Q (Ed. ). Fundamentals of agricultural and field robotics. Cham: Springer. 2021; pp.267‒295.
Pan L Q, Zhang Q, Zhang W, Sun Y, Hu P C, Tu K. Detection of cold injury in peaches by hyperspectral reflectance imaging and artificial neural network. Food Chemistry, 2016; 192: 134–141.
Zhang Z, Lu Y Z, Lu R F. Development and evaluation of an apple infield grading and sorting system. Postharvest Biology and Technology, 2021; 180: 111588.
Zhang B H, Gu B X, Tian G Z, Zhou J, Huang J C, Xiong Y J. Challenges and solutions of optical-based nondestructive quality inspection for robotic fruit and vegetable grading systems: A technical review. Trends in Food Science & Technology, 2018; 81: 213–231.
Zujevs A, Osadcuks V, Ahrendt P. Trends in robotic sensor technologies for fruit harvesting: 2010–2015. Procedia Computer Science, 2015; 77: 227–233.
Kondo N, Ting K C. Robotics for bioproduction systems. Michigan: American Society of Agricultural Engineers. 1998; 325p.
Bhargava A, Bansal A. Fruits and vegetables quality evaluation using computer vision: A review. Journal of King Saud University-Computer and Information Sciences, 2021; 33(3): 243–257.
Ding C Q, Feng Z, Wang D C, Cui D, Li W H. Acoustic vibration technology: Toward a promising fruit quality detection method. Comprehensive Reviews in Food Science and Food Safety, 2021; 20(2): 1655‒1680.
Shu F W, Lesur P, Xie Y X, Pagani A, Stricker D. SLAM in the field: An evaluation of monocular mapping and localization on challenging dynamic agricultural environment. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA: IEEE, 2021; pp.1760‒1770 .
Matiacevich S, Silva P, Enrione J, Osorio F. Quality assessment of blueberries by computer vision. Procedia Food Science, 2011; 1: 421–425.
Ismail N, Malik O A. Real-time visual inspection system for grading fruits using computer vision and deep learning techniques. Information Processing in Agriculture, 2022; 9(1): 24–37.
Hu Z L, Tang J S, Zhang P, Jiang J F. Deep learning for the identification of bruised apples by fusing 3D deep features for apple grading systems. Mechanical Systems and Signal Processing, 2020; 145: 106922.
Gongal A, Amatya S, Karkee M, Zhang Q, Lewis K. Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 2015; 116: 8–19.
Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, et al. SSD: Single shot multibox detector. Computer Vision–ECCV2016, 2016; 9905: 21–37.
Redmon J, Farhadi A J A P A. Yolov3: An incremental improvement. arXiv e-prints, 2018; 02767.
Dubey S R, Jalal A S. Apple disease classification using color, texture and shape features from images. Signal, Image and Video Processing, 2016; 10(5): 819–826.
Moallem P, Serajoddin A, Pourghassem H. Computer vision-based apple grading for golden delicious apples based on surface features. Information Processing in Agriculture, 2017; 4(1): 33–40.
Dorj U-O, Lee M, Yun S-s. An yield estimation in citrus orchards via fruit detection and counting using image processing. Computers and Electronics in Agriculture, 2017; 140: 103–112.
Santos Pereira L F, Barbon Jr S, Valous N A, Barbin D F. Predicting the ripening of papaya fruit with digital imaging and random forests. Computers and Electronics in Agriculture, 2018; 145: 76–82.
Rong D, Rao X Q, Ying Y B. Computer vision detection of surface defect on oranges by means of a sliding comparison window local segmentation algorithm. Computers and Electronics in Agriculture, 2017; 137: 59–68.
Nandi C S, Tudu B, Koley C. A machine vision technique for grading of harvested mangoes based on maturity and quality. IEEE Sensors Journal, 2016; 16(16): 6387–6396.
Tan K, Lee W S, Gan H, Wang S W. Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes. Biosystems Engineering, 2018; 176: 59–72.
Hadimani L, Garg N M. Automatic surface defects classification of Kinnow mandarins using combination of multi-feature fusion techniques. Journal of Food Process Engineering, 2021; 44(1): e13589.
Sabzi S, Abbaspour-Gilandeh Y, García-Mateos G. A new approach for visual identification of orange varieties using neural networks and metaheuristic algorithms. Information Processing in Agriculture, 2018; 5(1): 162–172.
Wan P, Toudeshki A, Tan H, Ehsani R. A methodology for fresh tomato maturity detection using computer vision. Computers and Electronics in Agriculture, 2018; 146: 43–50.
De Luna R G, Dadios E P, Bandala A A, Vicerra R R A. Tomato growth stage monitoring for smart farm using deep transfer learning with machine learning-based maturity grading. Agrivita, 2020; 42(1): 24–36.
Korchagin S A, Gataullin S T, Osipov A V, Smirnov M V, Suvorov S V, Serdechnyi D V. Development of an optimal algorithm for detecting damaged and diseased potato tubers moving along a conveyor belt using computer vision systems. Agronomy, 2021; 11(10).
Abayomi-Alli O O, Damaševičius R, Misra S, Maskeliūnas R. Cassava disease recognition from low-quality images using enhanced data augmentation model and deep learning. Expert Systems, 2021; 38(7): e12746.
Tian Y N, Yang G D, Wang Z, Li E Liang Z Z. Detection of apple lesions in orchards based on deep learning methods of CycleGAN and YOLOV3-Dense. Journal of Sensors, 2019; 2019: 7630926.
da Costa A Z, Figueroa H E H, Fracarolli J A. Computer vision based detection of external defects on tomatoes using deep learning. Biosystems Engineering, 2020; 190(C): 131–144.
Zhu L, Spachos P. Support vector machine and YOLO for a mobile food grading system. Internet of Things, 2021; 13: 100359.
Behera S K, Rath A K, Sethy P K. Maturity status classification of papaya fruits based on machine learning and transfer learning approach. Information Processing in Agriculture, 2021; 8(2): 244–250.
Zhang W Z, Tan A, Zhou G X, Chen A, Li M X, Chen X, et al. A method for classifying citrus surface defects based on machine vision. Journal of Food Measurement and Characterization, 2021; 15(3): 2877–2888.
Qiao J, Sasao A, Shibusawa S, Kondo N, Morimoto E. Mapping yield and quality using the mobile fruit grading robot. Biosystems Engineering, 2005; 90(2): 135–142.
Qiao J, Sasao A, Shibusawa S, Kondo N, Morimoto E. Mobile fruit grading robot (Part 1): Development of a robotic system for grading sweet peppers. Journal of JSAM, 2004; 66(2): 113‒122.
Bac C W, Hemming J, van Tuijl B A J, Barth R, Wais E, van Henten E J. Performance evaluation of a harvesting robot for sweet pepper. Journal of Field Robotics, 2017; 34(6): 1123–1139.
Hsieh K-W, Huang B-Y, Hsiao K-Z, Tuan Y-H, Shih F-P, Hsieh L-C, et al. Fruit maturity and location identification of beef tomato using R-CNN and binocular imaging technology. Journal of Food Measurement and Characterization, 2021; 15(6): 5170–5180.
Hayashi S, Shigematsu K, Yamamoto S, Kobayashi K, Kohno Y, Kamata J, et al. Evaluation of a strawberry-harvesting robot in a field test. Biosystems Engineering, 2010; 105(2): 160–171.
Xiong Y, Peng C, Grimstad L, From P J, Isler V. Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper. Computers and Electronics in Agriculture, 2019; 157: 392–402.
Lehnert C, English A, Mccool C, McCool C, Tow A W, Perez T. Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robotics and Automation Letters, 2017; 2(2): 872–879.
Yaguchi H, Nagahama K, Hasegawa T, Inaba M. Development of an autonomous tomato harvesting robot with rotational plucking gripper. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea: IEEE, 2016; pp.652‒657.
Arad B, Balendonck J, Barth R, Ben-Shahar O, Edan Y, Hellström T, et al. Development of a sweet pepper harvesting robot. Journal of Field Robotics, 2020; 37(6): 1027–1039.
Zhang Z, Igathinathane C, Li J, Chen H, Lu Y, Flores P. Technology progress in mechanical harvest of fresh market apples. Computers and Electronics in Agriculture, 2020; 175: 105606.
Jadhav T, Singh K, Abhyankar A. Volumetric estimation using 3D reconstruction method for grading of fruits. Multimedia Tools and Applications, 2019; 78(2): 1613–1634.
Reese D Y, Lefcourt A M, Kim S M, Lo Y M. Whole surface image reconstruction for machine vision inspection of fruit. In: Optics for natural resources, agriculture, and foods II, Boston, MA, United States: International Society for Optics and Photonics, 2007 .
Amirulah R. Real time visual system for starfruit maturity index classification. MS dissertation. Johor : Universiti Teknologi Malaysia, 2012; 12. 94p.
Benelli A, Cevoli C, Fabbri A. In-field hyperspectral imaging: An overview on the ground-based applications in agriculture. Journal Of Agricultural Engineering, 2020; 51(3): 129–139.
Wendel A, Underwood J, Walsh K. Maturity estimation of mangoes using hyperspectral imaging from a ground based mobile platform. Computers and Electronics in Agriculture, 2018; 155: 298–313.
Benelli A, Cevoli C, Ragni L, Fabbri A. In-field and non-destructive monitoring of grapes maturity by hyperspectral imaging. Biosystems Engineering, 2021; 207: 59–67.
Sun Y, Wang Y H, Xiao H, Gu X Z, Pan L Q, Tu Kang. Hyperspectral imaging detection of decayed honey peaches based on their chlorophyll content. Food Chemistry, 2017; 235: 194–202.
Tian X, Li J B, Wang Q Y, Fan S X, Huang W Q. A bi-layer model for nondestructive prediction of soluble solids content in apple based on reflectance spectra and peel pigments. Food Chemistry, 2018; 239: 1055–1063.
Gao Z M, Shao Y Y, Xuan G T, Wang Y X, Liu Y, Han X. Real-time hyperspectral imaging for the in-field estimation of strawberry ripeness with deep learning. Artificial Intelligence in Agriculture, 2020; 4: 31–38.
Munera S, Gómez-Sanchís J, Aleixos N, Vila-Francés J, Colelli G, Cubero S, et al. Discrimination of common defects in loquat fruit cv. ‘Algerie’ using hyperspectral imaging and machine learning techniques. Postharvest Biology and Technology, 2021; 171: 111356.
Zhang B, Xie Y, Zhou J, Wang K, Zhang Z. State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A review. Computers and Electronics in Agriculture, 2020; 177: 105694.
Tian S J, Xu H R. Mechanical-based and optical-based methods for nondestructive evaluation of fruit firmness. Food Reviews International, 2023; 39(7): 4009–4039.
Li T, Sun X G, Shu X, Wang C K, Wang Y F, Chen G, et al. Robot grasping system and grasp stability prediction based on flexible tactile sensor array. Machines, 2021; 9(6): 9060119 .
Xie H, Liu H B, Luo S, Seneviratne L D, Althoefer K. Fiber optics tactile array probe for tissue palpation during minimally invasive surgery. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan: IEEE, 2013; pp.2539‒2544 .
Chen L L, Feng J Y, Li T K, Li X F, Zhang J H. High-Tactile Sensitivity of Piezoresistive Sensors With a Micro-Crack Structure Induced by Thin Film Tension. IEEE Electron Device Letters, 2019; 40(9): 1519–1521.
Krishna G M, Rajanna K. Tactile sensor based on piezoelectric resonance. IEEE Sensors Journal, 2004; 4(5): 691–697.
Hoshi T, Shinoda H. Tactile sensing using nonlinear elasticity. In SICE Annual Conference 2005, Okayama University, Japan, 2005 .
Andrussow I, Sun H B, Kuchenbecker K J J, Martius G. Minsight: A fingertip-sized vision-based tactile sensor for robotic manipulation. Advanced Intelligent Systems, 2023; 5(8): 2300042.
Blanes C, Mellado M, Beltrán P. Tactile sensing with accelerometers in prehensile grippers for robots. Mechatronics, 2016; 33: 1‒12.
Blanes C, Ortiz C, Mellado M, Beltrán P. Assessment of eggplant firmness with accelerometers on a pneumatic robot gripper. Computers and Electronics in Agriculture, 2015; 113: 44–50.
Bandyopadhyaya I, Babu D, Kumar A, Roychowdhury J. Tactile sensing based softness classification using machine learning. In 2014 IEEE International Advance Computing Conference (IACC), Gurgaon, India: IEEE, 2014; pp.1231‒1236 .
Hu G R, Zhang E Y, Zhou J G Zhao J, Gao Z N, et al. Infield apple detection and grading based on multi-feature fusion. Horticulturae, 2021; 7(9): 276 .
Spiers A J, Liarokapis M V, Calli B, Dollar A M. Single-grasp object classification and feature extraction with simple robot hands and tactile sensors. IEEE Transactions on Haptics, 2016; 9(2): 207–220.
Zhou J, Meng Y M, Wang M J, Memon M S, Yang X R, et al. Surface roughness estimation by optimal tactile features for fruits and vegetables. International Journal of Advanced Robotic Systems, 2017; 14(4): 1729881417721866.
Scimeca L, Maiolino P, Cardin-catalan D, del Pobil A P, Morales A, Iida F. Non-destructive robotic assessment of mango ripeness via multi-point soft haptics. In: 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada: IEEE, 2019; pp.1821‒1826 .
Alfadhel A, Kosel J. Magnetic nanocomposite cilia tactile sensor. Advanced Materials, 2015; 27(47): 7888–7892.
Ribeiro P, Cardoso S, Bernardino A, Jamone L. Fruit quality control by surface analysis using a bio-inspired soft tactile sensor. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA: IEEE, 2021; pp.8875‒8881.
Carvalho M, Ribeiro P, Romão V, Cardoso S. Smart fingertip sensor for food quality control: Fruit maturity assessment with a magnetic device. Journal of Magnetism and Magnetic Materials, 2021; 536: 168116.
Ward-Cherrier B, Pestell N, Cramphorn L, Winstone B, Giannaccini M E, Rossiter J, et al. The TacTip family: Soft optical tactile sensors with 3D-printed biomimetic morphologies. Soft Robotics, 2018; 5(2): 216–227.
Dong S, Yuan W Z, Adelson E H. Improved gelsight tactile sensor for measuring geometry and slip. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada: IEEE, 2017 .
Donlon E, Dong S Y, Liu M, Li J H, Adelson E, Rodriguez A. GelSlim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, Nevada, USA: IEEE/RSJ, 2018 .
Lambeta M, Chou P W, Tian S, Yang B, Maloon B, Most V R, et al. DIGIT: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 2020; 5(3): 3838–3845.
Wang H, Peng J, Xie C, et al. Fruit quality evaluation using spectroscopy technology: A review. Sensors, 2015; 15(5): 11889–11927.
Wang D C, Ding C Q, Feng Z, Ji S Y, Cui D. Recent advances in portable devices for fruit firmness assessment. Critical Reviews in Food Science and Nutrition, 2023; 63(8): 1143–1154.
Fan S X, Li J B, Zhang Y H, Tian X, Wang Q Y, He X, et al. On line detection of defective apples using computer vision system combined with deep learning methods. Journal of Food Engineering, 2020; 286: 110102 .
Cortés V, Blanes C, Blasco J, Ortíz C, Aleixos N, Mellado M, et al. Integration of simultaneous tactile sensing and visible and near-infrared reflectance spectroscopy in a robot gripper for mango quality assessment. Biosystems Engineering, 2017; 162: 112–123.
Zhao M, Peng Y L, Li L. A robot system for the autodetection and classification of apple internal quality attributes. Postharvest Biology and Technology, 2021; 180: 111615.
Borba K R, Aykas D P, Milani M I, Colnago L A, Ferreira M D, Rodriguez-Saona L E. Portable near infrared spectroscopy as a tool for fresh tomato quality control analysis in the field. Applied Sciences, 2021; 11(3209): 11073209 .
Yu Y, Zhang Q L, Huang J P, Zhu J, Liu J W, et al. Nondestructive determination of SSC in Korla fragrant pear using a portable near-infrared spectroscopy system. Infrared Physics & Technology, 2021; 116: 103785.
Sohaib Ali Shah S, Zeb A, Qureshi W S, Malik A U, Tiwana M, Walsh K, et al. Mango maturity classification instead of maturity index estimation: A new approach towards handheld NIR spectroscopy. Infrared Physics & Technology, 2021; 115: 103639.
Das A J, Wahi A, Kothari I, Raskar R. Ultra-portable, wireless smartphone spectrometer for rapid, non-destructive testing of fruit ripeness. Scientific Reports, 2016; 6(1): 32504.
Pissard A, Marques E J N, Dardenne P, Lateur M, Pasquini C, Pimentel M F, et al. Evaluation of a handheld ultra-compact NIR spectrometer for rapid and non-destructive determination of apple fruit quality. Postharvest Biology and Technology, 2021; 172: 111375.
Blakey R J. Evaluation of avocado fruit maturity with a portable near-infrared spectrometer. Postharvest Biology and Technology, 2016; 121: 101–105.
Subedi P P, Walsh K B. Assessment of avocado fruit dry matter content using portable near infrared spectroscopy: method and instrumentation optimisation. Postharvest Biology and Technology, 2020; 161: 111078.
Gong X Y, Tang M, Gong Z J, Qiu Z P, Wang D M, Fan M K. Screening pesticide residues on fruit peels using portable Raman spectrometer combined with adhesive tape sampling. Food Chemistry, 2019; 295: 254–258.
Guo W C, Li W Q, Yang B, Zhu Z Z, Liu D Y, Zhu X H. A novel noninvasive and cost-effective handheld detector on soluble solids content of fruits. Journal of Food Engineering, 2019; 257: 1–9.
Sarkar S, Basak J K, Moon B E, Kim H T. A comparative study of PLSR and SVM-R with various preprocessing techniques for the quantitative determination of soluble solids content of hardy kiwi fruit by a portable Vis/NIR spectrometer. Foods, 2020; 9(8): 9081078.
Yang B, Guo W C, Li W Q, Li Q Q, Liu D Y, Zhu X H. Portable, visual, and nondestructive detector integrating Vis/NIR spectrometer for sugar content of kiwifruits. Journal of Food Process Engineering, 2019; 42(2): e12982.
Mishra P, Marini F, Brouwer B, Roger J M, Biancolillo A, Woltering E, et al. Sequential fusion of information from two portable spectrometers for improved prediction of moisture and soluble solids content in pear fruit. Talanta, 2021; 223: 121733.
Li M, Han D H, Liu W. Non-destructive measurement of soluble solids content of three melon cultivars using portable visible/near infrared spectroscopy. Biosystems Engineering, 2019; 188: 31–39.
Mishra P, Woltering E, El Harchioui N. Improved prediction of ‘Kent’ mango firmness during ripening by near-infrared spectroscopy supported by interval partial least square regression. Infrared Physics & Technology, 2020; 110: 103459.
Sulistyo S B, Siswantoro, Margiwiyatno A, Masrukhi, Mustofa A, Sudarmaji A, et al. Handheld arduino-based near infrared spectrometer for non-destructive quality evaluation of siamese oranges. In The 2nd International Conference on Sustainable Agriculture for Rural Development 2020, Purwokerto, Indonesia: IOP Publishing, 2021 .
Walsh K B, Blasco J, Zude-Sasse M, Sun X D. Visible-NIR ‘point’ spectroscopy in postharvest fruit and vegetable assessment: The science behind three decades of commercial use. Postharvest Biology and Technology, 2020; 168: 111246.
Yun Y-H, Bin J, Liu D-L, Xu L, Yan T L, Cao D S, et al. A hybrid variable selection strategy based on continuous shrinkage of variable space in multivariate calibration. Analytica Chimica Acta, 2019; 1058: 58–69.
Walsh K B, Mcglone V A, Han D H. The uses of near infra-red spectroscopy in postharvest decision support: A review. Postharvest Biology and Technology, 2020; 163: 111139.
Zheng W, Bai Y H, Luo H, Li Y H, Yang X, Zhang B H. Self-adaptive models for predicting soluble solid content of blueberries with biological variability by using near-infrared spectroscopy and chemometrics. Postharvest Biology and Technology, 2020; 169: 111286 .
Mishra P, Passos D. A synergistic use of chemometrics and deep learning improved the predictive performance of near-infrared spectroscopy models for dry matter prediction in mango fruit. Chemometrics and Intelligent Laboratory Systems, 2021; 212: 104287.
Mishra P, Passos D. Multi-output 1-dimensional convolutional neural networks for simultaneous prediction of different traits of fruit based on near-infrared spectroscopy. Postharvest Biology and Technology, 2022; 183: 111741.
Rong D, Wang H Y, Ying Y B, Zhang Z Y, Zhang Y S. Peach variety detection using VIS-NIR spectroscopy and deep learning. Computers and Electronics in Agriculture, 2020; 175: 105553.
Xu L X, Xie J, Cai F H, Wu J J. Spectral classification based on deep learning algorithms. Electronics, 2021; 10(16): 10161892 .
Li L, Huang W Q, Wang Z L, Liu S Q, He X, Fan S X, et al. Calibration transfer between developed portable Vis/NIR devices for detection of soluble solids contents in apple. Postharvest Biology and Technology, 2022; 183: 111720.
Silwal A, Karkee M, Zhang Q. A hierarchical approach to apple identification for robotic harvesting. Transactions of the ASABE, 2016; 59(5): 1079–1086.
Copyright (c) 2024 International Journal of Agricultural and Biological Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.