Tracking and recognition algorithm for a robot harvesting oscillating apples

Qinghua Yang, Chen Chen, Jiayu Dai, Yi Xun, Guanjun Bao

Abstract


Apple fruits on trees tend to swing because of wind or other natural causes, therefore reducing the accuracy of apple picking by robots. To increase the accuracy and to speed up the apple tracking and identifying process, tracking and recognition method combined with an affine transformation was proposed. The method can be divided into three steps. First, the initial image was segmented by Otsu’s thresholding method based on the two times Red minus Green minus Blue (2R-G-B) color feature; after improving the binary image, the apples were recognized with a local parameter adaptive Hough circle transformation method, thus improving the accuracy of recognition and avoiding the long, time-consuming process and excessive fitted circles in traditional Hough circle transformation. The process and results were verified experimentally. Second, the Shi-Tomasi corners detected and extracted from the first frame image were tracked, and the corners with large positive and negative optical flow errors were removed. The affine transformation matrix between the two frames was calculated based on the Random Sampling Consistency algorithm (RANSAC) to correct the scale of the template image and predict the apple positions. Third, the best positions of the target apples within 1.2 times of the prediction area were searched with a de-mean normalized cross-correlation template matching algorithm. The test results showed that the running time of each frame was 25 ms and 130 ms and the tracking error was more than 8% and 20% in the absence of template correction and apple position prediction, respectively. In comparison, the running time of our algorithm was 25 ms, and the tracking error was less than 4%. Therefore, test results indicate that speed and efficiency can be greatly improved by using our method, and this strategy can also provide a reference for tracking and recognizing other oscillatory fruits.
Keywords: apple picking robot, tracking and recognition algorithm, oscillating apple, Hough transform, pyramid LK optical flow algorithm, affine transform, template matching
DOI: 10.25165/j.ijabe.20201305.5520

Citation: Yang Q H, Chen C, Dai J Y, Xun Y, Bao G J. Tracking and recognition algorithm for a robot harvesting oscillating apples. Int J Agric & Biol Eng, 2020; 13(5): 163–170.

Keywords


apple picking robot, tracking and recognition algorithm, oscillating apple, Hough transform, pyramid LK optical flow algorithm, affine transform, template matching

Full Text:

PDF

References


Jiang G Q, Zhao C J. Apple recognition based on machine vision. 2012 International Conference on Machine Learning and Cybernetics, Xi’an: IEEE, 2012; pp.1148–1151.

Xu Y, Li Y H, Song H B, He D J. Segmentation method of overlapped double apples based on Snake model and corner detectors. Transactions of the CSAE, 2015; 31 (1): 196–203. (in Chinese)

Lyu J D, Zhao D A, Ji W, Ding S H. Recognition of apple fruit in natural environment. Optik, 2016; 127(3): 1354–1362.

Shamshiri R R, Hameed I A., Karkee M, Weltzien C. Robotic harvesting of fruiting vegetables: A simulation approach in V-REP, ROS and MATLAB. In Automation in Agriculture-Securing Food Supplies for Future Generations. IntechOpen, 2018. doi: 10.5772/intechopen.73861.

Shamshiri R R, Weltzien C, Hameed I A., Yule I J, Grift T E, Balasundram S K, et al. Research and development in agricultural robotics: A perspective of digital farming. Int J Agric & Biol Eng, 2018; 11(4): 1–14.

Liu Z D, Lyu Z Q, Zheng W X, Zhang W Z, Cheng X X. Design of obstacle avoidance controller for agricultural tractor based on ROS. Int J Agric & Biol Eng, 2019; 12(6): 58–65.

Wang D D, Song H B, Yu X L, Zhang W Y, Qu W F, Xu Y. An improved contour symmetry axes extraction algorithm and its application in the location of picking points of apples. Spanish Journal of Agricultural Research, 2015; 13(1): e0205. doi: 10.5424/sjar/ 2015131-6181.

Yang Q H, Chang C, Bao G J, Fan J, Xun Y. Recognition and localization system of the robot for harvesting Hangzhou White Chrysanthemums. Int J Agric & Biol Eng, 2018; 11(1): 88–95.

Ji W, Zhao D A, Cheng F Y, Xu B, Zhang Y, Wang J J. Automatic recognition vision system guided for apple harvesting robot. Computers & Electrical Engineering, 2012; 38(5): 1186–1195.

Yang Q H, Luo S L, Chang C, Xun Y, Bao G J. Segmentation algorithm for Hangzhou white chrysanthemums based on least squares support vector machine. Int J Agric & Biol Eng, 2019; 12(4): 127–134.

Jia W K, Zhao D A, Liu X Y, Tang S P, Ruan C Z, Ji W. Apple recognition based on K-means and GA-RBF-LMS neural network applicated in harvesting robot. Transactions of the CSAE, 2015; 31(18): 175–183. (in Chinese)

Linker R, Cohen O, Naor A. Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 2012; 81: 45–57.

Yang Q H, Jia M M, Xun Y, Bao G J. Detection of egg stains based on local texture feature clustering. Int J Agric & Biol Eng, 2018; 11(1): 199–205.

Gongal A, Silwal A, Amatya S, Karkee M, Zhang Q, Lewis K. Apple crop-load estimation with over-the-row machine vision system. Computers and Electronics in Agriculture, 2016; 120: 26–35.

Nguyen T T, Vandevoored K, Wouters N, Kayacan E, De B J G, Saeys W. Detection of red and bicoloured apples on tree with an RGB-D camera. Biosystems Engineering, 2016; 146: 33–44.

Malik M H, Zhang T, Li H, Zhang M, Shabbir S, Saeed A. Mature tomato fruit detection algorithm based on improved HSV and watershed algorithm. IFAC Papersonline, 2018; 51(17): 431–436.

Song H B, He D J, Pan J P. Recognition and localization methods of occluded based on convex hull theory. Transactions of the CSAE, 2012; 28(22): 174–180. (in Chinese)

Song H B, Zhang C D, Pan J P, Yin X, Zhuang Y B. Segmentation and reconstruction of overlapped apple images based on convex hull. Transactions of the CSAE, 2013; 29(3): 163–168. (in Chinese)

Li L J, Yang H J. Revised detection and localization algorithm for camellia oleifera fruits. Transactions of the CSAM, 2016; 47(12): 285–292, 346. (in Chinese)

Liu W H, Sui Q M. Automatic segmentation of overlapping powder particles based on concavity points. Journal of Electronic Measurement and Instrumentation, 2010; 24(12): 1095–1100. (in Chinese)

Xie Z H, Ji C Y, Guo X Q, Zhu S X. Detection and location algorithm for overlapped fruits based on concave spots searching. Transactions of the CSAM, 2011; 42 (12): 191–196. (in Chinese)

Liu Y K, Gao H L, Zhang J. Improvement of apple picking system design and visual algorithms based on open source project. Jiangsu Agricultural Sciences, 2019; 47(4): 187–192. (in Chinese)

Zhao D A, Shen T, Chen Y, Jia W K. Fast tracking and recognition of overlapping fruit for apple harvesting. Transactions of the CSAE, 2015; 31(2): 22–28. (in Chinese)

Lyu J D, Zhao D A, Ji W. Fast tracking recognition method of target fruit for apple harvesting robot. Transactions of the CSAM, 2014; 45(1): 65–72. (in Chinese)

Feng W, Zhao D A. Fast tracking and recognition of apple harvesting robot for target fruit. Information Technology, 2018; 5: 5–9. (in Chinese)

Otsu N. A Threshold selection method from gray-level histograms. IEEE Trans Systems Man & Cybernetics, 1979; 9(1): 62–66.

Mai C Y, Zheng L H, Xiao C Y, Li M Z. Comparison of apple recognition methods under natural light. Journal of China Agricultural University, 2016; 21(11): 43–50. (in Chinese)

Shi J B, Tomasi C. Good features to track. 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Seattle: IEEE, 1994; pp.593–600.




Copyright (c) 2020 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office