Detection method for the cucumber robotic grasping pose in clutter scenarios via instance segmentation

Fan Zhang, Zuyu Hou, Jin Gao, Junxiong Zhang, Xue Deng

Abstract


The application of robotic grasping for agricultural products pushes automation in agriculture-related industries. Cucumber, a common vegetable in greenhouses and supermarkets, often needs to be grasped from a cluttered scene. In order to realize efficient grasping in cluttered scenes, a fully automatic cucumber recognition, grasping, and palletizing robot system was constructed in this paper. The system adopted Yolact++ deep learning network to segment cucumber instances. An early fusion method of F-RGBD was proposed, which increases the algorithm's discriminative ability for these appearance-similar cucumbers at different depths, and at different occlusion degrees. The results of the comparative experiment of the F-RGBD dataset and the common RGB dataset on Yolact++ prove the positive effect of the F-RGBD fusion method. Its segmentation masks have higher quality, are more continuous, and are less false positive for prioritizing-grasping prediction. Based on the segmentation result, a 4D grab line prediction method was proposed for cucumber grasping. And the cucumber detection experiment in cluttered scenarios is carried out in the real world. The success rate is 93.67% and the average sorting time is 9.87 s. The effectiveness of the cucumber segmentation and grasping pose acquisition method is verified by experiments.
Keywords: Clutter scenarios, Cucumber grasp, Convolutional neural network, Instance segmentation
DOI: 10.25165/j.ijabe.20231606.7542

Citation: Zhang F, Hou Z Y, Gao J, Zhang J X, Deng X. Detection method for the cucumber robotic grasping pose in clutter scenarios via instance segmentation. Int J Agric & Biol Eng, 2023; 16(6): 215–225.

Keywords


Clutter scenarios, Cucumber grasp, Convolutional neural network, Instance segmentation

Full Text:

PDF

References


Cheng W-C, Hsiao H-C, Lin Y-P, Liu Y-H. Robotic arm pick-and-place system for L-shaped water pipe joint with arbitrary angle. In: 2020 International Automatic Control Conference （CACS）, Hsinchu: IEEE, 2020; pp.1–5.

Triantafyllou P, Mnyusiwalla H, Sotiropoulos P, Roa M A, Russell D, Deacon G. A Benchmarking Framework for systematic evaluation of robotic pick-and-place systems in an industrial grocery setting. In: 2019 International Conference on Robotics and Automation （ICRA）, Montreal: IEEE, 2019; pp.6692–6698.

Ye H J, Liu C Q, Niu P Y. Cucumber appearance quality detection under complex background based on image processing. Int J Agric & Biol Eng, 2018; 11（4）: 193–199.

Feng Q C, Zou W, Fan P F, Zhang C F, Wang X, Design and test of robotic harvesting system for cherry tomato. Int J Agric & Biol Eng, 2018; 11（1）: 96–100.

Cornelius H, Kragic D, Eklundh J O. Object and pose recognition using contour and shape information. In: ICAR '05 Proceedings 12th International Conference on Advanced Robotics, 2005; pp.613–620.

Rahardja K, Kosaka A. Vision-based bin-picking: recognition and localization of multiple complex objects using simple visual cues. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96, Osaka: IEEE, 1996; pp.1448–1457.

Hinterstoisser S, Cagniart C, Ilic S, Sturm P, Navab N, Fua P, et al. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012; 34（5）: 876–888.

Liu W H, Pan Z Y, Liu W J, Shao Q Q, Hu J, Wang W M, et al. Deep learning for picking point detection in dense cluster. In: 2017 11th Asian Control Conference （ASCC）, Gold Coast: IEEE, 2017; pp.1644-1649.

Redmon J, Angelova A. Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation （ICRA）, Seattle: IEEE, pp.1316–1322.

Kehl W, Manhardt F, Tombari F, Ilic S, Navab N. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In: 2017 IEEE International Conference on Computer Vision （ICCV）, Venice: IEEE, 2017; pp.1530–1538.

Mao S H, Li Y H, Ma Y, Zhang B H, Zhou J, Wang K. Automatic cucumber recognition algorithm for harvesting robots in the natural environment using deep learning and multi-feature fusion. Computers and Electronics in Agriculture, 2020; 170: 105254.

Liu X Y, Zhao D A, Jia W K, Ji W, Ruan C Z, Sun Y P. Cucumber fruits detection in greenhouses based on instance segmentation. IEEE Access, 2019; 7: 139635–139642.

Zhao W R, Wang J C, Chen W D, Huang Y. Planning for grasping cluttered objects based on obstruction degree. International Journal of Advanced Robotic Systems, 2021; 18（6）: 17298814211040632.

Tu S Q, Pang J, Liu H F, Zhuang N, Chen Y, Zheng C, et al. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 2020; 21: 1072–1091.

Sa I, Ge Z Y, Dayoub F, Upcroft B, Perez T, McCool C. DeepFruits: A fruit detection system using deep neural networks. Sensors, 2016; 16（8）: 1222.

Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 2008; 77: 157–173.

Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common objects in context. In: Computer Vision – ECCV 2014; 8693: 740–755.

Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision （ICCV）, Seoul: IEEE, 2019; pp.9156–9165.

Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT++: Better real-time instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022; 44（2）: 1108–1121.

Lin T-Y, Dollar P, Girshick R, He K M, Hariharan B, Belongie S. Feature Pyramid Networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）, Honolulu: IEEE, 2017; pp.936-944.

He K M, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision （ICCV）, Venice: IEEE, 2017; pp.2980-2988.

Wu M H, Yue H H, Wang J, Huang Y X, Liu M, Jiang Y H, et al. Object detection based on RGC mask R-CNN. IET Image Processing, 2020; 14（8）: 1502–1508.

Cheng F-T, Hour T-L, Sun Y-Y, Chen T-H. Study and resolution of singularities for a 6-DOF PUMA manipulator. IEEE Transactions on System, Man, Cybernetics, 1997; 27（2）: 332–343.

Yu S S, Wang C, Wen C L, Cheng M, Liu M H, Zhang Z H, et al. LiDAR-based localization using universal encoding and memory-aware regression. Pattern Recognition, 2022; 128: 108685.




Copyright (c) 2023 International Journal of Agricultural and Biological Engineering

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

2023-2026 Copyright IJABE Editing and Publishing Office