Obstacle avoidance transplanting method on Kinect visual processing

In modern facility agriculture, to improve the quality and efficiency of transplanting, the application of transplanting robots based on visual processing is becoming more and more widespread. In order to reduce the damage to plants during the transplanting process and reduce the damage rate of plant stems, leaves and substrates, a transplanting method based on Kinect visual processing combined with an inclined transplanting manipulator was proposed. In the research, the Kinect visual processing was used to obtain and process the seedling height information and leaf edge information, and the working coordinate system of the transplanting manipulator was established and applied to plan the obstacle avoidance path. Combined with the oblique manipulator, the obstacle avoidance transplanting method was proposed. Through the structural design and force analysis of the seedling transplanting device, the key parameters that affect the transplanting quality were obtained, and the optimal transplanting performance parameters were obtained through experiments. In the experiment, with the aid of the Kinect vision processing system, the designed obstacle avoidance transplanting manipulator had a leaf damage degree of 4.70%, a stem bending rate of 16.67%, substrate integrity of 83.45% and a transplanting quality parameter of 87.36%. The time for a single seedling transplanting was (8.32±0.40) s. The experiment result proves that the obstacle avoidance transplanting method based on Kinect visual processing can effectively reduce the damage to seedlings when ensuring the transplanting efficiency.


Introduction 
In facility horticulture, transplanting equipment is very important for the entire vegetable production line, and its degree of mechanization and intelligence reflects the advanced degree of the entire vegetable production process [1,2] . At this stage, there are many studies that apply image processing and machine vision to intelligent transplanting. Netherlands, United States and other European and American countries are very mature in the field of facility horticulture. For example, Vissor Horti systems, TTA transplanting, and Flier Systems are very advanced companies in facility horticulture transplanting equipment [3] . The research in the field of facility horticulture transplanting equipment has three main parts. The first part is the research on the key structure design of transplanting equipment, the second part is the research on seedling cultivation conditions during transplanting, and the third part is the integration of image processing and visual technology into transplanting to complete research on seedling detection and intelligent path planning. Tong [4] designed a kind of obstacle avoidance transplanting manipulator and planned its working trajectory according to the working characteristics of the transplanting manipulator to reduce the damage of seedlings. Jin et al. [5] applied Fuzzy C-means (FCM) method to identify the degree of seedlings growth condition and remove inferior seedlings and replenish seedlings. In addition, Yang et al. [6] and other experts combined image processing and analysis based on the acquired images and seedling modeling. Feng et al. [7,8] have conducted many studies on the structural design of fruit and vegetable transplanting equipment and low-damage transplanting robots. They have designed and tested key components such as the end effector of transplanting robots and obtained suitable design parameters. In addition, many experts integrated image processing and robot vision technology into the transplanting production environment during the transplanting process and conducted obstacle-avoidance transplanting by processing seedling images. Yu et al. [9] incorporated image processing technology into parallel transplanting through inexpensive cameras. Based on machine vision, Sun et al. [10] used CCD industrial cameras to accurately identify seedlings, so that transplanting robots can carry out accurate transplanting. At this stage, image processing is mainly used for seedling health identification and positioning during transplanting. There are few studies on its application in obstacle avoidance transplanting to reduce the damage of seedling. The main research on reducing the damage of seedling is reflected in the end effector structure design and transplanting path planning. Therefore, accelerating the modernization and mechanization of facility horticulture, researching the mechanization and intelligent production equipment of transplanting production links, and applying technology based on image processing visual analysis to formal production activities are important tasks that the authors are currently facing. This article mainly studies the application of image processing technology in obstacle avoidance transplanting to reduce the damage of seedling.
To reduce the damage to the seedlings during the transplanting caused by the unreasonable design of seedling transplanting structure and path planning, a method of transplanting obstacle avoidance based on image processing is studied. By studying the physical property of seedlings, the obstacle avoidance principle of the Kinect visual processing system was studied, an obstacle avoidance path planning was conducted, and an obstacle transplanting manipulator was designed. Moreover, a method for expressing the parameter of the overall damage degree of seedlings was proposed.
The experiments were designed to obtain suitable seedling transplanting parameters. This study provides a theoretical basis and method for further design of obstacle avoidance transplanting machinery and equipment.

Experiment conditions
The main targets of this experiment are pepper seedlings, which are about four weeks old, and grow to the 3-4 leaf stage reaching the transplanting requirement, as shown in Figure 1. The main proportion of its substrate widely used in agriculture is peat: coconut bran: perlite of 3:1:1. The main plug tray used for pepper seedlings is 21-hole (3×7) polyvinyl chloride (PVC) standard plug tray. The main outer dimensions of the plug tray are 280 mm×540 mm. Individually, the top diameter of the hole is 62 mm, the bottom diameter is 30 mm, and the height is 65 mm. According to the research of Li et al. [11] , the most versatile seedling cultivation temperature is 22°C and the substrate humidity is 81%. All the experimental cultivation parameters in the article adopted the standard mentioned above.

Figure 1 Plug planting pepper seedlings
By measuring the physical size of 42 groups of pepper seedlings that cultivated in the tray for 30 days, the height of seedlings is 70.38-108.26 mm, the length of underground stems and roots is 28.56-47.38 mm, and the diameter of root distribution is 15.76-19.01 mm.

Design of transplanting experiment platform
In order to complete the reliability test of the transplanting method based on Kinect visual processing, a transfer test platform was designed, as shown in Figure 2. The manipulator used on the platform is the WY760A manipulator (Dex Intelligent Technology, Beijing), which is mainly driven by 6 servo motors as joints, and the motion of the robot arm is driven by the industrial computer. The end effector uses the SC32-50S cylinder as the driver and the OTS_550 air compressor as the power source. The entire transplanting manipulator test bed is programmed by adjusting the position of the industrial manipulator to keep the inclination angle of the seedling picking device unchanged during the movement, so as to achieve the purpose of simulating the movement process of the seedling picking device on the gantry transplanting machine.  Methods of vision processing and obstacle avoidance path planning

Kinect vision processor
In the process of transplanting, to complete intelligent obstacle avoidance transplanting, the seedling height and seedling edge were collected to establish the coordinate system of seedlings structure [12] . By researching the method of identifying edge images of seedling [13] , the Kinect vision processor [14] developed by Microsoft was selected to process the seedlings' images.
In this design, Kinect 1.0 RGBD vision processor (Microsoft Corporation, USA) was used. The vision processor contains two depth-sensing cameras and one RGB infrared camera. Its stable and effective sensing distance is 500-4 000 mm [15] , as shown in Figure 3. Two depth-sensing cameras can complete the collection of seedling height information, and RGB infrared sensor cameras can complete the collection of seedling RGB images. The collected information is transmitted to a personal computer (PC) through the dedicated data line of a conversion connector of Kinect, and then processed by Kinect Fusion (Microsoft Corporation, USA) on the PC.

Principles of visual processing
In order to complete the processing of the information acquired by the Kinect vision processor, Kinect Fusion vision processing system developed based on the C# language was used as shown in Figure 4 [16] . The system first analyzes the original depth map, then establishes a temporary coordinate system according to its position, and calculates the position of the point cloud voxel in the current temporary coordinate system. Then, the position and posture of the current camera are calculated based on the calculated point cloud coordinates of the current frame and the predicted point cloud coordinates of the previous frame. The total coordinate system is established through cross-analysis of two sets of coordinate systems. Then the truncated signed distance function (TSDF) [17] value is updated according to the camera position and posture, the point cloud is fused, and finally, the depth information of each pixel is estimated according to the TSDF value. The signed distance function (SDF) value describes the distance from the point to the surface. It is 0 on the surface, positive on one side of the surface, and negative on the other side. TSDF only considers the SDF value in the neighborhood of the surface. If the maximum value of the neighborhood is maxed truncation, the actual distance will be divided by the value of max truncation to achieve the purpose of normalization. So, the value of TSDF is from −1 to +1. Graphics processing unit (GPU) was used to process each voxel in parallel. First, each voxel is projected to the camera according to the calculated camera position and posture. If the voxel is within the camera's viewing cone, there will be a pixel corresponding to it. Di(p) is the actual measurement value of the distance from the voxel on the object surface to the camera, and the difference between the voxel and camera is the SDF value. In this experiment, the coordinate system established by Kinect Fusion is the total coordinate system. The minimum value of depth of voxel is derived as the main parameter for obstacle avoidance in the vertical direction to ensure that the end effector of the transplanting manipulator will not damage the seedlings vertically during the transplanting process.
The RGB infrared camera acquired the plug image as shown in Figure 5a. A fixed threshold method was used to perform image segmentation [18] . Based on the calculation of multiple images, the best threshold for segmenting plug trays and seedlings was obtained. According to the calculated threshold values [19] , the leaves and the plug tray were respectively segmented from the background to obtain threshold segmented images. After the image was subjected to threshold image segmentation, the RGB camera source image was transmitted to the HSV (Hue, Saturation, Value) image as shown in Figure 5b. Then the edge information of the tray was extracted by the Canny operator detection method and the tray edge processing image was obtained as shown in Figure 5c. Each hole was accurately positioned according to the image. Then through Canny operator detection method, the leaf edge information of the seedlings was extracted as shown in Figure 5d. The image was analyzed to find the position of the leaf edge of each row of seedlings. By calculating the edge coordinates of the seedlings and embedding them in the total coordinate system established before, the positioning of the plug tray and the acquisition of the edge information of the seedling leaves were completed.
Note: ICP represents iterative closest point; TSDF represents truncated signed distance function.

Obstacle avoidance transplanting path planning
To plan the path of the seedling end effector, the obtained RGB pixel coordinates containing depth information need to be converted into world coordinates. First, the pixel coordinate was converted to the image coordinate. In the pixel coordinate system, the measurement unit of the image size is the number of pixels, and the origin of the pixel coordinate system is the upper left point of the image. In the image coordinate system, the unit of measurement is mm, and the coordinate origin is the intersection of the optical axis and the image. The two coordinate systems were on the same plane, but the units of measurement were different. Therefore, the conversion equation is as follows: ( 1) where, (u, v) is the coordinate of the pixel in the pixel coordinate system; (x, y) is the coordinate of the pixel in the image coordinate system; (u 0 , v 0 ) is the origin of the image coordinate system in the pixel coordinate system; d x is the pixel width; d y is the pixel height.
After obtaining the image coordinates, the image coordinates were converted to camera coordinates. This process was a process of converting from 2D to 3D. Since the depth value of the pixel was obtained by the depth camera, the camera coordinates of the height extreme point and the edge extreme point can be directly calculated by the principle of similar triangles. The equation is as follows: ( 2) where, f is the focal length of the RGB camera, mm; (x c , y c , z c ) is the coordinates of the extreme point of the image in the camera coordinate system, mm.
The origin of the world coordinate system was set as the midpoint between two outermost points of two seedling fetching needles of transplantation end effectors after the WY760A manipulator was reset to zero. The origin of the camera coordinate system was set to the optical center of the Kinect RGB camera. The world coordinate system and the camera coordinate system used the same coordinate axes. The optical axis direction was the positive direction of the Z-axis, the camera movement direction along the slide was the positive direction of the X-axis, and the direction from the bottom of the camera to the top of the camera was the positive direction of the Y-axis. Because the conversion between the world coordinate system and the camera coordinate was a rigid conversion, the conversion can be completed after determining the conversion matrix R and the translation vector (x wc , y wc , z wc ). The conversion equation is as follows: where, (x w , y w , z w ) is the coordinates of the extreme point of the image in the world coordinate system, mm. After transmitting the pixel coordinate to the world coordinate, the industrial control computer processed the edge information and the height information, then planned the path of the seedling harvesting device to achieve obstacle avoidance, the path planning process is shown in Figure 6.

.1 Structural design of end effector
The main seedling collection equipment at this stage is divided into shovel type, needle type and horseshoe type [20] . Compared with the needle-type transplanting manipulator, the shovel-type and horseshoe-type seedling taking devices have a larger volume, require more space for operation, and have a higher damage rate to the seedling substrate. The needle-type seedling harvesting device has a small volume. In this experiment, since the main component of the matrix used is clay, the shortcoming of low friction can also be compensated. Therefore, a needle-type seedling harvesting device was selected for this test.
According to the selected size of the 21 holes tray, the hole depth is 65 mm. The length of the seedling needle into the soil is about 75%-85% of the depth of the hole [21] . For the convenience of later experiments, the length of the needle is 56 mm. Comprehensively considering the surface friction coefficient and stiffness factors, stainless steel was collected as the material of seedling needles. In order to ensure that the seedling taking device does not interfere with the seedling stem, the size of the fixed block of the seedling taking needle is 15×20 mm. Compared with other seedling taking devices, the seedling taking device in the design is in an oblique state to avoid taking seedlings vertically as shown in Figure 7. Through the transplanting path planning, the end effector can avoid contacting with the seedling leaves and stems during the seedling transplanting process thereby reducing the damage to the leaves and stems of seedlings. The axonometric drawing of end effector is shown in Figure 8.

Mechanical analysis of end effector
The force analysis of the end effector is shown in Figure 9. After inserting the needles into the substrate, the needles are divided into two groups, and the needles in each group are parallel. So, the force analysis can be carried out based on two needles. In the process of fetching seedlings, the seedling substrate is subjected to the frictional forces F f1 and F f2 , generated by the pressure F N1 and F N2 between the steel and substrate. Meanwhile, the substrate is subjected to the longitudinal resistance N combined by the gravity G of substrate and the adhesion Z between the plug and the substrate. In the process of fetching seedlings, the formulas of the stress conditions that need to be satisfied are as follows. 11 22 where, F T is pulling force required for extracting seedlings, N; μ is the coefficient of friction between steel needle and substrate.

Experiment factors
Penetration depth d, inclination angle α and insertion point distance l (Figure 9) will affect the stress of the seedling harvesting needle during the transplanting process [22][23][24] . The insertion depth is obtained from the previous data, and the range is 75%-85% of the hole depth. If the depth is too shallow, the deep matrix cannot be fixed and the matrix is scattered. If the depth is too deep, it will touch the bottom of the tray and as a result, the steel needle will be broken. Thus the depth of insertion of the steel needle is 48-56 mm. According to the insertion depth of the steel needle, the inclination angle α was analyzed. When the inclination angle is less than 10°, the friction force between the steel needle and the substrate is too small, which will lead to the failure to fetch seedlings. When the inclination angle is greater than 16°, the steel needle will contact the side wall of the hole and cause the plug to break. Therefore, the selection range of the inclination angle is 10°-16°. According to the overall design of the seedling harvesting manipulator, it is assumed that the seedlings are all planted in the center of the plug tray. When the insertion point distance is greater than 16 mm, the seedling needle fixing device will interfere with the seedling stem and damage the stems, affecting the transplanting quality. So the distance range of the insertion point is 10-16 mm.

Experiment evaluation index
The evaluation of the transplanting quality is mainly based on the damage degree of the seedlings and the integrity of the substrate during the transplanting process [25,26] . Therefore, the leaf integrity of the seedlings, the degree of stem bending and the transplanting substrate integrity were used as the indicators shown in Figure 10. Because the stems of seedlings have their own elasticity, a small inclination angle will not cause plastic deformation of the seedlings. According to the experimental analysis, when the angle of inclination of the seedling stem exceeds 20°, the seedling will be permanently damaged. Therefore, the angle of the stem bent over 20° was used as the evaluation index.
In order to judge the damage rate of seedlings, the transplanting quality parameter (TQP) was proposed.
By comparing the damaged area of the seedling leaves, the ratio of excessive bending of the seedling stems and the ratio of the loss of the seedling substrate before and after transplanting, the damage of the seedlings during the transplanting process is judged. The calculation formula is as follows.
where, V R1 is the leaf area after transplanting, mm; V 1 is the leaf area before transplanting, mm; Q B is the number of potted seedling stems bending over 20°; Q is the total number of transplanting; V R2 is the cross-sectional area of substrate after transplanting, mm 2 ; V 2 is the cross-sectional area of substrate before transplanting, mm 2 . Figure 10 Transplanting quality evaluation index

End effector performance experiment design
In the experiment, the penetration depth d, the inclination angle α and the distance l of the insertion point of the seedling needles were taken as the influencing factors of the transplanting performance, and the transplanting quality parameter was used as test indicator. The factor coding table is shown in Table 1.
Before each group of experiments, the robotic arm was adjusted to reach the experimental demand position. The air compressor air pressure parameters were adjusted by controlling the solenoid valve on the air compressor to stabilize the cylinder movement speed at 1 m/s. Each group used the appropriate age seedlings cultivated in 21 plug trays, and was repeated 7 times. Before the experiment, the leaf area of the seedlings and the inclination angle of the seedling stems were measured by using the RGB camera. The experiment was carried out within ten minutes after the seedlings were taken out of the incubator to ensure the experiment environment was fixed at a temperature of 22°C and a substrate humidity of 81%, which was the same as the cultivation environment. According to the planned path of the transplanting manipulator, each group carried out 7 repeated tests. After completing the experiment, the cross-sectional area of the substrate, the damaged area of the leaves, the inclination angle of the seedling stem and the time of single transplanting were recorded. After removing the data groups with obvious error caused by measuring error, the remaining data was calculated. Then the transplanting quality parameter was calculated according to Equation (5).

Results and discussion
By conducting the orthogonal test with three factors and four levels, the analysis of variance (ANOVA) table obtained by SPSS 16.0 is shown in Table 2. According to the table, the p-values for depth of penetration d, the angle of inclination α and the distance l of the insertion point are all less than 0.01, proving that the three selected experimental factors all have significant influences on the transplanting quality parameter. Each factor was analyzed separately (Table 3). When the insertion depth d is 48 mm (l is 10 mm, α is 14°), the best transplant quality parameter is obtained, indicating that the insertion depth of the needle should not be too deep. If it is too deep, the needles may damage the internal adhesion of the substrate and cause the substrate to fall vertically. In the future, a similar design should ensure that the insertion depth of the needle is reduced under the premise of sufficient pulling force on the seedling. When the insertion point distance l reaches 10 mm (d is 48 mm, α is 14°), the best transplanting quality parameter is obtained. This result is the same as the previous result of the substrate stress analysis and transplanting obstacle avoidance strategy. If the end effector is too close to the seedling, the end effector will bend the seedling stem. If it is too far, it will cause the insertion position of the seedling needle to deviate from the center of the substrate and cause the substrate to fall. When the inclination angle α reaches 14° (l is 10 mm, d is 48 mm), the best transplanting quality parameter is obtained, which proves that when the inclination angle is too small, the design purpose of the inclined manipulator cannot be achieved, which will cause the seedling needle to be damaged during the seedling harvesting process. When the inclination angle is too large, the needle will break the substrate and cause the substrate to fall laterally. According to the data of the experiment, the significance order of influence factors is inclination angle α>probing depth d>insertion point distance l. Finally, when the insertion depth d is 54 mm, the insertion point distance l is 10 mm, and the inclination angle α is 14°, the transplanting manipulator has the optimal transplanting performance. The obstacle avoidance seedling transplanting mechanism under this condition has a leaf damage of 4.70%, a seedling stem bending rate of 16.67%, and a substrate integrity of 83.45%. The seedling transplanting quality parameter was 87.36%. According to the measurement of 112 sets of transplanting data, the single seedling transplanting time was (8.32±0.40) s.
The verification tests were carried out by using the optimal parameter combination obtained from the former experiment. In the verification tests result, the obstacle avoidance seedling transplanting method had leaf damage of 4.47%, a seedling stem bending rate of 15.98%, and a substrate integrity of 81.05%. The quality parameter of seedling transplanting was 85.78% and the average transplanting time was 8.29 s. The error between the verification and the previous test data is within ±5%, proving that the data obtained before is reliable. The structure design and performance experiment data of the seedling transplanting end effector in the article can provide theoretical reference and data support for the subsequent structure design of multi-manipulator seedling transplantation machine. This method can be applied to multi-manipulator seedling transplantation machinery, and effectively reduce the damage of the end effector to the seedlings, so as to improve the success rate of seedling transplantation and the efficiency of seedling transplantation.

Conclusions
(1) Aiming at the problem that the transplanting equipment is easy to damage the seedlings during the transplanting process, a transplanting method of an inclined transplanting robot combined with the Kinect visual processing system was designed, and the seedling height information and edge information were obtained through the Kinect visual processing system. After the information was processed, the coordinate system was established. The manipulator was controlled by the industrial computer, and the coordinate system is converted to achieve the obstacle avoidance function during the transplanting process.
(2) The key parameters affecting the transplanting quality were obtained by force analysis of the seedling transplanting end effector. Transplanting quality parameter (TQP) was proposed to judge the damage rate of seedlings. The optimal transplanting performance parameter was obtained through experiments. In the experiment, the obstacle avoidance transplanting manipulator had a leaf damage degree of 4.70%, a stem bending rate of 16.67%, a substrate integrity of 83.45% and a TQP of 87.36%. The time for a single seedling transplanting was (8.32±0.40) s.