Potential analysis of an automatic transplanting method for healthy potted seedlings using computer vision

Healthy seedlings transplanting is an important process in the production of vegetables and economic crops, and the transplanting quality directly affects crop yield. Automatic seedlings transplanting can improve the transplanting efficiency of seedlings. A physical prototype of potted seedling automatic transplanting with a conveyor and a transplanting end-effector was developed in the previous study. This work proposed an automatic transplanting method of healthy potted seedlings, which mainly included a detection part of seedlings growth status and a visual servo control part with the purpose of automatic transplanting seedlings high-efficiently. The seedlings and the tray cell were simultaneously detected for identifying healthy seedlings, damaged seedlings and empty cells using a machine vision algorithm when the tray was moving on the conveyor line. The visual servo model was applied to enable the collaborative operation of the machine vision and the end-effector for determining the position and attitude of grasping seedlings. The experimental results showed that the accuracy rates of the identification of empty tray cells, healthy seedling and unhealthy seedling were 96.42%, 98.77% and 89.95%, respectively. Under the successful identification of the healthy seedling, the accuracy rate of grasping seedling was 96.38%. It indicated that the proposed method can effectively transplant seedlings.


Introduction 
Seedling transplanting is an important process in the production of economic crops such as vegetables and cotton. The seedling transplanting is also an important agronomic measure to increase agricultural income [1] . The technical idea of seedling transplanting is to cultivate crop seeds into seedlings to be transplanted in a controllable environment, and then transplant the seedlings to farmland. The seedling transplanting can improve the ability for resisting disasters and stress of crops, which has the effect of increasing production and improving quality [2][3][4] .
Seedling raising methods include bare root seedling, paper bowl seedling, and potted seedling. With the development of modern agricultural engineering technology, the method of seedling raising has gradually changed from manual mode to automatic mode. Nowadays, vegetable planting mostly adopts the method of hole plate seedling cultivation, which consists of the factory intensive seedling cultivation, the seedlings commercialization and transplanting the whole plate seedlings to the field by the machine [5][6][7] . The final purpose of the whole process is to transplant high-quality bowl seedlings into the field using an automatic way according to the agricultural standards. However, from factory intensive seedling raising to field transplanting, the ideal states that every cell in the seedling tray has the seedling and every seedling is healthy cannot be guaranteed due to limiting factors such as germplasm, management, packing, transportation [8] .
Machine vision as an intelligent technology has been applied to various fields [9][10][11] . Many studies have reported different kinds of systems for greenhouse seedling transplanting [12][13][14][15] . For example, the research group of Jin et al. [16,17] previously developed automatic transplanting devices for potted vegetable seedlings and analyzed the performance of their working. However, most studies only focused on the seedling quality detection or the seedling grasp end-effector designing. Few studies discussed the integration of seedling detection and end-effector for determining position and attitude of seedling grasp.
At the aspect of seedling quality detection, morphology and robust characteristics of potted seedlings are considered as the key parameters of detection. Methods based on machine vision technology and hyperspectral technology were used in seedling quality detection. The quality of potted seedlings was measured using a machine vision system [18] . Similarly, the quality of transplanting and the growth state of potted seedlings were assessed by the machine vision system [19] . The multimodal segmentation algorithm was proposed to measure the canopy leaf area of a single tomato seedling quickly and accurately [20] . The analysis of the images acquired by a charge-coupled device (CCD) vision system was implemented for comparing the number of pixels in the cotyledons of young seedlings with the preset threshold, the transplanting efficiency could be improved based on the visual information of potted seedlings [21] . The visual recognition system was used to determine the growth direction of the cotyledon of the young seedling, thus the damaged seedlings could be avoided [22] . Some ideal detection results were obtained in some reports. Vigneau et al. [23] used hyperspectral image technology to study wheat leaves of seedlings at 400-1000 nm. It was found that the relationship between the content of nitrogen element in leaves and the spectrum was Closest. Angel et al. [24] applied hyperspectral imaging system to detect herbaceous plants. Each pixel in the hyperspectral image was detected one by one, and the recognition rate of scab was up to 97.1%. The CCD vision-based method may have the advantages of fast detection speed and low cost compared with the hyperspectral image-based method.
The performance of the seedling transplanting parts determines the quality, reliability, and efficiency of transplanting. The key to the transplanting is the research on the transplanting performance and the grasping mechanism of potted seedlings. Kutz et al. [25] took a PUMA560 robot as the carrier, and developed an automatic grasping and transplanting robot. The survival rate of seedlings after transplanting reached 96%. Ji et al. [26] used visual sensor to detect the position of the seedling, and used the force sensor to detect the clamping force exerted by the holding mechanism on the seedling, which could reduce the damage caused by the holder on the seedling during the holding process. A pair of photoelectricity sensors were installed in the seedling guide tube to detect the seedling stem, which could detect the lack and leakage of seedlings in the process of seedling transplanting [27] . A full-automatic system of single line seedling operation was developed, which has the functions of seedling collection, seedling planting, seedling tray transportation, planting depth adjustment, and plant spacing adjustment [28] .
The researches of the transplanting mechanism mostly stayed in the design stage of a full-automatic or semi-automatic mechanism. If the mechanism could not perceive the status of potted seedlings, it would easily cause the problem of damaging seedlings and affect the transplanting quality. However, few studies were reported about the transplanting methods that could adjust the position and attitude of end-effector to grasp seedlings based on detecting the growth state of potted seedlings. This study proposed an intelligent transplanting method of greenhouse potted seedlings with the aim of detecting seedling quality, identifying the empty cell of the tray, and determining the position and attitude of grasping seedlings using machine vision technology. The innovations of the proposed system are: (1) A vision algorithm called the detection algorithm of Grid Coordinate-Based Overlapping Image (GCOI) was developed for simultaneously identifying healthy seedlings, damaged seedlings and empty tray cells by using charge-coupled device (CCD) images analysis.
(2) A vision servo modeling including a transmission controller and a rotation controller was established for determining the position and attitude of the end-effector grasping seedling.

Image acquisition
As shown in Figure 1, the proposed intelligent transplanting system of vegetable seedlings mainly consists of a conveyor unit, a visual detection unit, and the transplanting unit.
The conveyor unit was mainly responsible for the position movement of the tray seedlings, which was composed of a conveyor belt and a frame. The conveyor belt was installed on the frame and driven by a stepper motor. The visual detection unit was installed on top of the conveyor belt. In order to ensure that the visual detection unit can collect effective image information of the tray seedling, the conveyor belt was equipped with a guide rail and a guide wheel. When the tray with seedlings passes through the guide rail area, the horizontal placement posture of the tray would be automatically corrected. The edge line of the tray and the edge line of the conveyor belt would be parallel after the correction. The end of the conveyor belt was equipped with a potted seedling recovery box, which was used for the removal and recovery of the seedlings of poor quality. The image of the tray seedlings was obtained by a charge-coupled device (CCD) camera (model MV-VD120SC, supplied by Microvision Company in Xi'an, China). The CCD camera was connected to a personal computer (PC) by a cable with one USB2.0 interface. The images of the tray seedlings obtained by the CCD camera were stored in the PC, which had 8 GB RAM, an Intel Core i5-4590 CPU, and a Windows 7 operating system. The software system of the image processing system running on the PC was OpenCV 3.0, TensorFlow, and Matlab 8.3. The transplanting unit including a gantry and an end-effector was mainly to carry out the sorting operation of the seedlings. A manipulator moved on a gantry according to the seedling information, and an end-effector equipped at the end of the manipulator was applied to grasp the target seedling. The actions of the gantry and the manipulator were controlled by a programmable logic controller (PLC).
The height and width of the gantry were 1020 mm and 1100 mm, respectively. The end-effector was vertically installed on the Z-axis mechanical arm of the gantry, and the end of the end-effector clamp was 400 mm away from the upper surface of the conveyor belt. According to the physical dimension information of the tray, the Z-axis and the X-axis movable distance of the gantry were 395 mm and 900 mm, respectively. The Y-axis was not designed in the system because that the seedlings were transported to the accessible range of the end-effector under the gantry crane. A rotational degree of freedom (DOF) was added at the connection between the end actuator and the Z-axis. The maximum opening and closing distance of the seedling needle was set as 30 mm, and the thickness of the seedling needle was 4 mm.

System operation flow
The operation flowchart of the proposed intelligent transplanting system is shown in Figure 2. Two triggers were set in the whole process. The tray seedlings were fed on the conveyor belt, then the tray with seedlings moved following the conveyor belt. When the tray was transported to the first trigger, the belt stopped for 1s and the vision detection unit was triggered to acquire the image information of the tray seedlings. The vision detection of seedlings growth status was carried out, where the seedling detection algorithm will be specified in the next section. Then the tray continued to move following the conveyor belt. When the second trigger was triggered by the tray, the belt stopped for 1s and waited for the results of seedlings identification. During tray movement from the first trigger to the second trigger, the detection information of the tray seedlings was transmitted to the sorting and transplanting unit. The position and attitude of the end-effector were determined based on the information of the healthy seedlings. Then the end-effector was driven by the visual information to grasp the healthy seedling. Therefore, the seedling transplanting could be accomplished. The determination algorithm will be specified in the next section.

Computer vision-based seedlings detection
In this section, a computer vision-based algorithm of tray seedlings detection is described in detail. An algorithm of seedling detection was proposed, which is called Grid Coordinate-Based Overlapping Image (GCOI) detection stated as follows.
After image acquisition of the whole tray seedlings, the original color image I was enhanced by using histogram equalization of three colors channels. The enhanced image was recorded as I 1 . Then, I 1 was segmented by the color cluster-based k-means algorithm. The cluster center was selected to be 3. The red part and the green part in I 1 , that were the tray part and the seedling part, were extracted, respectively. They were recorded as I 2 , I 3 , respectively. The edge detection algorithm and the morphology operation were applied in the image I 2 for the edge extraction of the tray cell.
Taking the geometric center of the tray cell in the upper left corner of the image I 2 as the coordinate origin, it was recorded as O 1  Empty tray cell detection: Overlap the image I 4 on the image I 3 . A window of 100×1 00 size was selected as the seed window, whose size was determined based on the pixel numbers of the tray cell. 21 seed windows were assigned to O 1 to O 21 . Seed window growing was implemented twice. The seed window grew 100 pixels at one time. Green part detection inside the window was implemented when the seed window was growing. If the green part cannot be detected in the seed window at any one time of the twice seed growing, the tray cell corresponding to the seed window was empty. And the positions of the empty tray cells were recorded by recording the corresponding positions of O.
Quality detection of potted seedling: A seedling with five large leaves was considered as a healthy seedling. The seedlings with other growth statuses were considered as unhealthy seedlings. In 30 images I 3 , 150 healthy seedling images and 50 unhealthy seedling images were manually extracted, respectively.

Visual servo control model
After the identification of the healthy seedlings, a translation controller and a rotation controller were developed for the end-effector to grasp the healthy seedling.
The specific construction process of the control operators is as follows.  Figure 3, the yellow frame is the schematic diagram of the tray cell edge, and the center O i of the frame can be obtained by the method of the previous section. The O i point was selected to be the target point of the tool center point (TCP), which is also the aim point that the end-effector move to along the X-axis of the gantry. Like the previous statement, a healthy seedling usually has five large leaves. The five leaves are connected together by their stems. The stems are stored in the image in the form of lines. Hough transform was used to detect straight lines in the leaf binary image. If there were two straight lines detected in the image, the end-effector would rotate so that the connection line of the end-effector two fingers would be parallel to the angular bisector of the obtuse angle formed by the detected two lines. According to the principle of robot kinematics, the transform matrix of the end-effector could be obtained as shown in Equation (1).
where, x i is the coordinate of the end-effector on the X-axis; x j is the coordinate of the point in the tray cell coordinate system; θ is the obtuse angle formed by the detected two lines.
If more than two straight lines were detected in the image, the extra straight lines were generally the stems of small leaves, which were in the middle of a group of large leaves. Thus, the connection line of the end-effector two fingers would be parallel to the angular bisector of the acute angle formed by the detected two major straight lines. The transform matrix was shown as Equation (2).

Tray seedlings samples preparation
In the experiment, the seedlings of capsicum were selected as the samples. The variety of capsicum was Hangjiao 7. The seedlings were raised in the tray with 21 cells. The volume ratio of the matrix was that vermiculite: Perlite: peat was equal to 1:3:6. After the matrix mixing evenly, the trays were seeded one by one cell. The humidity and temperature of the incubator were set according to the standard of factory seedling cultivation. The growth and development form of the seedlings in the disc has five leaves, and the root system can wrap the substrate to meet the conditions of transplanting at the right age, so it is suitable for transplanting. The 25 trays filled with capsicum seedlings after 45 d cultivation were fed on the conveyor belt. Then the proposed method was verified on the 25 trays one by one. The statistical data was recorded for the potential analysis of the proposed method.

Growth status detection of potted seedlings
The growth status detection process of the potted seedling was seen in Figures 4a and 4b presented the original image and the extracted tray part, respectively. Figures 4c and 4d showed the extracted seedlings part and the grid image, respectively. The detection results of the empty tray cell, the healthy seedlings and unhealthy seedlings are shown in Figures 4e and 4f, respectively. The identified empty tray cell, the healthy seedlings, and the unhealthy seedlings were remarked by label numbers of different colors.
True positive is the statistical value that predicts the target as the target. False positive is the statistical value that predicts the non-target as the target. False negative is the statistical value that predicts the target as the non-target. Statistical data of three different kinds of targets obtained by using the proposed method was recorded in Table 1. In the algorithm test, the total numbers of the empty cells, the healthy seedlings, and the unhealthy seedlings were 55, 440 and 30 in the total 25 trays, respectively. For the identification of the empty cells, the obtained results were that true positives rate, false positives rate and false negatives rate were 94.54%, 3.51% and 5.46%, respectively. In the tested 440 healthy seedlings, true positives rate, false positives rate and false negatives rate were 94.77%, 1.18% and 5.23% respectively. The results of the unhealthy seedlings, which were 86.67% true positives rate, 9.68% false positives rate and 13.33% false negatives rate, were presented in the table. In the identifications of three targets, the precisions of the proposed algorithm were 96.42%, 98.77% and 89.95%, respectively. And the recalls for three targets were 94.54%, 94.77% and 86.67%.    Figures 5a and 5b showed the grasp implementation and the grasp result, respectively. With the aim of illustrating the performance of the proposed controller, statistical data of grasp performance was recorded under the successful vision detection condition. The true positives, the false positives, and the false negatives mean grasp without seedling leaves damage, grasp other things rather than healthy seedlings and grasp with leaves damage, respectively. In the tested 440 healthy seedlings, the true positives rate was 94.77%, which meant 417 successfully identified healthy seedlings. In the grasp experiment of 417 seedlings, the results were that 95.93% true positives rate, 3.60% false positives rate and 4.07% false negatives rate, were presented in the table. The precision and the recalls were 96.38% and 95.93%, respectively.

Comparative experiment
Faster RCNN, CNN deep learning network, the method in literature [19] and the methods proposed in this paper were respectively used to conduct the identification experiment for 30 trays of seedlings, then the results of experiments were compared. The data is shown in Table 2. The 30 trays of seedlings include 130 empty cells, 40 unhealthy seedlings, and 460 healthy seedlings. 125 empty cells, 451 healthy seedlings, and 35 unhealthy seedlings were identified by the proposed method. Using Faster RCNN, the correct identification numbers of empty cells, healthy seedlings and unhealthy seedlings are 120, 438, and 29, respectively. The correct identification numbers of empty cells, healthy seedlings and unhealthy seedlings are 117, 431, and 28 based on CNN. Using the method in reference [19], the correct identification numbers of empty cells, healthy seedlings and unhealthy seedlings are 109, 423, and 22, respectively.

Efficiency of the proposed method
The time consumed by the proposed method mainly occurred during the growth status detection of the potted seedlings and the pose determination of the end-effector. The time consumed by these two processes is about 875 ms, which is the processing time of the whole tray seedlings. The reciprocating motion of the end-effector takes 400 ms. Since the conveyor belt will stop for 1s to process image information, the system will be able to complete the grasping of three seedlings in the next one second. It shows that the design of tray with three seedlings in one row is reasonable.

Discussions
To exactly grasp seedlings, a GCOI algorithm was firstly proposed for the identification of the empty tray cell, the healthy seedlings, and the unhealthy seedlings. For the empty tray cell identification, 94.54% true positives rate showed that the proposed secondary growth of the seed window could effectively judge whether one tray cell was an empty tray cell. However, the rate that other non-empty cells were identified as empty cells was 3.51%, which mostly happened in the tray cell including unhealthy seedlings. The growth of unhealthy seedlings deviated from the central line of the tray cell. The leaf surface shrank and extended to the adjacent tray cell, which might be the reason for wrong identification of the empty tray cells. The reason for the 5.46% leakage identification rates of the empty tray cell might be that the leaves of healthy seedlings in the adjacent tray cell extended to the center of the empty cell so that the empty cell was not identified. For the healthy seedling identification, 94.54% healthy seedlings in 440 samples were correctly identified, which implied Faster RCNN model was potential. And it was also confirmed by 1.18% false healthy seedling identification and 5.23% leakage healthy seedling identification. The large overlapped area of healthy seedling leaves of adjacent tray cells explained 5.23% false negatives rate. For the unhealthy seedling identification, although true positives rate was 86.67%, 26 correct identifications could explain that the identification was effective compared with 30 total samples. Most reasons for 9.68% false positives rate and 13.33% false negatives rate were the overlapping and occlusion.
In the seedling grasp experiment, 95.93% of the tested samples were without damage by using the proposed controller. However, due to mechanical structure matching error, 3.60% of the total samples were wrong to target grasping or correct grasping with damaged leaves. Meanwhile, the occurrence of this kind of error was often accompanied by the overlapping of adjacent tray cell leaves. In 4.07% false negatives rate, mechanism failure and communication failures contributed to the error rate.
The results show that the proposed method in this study is superior to other methods in the recognition of empty units, healthy seedlings, and unhealthy seedlings in the comparison. The proposed method not only absorbs the advantages of the deep learning module, but also develops a visual processing algorithm with a high success rate of recognition for the characteristics of tray seedlings, which illustrates the advantages of the proposed method.

Conclusions
This study developed an intelligent transplanting system for greenhouse potted seedlings based on machine vision. The system mainly includes the transmission system, the vision detection system and the transplanting system. A GCOI (Grid Coordinate-Based Overlapping Image) algorithm was proposed in the vision detection system for identification of the empty tray cell, the healthy seedlings, and the unhealthy seedlings. A translation controller and a rotation controller were developed in the transplanting system with the aim of locating the system end-effector to the target seedling for grasping. The performance of the system was tested by using the potted seedling images from 25 trays. The main results are shown as the following: (1) The proposed system adopts the intelligent way of combining the vision system and the grasp system to transplant seedlings.
(2) The vision detection part of the system uses a proposed GCOI algorithm to identify empty tray cells, healthy seedlings and unhealthy seedlings, by using which, the identification precisions of three targets were 96.42%, 98.77%, and 89.95%, respectively.
(3) Under the successful identification of the healthy seedlings condition, the successful grasp rate without damage seedlings is 95.93%. The precision of the grasp seedling is 96.38%. It implied that the proposed controller is effective.
(4) The overlapping and occlusion of leaves in the adjacent tray cells contributed to false positives and false negatives.
Therefore, the developed intelligent transplanting system can effectively identify the empty tray cell, the healthy seedlings, and the unhealthy seedlings, and grasp seedlings without damage. In future research, dynamic transplanting and transplanting under overlapping and occlusion conditions will be important topics of our research.