Measurement of the distance from grain divider to harvesting boundary based on dynamic regions of interest

: Combine harvesters need to work along the crop boundary line during operation. Lateral deviation (the distance between the grain divider and the harvesting boundary line) is important navigation information. Improving the measurement accuracy and real-time performance of lateral deviation is an effective way to improve navigation accuracy. Aiming at the problems of poor real-time performance and low measurement accuracy of existing lateral deviation measurement methods, a method of dynamic selection of the region of interest in the process of image processing was proposed and verified by field experiments. The calculation of lateral deviation includes the following stages: analyzing the average gray value of each column in the field image; drawing dynamic region of interest using maximum average gray value; extracting the rice boundary line by using the probabilistic Hough transform algorithm; predicting the location of the boundary line by using the Kalman filter algorithm; measuring the lateral deviation by using the inverse perspective transform algorithm. The analysis of images under different rice fields showed that the method can effectively identify crop boundary lines. According to the test results of calling cameras in different installation positions, the highest extraction success rate of the boundary line was 96.9%, the average success rate was 94.8%, and the speed of real-time measurement of lateral deviation was 0.065 s/frame. When the driving speed was 0.4 m/s, the detection error of linear tracking detection was less than 4.3 cm. With the increase of speed, the error gradually increased. The algorithm has a good real-time performance and high accuracy during low-speed driving.


Introduction 
Grain production plays a vital role in the global economy [1] . With the continuous advancement of agricultural modernization, agricultural production is increasingly developing in intelligence, high reliability, and low cost [2,3] , and automatic navigation has become the inevitable trend of its development [4,5] . The distance from the crop boundary to the divider is an essential parameter for visual navigation. Besides, it is also used to obtain the actual cutting width. To maintain a high-efficiency working state, the intelligently combined harvester works in the full cutting condition in the process of automatic driving. The deviation of the combine harvester is unavoidable during driving. The navigation system needs to correct the deviation constantly.
The real-time measurement of lateral deviation of the combine harvester in automatic driving is of great significance to improve work efficiency and path planning [6] .
Inverse perspective mapping has been widely used in obstacle detection, distance detection, and so on. The image obtained from a monocular camera can be used for distance measurement after inverse perspective mapping [7] . The primary function of inverse perspective mapping is to map the image from 2D space to the world coordinate system [8] . Linear scaling exists between the distance measurement on the image and the distance in the real three-dimensional world. Therefore, the target of interest can be measured to provide environmental information for the intelligent navigation system to assist smart driving. The general processing flow includes: inverse perspective mapping to eliminate perspective deformation and obtain the road plane top view; calibrate the linear relationship between the top view and the real world; detect the distance of interesting objects in real-time.
At present, people often use ultrasonic to measure the cutting width. However, the resolution and anti-interference performance of ultrasonic need to be improved [9] . Tuohy et al. [10] designed a distance measurement algorithm based on the inverse perspective mapping. The distance from the target to the driver was detected by the linear correspondence between the aerial view's pixels and the world coordinate system. As a simple and effective method, the Hough transform is widely used to detect boundary lines [11,12] . Domestic and foreign scholars also use the Kalman algorithm to detect and track lane lines [13,14] .
To provide accurate deviation information for the automatic driving system of the combine harvester, a method of calculating the lateral deviation of the region of interest (ROI) was proposed in this study.
The image processing method of dynamically selecting the ROI and the inverse perspective mapping method which is unable to accurately obtain the camera installation angle were studied to improve the accuracy of the real-time operation boundary line detection. According to the change of spatial coordinates, the lateral deviation was calculated. The method proposed in this study provides an important reference for the autonomous navigation of combine harvesters.

Materials and methods
The experimental equipment included a combine harvester (HaoLong 4lz-7c, Jiangsu World Agriculture Machinery Co., Ltd., China), two monocular cameras (DT1080, Jieruiweitong Ltd., China) used to collect the operation image of the combine harvester, a Raspberry Pi used to process the captured images and calculate the lateral deviation [15] , a 7-inch display screen used to display the processing effect and a mobile power supply used to power the Raspberry Pi. The experimental platform is shown in Figure 1.
According to the different harvest directions, the camera near the harvesting boundary was called. The vertical distance between the camera and the ground was 280 cm, and the distance between the camera and the divider was 35 cm. The image was transmitted through a USB serial port, and the pixel size of the image was 620×480. The average height of rice plants was about 110 cm. To improve the adaptability of the harvest boundary recognition algorithm in different scenes, color space contrast experiments were carried out on the images collected in Zhangjiagang City and Danyang City, Jiangsu Province, China. In the experiment of measuring lateral deviation, the artificial boundary was set on the road surface to simulate the field test, and then the field test was carried out in the Danyang test field. The forward speed was set to 0.4 m/s, 0.8 m/s, and 1.2 m/s to analyze the lateral deviation measurement results at different speeds. In the field experiment, the images of two cameras were distinguished and compared.

Application of inverse perspective mapping in distance measurement
Due to the influence of the relative position between the camera and its shooting object in three-dimensional space and the external parameters of the camera, the perspective distortion of the image is noticeable. To eliminate the perspective deformation, it is necessary to use the inverse perspective mapping algorithm to correct the image perspective. The general inverse perspective mapping steps are as follows: Step 1: Read the inline model and distortion model from the camera.
Step 2: Find the known object on the ground plane. At least four points are obtained with sub-pixel accuracy.
Step 3: Find the coordinates of the four vertices of the image, and calculate the perspective mapping matrix H according to the corresponding relationship between the image coordinates and the actual coordinates. The coordinate relationship is shown in Equation (1): where, x i and y i are the image coordinates; x i ' and y i ' are the world coordinates corresponding to the points on the image; t i is the scale relationship between them.
Step 4: Use matrix H to do pixel interpolation, transform the image to obtain the aerial view.
The lateral deviation is only related to the position of the operation boundary line and the position of the combine harvester. To obtain the top view, pixel interpolation was needed for the whole image. The two points on the boundary equation extracted from the original image were restored from the image coordinate system to the world coordinate system by using the calculated perspective mapping matrix. The real-time performance of the algorithm was improved by reducing the entire image transformation process in the inverse perspective process.

Calculation of perspective matrix
The inverse perspective mapping method based on corresponding point transformation was adopted. The image restoration process had many problems: 1. Accurate measurement of the camera installation angle; 2. Image contains information on different planes in space; 3. Individual lines cannot be accurately identified during the calibration process. According to the perspective transformation projection properties of any group of parallel lines intersecting at a point in the space, the corresponding point was selected with the help of the shadow line, and the physical size of the object performed the restoration. The process of obtaining inverse perspective mapping parameters was as follows: 1) Adjust and fix the camera installation angle; prepare a rectangular calibration plate, mark the calibration plane as ABCD, the physical size of AB side is ab, and the physical size of BC side is bc; measure the vertical height from the ground to the ear top in multiple groups of natural conditions, calculate the average value and record it as h 1 . Select the plane A'B'C'D' with four edges of the calibration plate ultimately detected, and record the plane height to the ground as h 2 .
2) As shown in Figure 2a, plane ABCD is h 1 from the ground; the plane A'B'C'D' is h 2 from the ground; the plane ABCD and A'B'C'D' are parallel to the ground during the calibration process; during the calibration process, keep the AB side perpendicular to the longitudinal axis of the harvester; AA' is perpendicular to the ground. ABCD−A'B'C'D' forms a cuboid; 3) According to the characteristics of the vanishing line, determine the three vanishing points P 1 , P 2 , P 3 Figure 2b shows the calibration effect after inverse perspective mapping. EF is lateral deviation. Rotate the image to the level of AB and the x-axis of the image coordinate system. Take CD in the image coordinate system as the standard, mark the coordinates of points C and D as (u 3 , v 3 ), (u 4 , v 4 ). According to the physical size of the rectangular calibration plate, the coordinates of point A (u 1 , v 1 ) and the coordinates of point B (u 3 , v 3 ) are obtained as follows: where, ab and bc are the physical sizes of AB and BC sides of the calibration plate. Open Access at https://www.ijabe.org Vol. 14 No.
where, k=a 7 m i +a 8 n i +1, the relation between a point (m, n) and the corresponding (u, v) on the original graph satisfy the following equation: There are eight unknowns in the inverse perspective mapping matrix. Given the coordinates of four points on the original image and the inverse perspective mapping image, the inverse perspective mapping matrix can be obtained by substituting the coordinates into Equation (4).

Determination of crop boundary
The agricultural visual navigation system works under natural light conditions affected by light, shadow noise, and other factors. The natural light conditions reduce the accuracy of harvesting boundary line identification and directly affect the operation effect of the agricultural guidance system. Therefore, it is necessary to select appropriate color space for subsequent image processing to improve the adaptability of the visual navigation system to illumination changes.
HSV and YCrCb are chosen for comparison.
These color spaces are composed of three channels [16] . Select appropriate color channels and separate them from speeding up image processing and reduce the impact of lighting. The images corresponding to different color space channels were obtained, and threshold processing was carried out for observation convenience.
Wu et al. [17] selected the binary image of the R channel for analysis according to the distribution map of white background points in the vertical direction. It was determined that the right intersection point of the number curve of white background points and its average value was a straight-line position.
Scan the image column by column from left to right, and calculate the pixel average value of each column in the Cb channel image. The distribution curve is shown in Figure 3a. The maximum pixel average value was 123.4388, and the column number corresponding to the maximum value is 261. In Figure 3b

Boundary extraction based on dynamic ROI
To improve the detection accuracy and speed up the image processing, this study used the method of dynamically obtaining the ROI. The specific operation was as follows: 1) The column number m corresponding to the possible position of the operation boundary line in the image coordinate system was obtained; 2) If the left camera is called, it means that the uncultivated area is on the left side of column M, and the harvested area is on the right side of column M. Tire imprint and stubble may be detected on the right side of column M, which affect the accurate extraction of harvesting boundary. Therefore 3) Similarly, if the right camera is called, α is set to 60 and β to 40.
The dynamic selected regions of interest contain crop row boundaries, which can effectively reduce the impact of shadow and other environments. A blue rectangle represents the ROI.
The Bilateral filter is a nonlinear filter that has edge-preserving, noise reduction, and smoothing advantages. These advantages are conducive to subsequent image processing.
The adaptive maximum between-class variance method was used to segment the image. After segmentation, the pixels on the edge of the harvesting boundary line had an apparent mutation.
The morphological operation was used to remove the interference points and connect the boundary lines. In general, the 'edge' in image edge detection refers to the position where the pixel value changes dramatically. Canny double threshold operator has a high signal-noise ratio (SNR), accurate positioning, and reduces the possibility of false edge and noise. It can effectively detect the edge of the rice boundary line.
Due to the influence of image noise, irregular edges, and other factors, the boundary was not a straight line. Therefore, the probability Hough transform was used to detect the harvesting boundary line [18] . The feature was that the method of random extraction was adopted when selecting the appropriate points. When the number of line fitting times reached the set threshold, the line fitting was completed [19] .
The two endpoints (m 5 , n 5 ) and (m 6 , n 6 ) of the straight line can be obtained by Hough transform. Two new points (u 5 , v 5 ), (u 6 , v 6 ) after perspective transformation can be obtained by Equation (3).
Due to the spatiotemporal continuity of the sequence images (the correlation between the positions of crop boundary lines detected between two consecutive photos), the detection boundary line position changes little. Therefore, the information obtained from the previous frame was used to predict the position of the boundary line in the current frame, instead of calculating the average value of column pixels every time, to improve the processing speed and robustness of the algorithm. In Reference [20], the Kalman filter was used to track the detected lane line, enhancing the robustness of the algorithm. Kalman filter is a linear minimum variance estimation, which adopts a real-time recursive algorithm and introduces state variables. Based on the state transition equation of the system itself, the optimal estimation of the current time was estimated according to the estimation value of the previous time and the current measurement value. The state equation was used to describe the dynamic variation of the estimated quantity. The state equation and measurement equation of the discrete linear system at TK time are as follows: where, X k is the system state variable; Φ k,k-1 is the transfer matrix of the system; Γ k,k-1 is the system noise driving matrix; Z k is the observation variable; H k is the measurement matrix; W k is the system noise sequence; V k is the measurement noise sequence. The subscript k represents time k, and the subscript k, k−1 denotes from time k to time k−1.
The updating process of the discrete Kalman filter is as follows: (1) Update status, ,  (10) where, X is the state estimation of the system; P k is the loud error covariance matrix of the system state; K k is the Kalman gain; I is the unit matrix; Q k is the variance matrix of the system process noise W k ; R k is the variance matrix of the measurement noise V k . Based on the assumption that the detected harvesting boundary line is straight, the slope and intercept can be used to describe a straight line. The two parameters and their change rate were used as the state vector and observation vector of the tracking system. To observe each state variable, the observation matrix is taken as follows: Kalman filter is used to predict the boundary line parameters, and the ROI is dynamically selected to improve image processing speed.

Calculation of lateral deviation
The transverse distance between the harvester and the harvesting boundary line is the vertical distance between the fixed point of the harvester body and the harvesting boundary line detected in this study. As shown in Figure 2b, select the reference point E in the image plane, and take a straight line EE' perpendicular to the ground in the calibration process, and record the distance between the transverse direction point E of the harvester and the detect harvesting boundary line as EG, then the physical size ef of EF can be obtained as follows: where, ab is the physical size of AB side of the calibration plane, cm; AB is the length of the calibration plane AB in the image plane after inverse perspective mapping, pixel; θ is the angle between the harvester and the harvesting boundary line in the image plane after inverse perspective mapping, (°).
To verify the accuracy of the algorithm, the lateral deviation was also measured manually. Positioning rods were placed in the rice field to form a reference line. The actual lateral deviation is calculated as where, d 1 is the distance from the baseline to the crop boundary before, cm; d 2 is the distance after harvesting, cm. The theoretical cutting width of the World 4lz-7c longitudinal axial flow grain harvester selected in this study is 2.6 m.

Results and discussion
The field experiment of rice was conducted in Chenshan Village, Danyang City, Jiangsu Province, China, on November 20, 2019. The fixed camera installation height was about 3 m from the ground. The forward-looking distance of the adjusted image is about 10 m. Experimental conditions and the image recognition efficiency of the harvesting boundary line are shown in Table 1.
The calibration was carried out on the plane with H 1 and H 2 of 0.85 m and 1.40 m, respectively. The reverse perspective mapping rectangle T l of the left camera and the reverse perspective mapping T r matrix of the right camera are as follows: It can be seen from Figure 4 that after using the inverse perspective mapping, the header part of the harvester and the harvesting boundary line is not deformed, and the shape information can be restored, which is conducive to the subsequent detection of the deflection angle and transverse distance between the harvester and the harvesting boundary line. Two common color models, HSV and YCrCb, were compared in this study. The two-color spaces were composed of three channels. To speed up the processing and reduce the influence of illumination, it is necessary to select a reasonable channel and separate them. The images corresponding to different color space channels were obtained. To facilitate observation, threshold processing is carried out, and the processing effect is shown in Figure 5.   Figure 5 Images of different color channels after binarization Taking the right camera of the harvester as an example, a frame was selected for processing. Figure 6a is a Cb channel image of Figure 6a. Figure 6b shows the binarization image without bilateral filtering; Figure 6c shows the binarization image processed by bilateral filtering. The comparison shows that after bilateral filtering, the image noise is significantly reduced, and the boundary information is retained completely.
The opening and closing operations are combined to eliminate small objects and fill small holes to smooth the contour of the image. As shown in Figure 7a, the interference points are removed, and the boundary is effectively connected. Canny double threshold operator with high SNR was used for edge detection (low threshold was 100, the threshold was 200). The edge of the harvesting boundary line can be effectively detected, as shown in Figure 7b. After the ROI was selected, the slope of the line was filtered, and the slope range was (−135, 45). The lines detected using probabilistic Hough transform are shown in the green line in Figure 7c.
To verify the effectiveness of the Kalman filter in selecting dynamic regions of interest, the images collected in Danyang paddy rice field were analyzed. Figure 8 shows the effects of boundary line detection under different harvesting directions. According to the statistics of the pictures collected, i=100, y 1 =100, y 2 =200, has a good image processing effect. Set the initial lateral distance as 30 cm and calculate the lateral distance with Equation (11) at 0.4 m/s, 0.8 m/s, and 1.2 m/s. The flat road and field test results are shown in Table 2.

Conclusions
Based on extracting the boundary line of rice, this study presents a calculation method of the lateral distance deviation during the combine harvester operation based on dynamic ROI. Firstly, the combine harvester parameters and the height of the crop were calibrated, and the inverse perspective matrix was calculated. According to the operation direction, the appropriate camera was called to collect images. The average value of pixels in each column of the Cb channel was calculated, and the maximum value was located in the position of the rice boundary line. Then the ROI was selected dynamically, the accurate boundary line was extracted, and the lateral deviation of the combine harvester was calculated. The results show that the algorithm can calculate the lateral deviation accurately.
Compared with the ultrasonic measurement method, the proposed method has no blind area and is not easily disturbed. Besides, it can provide angle information for automatic driving.
To speed up the processing, the inverse perspective matrix was used to restore two points on the boundary line, which reduces the amount of calculation. Furthermore, the Kalman filter was used to predict the boundary line parameters and calculate the possible position of the boundary line. The average processing speed of each image was increased from 0.176 087 ms to 0.064 547 ms, and the processing speed was increased by 63.3%, which meets the needs of real-time navigation. The field test results showed that the lateral distance measurement error was less than 4.30 cm when the operating speed was 0.4 m/s, the lateral distance measurement error was less than 6.70 cm when the operating speed was 0.8 m/s, and the operation speed was less than 0.8 m/s, which can meet the needs of the navigation system.