Real-time grain breakage sensing for rice combine harvesters using machine vision technology

: Breakage rate is one of the most important indicators to evaluate the harvesting performance of a combine harvester. It is affected by operating parameters of a combine such as feeding rate, the peripheral speed of the threshing cylinder and concave clearance, and shows complex non-linear law. Real-time acquisition of the breakage rate is an effective way to find the correlation of them. In addition, real-time monitoring of the breakage rate can help the driver optimize and adjust the operating parameters of a combine harvester to avoid the breakage rate exceeding the standard. In this study, a real-time monitoring method for the grain breakage rate of the rice combine harvester based on machine vision was proposed. The structure of the sampling device was designed to obtain rice kernel images of high quality in the harvesting process. According to the working characteristics of the combine, the illumination and installation of the light source were optimized, and the lateral lighting system was constructed. A two-step method of “color training-verification” was applied to identify the whole and broken kernels. In the first step, the local threshold algorithm was used to get the edge of kernel particles in a few training images with binary transformation, extract the color spectrum of each particle in color-space HSL and output the recognition model file. The second step was to verify the recognition accuracy and the breakage rate monitoring accuracy through grabbing and processing images in the laboratory. The experiments of about 2300 particles showed that the recognition accuracy of 96% was attained, and the monitoring values of breakage rate and the true artificial monitoring values had good trend consistency. The monitoring device of grain breakage rate based on machine vision can provide technical supports for the intellectualization of combine harvester.


Introduction
Combine harvester operate all over the world, harvesting different crops under different environmental conditions [1] . The working process of a combine is a complex non-linear process, which is influenced by a wide range of parameters, such as crop yield, climate, threshing and cleaning settings [2] . In order to optimize the internal settings of a combine harvester, it is essential to monitor the combine performance information from the harvesting process. The main performance criteria of a combine harvester are grain purity, damage to kernels and processing loss [3] . The breaking rate refers to the mechanical damage caused by the action on the kernel during harvesting, which is mainly formed in the time of threshing. The formation mechanism is that the mechanical action intensity is greater than the threshing work of the crop. On the one hand, the increased broken rate might make the harvested grains easy to mildew and difficult to store. On the other hand, the higher breakage rate reflects the waste of machine energy, especially the increase of threshing energy consumption. Many factors are affecting the kernel damage rate. Increase in kernel breakage and a decrease in seed germination due to a decrease in forwarding speed, increase in cylinder rotation and decrease in clearance between cylinder and concave [4] . In addition, the impact and extrusion of the grain conveyor will also lead to kernel broke up, such as the grain lifting auger that conveys kernels into the grain bin. The real-time monitoring of the grain breakage rate can provide the basis for the relevant operating parameters of the threshing system, reduce the grain breakage rate and the energy consumption, and improve the operational performance of the combine harvester.
In the literature, some researches can be found engaging in grain damage auto-detection technology [5][6][7][8][9][10] . Georg et al. [8] developed an automatic measuring system for whole and broken kernels in which a digital image processing method was chosen as a sensor. Jahari et al. [9] developed a machine vision system with a double lighting device that gives promising results to detect and measure undesirable objects and damaged grain in harvested grain paddy. However, a real-time monitoring device for the kernel breakage rate is not yet developed for practical use so far.
It was effective to install a sampling box at the grain outlet in the grain bin and collect materials (a mixture of whole and damaged kernels). After grabbing an image of the collected samples of kernels, machine vision technology was utilized to extract the color spectrum of connected particles segmented from the image, and data mining technology was involved to identify whole and damaged kernels and the breakage rate of combine harvester was obtained in real-time. The real-time monitoring technology of the grain breakage rate based on machine vision must meet the following requirements. First of all, it is necessary to ensure that kernel-sample be collected and discarded from the grain flow smoothly and periodically by the sampling device and that no blockage occurs. Secondly, it is necessary to provide a closed space for the camera and the light source to generate the appropriate light field for the camera to grab images. Thirdly, an accurate and efficient image processing algorithm should be developed to extract the features of connected domains of kernels in an image, and recognize whole and broken kernels according to the extracted features.
In this paper, the structure design of a sampling box, the optimization design of the light source, and the recognition algorithm of whole-broken kernels based on the color spectrum were introduced. The color spectrum of the connected domain was extracted in the color space HSL, and the "whole/broken kernel classification model" was established. The accuracy of the model was verified by bench tests of about 2300 kernel particles.

Developed equipment
The grains were lifted and transported into the grain-bin by a grain elevator and an auger when harvesting in a field with a forward speed of about 1 m/s on GK 100 Lovol Gushen combine harvester, as shown in Figure 1a. A large number of grains reached the outlet of the auger and were thrown out by the auger. Some grains fell into the sampling box. Several seconds later (the time varies according to the flow rate of the combine), the sampling box was filled with grains. The camera in the box grabbed an image from the visual window and sent it to the image processing unit, and the whole and broken kernels were recognized by the machine vision algorithm, and then the breakage rate was calculated. The image acquisition in the combine grain bin is based on Mind Vision GE200GC-T global shutting camera, with a resolution of 1600×1200 pixels and the capture rate of 60 frames/s. The virtual instrument development platform LabVIEW was utilized, and an image processing algorithm was developed on the Windows system in an industrial computer which was DTB-1222-J1900, Dongtiantech.

Parameters of the sampling box
According to Chen et al. [11] , when rice is harvested by a combine harvester, the crops are fed into the header, transported by the conveying trough, and then separated and cleaned by the threshing and cleaning units. The light impurities are blown out by the airflow. The heavy kernels are deposited and then sent into the grain bin through the elevator. It is almost impossible to install a grain sampling device inside a combine where the contact force, vibration and airflow interfering with each other. When the grain flow enters the grain bin from the auger, the flow rate is large. Therefore, installing a sampling box at the outlet of the auger in the grain bin will be convenient for collecting grains in real time [11] . Jahari et al. [9] proposed an experimental prototype and claimed that more accurate combine harvester operation settings by monitoring the harvested grain as it enters the grain tank could be established in the future [9] . In this section, a sampling box with a transparent visual window and a camera inside was presented as shown in Figure 2a. The camera "observed" the kernels as they flow through the sample cavity. In addition, the camera needed to capture images in a stable environment, with no interruption of natural light. Therefore, covers were used to enclose light source, which was fixed on the two lateral sides and the bottom, and meanwhile waterproof and dust-proof effects were acquired, as shown in Figure 2b flowing through. After a certain period, the sample cavity is filled with kernels, and the camera grabs an image. Then the electromagnet is electrified and the choke plate is pulled out to let the kernels flow out of the cavity. At the same time, the spring is compressed. When the grains in the cavity are almost flowed out, the electromagnet is cut off and the spring pops out, driving choke plate reset and sampling action continues.
The size of the enclosed space with the camera and the light source inside is 180 mm×180 mm×75 mm (length×width×height). The size design is based on the focal length of the camera. The size of the sampling cavity is 180 mm×40 mm×75 mm (length× width×height). The size is based on the number of kernels required in an image. The number of kernels in a single photograph taken by the camera is set to no less than 200. Therefore, the size of the visual window is designed to be 120 mm×50 mm (length×width). The sample cavity with the length and height of 180 mm×75 mm (larger than the visual window) provides enough installation location for the visual window. The size of the sampling box is 300 mm×180 mm× 90 mm (length×width×height, including electromagnets and hanging ears, as shown in Figure 2).

Light source
A light field can be essentially viewed as an array of images captured by a grid of cameras towards the scene [12] . LEDs were chosen as the light source of the "black box" (the enclosed space of the camera and the light source), to provide narrow-spectrum or white light. The tiny light-emitting structure of LED beads formed high-brightness spots on the visual window, which was not conducive to image processing subsequent, and thus the surface light source of a long strip was chosen [13] . Since Super-rice was chosen as the test object, it was hoped that the broken white and whole yellow kernels in the visual window could reflect more light and be more clearly "seen" and "classified" by the camera [14,15] . In addition, the light distribution was supposed to be homogeneous to avoid local over the brightness of the image caused by the reflection highlights of transparent visual windows [16][17][18][19] . Therefore, it is necessary to optimize the angle and distance of the surface light source illuminating at the visual window [20,21] .
The long strip LEDs with blue (450-480 nm), white, red (650-700 nm), green (555-570 nm) and yellow (585 nm) wavelengths were chosen and verified in the illuminating experiments. The images of a grain sample irradiated by different wavelength light sources were captured, as shown in Figure 3. It can be seen that under the illumination of blue, red, green and yellow light sources, broken grains and intact grains cannot be well distinguished. The contrast between broken and intact grains is better under white light because broken grains are white, and the average contrast can be relatively good by using white light.
The shape size of the LED is 60 mm×15 mm, the luminous surface size is 37 mm×10 mm, the rated voltage is 12 V, the luminous flux is 200 lumens, the color rendering index CRI is 80. They were pasted at 20 mm on both sides of the visual window.

Software
The camera operation parameters were adjusted in order to capture images of good quality. The lens was focused and the gain was adjusted to ensure objects could be easily distinguished from the background. The color images had a 10-bit depth for the components. The data were stored in a JPEG format and transferred to the computer.
On the LabVIEW platform, image capturing, enhancement, segmentation and classification of connected particles were processed, and the flow diagram as shown in Figure 4. The sampling box conducted two actions of sampling and discarding within 5 s, and the capturing period of the camera was set to 5 s. In every 5 s the grains were collected and discarded once, and an image was captured and processed. The gray image was extracted through the Green channel signal of the color image. A super rice grain occupies about 60×60 (pixels) image area, and the window size of the local threshold was set to 60×60 (pixels). The colorful images of kernels were extracted by mask method, and the number of extracted kernels was counted and stored as N. The color spectrum of each kernel was calculated, and the distances between the kernel spectrum and the spectrums in the "broken/whole kernel monitoring model" were calculated. The kernel was classified as broken or whole kernels depending on the calculated distances. After classifying all the connected particles in an image, the grain breakage rate was obtained.

Recognition model and validation
Based on the HSL color space, a "broken/whole kernel recognition model" was established. In HSL color space, H is the hue, which specifies the base color in a certain area; S is the saturation, which specifies the brightness of color; L is the luminance, which specifies brightness relative to white. Color spectrum classification is usually divided into two steps: color spectrum learning and matching. In the color spectrum learning stage, the learning algorithm quantifies the color information of the kernel image, generates the color feature vector, the so-called "color spectrum". The file composed of different kinds of image's color spectrum is the "color feature model". In the color spectrum matching stage, the algorithm compares the color feature vectors of the template in the model and the kernel image to be classified, calculates the Manhattan Distance between them, finally, a value is rated representing the difference between 0 and 1000, in which 0 means no similarity between the chromatograms, and 1000 means perfect matching.
Two steps of "color spectrum training-verification" were used to recognize the whole and broken kernel.
The first step was "color spectrum training" according to the training image set. 223 kernel images were selected, in which thirty kernels were classified and marked as "broken" artificially, and the rest were "whole". The color spectrum of each image was calculated to form a spectrum file, which is named as "the broken/whole kernel monitoring model".
The second step was to collect some images as a test set to verify the accuracy of the kernel monitoring model. In the laboratory, the "whole and broken" mixtures were filled into the sampling box and images were captured. Artificial and machine vision methods were used to identify the number of connected domains stored as N (about 230 in an image), and classes labeled as broken or whole. The number of broken grains B a (artificial) and B m (by machine vision) were counted. As shown in Figure 5, a grabbed image is in the left, and the three kernel images of connected domains segmented from the grabbed image are in the right. The three kernel images were classified as "broken kernel" both artificially and by machine vision, which meant the three kernel images were classified correctly. The machine recognized the whole grain as broken, or the broken grain as a whole, which were considered to be wrong, and the number of errors were counted and stored as N w . After the recognition of all connected components, the recognition error of the machine vision algorithm was calculated and the monitoring accuracy of the breakage rate was calculated by, where, N is the number of connected domains in a grabbed image; η r is the recognition error of the machine vision algorithm; α is the breakage rate calculated artificially; β is the breakage rate calculated by the algorithm; γ is the monitoring accuracy of the breakage rate; N w is the number of recognition errors with machine vision; B a is the number of broken grains classified artificially; B m is the number of broken grains classified by machine vision. The above process was repeated ten times and about 2300 connected domains were classified artificially and by machine vision, and the recognition accuracy and monitoring accuracy were validated.

Results and discussion
A total of 232 kernel images were segmented from the grabbed image in Figure 5. The kernels in the connected domains could be clearly seen and identified (kernels in the middle of the rectangular images), as shown in Figure 6. It took 5-6 min to identify all the kernels in a grabbed image artificially, while about 3 s to recognize the kernels by machine vision. Parts of the results were shown in Table 1 in which "1" represents a whole kernel and "0" a broken one. Comparing the classification results of 232 kernel images artificially and by machine vision, seven were classified incorrectly, 225 were classified correctly, and the recognition accuracy of the machine vision algorithm was 97.0%. Figure 6 shows 22 kernel images, and the corresponding results of artificial and machine vision were listed in Table 1.
These 22 machine vision classifications are all correct. Segmented kernel images (partly) In order to verify the accuracy of the algorithm ten images were captured, and the recognition results of Machine-Vision were compared with the actual artificial results, and the error of machine vision recognition η m was calculated, as shown in Table 2. The results of artificial recognitions were taken as the true values, and the number of error classifications by machine vision were counted and stored. Errors were divided into two groups: one was to misidentify a broken kernel as a whole one, the other a whole kernel as a broken one. Table 2 shows that the recognition error of each image was less than 4% for the breakage rate from 6% to 14%, which is to say that the recognition accuracy is no less than 96%. Furthermore, the monitoring accuracy of the breakage rate was verified. The breakage rates by artificial and machine vision methods were compared, and monitoring accuracy of the breakage rate was obtained, as shown in Figure 7. It can be seen from Figure 7 that the breakage rate obtained by machine vision has a good trend correlation with the true value of artificial recognition.

Conclusions
A real-time monitoring device for the rice kernel breakage rate based on machine vision was developed, including a sampling box, light source, camera, image processing and software. The sampling box was installed at the grain outlet in the grain bin, where grain samples were collected in real-time and periodically. The camera fixed inside the sampling box grabbed the images of the grains. The image processing algorithm extracted the color spectrums of kernel images in HSL space to identify the broken and whole kernels, and the breakage rate was calculated. The recognition error was less than 4% for the breakage rate from 6% to 14%, and the monitoring value of the breakage rate has a good trend correlation with the artificial value.