Comparison of manual and automatic barcode detection in rough horticultural production systems

: Automation of production in the nurseries of flower producing companies using barcode scanners have been attempted but with little success. Stationary laser barcode scanners which have been used for automation have failed due to the close proximity between the barcode and the scanner, and factors such as speed, angle of inclination of the barcode, damage to the barcode and dirt on the barcode. Furthermore, laser barcode scanners are still being used manually in the nurseries making work laborious and time consuming, which leading to reduced productivity. Therefore, an automated image-based barcode detection system to help solve the aforementioned problems was proposed. Experiments were conducted under different situations with clean and artificially soiled Code 128 barcodes in both the laboratory and under real production conditions in a flower producing company. The images were analyzed with a specific algorithm developed with the software tool Halcon. Overall the results from the company showed that the image-based system has a future prospect for automation in the nursery.


Introduction 
Automation of production systems is continually developing in horticulture, logistics and many other fields to increase production, productivity and profitability.Automatic Identification Data Capture (AIDC) such as barcodes, Radio Frequency Identification (RFID), smart cards etc., have been developed to replace manual data collection and to provide an accurate, quick, and efficient means of capturing and storing data [1][2][3] .Barcodes are the most common of the AIDCs used in the last 5 decades [4,5] .Barcodes are simply the machine-readable vertical black strips with white spaces which are printed and found on most products [6][7][8] .Advancement in barcode technology has led to two-dimensional (2D) barcodes being developed [5,9,10] .Barcodes have been used extensively in horticultural production systems to eliminate the laborious and time-consuming process of manual data entry and to capture information about plants and products to which they are attached [6,11] .Flower producing companies mainly use barcodes in their production to track planting trays containing flowers and cuttings, gather information on the availability and re-ordering of planting stock, and store information on the stock for planting and distribution.Furthermore, at the points of sale (POS) barcodes are used to check theft and speed up sales [12][13][14] .All these practices with barcodes are done manually using a laser barcode scanner.
For the flower producing companies it is important to have the barcode data to know how they are faring in their production process and where to make changes and improvements.
However, full automation in the area of planting flower cuttings in greenhouses is still a big challenge as the conventional systems of using laser barcode scanners are still being employed.These barcode scanners require a line of sight for identification.Therefore, one has to get close to the barcodes with the scanner before the sensor can generate a reading.Also dirty, damaged, faded and multiple barcodes cannot be read or are difficult to read, and there is also the risk of losing some barcodes in the production process [9,10,[15][16][17][18][19] .Furthermore, attempts to use barcodes to track trays in large nursery production systems have failed as soil and water often covered the barcodes, preventing successful scanning.Also the growth of the plants leads to an increase in the crop canopy which sometimes covers the barcodes, resulting in reduction in successful barcode scans and thereby making identification difficult [20,21] .
Over the years several barcode recognition methods have been developed to help in the processes of automation and detection.There are different localization and re-identification methods due to accuracy and speed.However, there are problems associated with each different type of barcode.There is a continuing need for improved barcode localization.High speed processing activities such as automated production and conveyor belts where, a missed detection results in loss of profit, requires automatic barcode detection with great accuracy [8,22] .Different image processing methods and techniques have been used to decode barcodes in the last four decades.Hough transform [11,23,24] .or mathematical methods like morphological operations detection based on extraction from the image of various texture-like properties such as erosion, dilation, opening, closing etc. [25][26][27][28] have all been used successfully in decoding barcodes.However, these methods have been used mostly in combination with each other.In recent years deep learning and artificial intelligence with Hough transform and morphological operations is increasingly being used to localize and decode barcodes [5,[29][30][31] .The Parallel Line Segment Detector with Hough transform and morphological operations have also been used to decode barcodes in real-time [7,32] .Furthermore, the Zamberletti algorithm has recently been used to detect multiple 1D and 2D barcode images [8] .Most of these methods have tried to localize barcodes in real-time when the images are stationary rather than in motion.There is still the increasing need to capture barcodes in motion.In summary, according to literature and practice, speed, dirt, moisture, uneven illumination and complex background hinder successful barcode detection.Therefore, there is a need to find appropriate solutions to overcome these problems.
The objective of this research was to provide a robust system for successful automatic detection of barcodes in horticultural production.This was achieved by comparing a hand-based barcode laser detection system to an image-based barcode detection system to determine which provides more reliable, stable and faster results under various conditions.We proposed a new image processing algorithm for robust barcode detection and evaluation using Halcon.
For evaluation of the proposed system, experiments were conducted under real horticultural production conditions in a flower producing company (Brandkamp GmbH, Isselburg-Anholt, Germany).

Materials and methods
In the experiments conducted, two types of barcodes Code 128 were chosen.One shows the variety, factory week and tray type of the cutting to be planted "Plant Barcode (PB)" (Figure 1a).The other type of barcode is the Personal Number Barcode (PNB) (Figure 1b), which identifies the worker who planted the tray, making it possible to trace back to that person in case of any problem during production.A monochrome USB industrial camera (DMK 41BU02.H, The Imaging Source Europe GmbH, Bremen, Germany) with a CS-IR lens (H3Z 4512, The Imaging Source Europe GmbH, Bremen, Germany) were used for all the experiments.The MVTec Halcon 11.0.1 (CGI Systems GmbH, Seeshaupt, Germany) image processing software was used to decode the barcodes.Statistical analyses for the experiments were done using R-Statistical Package 3.4.4(http://cran.r-project.org/).Confidence intervals of 95% for the difference of proportions were estimated to determine the proportion of successful readings.A Mosaic plot to create a color chart using ggplot2 (http://ggplot2.org/)was used to determine the percentage of success and failure in barcode detection.Pairwise comparison tests using the equality differences of proportions were completed to compare pairs of treatments using Pearson Chi Square test.The raw p-value and the adjusted p-values for multiple testing were conducted using the Holm method.Graphs were created using SigmaPlot 12 (Systat Software Inc, USA).

Experimental setup for laboratory and company
400 replications of each experiment were completed at the laboratory of the Department of Biosystems Engineering, Leibniz University of Hannover, and at Brandkamp GmbH.The effects of angle placement and conveyor belt speed on barcode detection were measured in the laboratory.Seven different conveyor belt speeds (0.03, 0.07, 0.11, 0.17, 0.21, 0.26, 0.30 m/s) were used for the clean barcode experiments in both the laboratory and company.However, for the treatments with dirt and water in the company only two speeds (0.11 and 0.15 m/s) were used.
The laboratory experiment setup consisted of two parallel conveyor belts on top of which are movable plates.At each end of the conveyor belts are two plates that rotate in opposite directions.These act as a switching mechanism between the two belts (Figure 2).The system forms a closed loop with an adjustable front conveyor belt on which a shading system was constructed.The maximum test speed of the system is 0.30 m/s.This was used to determine whether it was possible to automate barcode detection at this speed.The test system was controlled by the SPS Control System (Beckhoff Automation GmbH & Co. KG, Lübeck, Germany) located in front of the conveyor belt (Figure 2).Shading of the image-based barcode recognition system to reduce the effect of reflection on barcode detection used a black fine-twill viscose material which was selected for its superior shading capability from among 8 different black fabrics.3 Osram L 18 W/840 Lumilux 59cm -Cool White fluorescent tubes (Osram GmbH, Munich, Germany) were used for lighting the system (Figure 2).Only clean PB labels were used in the laboratory experiments.The second part of the experiments were conducted in the flower producing company (Brandkamp GmbH, Isselburg-Anholt, Germany) to see the effect of speed, dirt and moisture on automation of barcode detection in real production conditions.The setup of the company is as shown below (Figure 2).Both PB and PNB labels were used for these experiments (Figure 1).
Fresh cuttings of various flowers such as Chrysanthemum, Impatiens, Fuchsia etc. were produced in the Figure 2 Laboratory and company image-based and hand-held barcode recognition system company.These cuttings were brought in the morning in plastic bags, moistened to keep them fresh and then planted manually in the trays already filled with soil.The company's current procedure is to scan the barcodes while stationary with a Datalogic Memor Mobile barcode scanner (Opal Associates GmbH, Munich, Germany) before they are moved on a conveyor belt.For our experiments an additionally image processing system with camera was installed on the conveyor belt system to read the labels (Figure 2).A summary of the test materials and general parameters used for all the experiments is shown below (Table 1).Note: * The x and y direction is in the view plane of the camera.

Robustness of the automated image-based barcode detection system
To determine the robustness of the image-based system, various dirt (artificially soiled) treatments were applied to the surface of both types of barcodes.20 g of Potgrond P (Klasmman-Deilmann GmbH, Geeste, Germany) soil was put in a 28 cm plant pot saucer and mixed with 20 mL of water.This was then rubbed randomly on all the barcodes as shown (Figure 3a and   b) to get a slightly dirty surface; i.e., 10%-20% of the surface was dirty.The mixed soil medium was also vigorously applied to the surface of the slightly dirtied barcodes to make the barcodes extremely dirty; i.e., 50%-60% of the surface was dirty (Figure 3c  and d).Finally, 20 g of planting soil medium was put in a 1 L spray bottle and 750 mL water was added, stirred to mix well and sprayed on the dirtied barcodes to see the effect of moisture on barcode detection.

Proposed algorithm
The proposed algorithm (Figure 4) which was based on mathematical morphological methods was used for all the experiments.The system was divided into two main parts: preprocessing and detection.Images were initially captured using the DMK 41BU02.H CCD USB 2.0 Monochrome Industrial Camera, the CS mount lens (H3Z 4512 CS-IR) and the image acquisition software IC Capture 2.2 (The Imaging Source Europe GmbH, Bremen, Germany).IC Capture allows the images to be captured from the camera and saved in three different ways.The images can be saved manually as single images, manually as image sequences and finally image sequences can be saved via a timer.The images were grabbed asynchronously so that while an old image is being processed a new one can also be grabbed in the process.Also, the images were grabbed asynchronously so that images can be grabbed and stored intermittently when they become available.The RGB image was then converted into three one-channel images with the same definition domain using the function decompose3 in Halcon.The three channels of the image were passed as three separate images on input and output so that only the channel of interest was chosen for further processing.The best image access channel was then selected.The third image channel was chosen for all the experiments.The selected images were then smoothed to reduce noise using the Gaussian filter of size 11×11 where σ = 2.550.
The second step used morphological operators to find the region containing the code.The smooth (Gauss) image was segmented into regions of the same intensity using the morphological operator region growing.The region boundaries were then smoothed and the small gaps between adjacent regions and holes smaller than the structure element were closed using the morphological operator closing circle.The regions of interest were then selected according to shape and merged together using the operators selected shape and union respectively.Further smoothing of the region boundaries and removing of regions smaller than the structure element were opened using the morphological operator opening circle.The shape and size of the region of interest was maintained by performing a rigid affine transformation using a rotation matrix and a translation vector from a point and two corresponding angles, and a scaling by scale factors along the x-axis and y-axis based on the following equation: cos( ) sin( ) 0 2 sin( ) cos( ) 0 0 0 1 where, HomMat2D is the homogeneous 2D transformation matrix, Phi is the rotation angle; Tx is the translation along x-axis; Ty is the translation along y-axis; Sx is the scale factor along x-axis and Sy is the scale factor along y-axis.The final part of the system was to decode the image data by performing a 2D Homogeneous and Affine Transformation on the image to enable detection from various angles.The image was first scaled, then rotated and finally translated before the code was detected.This was done by using the shape-based matching algorithm to find the region of interest (scan lines) on the barcode (Table 2).If the image was decoded a green rectangle and the number on the barcode was displayed.The result was then stored in a data sheet in the form of the company's name, type of production, production speed, date and time when it was captured.However, if the image was not decoded, no rectangle was displayed and the result was stored as zero.Three different angle combinations (60°, 75° and 90°) and 400 replications were used.The color chart (Figure 5) shows the influence of different speeds and angle placements on detection and reading of clean plant barcodes (PB).Also shown is the result for static barcodes at various angle combinations.The most critical angle is the rotation in the z-plane.Z-plane rotation decreases the apparent size of the barcode for the camera, which is a critical consideration when dealing with barcode reading by image processing.The system detected 100% of the barcodes in a static position (Figure 5).At angle combinations of 90°.90°.90°where the barcode was held perpendicular to the camera, there was 100% detection for speeds of 0.03, 0.07, 0.11, 0.17 & 0.21 m/s.There was 95% detection at the highest speed, 0.30 m/s (Figure 6).This demonstrated that speed did not significantly affect readings at this combination of angles.
However, at angle combinations of 60°.60°.60°,75°.75°.75°,60°.90°.75° and 60°.60°.75°there was successful detection only at the lowest speed of 0.03 m/s (Figure 5).Thus, as speed increases and the angle of inclination of the barcodes in relation to the camera decreases from 90° to 60°, the percentage of success of barcode detection decreases.Conversely as speed decreases and the angle increases, the rate of successful detection improves.
Figure 5 The effect of speed and angle of barcode placement on barcode detection (n = 400) Figure 6 The effect of speed on barcode detection in the lab and company (n = 400) Static and dynamic (in motion) experiments have been conducted for barcode detection using an omnidirectional camera for automated guided vehicle (AGV) with different barcode sizes [33] .The dynamic experiments were conducted with multiple runs where the algorithm must decode four different barcodes located along the pathway of the AGV.Due to the slower frame rate of the omnidirectional camera, the speed of the AGV was limited to 0.32 m/s resulting in a longer processing time of the algorithm, even though the AGV could move at a speed of 2.2 m/s [33] .100% of the largest size of barcode was detected while the AGV was in motion for the 4 different runs.The detection and score decrease for smaller barcodes, with no detection for the smallest barcode size.The dynamic experiments showed that the frame rate of the camera, the barcode size and the processing time affected the detection of the barcodes [9,10,33] .In our dynamic experiments, as the speed increased above 0.21 m/s the detection rate decreased (Figure 6).
The static experiments conducted with the barcodes placed in front of and on the right or left sides of the AGV with the barcodes either lying flat or standing showed that larger barcode sizes had better detection rates than smaller barcode sizes.100% of the bigger barcode sizes placed at the side of the AGV were detected.0% of the smallest barcode sizes were detected.For barcodes placed in front of the AGV there was 100% detection for only the biggest barcode size.There was no detection for the rest of the barcode sizes.Therefore, size affects the speed of detection of the barcode.Also larger barcodes have wider bar arrangements and even in the midst of noise are clearly mapped into image pixels, resulting in better detection compared to the other sizes [9,10,33] .In our static experiments the width and size of the scan lines of the barcodes did not affect detection although the size of the plant barcodes (PB) (31mm × 7mm) was slightly smaller than that of the personnel number barcodes (PNB) (34 mm × 10 mm).

Influence of speed on automated image-based barcode detection in the laboratory and the company
Various speed treatments were tried to see how they affect barcode detection with the image-based detection system in the laboratory and in real production conditions.The bar chart (Figure 6) shows the percentage of successfully decoded barcodes in the laboratory and company.As the speed increases the success of barcode detection decreases for both types of barcodes (PB and PNB). Figure 6 shows the influence of different speeds on barcode detection of clean PB (laboratory and company experiment) and PNB (company experiment) labels.The Pearson Chi 2 test using the Holm-adjustment method for multiple comparison tests showed that there were no significant differences among all the clean barcodes in the laboratory and the two types of barcodes used in the company at 0.03, 0.07 & 0.11 m/s.However, there were significant differences among the results in the laboratory and company at 0.17, 0.21, 0.26 & 0.30 m/s.As the speed of the automated image-based barcode detection system in both the laboratory and company increased the detection of the barcodes decreased significantly.
There was 100% detection of clean PB at more of the operating speeds of the system used in the laboratory than in the company.The system of the company is inclined at an angle approximately 45° as the planting tray passes in the FOV (field of view) of the camera.However, in the laboratory, the operating system is not inclined and thus the barcode image is perpendicular to the FOV of the camera.Barcode orientation and inclination has been known to affect detection of 1D barcodes.Therefore, as speed increased detection decreased and the greater success in detecting barcodes in the laboratory as compared to the company, can be attributed to this [34,35] .Also due to the wider bar arrangement of scan lines of the PNB as compared to the PB (Figure 1) used in the company, there was better detection of PNB in the company at the various operating speeds (Figure 6).Consequently, as speed increases there was a better detection of the PNB [9,10,30,33,36] .
Also the frame rate of the camera and operating system of the computer has been known to affect the speed of barcode detection in production systems.In the dynamic experiments using the omnidirectional vision system the operating speed of the AGV was limited to 0.32 m/s due to the slower frame rate of the camera resulting in a longer processing time for the image due to the computer processor [33] .In our experiments, a camera with a frame rate of 15 fps and a PC with Pentium®Dual-Core ™ E5800 3.3 GHz CPU, 2 GB RAM Windows 7 Enterprise 32-bit Operating System was used.This can possibly be one of the reasons why there was reduction in barcode detection in our experiments when the test speed of the automated image-based system increased from 0.21 to 0.30 m/s (Figure 6).A modern camera with a very high frame rate and a PC with a high speed processor should therefore be tested to see if it will result in better barcode detection.Also, external light affects the detection of the barcodes because it affects the reflection of scan lines of the barcode.Shading was done in order to prevent the influence of external light but a hundred percent shading was not achieved.This is because the system requires openings in the shading system so that the trays can pass on the conveyor belt.These openings inevitably allowed some external light to enter.How much this external light affected the detection was not measured and should be examined or studied in the nearest future.

Robustness of the automated image-based barcode detection system
Various dirt treatments were used to see how they affect barcode detection with the automated image-based detection in real production conditions.The bar chart (Figure 7) shows the influence of speed, dirt and moisture on successfully decoded dirty PB and PNB in the company.Dirt and speed had an effect on the barcode detection as the barcodes that were slightly dirtied had better detection than those that were extremely dirtied.There was 100% detection for slightly dirty PB at 0.11 m/s and 90% at 0.15 m/s.For both extremely dirty PB and dirty water PB there was 90% detection at 0.11 m/s and 78% and 93% at 0.15 m/s respectively.For PNB there was 100% detection for both slightly dirty and dirty water at both speeds and 90% detection for extremely dirty at both speeds.Thus, extremely dirty barcode surfaces influenced the detection of the barcodes while moisture and slightly dirty barcode surfaces did not significantly affect detection.Table 3 below shows the comparison among the various dirt treatments, barcode types (PB and PNB) and the two speeds (0.11 & 0.15 m/s) used in the company.The Pearson Chi 2 test using the Holm-adjustment method for multiple comparison tests showed that there were significant differences among slightly dirty (SD) and dirty water (DW) and SD and extremely dirty (ED) at 0.11 m/s when PB were used.There was no significant difference between ED and DW at 0.11 m/s when PB was used.At 0.15 m/s there were significant differences among all the dirt treatments of PB.For PNB at 0.11 and 0.15 m/s there was no significant difference between SD and DW but there were significant differences among ED and DW and SD and ED.This showed that dirt on the surface of the barcode and speed of the automated image-based barcode detection system in company significantly affects barcode detection.
For nearly all the treatments there was either no difference at all or the observed differences were so highly significant that the p-values are practically equal to 0 (Table 3).PNB had the best detection at the two speeds of all the dirt treatments which could be due to its slightly wider bar arrangement and size compared to that of the PB [9,10,[17][18][19]34,37] .
Operational errors such as distorted labels and wrong position of barcodes also play a significant role in automated barcode detection systems.Operational errors from handling of the barcodes, such as improper fixing of barcodes, dirt on the barcodes and the use of wrong barcodes, i.e., barcodes without scan lines (as some are used in the company for labeling), all affect successful automated barcode detection.Operational errors can lead to 64% reduction in barcode detection [38] .This is a significant loss in barcode detection and as such during production all efforts must be put in place to minimize these errors as it affects the profit of the production system.There was always a 100% detection of barcodes using the Datalogic Memor X3 mobile barcode scanner irrespective of the treatments used although the more extreme the treatment (extremely dirty) the longer it took to detect the barcode.Readings for the barcodes using the hand laser scanner ranged between 1 to 5 seconds depending on the treatment on the barcode, as line of sight is required for successful scanning.However for the automated image-based system it took 1 second for readings to be detected as there was no need for a direct line of sight to the barcode before detection.The proposed system was robust and faster in detecting dirty barcodes than with the hand laser scanner.

Conclusions
The image-based barcode detection system demonstrated that both clean and dirty barcodes can be successfully decoded in real production conditions at the company's production speed of 0.11 m/s.However, extremely dirty barcodes affected the detection of the plant barcodes more than the personnel number barcodes.The automated image-based barcode system has good future prospects over the hand-based system.Automated image-based barcode detection and reading can be implemented at a speed range of 0.11 to 15 m/s in a horticultural production system to increase the productivity and profitability of the company.For a successful implementation of the image-based system for automation, the following points must all be considered for reliable and stable results: a good lighting system, shading from interaction of external light and angle of inclination of barcodes in relation to the camera, barcode size, camera resolution and frame rate, width of the bar arrangement on the barcode and operational errors from workers.
QR Codes are now widely used in horticulture production and carry a lot of information with a high data capacity.Future studies using QR Code should also be carried out using the image-based system and compared to the 1D (Code 128) barcode to see which works best with the image-based system.This will help small flower producing companies save costs as printing of QR Codes is much cheaper than using RFID.There are plans to implement the proposed system in the flower production company Brandkamp GmbH in the near future.

Figure 1 (
Figure 1 (a) Barcode showing tray type, variety and factory week; and (b) Barcode showing personnel number

Figure 3
Figure 3 Dirt treatment of barcodes HomMat2DIdentity is the homogeneous transformation of the identical 2D transformation; S is the scalar factor; P the fixed point of the transformation; ScaleR is Sx, ScaleC is Sy, the Angle is Phi; Row is the translation vector Tx and Column the translation vector Ty.

Figure 4 Flow3 Results and discussion 3 . 1
Figure 4 Flow Chart of the image processing system 3 Results and discussion 3.1 Influence of speed and angle placement and other factors on barcode detection The experiment to determine the effect of speed and angle placement on barcode detection used 7 different speeds with 0 m/s indicating a stationary barcode.Three different angle combinations (60°, 75° and 90°) and 400 replications were used.The color chart (Figure5) shows the influence of different speeds and angle placements on detection and reading of clean plant barcodes (PB).Also shown is the result for static barcodes at various angle combinations.The most critical angle is the rotation in the z-plane.Z-plane rotation decreases the apparent size of the barcode for the camera, which is a critical consideration when dealing with barcode reading by image processing.The system detected 100% of the barcodes in a static position (Figure5).At angle combinations of 90°.90°.90°where the barcode was held perpendicular to the camera, there was 100% detection for speeds of 0.03, 0.07, 0.11, 0.17 & 0.21 m/s.There was 95% detection at the highest speed, 0.30 m/s (Figure6).This demonstrated that speed did not significantly affect readings at this combination of angles.However, at angle combinations of 60°.60°.60°,75°.75°.75°,60°.90°.75° and 60°.60°.75°there was successful detection only at the lowest speed of 0.03 m/s (Figure5).Thus, as speed increases and the angle of inclination of the barcodes in relation to the camera decreases from 90° to 60°, the percentage of

Figure 7 Table 3
Figure 7 The effects of dirt and speed on barcode detection in the company (n = 400) Table 3 Comparison of dirt treatments of the plant barcodes (PB), worker barcodes (PNB) and speed at 95% confidence intervals using Pearson Chi 2 test and the Holm-adjustment method for multiple comparison tests p.val.adj p.val.raw comparison barcode Speed/m• s -1