1D Barcode Scanning Accuracy Benchmark and Comparison
Introduction
Accuracy is a key metric of barcode reading SDKs. Given an image, a barcode engine should correctly detect whether one or more barcodes exist. If barcodes exist, the percentage of correct values returned dictates the accuracy.
As a software developer, choosing a barcode engine that has the best speed, accuracy, robustness, and customer support becomes a crucial step in the success of a barcode scanner application. In this article, we are going to discuss evaluation of barcode scanning accuracy and tips on achieving a higher accuracy.
This article is Part 2 in a 5-Part Series.
How to Evaluate Barcode Scanning Accuracy
There are two steps to read a barcode:
- Detection (AKA localization) - Upon initiating a scan, a barcode engine scans the entire width of an image trying to identify whether there are barcode candidates. One or more coordinates of quadrilaterals are returned.
- Decode (AKA recognition) - Once the regions of interest are identified, decoding is then initiated. In this step, the barcode scanner tries to unravel the encoded data.
Corresponding to the two steps, some datasets mark up the coordinates of the barcode candidates, while others mark up the barcode encoded values. Both approaches can be used to compare different barcode scanning engines.
This article uses the barcode values to measure barcode scanning accuracy. We will discuss the datasets in the following section.
Barcode Engines
In this comprehensive analysis, we put compare Dynamsoft’s barcode SDK against another commercial barcode SDK and two popular open source solutions, Zbar and ZXing.
Datasets Used For Comparison
We have compiled a table of existing publicly available barcode datasets.
| Dataset | Artelab Medium Barcode 1D Collection | Muenster BarcodeDB | DEAL Lab’s Barcode Dataset |
|---|---|---|---|
| Symbologies | EAN13 | EAN13, UPC-A | EAN13 |
| Include images with multiple barcodes | No | No | No |
| Number of images | 430 | 1054 | 2000 |
| Image quality | Barcodes are relatively large compared to the image size | Barcodes are relatively large compared to the image size | Resolutions range from 400x217 to 3480x4640 |
| Web page | http://artelab.dista.uninsubria.it/downloads/datasets/barcode/medium_barcode_1d/medium_barcode_1d.html | https://www.uni-muenster.de/PRIA/en/forschung/index.shtml | https://www.resl.kaist.ac.kr/doc/datasets |
Metrics to Evaluate Accuracy
In this article, we use precision and reading rate to evaluate the accuracy.
The precision is the number of correct results divided by the number of all results. A precision of 100% means each of the results is correct, although some barcodes might be missed.
Precision = correct results / all returned results
The reading rate is the number of correct results divided by the number of all barcodes. A precision of 100% means that all the barcodes were recognized successfully, although there might be wrong values mixed in with the results. For example, if a barcode is mistakenly recognized as two separate ones, this behavior would not lower the reading rate.
Reading rate = correct results / all barcodes
Performance Test Results
We ran a performance test on the three datasets mentioned above. The annotations are examined and corrected manually to avoid errors.
Note:
- If a barcode is recognized as UPC-A and the ground truth is in EAN13 format (the EAN13 format has one extra leading zero), it is considered as correct.
- Some images contain multiple barcodes while in the annotation, only one barcode is annotated. If the SDK reads the unannotated barcode, it is not considered as mis-read.
Performance Test Result on Artelab Medium Barcode 1D Collection
The Artelab Medium Barcode 1D Collection has two sets of images. Dataset 1 contains pictures of barcodes taken from devices with the autofocus (AF) function. Dataset 2 contains pictures taken by devices without AF. Each of the datasets has 215 images.
The test result on dataset 1:
| SDK | Dynamsoft Barcode Reader v9.6.10 | Commercial SDK A v5.19 | ZXing-CPP v1.2.0 | pyZbar v0.1.8 |
|---|---|---|---|---|
| Correctly recognized | 215 out of 215 | 197 out of 215 | 177 out of 215 | 193 out of 215 |
| Misread | 0 | 0 | 1 | 1 |
| Reading rate | 100% | 91.63% | 82.36% | 89.77% |
| Precision | 100% | 100% | 99.44% | 99.48% |
Sample image:
The test result on out-of-focus images dataset 2:
| SDK | Dynamsoft Barcode Reader v9.6.10 | Commercial SDK A v5.19 | ZXing-CPP v1.2.0 | pyZbar v0.1.8 |
|---|---|---|---|---|
| Correctly recognized | 176 out of 215 | 169 out of 215 | 22 out of 215 | 30 out of 215 |
| Misread | 0 | 1 | 2 | 8 |
| Reading rate | 81.86% | 79.07% | 10.23% | 13.95% |
| Precision | 100% | 100% | 91.67% | 78.95% |
Sample image:
.jpg)
Performance Test Result on Muenster BarcodeDB
The dataset contains 1054 valid barcode images in real-life. It covers typical distortions, such as inhomogeneous illumination, reflections, or blurriness due to camera movement. Here is the detailed performance test result:
| SDK | Dynamsoft Barcode Reader v9.6.10 | Commercial SDK A v5.19 | ZXing-CPP v1.2.0 | pyZbar v0.1.8 |
|---|---|---|---|---|
| Correctly recognized | 1022 out of 1054 | 984 out of 1054 | 792 out of 1054 | 744 out of 1054 |
| Misread | 0 | 0 | 1 | 34 |
| Reading rate | 96.96% | 93.26% | 75.14% | 70.59% |
| Precision | 100% | 100% | 99.87% | 95.63% |
Sample image:
Performance Test Result on DEAL Lab’s Barcode Dataset
Images in the DEAL Lab’s barcode dataset are EAN-13 single barcodes and have various sizes and formats. It has 2000 images for testing.
| SDK | Dynamsoft Barcode Reader v9.6.10 | Commercial SDK A v5.19 | ZXing-CPP v1.2.0 | pyZbar v0.1.8 |
|---|---|---|---|---|
| Correctly recognized | 1839 out of 2000 | 1221 out of 2000 | 1264 out of 2000 | 1452 out of 2000 |
| Misread | 0 | 0 | 26 | 9 |
| Reading rate | 91.95% | 61.05% | 63.20% | 72.60% |
| Precision | 100% | 100% | 97.98% | 99.38% |
Sample image:
Conclusion
We can see that the Dynamsoft Barcode Reader has the highest reading rate on the three datasets. The Dynamsoft Barcode Reader can read more blurred barcodes. You can check out this video to learn more:
Improve Accuracy Even Further
Confidence Level
Dynamsoft allows developers to refine barcode results by adjusting the confidence level, which determines whether the result should be kept and discarded. For general scenarios, the value of confidence level should be set to 30 of 100, which leads to a 99.873% barcode recognition accuracy according to the test report from Dynamsoft’s QA team. The confidence level may be varied based on different datasets of barcode images.
In addition to the confidence level, the following techniques are also applied to guarantee the barcode recognition accuracy:
- API for comparing adjacent frames
- Dynamsoft Camera Enhancer for filtering out blurry frames
- OCR of the accompanying text for verification
Verify the Results Between Adjacent Frames
Since version 8.6 for mobile, Dynamsoft Barcode Reader added a new method enableResultVerification to verify the barcode result between adjacent frames before returning it. The code for this method is as follows:
reader.enableResultVerification(true)
// To check the status of this mode:
boolean x = reader.getEnableResultVerificationStatus();
This will filter out the misread barcode results efficiently by comparing with an internal frame queue.
Filter Out Blurry Frames
Dynamsoft Camera Enhancer is a module that features camera control, image processing, and image enhancement. It mainly works as an extension of Dynamsoft Barcode Reader to filter out blurry images on Android and iOS (the JavaScript edition is coming soon). The higher quality of images you get, the more accurate the barcode recognition will be.
OCR of the Accompanying Text
Dynamsoft Label Recognition is an OCR SDK developed based on deep learning. It trained a Caffe model for getting a high text recognition rate. The barcode orientation returned by barcode localization is utilized by the OCR SDK to precisely recognize the accompanying text around a barcode. The barcode text result is then verified by comparing the OCR result.
Conclusion
Choosing the right SDK ensures a fast go-to-market timeline, cutting down development and maintenance costs. High scanning accuracy also means happy end-users and overall better customer satisfaction.
Dynamsoft Barcode Reader is a robust and reliable barcode SDK. It brings high accuracy for single barcode and multi-code recognition. Accuracy is the competitive advantage of Dynamsoft Barcode Reader. With a world-grade barcode detection algorithm, OCR algorithm, and camera enhancing functionality, Dynamsoft Barcode SDK is dependable and productive for personal developers and enterprises.