1D Barcode Scanning Accuracy Benchmark and Comparison

Introduction

Accuracy is a key metric of barcode reading SDKs. Given an image, a barcode engine should correctly detect whether one or more barcodes exist. If barcodes exist, the percentage of correct values returned dictates the accuracy.  

As a software developer, choosing a barcode engine that has the best speed, accuracy, robustness, and customer support becomes a crucial step in the success of a barcode scanner application. In this article, we are going to discuss evaluation of barcode scanning accuracy and tips on achieving a higher accuracy.

This article is Part 2 in a 5-Part Series.

How to Evaluate Barcode Scanning Accuracy

There are two steps to read a barcode: 

  1. Detection (AKA localization) - Upon initiating a scan, a barcode engine scans the entire width of an image trying to identify whether there are barcode candidates. One or more coordinates of quadrilaterals are returned. 
  2. Decode (AKA recognition) - Once the regions of interest are identified, decoding is then initiated. In this step, the barcode scanner tries to unravel the encoded data.

Corresponding to the two steps, some datasets mark up the coordinates of the barcode candidates, while others mark up the barcode encoded values. Both approaches can be used to compare different barcode scanning engines.

This article uses the barcode values to measure barcode scanning accuracy. We will discuss the datasets in the following section.

Barcode Engines 

In this comprehensive analysis, we put compare Dynamsoft’s barcode SDK against another commercial barcode SDK and two popular open source solutions, Zbar and ZXing.

Datasets Used For Comparison

We have compiled a table of existing publicly available barcode datasets.

Dataset Artelab Medium Barcode 1D Collection Muenster BarcodeDB DEAL Lab’s Barcode Dataset
Symbologies EAN13 EAN13, UPC-A EAN13
Include images with multiple barcodes No No No
Number of images 430 1054 2000
Image quality Barcodes are relatively large compared to the image size Barcodes are relatively large compared to the image size Resolutions range from 400x217 to 3480x4640
Web page http://artelab.dista.uninsubria.it/downloads/datasets/barcode/medium_barcode_1d/medium_barcode_1d.html https://www.uni-muenster.de/PRIA/en/forschung/index.shtml https://www.resl.kaist.ac.kr/doc/datasets

Metrics to Evaluate Accuracy

In this article, we use precision and reading rate to evaluate the accuracy. 

The precision is the number of correct results divided by the number of all results. A precision of 100% means each of the results is correct, although some barcodes might be missed.

Precision = correct results / all returned results

The reading rate is the number of correct results divided by the number of all barcodes. A precision of 100% means that all the barcodes were recognized successfully, although there might be wrong values mixed in with the results. For example, if a barcode is mistakenly recognized as two separate ones, this behavior would not lower the reading rate.

Reading rate = correct results / all barcodes

Performance Test Results

We ran a performance test on the three datasets mentioned above. The annotations are examined and corrected manually to avoid errors.

Note: 

  1. If a barcode is recognized as UPC-A and the ground truth is in EAN13 format (the EAN13 format has one extra leading zero), it is considered as correct.
  2. Some images contain multiple barcodes while in the annotation, only one barcode is annotated. If the SDK reads the unannotated barcode, it is not considered as mis-read.

Performance Test Result on Artelab Medium Barcode 1D Collection

The Artelab Medium Barcode 1D Collection has two sets of images. Dataset 1 contains pictures of barcodes taken from devices with the autofocus (AF) function. Dataset 2 contains pictures taken by devices without AF. Each of the datasets has 215 images.

The test result on dataset 1:

SDK Dynamsoft Barcode Reader v9.6.10 Commercial SDK A v5.19 ZXing-CPP v1.2.0 pyZbar v0.1.8
Correctly recognized 215 out of 215 197 out of 215 177 out of 215 193 out of 215
Misread 0 0 1 1
Reading rate 100% 91.63% 82.36% 89.77%
Precision 100% 100% 99.44% 99.48%

Sample image:

sample image

The test result on out-of-focus images dataset 2:

SDK Dynamsoft Barcode Reader v9.6.10 Commercial SDK A v5.19 ZXing-CPP v1.2.0 pyZbar v0.1.8
Correctly recognized 176 out of 215 169 out of 215 22 out of 215 30 out of 215
Misread 0 1 2 8
Reading rate 81.86% 79.07% 10.23% 13.95%
Precision 100% 100% 91.67% 78.95%

Sample image:

sample image

Performance Test Result on Muenster BarcodeDB

The dataset contains 1054 valid barcode images in real-life. It covers typical distortions, such as inhomogeneous illumination, reflections, or blurriness due to camera movement. Here is the detailed performance test result:

SDK Dynamsoft Barcode Reader v9.6.10 Commercial SDK A v5.19 ZXing-CPP v1.2.0 pyZbar v0.1.8
Correctly recognized 1022 out of 1054 984 out of 1054 792 out of 1054 744 out of 1054
Misread 0 0 1 34
Reading rate 96.96% 93.26% 75.14% 70.59%
Precision 100% 100% 99.87% 95.63%

Sample image:

sample image

Performance Test Result on DEAL Lab’s Barcode Dataset

Images in the DEAL Lab’s barcode dataset are EAN-13 single barcodes and have various sizes and formats. It has 2000 images for testing.

SDK Dynamsoft Barcode Reader v9.6.10 Commercial SDK A v5.19 ZXing-CPP v1.2.0 pyZbar v0.1.8
Correctly recognized 1839 out of 2000 1221 out of 2000 1264 out of 2000 1452 out of 2000
Misread 0 0 26 9
Reading rate 91.95% 61.05% 63.20% 72.60%
Precision 100% 100% 97.98% 99.38%

Sample image:

sample image

Conclusion

We can see that the Dynamsoft Barcode Reader has the highest reading rate on the three datasets. The Dynamsoft Barcode Reader can read more blurred barcodes. You can check out this video to learn more:

Improve Accuracy Even Further

Confidence Level

Dynamsoft allows developers to refine barcode results by adjusting the confidence level, which determines whether the result should be kept and discarded. For general scenarios, the value of confidence level should be set to 30 of 100, which leads to a 99.873% barcode recognition accuracy according to the test report from Dynamsoft’s QA team. The confidence level may be varied based on different datasets of barcode images.

In addition to the confidence level, the following techniques are also applied to guarantee the barcode recognition accuracy:

  • API for comparing adjacent frames
  • Dynamsoft Camera Enhancer for filtering out blurry frames
  • OCR of the accompanying text for verification

Verify the Results Between Adjacent Frames

Since version 8.6 for mobile, Dynamsoft Barcode Reader added a new method enableResultVerification to verify the barcode result between adjacent frames before returning it. The code for this method is as follows:

reader.enableResultVerification(true)
// To check the status of this mode:
boolean x = reader.getEnableResultVerificationStatus();

This will filter out the misread barcode results efficiently by comparing with an internal frame queue.

Filter Out Blurry Frames

Dynamsoft Camera Enhancer is a module that features camera control, image processing, and image enhancement. It mainly works as an extension of Dynamsoft Barcode Reader to filter out blurry images on Android and iOS (the JavaScript edition is coming soon). The higher quality of images you get, the more accurate the barcode recognition will be.

OCR of the Accompanying Text

Dynamsoft Label Recognition is an OCR SDK developed based on deep learning. It trained a Caffe model for getting a high text recognition rate. The barcode orientation returned by barcode localization is utilized by the OCR SDK to precisely recognize the accompanying text around a barcode. The barcode text result is then verified by comparing the OCR result.

Conclusion

Choosing the right SDK ensures a fast go-to-market timeline, cutting down development and maintenance costs. High scanning accuracy also means happy end-users and overall better customer satisfaction.  

Dynamsoft Barcode Reader is a robust and reliable barcode SDK. It brings high accuracy for single barcode and multi-code recognition. Accuracy is the competitive advantage of Dynamsoft Barcode Reader. With a world-grade barcode detection algorithm, OCR algorithm, and camera enhancing functionality, Dynamsoft Barcode SDK is dependable and productive for personal developers and enterprises.