QR Code Reading Benchmark and Comparison

Sep 06, 2021

 Barcode

•  BARCODE READER
•  QR CODE
•  BENCHMARK

QR Code is widely used in our everyday life. As a two-dimensional barcode, it can store more data than 1D barcodes. It can be captured using a camera and then be decoded using image processing methods.

Reading QR codes in the real world is a challenging job. For example, the following QR code is damaged and wrinkled, which is difficult to read.

There are many open-source and commercial libraries or SDKs which can read QR codes. A benchmark is needed to evaluate which one is more robust or suitable for a specific use case.

There aren’t many QR codes reading benchmarks. A comprehensive one is made by the author of the open-source computer vision library Boofcv. The author collected a dataset of QR code images and put them into categories. A performance test is then made on the dataset to evaluate 5 open-source libraries.

The dataset has 536 images containing 1232 QR codes and has 16 categories: blurred, bright_spots, brightness, close, curved, damaged, glare, high_version, lots, monitor, nominal, noncompliant, pathological, perspective, rotations and shadows. In this article, we are going to do a QR code reading benchmark using this dataset.

This article is Part 1 in a 5-Part Series.

• Part 1 - QR Code Reading Benchmark and Comparison
• Part 2 - 1D Barcode Scanning Accuracy Benchmark and Comparison
• Part 3 - What are the Best Data Matrix Reading SDKs?
• Part 4 - What are the Best PDF417 Reading SDKs?
• Part 5 - Multiple Barcodes Reading Benchmark and Comparison

Video:

Note: The article has been updated on 06/26/2023. The results in the video may have some slight differences.

Performance Test Tool

In order to run the benchmark, a performance test tool is written.

You can find its code here.

It is going to run on a PC device with an Intel i5-10400 CPU and 16GB memory.

Evaluated Libraries

We are going to evaluate 7 libraries and SDKs:

1. Dynamsoft Barcode Reader (version: 9.6.20, commercial)
2. Commercial SDK A (version: 6.16, commercial)
3. Commercial SDK B (version: 13.9, commercial)
4. BoofCV (version: 0.43.1, open-source)
5. ZXing (version: 3.5.1, open-source)
6. ZBar (version: pyzbar 0.1.9, open-source)
7. WeChat QR code detector in OpenCV (version: OpenCV 4.7.0.72, open-source)

If the SDK supports multiple barcode formats, its runtime settings are set to decode QR codes only.

Evaluation Metrics

The performance is evaluated by reading rate and runtime.

We can get the reading rate by dividing detected QR codes by total QR codes in images.

Reading rate = Detected QR codes / Total QR codes

How to infer that a QR code is detected? If there is a ground truth QR code which overlaps with the detected QR code, then it is considered detected.

Here is the simplified code for doing this:

function isDetected(barcodeResult){
  for (let j = 0; j < groundTruthList.length; j++) {
    const groundTruth = groundTruthList[j];
    let percent;
    const points1 = getPointsFromBarcodeResultResult(barcodeResult);
    const points2 = getPointsFromGroundTruth(groundTruth);
    percent = overlappingPercent(points1,points2);
    if (percent > 0.20) {
      return true;
    }
  }
  return false;
}

The overlapping percent is calculated with the following code:

export function overlappingPercent(pts1:Point[] ,pts2:Point[]) : number {
  const rect1 = getRectFromPoints(pts1);
  const rect2 = getRectFromPoints(pts2);

  let leftRect;
  let rightRect;
  if (rect1.left<rect2.left) {
    leftRect = rect1;
    rightRect = rect2;
  }else{
    leftRect = rect2;
    rightRect = rect1;
  }
  let upperRect;
  let lowerRect;
  if (rect1.top<rect2.top) {
    upperRect = rect1;
    lowerRect = rect2;
  }else{
    upperRect = rect2;
    lowerRect = rect1;
  }
  if (leftRect.right > rightRect.left && upperRect.bottom>lowerRect.top) {
    const overlappedX = Math.min(leftRect.right,rightRect.right) - rightRect.left;
    const overlappedY = Math.min(upperRect.bottom,lowerRect.bottom) - lowerRect.top;
    const overlappedArea = overlappedX * overlappedY;
    const area1 = rect1.width * rect1.height;
    const area2 = rect2.width * rect2.height;
    const smallerArea = Math.min(area1,area2);
    return overlappedArea/smallerArea;
  }else{
    return 0;
  }
}

The QR code’s text result is not considered since the libraries all have 100% precision.

Note:

ZXing only returns the three points of the position patterns in a QR code. Postprocessing is done to convert the three points to a polygon.

The runtime is calculated using the following code in a Python backend.

start_time = time.time()
results = self.reader.decode_bytes(file_bytes)
end_time = time.time()
elapsedTime = int((end_time - start_time) * 1000)

Detection Results

Reading rate in percent in categories:

Category	Dynamsoft	BoofCV	ZXing	ZBar	OpenCV WeChat	Commercial SDK A	Commercial SDK B
blurred	66.15	38.46	21.54	35.38	46.15	33.85	36.92
brightness	81.18	78.82	52.94	50.59	23.53	52.94	51.76
bright_spots	43.3	27.84	20.62	19.59	24.74	8.25	29.9
close	95	100	5	12.5	67.5	22.5	25
curved	70	56.67	31.67	35	43.33	31.67	36.67
damaged *	51.16	16.28	20.93	25.58	30.23	27.91	27.91
glare	84.91	32.08	20.75	35.85	58.49	43.4	20.75
high_version *	97.3	40.54	5.41	27.03	18.92	78.38	35.14
lots *	100	99.76	82.86	18.1	0	14.52	97.14
monitor	100	82.35	0	0	94.12	11.76	5.88
nominal	93.59	89.74	53.85	66.67	71.79	64.1	65.38
noncompliant *	92.31	3.85	15.38	50	76.92	61.54	11.54
pathological *	95.65	43.48	34.78	65.22	91.3	0	78.26
perspective	62.86	80	37.14	42.86	42.86	65.71	34.29
rotations	99.25	96.24	42.11	48.87	32.33	99.25	69.17
shadows	100	85	65	90	60	90	95
total average	83.29	60.69	31.87	38.95	48.89	44.11	45.04

1. Damaged QR codes are the ones with scratches, dents, etc.
2. Pathological QR codes have the markers intentionally corrupted. The markers are designed for the detection of QR Codes.
3. The lots category has multiple QR Codes in one image.
4. The QR Code symbology ranges from Version 1 to Version 40. The Version Information represents the number of cells (per side) making up the code. More on the QR Code versions.
5. Non-compliant QR Codes have customized designs, for example, colors, or a logo at the center.

Dynamsoft Barcode Reader Wins by a Large Margin

We can see that the Dynamsoft Barcode Reader ranks 1st in most of the categories. In some categories, for example, the blurred, curved, glare, and high version, the SDK is the only solution that has a rate higher than 67%.

It ranks 2nd in the close categories and 3rd in the perspective category.

Note: The reading rates of some categories are not very high, because the QR codes are in bad condition and not readable.

Download 30-Day Free Trial SDK

ZXing vs. ZBar vs. BoofCV vs. OpenCV WeChat

If open source is a must for your project, it’s a safer choice to go with BoofCV or OpenCV WeChat. These two are based on computer vision and are actively maintained. This could be why they outperform ZXing and ZBar in some challenging situations.

• OpenCV WeChat is robust. It has the highest success rate for damaged and pathological QR Codes.
• OpenCV WeChat also excels for non-compliant QR Codes, which is not surprising, as QR codes with colors, or a logo at the center, or other customized designs are ubiquitous in WeChat usage.
• If we cast aside the noncompliant and pathological codes, then BoofCV is a good free solution, especially when it comes to the “lots” category.

Runtime Results

Runtime per image (in milliseconds):

Engine	Result
Dynamsoft	195.01
Commercial SDK A	1400.43
Commercial SDK B	346.27
ZXing	179.57
ZBar	157.31
BoofCV	104.95
OpenCV Wechat	757.71

Conclusion

We can see that Dynamsoft Barcode Reader has the best QR code reading rate on the test set and its speed is fairly good. If conditions allow, it is better to choose it as the barcode reading SDK.

You can find the detailed benchmark results here.