QR Code Reading Benchmark and Comparison

QR Code is widely used in our everyday life. As a two-dimensional barcode, it can store more data than 1D barcodes. It can be captured using a camera and then be decoded using image processing methods.

Reading QR codes in the real world is a challenging job. For example, the following QR code is damaged and wrinkled, which is difficult to read.

Damaged QR code

There are many open-source and commercial libraries or SDKs which can read QR codes. A benchmark is needed to evaluate which one is more robust or suitable for a specific use case.

There aren’t many QR codes reading benchmarks. A comprehensive one is made by the author of the open-source computer vision library Boofcv. The author collected a dataset of QR code images and put them into categories. A performance test is then made on the dataset to evaluate 5 open-source libraries.

The dataset has 536 images containing 1232 QR codes and has 16 categories: blurred, bright_spots, brightness, close, curved, damaged, glare, high_version, lots, monitor, nominal, noncompliant, pathological, perspective, rotations and shadows. In this article, we are going to do a QR code reading benchmark using this dataset.

Performance Test Tool

In order to run the benchmark, a performance test tool is written.

The tool uses Python to run the test and provides a web interface. The Flask web framework is used.

You can find its code here.

It is going to run on a PC device with an Intel i5-10400 CPU and 16GB memory.

Evaluated Libraries

We are going to evaluate 7 libraries and SDKs:

  1. Dynamsoft Barcode Reader (DBR, version: 8.6, commercial)
  2. Commercial SDK A (version: Java 13.5, commercial)
  3. Commercial SDK B (version: Windows 5.19.3.10, commercial)
  4. BoofCV (version: pyboof 0.36.1, open-source)
  5. Zxing (version: 3.4.1, open-source)
  6. Zbar (version: pyzbar 0.1.8, open-source)
  7. WeChat QR code detector in OpenCV (version: OpenCV 4.5.3, open-source)

Evaluation Metrics

The performance is evaluated by reading rate and runtime.

We can get the reading rate by dividing detected QR codes by total QR codes in images.

Reading rate = Detected QR codes / Total QR codes

How to infer that a QR code is detected? If the value of IoU (Intersection over Union) between the detected QR code’s polygon and the ground truth’s is over 0, then the QR code is considered detected.

Here is the simplified code:

detected_ground_truth = []
for ground_truth in ground_truth_list:
    detected = False
    for detected_result in detected_result_list:
        if iou<=0.0: #if not overlapped
            continue
        else:
            detected = True
            break
    if detected:
        detected_ground_truth.append(ground_truth)

The IoU is calculated using the following code:

from shapely.geometry import Polygon

def calculate_polygon_iou(points1,points2):
    poly1 = Polygon(points1)
    poly2 = Polygon(points2)
    # ratio of intersection area to union area
    iou = poly1.intersection(poly2).area/poly1.union(poly2).area
    return iou

# convert localization results of x1, y1...x4, y4 to a list of points
def points_of_one_detection_result(result):
    p = [(float(result["x1"]), float(result["y1"])),
        (float(result["x2"]), float(result["y2"])),
        (float(result["x3"]), float(result["y3"])),
        (float(result["x4"]), float(result["y4"]))]  
    return p

The QR code’s text result is not considered since the libraries all have 100% precision.

Note:

Zxing only returns the three points of the position patterns in a QR code. Postprocessing is done to convert the three points to a polygon. Since the IoU threshold is low, this works okay.

Zxing detection result

The runtime is calculated using the following code.

start_time = time.time()
result_dict = self.reader.decode_file(os.path.join(self.img_folder,filename))
end_time = time.time()
elapsedTime = int((end_time - start_time) * 1000)

Detection Results

Reading rate in percent:

Categories Dynamsoft Commercial SDK A Commercial SDK B Zxing Zbar BoofCV OpenCV Wechat
blurred 69.23 38.46 26.15 23.08 38.46 40.00 55.38
bright_spots 38.14 30.93 6.19 21.65 19.59 23.71 30.93
brightness 81.18 52.94 32.94 54.12 51.76 78.82 68.24
close 92.50 25.00 35.00 5.00 12.50 77.50 65.00
curved 73.33 38.33 33.33 30.00 35.00 50.00 43.33
damaged * 53.49 27.91 23.26 20.93 25.58 16.28 48.84
glare 86.79 20.75 22.64 20.75 35.85 32.08 64.15
high_version * 97.30 35.14 59.46 5.41 27.03 40.54 18.92
lots * 60.48 97.86 1.90 82.86 18.10 99.76 0.00
monitor 100 5.88 5.88 0.00 0.00 64.71 94.12
nominal 91.03 62.82 51.28 51.28 66.67 89.74 80.77
noncompliant * 88.46 3.85 15.38 19.23 50.00 3.85 92.31
pathological * 100 78.26 78.26 34.78 65.22 43.48 91.30
perspective 68.57 34.29 51.43 37.14 42.86 82.86 42.86
rotations 99.25 67.67 63.91 42.86 49.62 96.99 82.71
shadows 100 95.00 80.00 65.00 90.00 85.00 85.00
total average 81.23 44.69 36.69 32.13 39.27 57.83 60.24
  1. Damaged QR codes are the ones with scratches, dents, etc.
  2. Pathological QR codes have the markers intentionally corrupted. The markers are designed for detection of QR Codes.
  3. The lots category has multiple QR Codes in one image.
  4. The QR Code symbology ranges from Version 1 to Version 40. The Version Information represents the number of cells (per side) making up the code. More on the QR Code versions.
  5. Non-compliant QR Codes have customized designs, for example, colors, or a logo at the center.

Reading rate Chart

Reading rate Chart

Dynamsoft Barcode Reader Wins by a Large Margin

We can see that the Dynamsoft Barcode Reader ranks 1st in most of the categories. In some categories, for example, the blurred, curved, glare, and high version, the SDK is the only solution that has a rate higher than 67%.

Blurred QR Codes Curved QR Codes QR Codes with Glare High-version QR Codes

It ranks 2nd in the noncompliant and perspective categories and 4th in the lots category.

Note: The reading rates of some categories are not very high, because the QR codes are in bad conditions and not readable.

ZXing vs. ZBar vs. BoofCV vs. OpenCV WeChat

If open source is a must for your project, it’s a safer choice to go with BoofCV or OpenCV WeChat. These two are based on computer vision and are actively maintained. This could be why they outperform ZXing and ZBar in some challenging situations.

  • OpenCV WeChat is robust. It has the highest success rate for damaged and pathological QR Codes.
  • OpenCV WeChat also excels for non-compliant QR Codes, which is not surprising, as QR codes with colors, or a logo at the center, or other customized designs are ubiquitous in WeChat usage.
  • If we cast aside the noncompliant and pathological codes, then BoofCV is a good free solution, especially when it comes to the “lots” category.

Runtime Results

Runtime per image (in milliseconds):

Engine Result
DBR 141.10
Commercial SDK A 314.04
Commercial SDK B 48.31
Zxing 181.68
Zbar 199.26
BoofCV 205.48
OpenCV Wechat 175.06

Times per Image Chart

Runtime per image by categories (in milliseconds):

Categories Dynamsoft Commercial SDK A Commercial SDK B Zxing Zbar BoofCV OpenCV Wechat
blurred 298.00 335.16 39.67 206.89 177.58 138.16 125.31
bright_spots 252.84 625.47 47.56 311.59 319.81 221.81 112.25
brightness 192.75 600.14 47.86 320.32 349.79 273.43 134.79
close 169.40 607.27 80.08 295.18 381.65 150.70 676.00
curved 188.76 300.82 43.40 173.44 220.28 103.42 129.54
damaged 200.03 205.49 42.05 84.41 96.76 46.14 138.35
glare 104.66 217.60 45.40 130.32 157.30 70.80 95.38
high_version 106.06 349.21 79.88 294.97 141.70 268.36 566.03
lots 275.43 851.14 110.14 1073.00 1477.57 6149.00 139.57
monitor 176.59 812.53 105.88 233.29 515.47 142.47 422.76
nominal 74.05 203.65 32.12 141.49 111.68 109.49 54.54
noncompliant 58.88 81.63 34.44 47.81 57.06 31.94 43.31
pathological 8.52 4.09 65.87 3.00 1.83 14.61 19.39
perspective 25.74 46.34 14.91 27.66 24.31 26.74 24.60
rotations 72.11 173.84 35.80 106.84 134.66 178.45 86.91
shadows 102.21 243.50 42.29 150.00 163.29 117.50 86.36
total average 144.13 353.62 54.21 225.01 270.67 502.69 178.44

Because Boofcv spent a lot of time on decoding the lots category, we separated it from the main chart.

Times Chart 1

Times Chart 2

Times Chart 3

Although the commercial SDK B has low reading rate on the dataset, it is the fastest. The Dynamsoft Barcode Reader ranks second on average.

Update Runtime Settings to Improve the Results

The benchmark is done without updating the SDKs’ default settings. We can update the runtime settings to improve the reading accuracy or speed.

Among the SDKs, only the Dynamsoft Barcode Reader provides rich parameters that users can customize and optimize for different usage scenarios for the best scanning performance.

The runtime settings of DBR can be represented and modified using a JSON template. The following is the default template used in DBR Python.

{
   "ImageParameter" : {
      "BarcodeColourModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "LightReflection" : 1,
            "Mode" : "BICM_DARK_ON_LIGHT"
         }
      ],
      "BarcodeComplementModes" : [
         {
            "Mode" : "BCM_SKIP"
         }
      ],
      "BarcodeFormatIds" : [ "BF_ALL" ],
      "BarcodeFormatIds_2" : null,
      "BinarizationModes" : [
         {
            "BlockSizeX" : 0,
            "BlockSizeY" : 0,
            "EnableFillBinaryVacancy" : 1,
            "ImagePreprocessingModesIndex" : -1,
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "BM_LOCAL_BLOCK",
            "ThresholdCompensation" : 10
         }
      ],
      "ColourClusteringModes" : [
         {
            "Mode" : "CCM_SKIP"
         }
      ],
      "ColourConversionModes" : [
         {
            "BlueChannelWeight" : -1,
            "GreenChannelWeight" : -1,
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "CICM_GENERAL",
            "RedChannelWeight" : -1
         }
      ],
      "DPMCodeReadingModes" : [
         {
            "Mode" : "DPMCRM_SKIP"
         }
      ],
      "DeblurLevel" : 9,
      "DeblurModes" : null,
      "DeformationResistingModes" : [
         {
            "Mode" : "DRM_SKIP"
         }
      ],
      "Description" : "",
      "ExpectedBarcodesCount" : 0,
      "FormatSpecificationNameArray" : null,
      "GrayscaleTransformationModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "GTM_ORIGINAL"
         }
      ],
      "ImagePreprocessingModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "IPM_GENERAL"
         }
      ],
      "IntermediateResultSavingMode" : {
         "Mode" : "IRSM_MEMORY"
      },
      "IntermediateResultTypes" : [ "IRT_NO_RESULT" ],
      "LocalizationModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "LM_CONNECTED_BLOCKS"
         },
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "LM_SCAN_DIRECTLY",
            "ScanDirection" : 0,
            "ScanStride" : 0
         },
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "LM_STATISTICS"
         },
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "LM_LINES"
         }
      ],
      "MaxAlgorithmThreadCount" : 4,
      "Name" : "CurrentRuntimeSettings",
      "PDFRasterDPI" : 300,
      "PDFReadingMode" : {
         "Mode" : "PDFRM_AUTO"
      },
      "Pages" : "",
      "RegionDefinitionNameArray" : null,
      "RegionPredetectionModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "RPM_GENERAL"
         }
      ],
      "ResultCoordinateType" : "RCT_PIXEL",
      "ReturnBarcodeZoneClarity" : 0,
      "ScaleDownThreshold" : 2300,
      "ScaleUpModes" : [
         {
            "Mode" : "SUM_AUTO"
         }
      ],
      "TerminatePhase" : "TP_BARCODE_RECOGNIZED",
      "TextAssistedCorrectionMode" : {
         "BottomTextPercentageSize" : 0,
         "LeftTextPercentageSize" : 0,
         "LibraryFileName" : "",
         "LibraryParameters" : "",
         "Mode" : "TACM_VERIFYING",
         "RightTextPercentageSize" : 0,
         "TopTextPercentageSize" : 0
      },
      "TextFilterModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "MinImageDimension" : 65536,
            "Mode" : "TFM_GENERAL_CONTOUR",
            "Sensitivity" : 0
         }
      ],
      "TextResultOrderModes" : [
         {
            "Mode" : "TROM_CONFIDENCE"
         },
         {
            "Mode" : "TROM_POSITION"
         },
         {
            "Mode" : "TROM_FORMAT"
         }
      ],
      "TextureDetectionModes" : [
         {
            "LibraryFileName" : "",
            "LibraryParameters" : "",
            "Mode" : "TDM_GENERAL_WIDTH_CONCENTRATION",
            "Sensitivity" : 5
         }
      ],
      "Timeout" : 10000
   },
   "Version" : "3.0"
}

You can learn more about the parameters of DBR here.

Modify the Template to Decode the Lots Category

In the benchmark, there are several files in the lots category that DBR cannot detect all the QR codes.

The reason is that the QR codes are relatively small and the image has a large resolution (3024x4032) while in DBR’s template, there is a ScaleDownThreshold parameter which is set to 2300 so that the image will be scaled down for a better decoding speed.

If we set the ScaleDownThreshold to 99999, we can decode all the QR codes.

Lots category

The reading rate for the lots category can increase from 60.48 to 100 using the modified template.

Modify the Template to Decode the Blurred Category

DBR comes with many image processing algorithms.

For blurred QR codes as below, we can use the GRAY_SMOOTH image processing method before deblurring to decode it.

"ImagePreprocessingModes" : [
    {
        "LibraryFileName": "",
        "LibraryParameters": "",
        "Mode": "IPM_GRAY_SMOOTH",
        "SmoothBlockSizeX": 3,
        "SmoothBlockSizeY": 3
    }
]

Blurred category

But if the QR codes are blurred too much, we cannot do much about this. DBR can only decode one extra image after using the modified template in the blurred category. We should try to capture clear QR code images in the first place. SDKs like the Dynamsoft Camera Enhancer can help on this aspect.

Conclusion

The Dynamsoft Barcode Reader has the best QR code reading accuracy on the test set. Its speed is good and has rich runtime settings to adapt to different scenarios. If conditions allow, it is better to choose DBR as the barcode reading SDK.

Download Dynamsoft Barcode Reader