Accelerating Android QR Code Detection with TensorFlow Lite

I conducted an experiment to evaluate QR code detection performance using YOLOv4 on Android. The results showed that running the YOLOv4 model on the CPU took over 100ms per inference, which fell short of my expectations. To improve inference speed, I decided to leverage the Android Neural Networks API (NNAPI) for hardware acceleration by training a TensorFlow Lite model.

This article is Part 6 in a 7-Part Series.

Steps to Train a QR Code Detection Model with TensorFlow Lite Model Maker

  1. Install TensorFlow Lite Model Maker:

     pip install tflite-model-maker

  2. Clone the GitHub repository containing the image dataset:

     git clone https://github.com/yushulx/barcode-qrcode-images.git

  3. Partition the image dataset into training and validation sets:

     cd tensorflow-lite
 python partition_dataset.py -x -i ../images -r 0.1 -o ./

  4. Follow the TensorFlow tutorial to create the following Python script for training the model:

     from tflite_model_maker.config import QuantizationConfig
 from tflite_model_maker.config import ExportFormat
 from tflite_model_maker import model_spec
 from tflite_model_maker import object_detector

 import tensorflow as tf
 assert tf.__version__.startswith('2')

 tf.get_logger().setLevel('ERROR')
 from absl import logging
 logging.set_verbosity(logging.ERROR)

 spec = model_spec.get('efficientdet_lite0')
 train_data = object_detector.DataLoader.from_pascal_voc(images_dir="train", annotations_dir="train", label_map={1: "QR_CODE"} )
 validation_data = object_detector.DataLoader.from_pascal_voc(images_dir="test", annotations_dir="test", label_map={1: "QR_CODE"} )
 model = object_detector.create(train_data, model_spec=spec, batch_size=8, train_whole_model=True, validation_data=validation_data)
 model.export(export_dir='.')

    EfficientDet-D0 offers comparable accuracy to YOLOv3 with less computational cost.

    about EfficientDet

Once the model is trained and saved, you can download the TensorFlow Lite example and replace the default model file with your custom-trained model.

Integrating the Model into the Android Project

  1. Place the model.tflite file into the assets folder.

  2. Update the TF_OD_API_MODEL_FILE path in DetectorActivity.java:

     private static final String TF_OD_API_MODEL_FILE = "model.tflite";

  3. Clear the contents of labelmap.txt and add the following:

     QR_CODE

Your QR code detection model is now ready to be tested on an Android device.

TensorFlow lite QR code detection

Building a QR Code Scanner with TensorFlow Lite

Now, let’s integrate TensorFlow Lite into our Android QR Code Scanner:

  1. Copy model.tflite and labelmap.txt to the assets folder.

  2. Add the TensorFlow Lite dependency to build.gradle:

     // Import the Task Vision Library dependency (NNAPI is included)
 implementation 'org.tensorflow:tensorflow-lite-task-vision:0.3.0'

  3. Copy Detector.java and TFLiteObjectDetectionAPIModel.java from the TensorFlow Lite example project to your project. Use the NNAPI delegate for TensorFlow Lite:

     private TFLiteObjectDetectionAPIModel(Context context, String modelFilename) throws IOException {
   modelBuffer = FileUtil.loadMappedFile(context, modelFilename);
   optionsBuilder = ObjectDetectorOptions.builder();
   optionsBuilder.setBaseOptions(BaseOptions.builder().useNnapi().build());
   objectDetector = ObjectDetector.createFromBufferAndOptions(modelBuffer, optionsBuilder.build());
 }

  4. Initialize the model detector in CameraXActivity.java:

     // Configuration values for the prepackaged QR Code model.
 private static final int TF_OD_API_INPUT_SIZE = 416;
 private static final boolean TF_OD_API_IS_QUANTIZED = true;
 private static final String TF_OD_API_MODEL_FILE = "model.tflite";
 private static final String TF_OD_API_LABELS_FILE = "labelmap.txt";
 // Minimum detection confidence to track a detection.
 private static final float MINIMUM_CONFIDENCE_TF_OD_API = 0.5f;
 private Detector detector;
 int cropSize = TF_OD_API_INPUT_SIZE;

 private void initTFDetector() {
     try {
         detector =
                 TFLiteObjectDetectionAPIModel.create(
                         this,
                         TF_OD_API_MODEL_FILE,
                         TF_OD_API_LABELS_FILE,
                         TF_OD_API_INPUT_SIZE,
                         TF_OD_API_IS_QUANTIZED);
         cropSize = TF_OD_API_INPUT_SIZE;
     } catch (final IOException e) {
         e.printStackTrace();
         Toast toast =
                 Toast.makeText(
                         getApplicationContext(), "Detector could not be initialized", Toast.LENGTH_SHORT);
         toast.show();
         finish();
     }
 }

  5. In the camera frame callback function, convert ImageProxy to Bitmap:

     analysisUseCase.setAnalyzer(cameraExecutor,
                                 imageProxy -> {
     Bitmap bitmap = ImageUtils.getBitmap(imageProxy);
     imageProxy.close();
 });

    Note: The bitmap has already been rotated, so there’s no need to rotate the QR code bounding box coordinates for rendering overlays.

    To understand the implementation of getBitmap(), please refer to BitmapUtils.java.

  6. Detect QR codes and draw bounding boxes:

     final List<Detector.Recognition> tfResults = detector.recognizeImage(bitmap);
 if (tfResults.size() > 0) {
     for (final Detector.Recognition tfResult : tfResults) {
         final RectF location = tfResult.getLocation();
         if (location != null && tfResult.getConfidence() >= MINIMUM_CONFIDENCE_TF_OD_API) {
             overlay.add(new BarcodeGraphic(overlay, location, null, isPortrait));
         }
     }
 }

  7. Decode QR codes using the Dynamsoft Barcode Reader API:

     TextResult[] results = null;
 try {
     PublicRuntimeSettings settings = reader.getRuntimeSettings();
     settings.barcodeFormatIds = EnumBarcodeFormat.BF_QR_CODE;
     settings.expectedBarcodesCount = tfResults.size();
     reader.updateRuntimeSettings(settings);
 } catch (BarcodeReaderException e) {
     e.printStackTrace();
 }

  8. Finally, test the QR code detection model on your Android device.

    TensorFlow Lite Android QR code scanner

    Running the TensorFlow Lite model for QR code detection is faster than YOLOv4, but still slower compared to using Dynamsoft Barcode Reader alone.

Source Code

https://github.com/yushulx/android-camera-barcode-mrz-document-scanner/tree/main/examples/9.x/camerax_ml_qr