Flutter OCR Recognition for Passport MRZ

Last week, I built a simple Android application with Dynamsoft camera SDK and OCR SDK to recognize passport MRZ. To facilitate Flutter development, I determined to make a Flutter plugin for wrapping Dynamsoft OCR SDK this week. Since there has been an official Flutter camera plugin available for download on pub.dev, I don’t need to put effort into the camera part.

Flutter Plugin for Dynamsoft OCR SDK

As always, the first step is to create the plugin project by running the following command:

flutter create --org com.dynamsoft --template=plugin --platforms=android -a java flutter_ocr_sdk

After that, we add the Dynamsoft maven repository and configure the OCR library in android/build.gradle:

rootProject.allprojects {
    repositories {
        maven {
            url "http://download2.dynamsoft.com/maven/dlr/aar"
        }
        google()
        jcenter()
    }
}

dependencies {
    implementation "com.dynamsoft:dynamsoftlabelrecognition:1.2.1@aar"
}

Then, we go to android/src/main/java/com/dynamsoft/flutter_ocr_sdk/FlutterOcrSdkPlugin.java and find the entry point onMethodCall(). We will process five method calls, including setOrganizationID, recognizeByFile, recognizeByBuffer, loadModelFiles, and loadTemplate:

  • setOrganizationID(): register a Dynamsoft account to get the organization ID in order to get authorization from Dynamsoft server.

      public void setOrganizationID(String id) {
          DMLTSConnectionParameters ltspar = new DMLTSConnectionParameters();
          ltspar.organizationID = id;
          mLabelRecognition.initLicenseFromLTS(ltspar, new DLRLTSLicenseVerificationListener() {
              @Override
              public void LTSLicenseVerificationCallback(boolean b, Exception e) {
                  if (e != null) {
                      Log.e("lts error: ", e.getMessage());
                  }
              }
          });
      }
    
  • recognizeByFile(): do MRZ OCR recognition by an image file.

      public String recognizeByFile(String fileName, String templateName) {
          JSONObject ret = new JSONObject();
          DLRResult[] results = null;
          try {
              results = mLabelRecognition.recognizeByFile(fileName, templateName);
              ret = wrapResults(results);
          } catch (LabelRecognitionException e) {
              Log.e(TAG, e.toString());
          }
          return ret.toString();
      }
    
  • recognizeByBuffer(): do MRZ OCR recognition by a byte array of image data.

      public String recognizeByBuffer(byte[] bytes, int width, int height, int stride, int format, String templateName) {
          JSONObject ret = new JSONObject();
          DLRResult[] results = null;
          DLRImageData data = new DLRImageData();
          data.bytes = bytes;
          data.width = width;
          data.height = height;
          data.stride = stride;
          data.format = format;
          try {
              results = mLabelRecognition.recognizeByBuffer(data, templateName);
              ret = wrapResults(results);
          } catch (LabelRecognitionException e) {
              Log.e(TAG, e.toString());
          }
          return ret.toString();
      }
    
  • loadModelFiles(): load the character model files trained based on deep learning.

      public void loadModelFiles(String name, byte[] prototxtBuffer, byte[] txtBuffer, byte[] characterModelBuffer) {
          try {
              mLabelRecognition.appendCharacterModelBuffer(name, prototxtBuffer, txtBuffer, characterModelBuffer);
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
    
  • loadTemplate(): load the template file which contains specific parameters for MRZ recognition.

      public void loadTemplate(String content) {
          try {
              mLabelRecognition.appendSettingsFromString(content);
          } catch (Exception e) {
              e.printStackTrace();
          }
      }
    

The final step is to define the corresponding Flutter methods in lib/flutter_ocr_sdk.dart.

Future<void> setOrganizationID(String id) async {
    await _channel.invokeMethod('setOrganizationID', {'id': id});
}

Future<String> recognizeByFile(String filename, String template) async {
    return await _channel.invokeMethod('recognizeByFile', {
      'filename': filename,
      'template': template,
    });
}

Future<String> recognizeByBuffer(Uint8List bytes, int width, int height,
      int stride, int format, String template) async {
    return await _channel.invokeMethod('recognizeByBuffer', {
      'bytes': bytes,
      'width': width,
      'height': height,
      'stride': stride,
      'format': format,
      'template': template,
    });
}

Future<void> loadModelFiles(String name, Uint8List prototxtBuffer,
      Uint8List txtBuffer, Uint8List characterModelBuffer) async {
    return await _channel.invokeMethod('loadModelFiles', {
      'name': name,
      'prototxtBuffer': prototxtBuffer,
      'txtBuffer': txtBuffer,
      'characterModelBuffer': characterModelBuffer,
    });
}

Future<void> loadTemplate(String template) async {
    return await _channel.invokeMethod('loadTemplate', {
      'template': template,
    });
}

According to the passport MRZ standard, we create a parser to extract passport information from the MRZ string:

static String parse(String line1, String line2) {
    // https://en.wikipedia.org/wiki/Machine-readable_passport
    String result = "";
    // Type
    String tmp = "Type: ";
    tmp += line1[0];
    result += tmp + '\n\n';

    // Issuing country
    tmp = "Issuing country: ";
    tmp += line1.substring(2, 5);
    result += tmp + '\n\n';

    // Surname
    int index = 5;
    tmp = "Surname: ";
    for (; index < 44; index++) {
      if (line1[index] != '<') {
        tmp += line1[index];
      } else {
        break;
      }
    }
    result += tmp + '\n\n';

    // Given names
    tmp = "Given Names: ";
    index += 2;
    for (; index < 44; index++) {
      if (line1[index] != '<') {
        tmp += line1[index];
      } else {
        tmp += ' ';
      }
    }
    result += tmp + '\n\n';

    // Passport number
    tmp = "Passport number: ";
    index = 0;
    for (; index < 9; index++) {
      if (line2[index] != '<') {
        tmp += line2[index];
      } else {
        break;
      }
    }
    result += tmp + '\n\n';

    // Nationality
    tmp = "Nationality: ";
    tmp += line2.substring(10, 13);
    result += tmp + '\n\n';

    // Date of birth
    tmp = line2.substring(13, 19);
    tmp = tmp.substring(0, 2) +
        '/' +
        tmp.substring(2, 4) +
        '/' +
        tmp.substring(4, 6);
    tmp = "Date of birth (YYMMDD): " + tmp;
    result += tmp + '\n\n';

    // Sex
    tmp = "Sex: ";
    tmp += line2[20];
    result += tmp + '\n\n';

    // Expiration date of passport
    tmp = line2.substring(21, 27);
    tmp = tmp.substring(0, 2) +
        '/' +
        tmp.substring(2, 4) +
        '/' +
        tmp.substring(4, 6);
    tmp = "Expiration date of passport (YYMMDD): " + tmp;
    result += tmp + '\n\n';

    // Personal number
    if (line2[28] != '<') {
      tmp = "Personal number: ";
      for (index = 28; index < 42; index++) {
        if (line2[index] != '<') {
          tmp += line2[index];
        } else {
          break;
        }
      }
      result += tmp + '\n\n';
    }

    return result;
}

How to load assets in Flutter?

To load model and template files in Flutter project, we use AssetBundle:

Future<String> loadAssetString(String path) async {
    return await rootBundle.loadString(path);
}

Future<ByteData> loadAssetBytes(String path) async {
return await rootBundle.load(path);
}

In addition, the files must be declared in pubspec.yaml.

Building a Flutter App to Recognize Passport MRZ

First, we download the model package and extract files to a model folder.

In example/pubspec.yaml, we add camera plugin and make model files accessible:

dependencies:
    camera:

flutter:
  assets:
    - model/
    - model/CharacterModel/

Now, let’s take a few steps to implement passport MRZ recognition in example/main.dart:

  1. Initialize the camera plugin and OCR plugin:

     final cameras = await availableCameras();
     final firstCamera = cameras.first;
    
     class MobileState extends State<Mobile> {
       CameraController _controller;
       Future<void> _initializeControllerFuture;
       FlutterOcrSdk _textRecognizer;
       final picker = ImagePicker();
        
       @override
       void initState() {
         super.initState();
    
         _controller = CameraController(
           widget.camera,
           ResolutionPreset.ultraHigh,
         );
        
         _initializeControllerFuture = _controller.initialize();
         _initializeControllerFuture.then((_) {
           setState(() {});
         });
    
         initBarcodeSDK();
       }
    
       Future<void> initBarcodeSDK() async {
         _textRecognizer = FlutterOcrSdk();
         _textRecognizer.loadModel('model/');
       }
     }
    
  2. Construct the UI layout with a camera view and a floating action button:

     @override
     Widget build(BuildContext context) {
         double width = MediaQuery.of(context).size.width;
         double height = MediaQuery.of(context).size.height;
         double left = 5;
         double mrzHeight = 50;
         double mrzWidth = width - left * 2;
         return Scaffold(
             body: Stack(children: [
                 getCameraWidget(),
                 Positioned(
                 left: left,
                 top: height - mrzHeight * 4,
                 child: Container(
                     width: mrzWidth,
                     height: mrzHeight,
                     decoration: BoxDecoration(
                     border: Border.all(
                         width: 2,
                         color: Colors.blue,
                     ),
                     ),
                 ),
                 )
             ]),
             floatingActionButton: FloatingActionButton(
                 child: Icon(Icons.camera),
                 onPressed: () async {
                 showDialog(
                     context: context,
                     builder: (BuildContext context) {
                         return Center(
                         child: CircularProgressIndicator(),
                         );
                     });
                 pictureScan();
                 },
             ),
         );
     }
    
  3. Take a picture to recognize the MRZ and show results on a new page:

     void pictureScan() async {
         final image = await _controller.takePicture();
         if (image == null) {
           Navigator.pop(context);
           return;
         }
         String ret = await _textRecognizer.recognizeByFile(image?.path, 'locr');
         String results = getTextResults(ret);
         Navigator.pop(context);
         Navigator.push(
           context,
           MaterialPageRoute(
             builder: (context) => DisplayPictureScreen(
                 imagePath: image?.path, barcodeResults: results),
           ),
         );
     }
        
     class DisplayPictureScreen extends StatelessWidget {
       final String imagePath;
       final String barcodeResults;
        
       const DisplayPictureScreen({Key key, this.imagePath, this.barcodeResults})
           : super(key: key);
        
       @override
       Widget build(BuildContext context) {
         return Scaffold(
           appBar: AppBar(title: Text('OCR')),
           body: Stack(
             alignment: const Alignment(0.0, 0.0),
             children: [
               Image.file(
                 File(imagePath),
                 fit: BoxFit.cover,
                 height: double.infinity,
                 width: double.infinity,
                 alignment: Alignment.center,
               ),
               Container(
                 decoration: BoxDecoration(
                   color: Colors.black45,
                 ),
                 child: Text(
                   barcodeResults,
                   style: TextStyle(
                     fontSize: 14,
                     color: Colors.white,
                   ),
                 ),
               ),
             ],
           ),
         );
       }
     }
    
  4. Run the passport MRZ recognition app:

     cd example
     flutter run
    

    Flutter OCR recognition for passport mrz

Source Code

https://github.com/yushulx/flutter_ocr_sdk

Search Blog Posts