How to Create a Flutter Document Rectification Plugin for Android and iOS

Previously, we created a Flutter document rectification plugin for web, Windows, and Linux. In this article, we will add support for Android and iOS. As a result, you will be able to create a document rectification app that corrects the perspective of an image of a document on all platforms.

Flutter Document Rectification SDK

https://pub.dev/packages/flutter_document_scan_sdk

Development Environment

  • Flutter 3.3.9
  • Minimum Android SDK Version: 21
  • Swift 5.0

Add Support for Android and iOS in Flutter Plugin

To add support for Android and iOS, run the following command in the root directory of the project:

flutter create --org com.dynamsoft --template=plugin --platforms=android,ios .

After generating the platform-specific code, update the pubspec.yaml file:

plugin:
    platforms:
      android:
        package: com.dynamsoft.flutter_document_scan_sdk
        pluginClass: FlutterDocumentScanSdkPlugin
      ios:
        pluginClass: FlutterDocumentScanSdkPlugin
      linux:
        pluginClass: FlutterDocumentScanSdkPlugin
      windows:
        pluginClass: FlutterDocumentScanSdkPluginCApi
      web:
        pluginClass: FlutterDocumentScanSdkWeb
        fileName: flutter_document_scan_sdk_web.dart

Since the Dart code and API remain unchanged, we only need to implement the native code for Android and iOS.

Linking Third-party Libraries for Android and iOS in a Flutter Plugin

Configure the Dynamsoft Document Normalizer SDK in Gradle and CocoaPods

For Android, open the android/build.gradle file and add the following dependency:

rootProject.allprojects {
    repositories {
        maven {
            url "https://download2.dynamsoft.com/maven/aar"
        }
        google()
        mavenCentral()
    }
}

dependencies { 
    implementation 'com.dynamsoft:dynamsoftdocumentnormalizer:1.0.20' 
}

For iOS, open the ios/flutter_document_scan_sdk.podspec file and add the following dependency:

s.dependency 'DynamsoftDocumentNormalizer', '1.0.20'

Write Platform-Specific Code in Java and Swift

We write Java code in the FlutterDocumentScanSdkPlugin.java file and Swift code in the SwiftFlutterDocumentScanSdkPlugin.swift file. The SDK consists of two packages: DynamsoftCore and DynamsoftDocumentNormalizer. The DynamsoftCore package contains the classes and interfaces related to license management and image data, and the DynamsoftDocumentNormalizer package contains the classes and interfaces for document rectification.

  1. Import the relevant SDK packages and classes:

    Android

     import com.dynamsoft.core.ImageData;
     import com.dynamsoft.core.CoreException;
     import com.dynamsoft.core.LicenseManager;
     import com.dynamsoft.core.LicenseVerificationListener;
     import com.dynamsoft.core.EnumImagePixelFormat;
     import com.dynamsoft.core.Quadrilateral;
    
     import com.dynamsoft.ddn.DocumentNormalizer;
     import com.dynamsoft.ddn.DetectedQuadResult;
     import com.dynamsoft.ddn.DocumentNormalizerException;
     import com.dynamsoft.ddn.NormalizedImageResult;
    

    iOS

     import DynamsoftCore
     import DynamsoftDocumentNormalizer
    
  2. Create an instance of the DocumentNormalizer class:

    Android

     private DocumentNormalizer mNormalizer;
     try {
         mNormalizer = new DocumentNormalizer();
     } catch (DocumentNormalizerException e) {
         e.printStackTrace();
     }
    

    iOS

     var normalizer: DynamsoftDocumentNormalizer = DynamsoftDocumentNormalizer()
    
  3. Set a license key to activate the SDK:

    Android

     LicenseManager.initLicense(
             license, activity,
                 new LicenseVerificationListener() {
                     @Override
                     public void licenseVerificationCallback(boolean isSuccessful, CoreException e) {
                         if (isSuccessful)
                         {
                             result.success(0);
                         }
                         else {
                             result.success(-1);
                         }
                     }
                 });
    

    The initLicense() method’s second parameter is the Activity object. To obtain the Activity object in the Flutter Android plugin, we use the ActivityAware interface. The FlutterDocumentScanSdkPlugin class implements the ActivityAware interface. The onAttachedToActivity method is called when the Activity object is created, and the onDetachedFromActivity method is called when the Activity object is destroyed.

     public class FlutterDocumentScanSdkPlugin implements FlutterPlugin, MethodCallHandler, ActivityAware {
        
         private void bind(ActivityPluginBinding activityPluginBinding) {
           activity = activityPluginBinding.getActivity();
         }
    
         @Override
         public void onAttachedToActivity(ActivityPluginBinding activityPluginBinding) {
           bind(activityPluginBinding);
         }
    
         @Override
         public void onDetachedFromActivity() {
           activity = null;
         }
     }
        
    

    iOS

     public class SwiftFlutterDocumentScanSdkPlugin: NSObject, FlutterPlugin, LicenseVerificationListener {
    
         ...
         DynamsoftLicenseManager.initLicense(license, verificationDelegate: self)
         ...
    
         public func licenseVerificationCallback(_ isSuccess: Bool, error: Error?) {
             if isSuccess {
                 completionHandlers.first?(0)
             } else{
                 completionHandlers.first?(-1)
             }
         }
     }
        
    
  4. Before calling the detection method, we can retrieve and configure parameters for the detection algorithm.

    Android

     // Get parameters
     try {
             parameters = mNormalizer.outputRuntimeSettings("");
     } catch (Exception e) {}
    
     // Set parameters
     try {
             mNormalizer.initRuntimeSettingsFromString(params);
     } catch (DocumentNormalizerException e) {}
    

    iOS

     // Get parameters
     let parameters = try? self.normalizer!.outputRuntimeSettings("")
    
     // Set parameters
     try? self.normalizer!.initRuntimeSettingsFromString(params)
    
  5. Call the detectQuad() method to identify the document edge:

    Android

     DetectedQuadResult[] detectedResults = mNormalizer.detectQuad(filename);
     if (detectedResults != null && detectedResults.length > 0) {
         for (int i = 0; i < detectedResults.length; i++) {
             Map<String, Object> map = new HashMap<>();
    
             DetectedQuadResult detectedResult = detectedResults[i];
             int confidence = detectedResult.confidenceAsDocumentBoundary;
             Point[] points = detectedResult.location.points;
             int x1 = points[0].x;
             int y1 = points[0].y;
             int x2 = points[1].x;
             int y2 = points[1].y;
             int x3 = points[2].x;
             int y3 = points[2].y;
             int x4 = points[3].x;
             int y4 = points[3].y;
    
             map.put("confidence", confidence);
             map.put("x1", x1);
             map.put("y1", y1);
             map.put("x2", x2);
             map.put("y2", y2);
             map.put("x3", x3);
             map.put("y3", y3);
             map.put("x4", x4);
             map.put("y4", y4);
                
             out.add(map);
         }
     }
    

    iOS

     let detectedResults = try? self.normalizer!.detectQuadFromFile(filename)
                    
     if detectedResults != nil {
         for result in detectedResults! {
             let dictionary = NSMutableDictionary()
                
             let confidence = result.confidenceAsDocumentBoundary
             let points = result.location.points as! [CGPoint]
                
             dictionary.setObject(confidence, forKey: "confidence" as NSCopying)
             dictionary.setObject(Int(points[0].x), forKey: "x1" as NSCopying)
             dictionary.setObject(Int(points[0].y), forKey: "y1" as NSCopying)
             dictionary.setObject(Int(points[1].x), forKey: "x2" as NSCopying)
             dictionary.setObject(Int(points[1].y), forKey: "y2" as NSCopying)
             dictionary.setObject(Int(points[2].x), forKey: "x3" as NSCopying)
             dictionary.setObject(Int(points[2].y), forKey: "y3" as NSCopying)
             dictionary.setObject(Int(points[3].x), forKey: "x4" as NSCopying)
             dictionary.setObject(Int(points[3].y), forKey: "y4" as NSCopying)
                
             out.add(dictionary)
         }
     }
     result(out)
    
  6. Call the normalizeFile() method to crop the document based on its corners and correct its perspective:

    Android

     Quadrilateral quad = new Quadrilateral();
     quad.points = new Point[4];
     quad.points[0] = new Point(x1, y1);
     quad.points[1] = new Point(x2, y2);
     quad.points[2] = new Point(x3, y3);
     quad.points[3] = new Point(x4, y4);
     mNormalizedImage = mNormalizer.normalize(filename, quad);
    
     if (mNormalizedImage != null) {
       ImageData imageData = mNormalizedImage.image;
       int width = imageData.width;
       int height = imageData.height;
       int stride = imageData.stride;
       int format = imageData.format;
       byte[] data = imageData.bytes;
       int length = imageData.bytes.length;
       int orientation = imageData.orientation;
     }
    

    iOS

     let points = [CGPoint(x: x1, y: y1), CGPoint(x: x2, y: y2), CGPoint(x: x3, y: y3), CGPoint(x: x4, y: y4)]
     let quad = iQuadrilateral()
     quad.points = points
    
     if self.normalizedImage != nil {
         let imageData: iImageData = self.normalizedImage!.image
         let width = imageData.width
         let height = imageData.height
         let stride = imageData.stride
         let format = imageData.format
         let data = imageData.bytes
         let length = data!.count
         let orientation = imageData.orientation
     }
    
  7. To transfer the image data of the rectified document to the Flutter side, we need to convert the image data (such as RGB888, Grayscale or binary ) to a RGBA byte array.

    • RGB888: A pixel is represented by three bytes. The order of the bytes is R, G, and B.
    • Grayscale: Every byte represents the grayscale value of a pixel.
    • Binary: Every byte represents 8 pixels. The value of each bit is 0 or 1.

    Android

     byte[] rgba = new byte[width * height * 4];
        
     if (format == EnumImagePixelFormat.IPF_RGB_888) {
       int dataIndex = 0;
       for (int i = 0; i < height; i++)
       {
           for (int j = 0; j < width; j++)
           {
               int index = i * width + j;
    
               rgba[index * 4] = data[dataIndex + 2];     // red
               rgba[index * 4 + 1] = data[dataIndex + 1]; // green
               rgba[index * 4 + 2] = data[dataIndex];     // blue
               rgba[index * 4 + 3] = (byte)255;                 // alpha
               dataIndex += 3;
           }
       }
     }
     else if (format == EnumImagePixelFormat.IPF_GRAYSCALED) {
       int dataIndex = 0;
       for (int i = 0; i < height; i++)
       {
           for (int j = 0; j < width; j++)
           {
               int index = i * width + j;
               rgba[index * 4] = data[dataIndex];
               rgba[index * 4 + 1] = data[dataIndex];
               rgba[index * 4 + 2] = data[dataIndex];
               rgba[index * 4 + 3] = (byte)255;
               dataIndex += 1;
           }
       }
     }
     else if (format == EnumImagePixelFormat.IPF_BINARY) {
       byte[] grayscale = new byte[width * height];
       binary2grayscale(data, grayscale, width, height, stride, length);
    
       int dataIndex = 0;
       for (int i = 0; i < height; i++)
       {
           for (int j = 0; j < width; j++)
           {
               int index = i * width + j;
               rgba[index * 4] = grayscale[dataIndex];
               rgba[index * 4 + 1] = grayscale[dataIndex];
               rgba[index * 4 + 2] = grayscale[dataIndex];
               rgba[index * 4 + 3] = (byte)255;
               dataIndex += 1;
           }
       }
     }
    
     void binary2grayscale(byte[] data, byte[] output, int width, int height, int stride, int length) {
       int index = 0;
    
       int skip = stride * 8 - width;
       int shift = 0;
       int n = 1;
    
       for (int i = 0; i < length; ++i)
       {
           byte b = data[i];
           int byteCount = 7;
           while (byteCount >= 0)
           {
               int tmp = (b & (1 << byteCount)) >> byteCount;
    
               if (shift < stride * 8 * n - skip)
               {
                   if (tmp == 1)
                       output[index] = (byte)255;
                   else
                       output[index] = 0;
                   index += 1;
               }
    
               byteCount -= 1;
               shift += 1;
           }
    
           if (shift == stride * 8 * n)
           {
               n += 1;
           }
       }
     }
    

    iOS

     var rgba: [UInt8] = [UInt8](repeating: 0, count: width * height * 4)
                        
     if format == EnumImagePixelFormat.RGB_888 {
         var dataIndex = 0
         for i in 0..<height {
             for j in 0..<width {
                 let index = i * width + j
                 rgba[index * 4] = data![dataIndex + 2]     // red
                 rgba[index * 4 + 1] = data![dataIndex + 1] // green
                 rgba[index * 4 + 2] = data![dataIndex]     // blue
                 rgba[index * 4 + 3] = 255                 // alpha
                 dataIndex += 3
             }
         }
     }
     else if (format == EnumImagePixelFormat.grayScaled) {
         var dataIndex = 0
         for i in 0..<height {
             for j in 0..<width {
                 let index = i * width + j
                 rgba[index * 4] = data![dataIndex]
                 rgba[index * 4 + 1] = data![dataIndex]
                 rgba[index * 4 + 2] = data![dataIndex]
                 rgba[index * 4 + 3] = 255
                 dataIndex += 1
             }
         }
     }
     else if (format == EnumImagePixelFormat.binary) {
         var grayscale: [UInt8] = [UInt8](repeating: 0, count: width * height)
            
         var index = 0
         let skip = stride * 8 - width
         var shift = 0
         var n = 1
            
         for i in 0..<length {
             let b = data![i]
             var byteCount = 7
             while byteCount >= 0 {
                 let tmp = (b & (1 << byteCount)) >> byteCount
                    
                 if (shift < stride * 8 * n - skip)
                 {
                     if (tmp == 1) {
                         grayscale[index] = 255
                     }
                     else {
                         grayscale[index] = 0
                     }
                     index += 1
                 }
                    
                 byteCount -= 1
                 shift += 1
             }
                
             if (shift == stride * 8 * n)
             {
                 n += 1
             }
         }
            
         var dataIndex = 0
         for i in 0..<height {
             for j in 0..<width {
                 let index = i * width + j
                 rgba[index * 4] = grayscale[dataIndex]
                 rgba[index * 4 + 1] = grayscale[dataIndex]
                 rgba[index * 4 + 2] = grayscale[dataIndex]
                 rgba[index * 4 + 3] = 255
                 dataIndex += 1
             }
         }
     }
    

Test the Flutter Document Rectification Plugin on both Android and iOS

flutter run

Document Edge Detection

Flutter document edge detection for Android and iOS

Document Perspective Correction

Flutter document perspective correction for Android and iOS

Source Code

https://github.com/yushulx/flutter_document_scan_sdk