How to Create a Flutter Document Rectification Plugin for Android and iOS
Previously, we created a Flutter document rectification plugin for web, Windows, and Linux. In this article, we will add support for Android and iOS. As a result, you will be able to create a document rectification app that corrects the perspective of an image of a document on all platforms.
This article is Part 3 in a 3-Part Series.
Flutter Document Rectification SDK
https://pub.dev/packages/flutter_document_scan_sdk
Development Environment
- Flutter 3.3.9
- Minimum Android SDK Version: 21
- Swift 5.0
Add Support for Android and iOS in Flutter Plugin
To add support for Android and iOS, run the following command in the root directory of the project:
flutter create --org com.dynamsoft --template=plugin --platforms=android,ios .
After generating the platform-specific code, update the pubspec.yaml
file:
plugin:
platforms:
android:
package: com.dynamsoft.flutter_document_scan_sdk
pluginClass: FlutterDocumentScanSdkPlugin
ios:
pluginClass: FlutterDocumentScanSdkPlugin
linux:
pluginClass: FlutterDocumentScanSdkPlugin
windows:
pluginClass: FlutterDocumentScanSdkPluginCApi
web:
pluginClass: FlutterDocumentScanSdkWeb
fileName: flutter_document_scan_sdk_web.dart
Since the Dart code and API remain unchanged, we only need to implement the native code for Android and iOS.
Linking Third-party Libraries for Android and iOS in a Flutter Plugin
Configure the Dynamsoft Document Normalizer SDK in Gradle and CocoaPods
For Android, open the android/build.gradle
file and add the following dependency:
rootProject.allprojects {
repositories {
maven {
url "https://download2.dynamsoft.com/maven/aar"
}
google()
mavenCentral()
}
}
dependencies {
implementation 'com.dynamsoft:dynamsoftdocumentnormalizer:1.0.20'
}
For iOS, open the ios/flutter_document_scan_sdk.podspec
file and add the following dependency:
s.dependency 'DynamsoftDocumentNormalizer', '1.0.20'
Write Platform-Specific Code in Java and Swift
We write Java code in the FlutterDocumentScanSdkPlugin.java
file and Swift code in the SwiftFlutterDocumentScanSdkPlugin.swift
file. The SDK consists of two packages: DynamsoftCore
and DynamsoftDocumentNormalizer
. The DynamsoftCore package
contains the classes and interfaces related to license management and image data, and the DynamsoftDocumentNormalizer
package contains the classes and interfaces for document rectification.
-
Import the relevant SDK packages and classes:
Android
import com.dynamsoft.core.ImageData; import com.dynamsoft.core.CoreException; import com.dynamsoft.core.LicenseManager; import com.dynamsoft.core.LicenseVerificationListener; import com.dynamsoft.core.EnumImagePixelFormat; import com.dynamsoft.core.Quadrilateral; import com.dynamsoft.ddn.DocumentNormalizer; import com.dynamsoft.ddn.DetectedQuadResult; import com.dynamsoft.ddn.DocumentNormalizerException; import com.dynamsoft.ddn.NormalizedImageResult;
iOS
import DynamsoftCore import DynamsoftDocumentNormalizer
-
Create an instance of the
DocumentNormalizer
class:Android
private DocumentNormalizer mNormalizer; try { mNormalizer = new DocumentNormalizer(); } catch (DocumentNormalizerException e) { e.printStackTrace(); }
iOS
var normalizer: DynamsoftDocumentNormalizer = DynamsoftDocumentNormalizer()
-
Set a license key to activate the SDK:
Android
LicenseManager.initLicense( license, activity, new LicenseVerificationListener() { @Override public void licenseVerificationCallback(boolean isSuccessful, CoreException e) { if (isSuccessful) { result.success(0); } else { result.success(-1); } } });
The
initLicense()
method’s second parameter is theActivity
object. To obtain theActivity
object in the Flutter Android plugin, we use theActivityAware
interface. TheFlutterDocumentScanSdkPlugin
class implements theActivityAware
interface. TheonAttachedToActivity
method is called when theActivity
object is created, and theonDetachedFromActivity
method is called when theActivity
object is destroyed.public class FlutterDocumentScanSdkPlugin implements FlutterPlugin, MethodCallHandler, ActivityAware { private void bind(ActivityPluginBinding activityPluginBinding) { activity = activityPluginBinding.getActivity(); } @Override public void onAttachedToActivity(ActivityPluginBinding activityPluginBinding) { bind(activityPluginBinding); } @Override public void onDetachedFromActivity() { activity = null; } }
iOS
public class SwiftFlutterDocumentScanSdkPlugin: NSObject, FlutterPlugin, LicenseVerificationListener { ... DynamsoftLicenseManager.initLicense(license, verificationDelegate: self) ... public func licenseVerificationCallback(_ isSuccess: Bool, error: Error?) { if isSuccess { completionHandlers.first?(0) } else{ completionHandlers.first?(-1) } } }
-
Before calling the detection method, we can retrieve and configure parameters for the detection algorithm.
Android
// Get parameters try { parameters = mNormalizer.outputRuntimeSettings(""); } catch (Exception e) {} // Set parameters try { mNormalizer.initRuntimeSettingsFromString(params); } catch (DocumentNormalizerException e) {}
iOS
// Get parameters let parameters = try? self.normalizer!.outputRuntimeSettings("") // Set parameters try? self.normalizer!.initRuntimeSettingsFromString(params)
-
Call the
detectQuad()
method to identify the document edge:Android
DetectedQuadResult[] detectedResults = mNormalizer.detectQuad(filename); if (detectedResults != null && detectedResults.length > 0) { for (int i = 0; i < detectedResults.length; i++) { Map<String, Object> map = new HashMap<>(); DetectedQuadResult detectedResult = detectedResults[i]; int confidence = detectedResult.confidenceAsDocumentBoundary; Point[] points = detectedResult.location.points; int x1 = points[0].x; int y1 = points[0].y; int x2 = points[1].x; int y2 = points[1].y; int x3 = points[2].x; int y3 = points[2].y; int x4 = points[3].x; int y4 = points[3].y; map.put("confidence", confidence); map.put("x1", x1); map.put("y1", y1); map.put("x2", x2); map.put("y2", y2); map.put("x3", x3); map.put("y3", y3); map.put("x4", x4); map.put("y4", y4); out.add(map); } }
iOS
let detectedResults = try? self.normalizer!.detectQuadFromFile(filename) if detectedResults != nil { for result in detectedResults! { let dictionary = NSMutableDictionary() let confidence = result.confidenceAsDocumentBoundary let points = result.location.points as! [CGPoint] dictionary.setObject(confidence, forKey: "confidence" as NSCopying) dictionary.setObject(Int(points[0].x), forKey: "x1" as NSCopying) dictionary.setObject(Int(points[0].y), forKey: "y1" as NSCopying) dictionary.setObject(Int(points[1].x), forKey: "x2" as NSCopying) dictionary.setObject(Int(points[1].y), forKey: "y2" as NSCopying) dictionary.setObject(Int(points[2].x), forKey: "x3" as NSCopying) dictionary.setObject(Int(points[2].y), forKey: "y3" as NSCopying) dictionary.setObject(Int(points[3].x), forKey: "x4" as NSCopying) dictionary.setObject(Int(points[3].y), forKey: "y4" as NSCopying) out.add(dictionary) } } result(out)
-
Call the
normalizeFile()
method to crop the document based on its corners and correct its perspective:Android
Quadrilateral quad = new Quadrilateral(); quad.points = new Point[4]; quad.points[0] = new Point(x1, y1); quad.points[1] = new Point(x2, y2); quad.points[2] = new Point(x3, y3); quad.points[3] = new Point(x4, y4); mNormalizedImage = mNormalizer.normalize(filename, quad); if (mNormalizedImage != null) { ImageData imageData = mNormalizedImage.image; int width = imageData.width; int height = imageData.height; int stride = imageData.stride; int format = imageData.format; byte[] data = imageData.bytes; int length = imageData.bytes.length; int orientation = imageData.orientation; }
iOS
let points = [CGPoint(x: x1, y: y1), CGPoint(x: x2, y: y2), CGPoint(x: x3, y: y3), CGPoint(x: x4, y: y4)] let quad = iQuadrilateral() quad.points = points if self.normalizedImage != nil { let imageData: iImageData = self.normalizedImage!.image let width = imageData.width let height = imageData.height let stride = imageData.stride let format = imageData.format let data = imageData.bytes let length = data!.count let orientation = imageData.orientation }
-
To transfer the image data of the rectified document to the Flutter side, we need to convert the image data (such as
RGB888
,Grayscale
orbinary
) to aRGBA
byte array.RGB888
: A pixel is represented by three bytes. The order of the bytes isR
,G
, andB
.Grayscale
: Every byte represents the grayscale value of a pixel.Binary
: Every byte represents 8 pixels. The value of each bit is0
or1
.
Android
byte[] rgba = new byte[width * height * 4]; if (format == EnumImagePixelFormat.IPF_RGB_888) { int dataIndex = 0; for (int i = 0; i < height; i++) { for (int j = 0; j < width; j++) { int index = i * width + j; rgba[index * 4] = data[dataIndex + 2]; // red rgba[index * 4 + 1] = data[dataIndex + 1]; // green rgba[index * 4 + 2] = data[dataIndex]; // blue rgba[index * 4 + 3] = (byte)255; // alpha dataIndex += 3; } } } else if (format == EnumImagePixelFormat.IPF_GRAYSCALED) { int dataIndex = 0; for (int i = 0; i < height; i++) { for (int j = 0; j < width; j++) { int index = i * width + j; rgba[index * 4] = data[dataIndex]; rgba[index * 4 + 1] = data[dataIndex]; rgba[index * 4 + 2] = data[dataIndex]; rgba[index * 4 + 3] = (byte)255; dataIndex += 1; } } } else if (format == EnumImagePixelFormat.IPF_BINARY) { byte[] grayscale = new byte[width * height]; binary2grayscale(data, grayscale, width, height, stride, length); int dataIndex = 0; for (int i = 0; i < height; i++) { for (int j = 0; j < width; j++) { int index = i * width + j; rgba[index * 4] = grayscale[dataIndex]; rgba[index * 4 + 1] = grayscale[dataIndex]; rgba[index * 4 + 2] = grayscale[dataIndex]; rgba[index * 4 + 3] = (byte)255; dataIndex += 1; } } } void binary2grayscale(byte[] data, byte[] output, int width, int height, int stride, int length) { int index = 0; int skip = stride * 8 - width; int shift = 0; int n = 1; for (int i = 0; i < length; ++i) { byte b = data[i]; int byteCount = 7; while (byteCount >= 0) { int tmp = (b & (1 << byteCount)) >> byteCount; if (shift < stride * 8 * n - skip) { if (tmp == 1) output[index] = (byte)255; else output[index] = 0; index += 1; } byteCount -= 1; shift += 1; } if (shift == stride * 8 * n) { n += 1; } } }
iOS
var rgba: [UInt8] = [UInt8](repeating: 0, count: width * height * 4) if format == EnumImagePixelFormat.RGB_888 { var dataIndex = 0 for i in 0..<height { for j in 0..<width { let index = i * width + j rgba[index * 4] = data![dataIndex + 2] // red rgba[index * 4 + 1] = data![dataIndex + 1] // green rgba[index * 4 + 2] = data![dataIndex] // blue rgba[index * 4 + 3] = 255 // alpha dataIndex += 3 } } } else if (format == EnumImagePixelFormat.grayScaled) { var dataIndex = 0 for i in 0..<height { for j in 0..<width { let index = i * width + j rgba[index * 4] = data![dataIndex] rgba[index * 4 + 1] = data![dataIndex] rgba[index * 4 + 2] = data![dataIndex] rgba[index * 4 + 3] = 255 dataIndex += 1 } } } else if (format == EnumImagePixelFormat.binary) { var grayscale: [UInt8] = [UInt8](repeating: 0, count: width * height) var index = 0 let skip = stride * 8 - width var shift = 0 var n = 1 for i in 0..<length { let b = data![i] var byteCount = 7 while byteCount >= 0 { let tmp = (b & (1 << byteCount)) >> byteCount if (shift < stride * 8 * n - skip) { if (tmp == 1) { grayscale[index] = 255 } else { grayscale[index] = 0 } index += 1 } byteCount -= 1 shift += 1 } if (shift == stride * 8 * n) { n += 1 } } var dataIndex = 0 for i in 0..<height { for j in 0..<width { let index = i * width + j rgba[index * 4] = grayscale[dataIndex] rgba[index * 4 + 1] = grayscale[dataIndex] rgba[index * 4 + 2] = grayscale[dataIndex] rgba[index * 4 + 3] = 255 dataIndex += 1 } } }
Test the Flutter Document Rectification Plugin on both Android and iOS
flutter run
Document Edge Detection
Document Perspective Correction