How to Build a Capacitor Plugin for Document Scanning

Nov 16, 2022

Capacitor is an open-source native runtime created by the Ionic team for building Web Native apps. We can use it to create cross-platform iOS, Android, and Progressive Web Apps with JavaScript, HTML, and CSS.

Using Capacitor plugins, we can use JavaScript to interface directly with native APIs. In this article, we are going to build a document scanning Capacitor plugin using Dynamsoft Document Normalizer SDK. Dynamsoft Document Normalizer can detect the borders of documents and run perspective transformation to get a corrected document image.

Build a Document Scanning Capacitor Plugin

Let’s do this in steps.

New Plugin Project

In a new terminal, run the following:

npm init @capacitor/plugin

We will be prompted to input relevant project info.

√ What should be the npm package of your plugin?
 ... capacitor-plugin-dynamsoft-document-normalizer
√ What directory should be used for your plugin?
 ... capacitor-plugin-dynamsoft-document-normalizer
√ What should be the Package ID for your plugin?

    Package IDs are unique identifiers used in apps and plugins. For plugins,
    they're used as a Java namespace. They must be in reverse domain name
    notation, generally representing a domain name that you or your company owns.

 ... com.dynamsoft.capacitor.ddn
√ What should be the class name for your plugin?
 ... DocumentNormalizer
√ What is the repository URL for your plugin?
 ... https://github.com/tony-xlh/capacitor-plugin-dynamsoft-document-normalizer

Create an Example Project

In order to test the plugin, we can create an example project.

Under the root of the plugin, create an example folder and start a webpack project.

git clone https://github.com/wbkd/webpack-starter
mv webpack-starter example # rename webpack-starter to example

Install the Capacitor plugin to the example project:

cd example
npm install ..

Then, we can run npm start to test the example project.

Next, we are going to implement the Web, Android and iOS parts of the plugin.

Web Implementation

Add Dynamsoft Document Normalizer as a Dependency

npm install dynamsoft-document-normalizer@1.0.10

Write Definitions

Define interfaces in src/definitions.ts. The DocumentNormalizerPlugin provides methods to initialize Dynamsoft Document Normalizer, update its settings and use it to detect and normalize documents.

import { DCEFrame } from "dynamsoft-camera-enhancer";
import { DetectedQuadResult } from "dynamsoft-document-normalizer";
import { Quadrilateral } from "dynamsoft-document-normalizer/dist/types/interface/quadrilateral";

export interface DocumentNormalizerPlugin {
  initialize(): Promise<void>;
  initLicense(options: {license: string}): Promise<void>;
  initRuntimeSettingsFromString(options: {template:string}): Promise<void>;
  detect(options:{source:string | DCEFrame}): Promise<{results:DetectedQuadResult[]}>;
  normalize(options:{source:string | DCEFrame, quad:Quadrilateral}): Promise<{result:NormalizedImageResult}>;
  /**
  * Web Only
  */
  setEngineResourcesPath(options:{path:string}): Promise<void>; 
}

export interface NormalizedImageResult {
  data: string;
}

Implement the Interfaces

Initialization.

The initialize method will create an instance of Dynamsoft Document Normalizer for use. We can set the license and engine resources path before creating an instance.

export class DocumentNormalizerWeb extends WebPlugin implements DocumentNormalizerPlugin {
  private normalizer:DocumentNormalizer | undefined;
  private engineResourcesPath: string = "https://cdn.jsdelivr.net/npm/dynamsoft-document-normalizer@1.0.10/dist/";
  private license: string = "DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==";
  async initialize(): Promise<void> {
    try {
      DocumentNormalizer.license = this.license;
      DocumentNormalizer.engineResourcePath = this.engineResourcesPath;
      this.normalizer = await DocumentNormalizer.createInstance();
    } catch (error) {
      throw error;
    }
  }
   
   
async initLicense(options: { license: string }): Promise<void> {
  this.license = options.license;
}

async setEngineResourcesPath(options: { path: string; }): Promise<void> {
  this.engineResourcesPath = options.path;
}}

Updating the runtime settings.

We can update the runtime settings of Document Normalizer using a JSON template to adjust its behaviors like which document color mode to use.

async initRuntimeSettingsFromString(options: { template: string }): Promise<void> {
  if (this.normalizer) {
    await this.normalizer.setRuntimeSettings(options.template);
  } else {
    throw new Error("DDN not initialized.");
  }
}

Detecting documents.

We can pass the image as a base64-encoded string or a DCEFrame from the JS version of Dynamsoft Camera Enhancer for document detection.

async detect(options: { source: string | DCEFrame }): Promise<{results:DetectedQuadResult[]}> {
  if (this.normalizer) {
    let detectedQuads = await this.normalizer.detectQuad(options.source);
    return {results:detectedQuads};
  } else {
    throw new Error("DDN not initialized.");
  }
}

Normalizing a document using a detected quadrilateral result.

We can pass the image as a base64-encoded string or a DCEFrame from the JS version of Dynamsoft Camera Enhancer along with the quadrilateral result from document detection to normalize the document image.

async normalize(options: { source: string | DCEFrame, quad:Quadrilateral}): Promise<{result:NormalizedImageResult}> {
  if (this.normalizer) {
    let result = await this.normalizer.normalize(options.source,{quad:options.quad});
    let normalizedResult:NormalizedImageResult = {
      data:result.image.toCanvas().toDataURL()
    }
    return {result:normalizedResult};
  } else {
    throw new Error("DDN not initialized.");
  }
}

Android Implementation

Add Dynamsoft Document Normalizer as a Dependency

Open android/build.gradle to add the Dynamsoft Document Normalizer dependency:

rootProject.allprojects {
    repositories {
        maven {
            url "https://download2.dynamsoft.com/maven/aar"
        }
    }
}


dependencies {
    // DDN
    implementation 'com.dynamsoft:dynamsoftdocumentnormalizer:1.0.10'
}

Implementation

Open DocumentNormalizerPlugin.java to implement the plugin.

Initialization.

private DocumentNormalizer ddn;
@PluginMethod
public void initialize(PluginCall call) {
    try {
        if (ddn == null) {
            ddn = new DocumentNormalizer();
        }
        call.resolve();
    } catch (DocumentNormalizerException e) {
        e.printStackTrace();
        call.reject(e.getMessage());
    }
}

License initialization.

@PluginMethod
public void initLicense(PluginCall call) {
    String license = call.getString("license","DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==");
    LicenseManager.initLicense(license, getContext(), new LicenseVerificationListener() {
        @Override
        public void licenseVerificationCallback(boolean isSuccess, CoreException error) {
            if(!isSuccess){
                error.printStackTrace();
                call.reject(error.getMessage());
            }else{
                call.resolve();
            }
        }
    });
}

Updating the runtime settings.

@PluginMethod
public void initRuntimeSettingsFromString(PluginCall call) {
    String template = call.getString("template");
    if (ddn != null) {
        try {
            ddn.initRuntimeSettingsFromString(template);
            call.resolve();
        } catch (DocumentNormalizerException e) {
            e.printStackTrace();
            call.reject(e.getMessage());
        }
    }else{
        call.reject("DDN not initialized");
    }
}

Detecting the document.

@PluginMethod
public void detect(PluginCall call) {
    String source = call.getString("source");
    source = source.replaceFirst("data:.*?;base64,","");
    if (ddn != null) {
        try {
            JSObject response = new JSObject();
            JSArray detectionResults = new JSArray();
            DetectedQuadResult[] results = ddn.detectQuad(Utils.base642Bitmap(source));
            if (results != null) {
                for (DetectedQuadResult result:results) {
                    detectionResults.put(Utils.getMapFromDetectedQuadResult(result));
                }
            }
            response.put("results",detectionResults);
            call.resolve(response);
        } catch (DocumentNormalizerException e) {
            e.printStackTrace();
            call.reject(e.getMessage());
        }
    }else{
        call.reject("DDN not initialized");
    }
}

Normalizing the document image.

@PluginMethod
public void normalize(PluginCall call) {
    JSObject quad = call.getObject("quad");
    String source = call.getString("source");
    source = source.replaceFirst("data:.*?;base64,","");
    if (ddn != null) {
        try {
            Point[] points = Utils.convertPoints(quad.getJSONArray("points"));
            Quadrilateral quadrilateral = new Quadrilateral();
            quadrilateral.points = points;
            NormalizedImageResult result = ddn.normalize(Utils.base642Bitmap(source),quadrilateral);
            Bitmap bm = result.image.toBitmap();
            JSObject response = new JSObject();
            JSObject resultObject = new JSObject();
            resultObject.put("data",Utils.bitmap2Base64(bm));
            response.put("result",resultObject);
            call.resolve(response);
        }catch (Exception e) {
            call.reject(e.getMessage());
        }
    }else{
        call.reject("DDN not initialized");
    }
}

A Utils class is created for wrapping the results and base64-bitmap conversion.

public class Utils {
    public static Bitmap base642Bitmap(String base64) {
        byte[] decode = Base64.decode(base64,Base64.DEFAULT);
        return BitmapFactory.decodeByteArray(decode,0,decode.length);
    }

    public static String bitmap2Base64(Bitmap bitmap) {
        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        bitmap.compress(Bitmap.CompressFormat.JPEG, 100, outputStream);
        return Base64.encodeToString(outputStream.toByteArray(), Base64.DEFAULT);
    }

    public static Point[] convertPoints(JSONArray pointsArray) throws JSONException {
        Point[] points = new Point[4];
        for (int i = 0; i < pointsArray.length(); i++) {
            JSONObject pointMap = pointsArray.getJSONObject(i);
            Point point = new Point();
            point.x = pointMap.getInt("x");
            point.y = pointMap.getInt("y");
            points[i] = point;
        }
        return points;
    }

    public static JSObject getMapFromDetectedQuadResult(DetectedQuadResult result){
        JSObject map = new JSObject ();
        map.put("confidenceAsDocumentBoundary",result.confidenceAsDocumentBoundary);
        map.put("location",getMapFromLocation(result.location));
        return map;
    }

    private static JSObject getMapFromLocation(Quadrilateral location){
        JSObject map = new JSObject();
        JSArray  points = new JSArray();
        for (Point point: location.points) {
            JSObject pointAsMap = new JSObject();
            pointAsMap.put("x",point.x);
            pointAsMap.put("y",point.y);
            points.put(pointAsMap);
        }
        map.put("points",points);
        return map;
    }
}

iOS Implementation

Add Dynamsoft Document Normalizer as a Dependency

Open CapacitorPluginDynamsoftDocumentNormalizer.podspec to add the Dynamsoft Document Normalizer dependency:

s.dependency 'DynamsoftDocumentNormalizer', '= 1.0.10'

Write Definitions

Open DocumentNormalizerPlugin.m to define the methods.

// Define the plugin using the CAP_PLUGIN Macro, and
// each method the plugin supports using the CAP_PLUGIN_METHOD macro.
CAP_PLUGIN(DocumentNormalizerPlugin, "DocumentNormalizer",
           CAP_PLUGIN_METHOD(initialize, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(initLicense, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(initRuntimeSettingsFromString, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(detect, CAPPluginReturnPromise);
           CAP_PLUGIN_METHOD(normalize, CAPPluginReturnPromise);
)

Implementation

Open DocumentNormalizerPlugin.swift to implement the plugin.

Initialization.

private var ddn:DynamsoftDocumentNormalizer!;
@objc func initialize(_ call: CAPPluginCall) {
    if ddn == nil {
        ddn = DynamsoftDocumentNormalizer()
    }
    call.resolve()
}

License initialization.

@objc func initLicense(_ call: CAPPluginCall) {
    let license = call.getString("license") ?? "DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ=="
    DynamsoftLicenseManager.initLicense(license, verificationDelegate: self)
    call.resolve()
}

Updating the runtime settings.

 @objc func initRuntimeSettingsFromString(_ call: CAPPluginCall) {
    let template = call.getString("template") ?? ""
    if ddn != nil {
        if template != "" {
            do {
                try ddn.initRuntimeSettingsFromString(template)
                call.resolve()
            }catch {
                print("Unexpected error: \(error).")
                call.reject(error.localizedDescription)
            }
        }else{
            call.reject("Empty template")
        }
    }else{
        call.reject("DDN not initialized")
    }
}

Detecting the document.

@objc func detect(_ call: CAPPluginCall) {
    var base64 = call.getString("source") ?? ""
    base64 = Utils.removeDataURLHead(base64)
    let image = Utils.convertBase64ToImage(base64)
    var returned_results: [Any] = []
    let results = try? ddn.detectQuadFromImage(image!)
    if results != nil {
        for result in results! {
            returned_results.append(Utils.wrapDetectionResult(result:result))
        }
    }
    call.resolve(["results":returned_results])
}

Normalizing the document image.

 @objc func normalize(_ call: CAPPluginCall) {
    do {
        var base64 = call.getString("source") ?? ""
        base64 = Utils.removeDataURLHead(base64)
        let image = Utils.convertBase64ToImage(base64)
        let quad = call.getObject("quad")
        let points = quad!["points"] as! [[String:NSNumber]]
        let quadrilateral = iQuadrilateral.init()
        quadrilateral.points = Utils.convertPoints(points)
           
        let normalizedImageResult = try ddn.normalizeImage(image!, quad: quadrilateral)
        let normalizedUIImage = try? normalizedImageResult.image.toUIImage()
        let normalziedResultAsBase64 = Utils.getBase64FromImage(normalizedUIImage!)
        call.resolve(["result":["data":normalziedResultAsBase64]])
    }catch {
        print("Unexpected error: \(error).")
        call.reject(error.localizedDescription)
    }
}

A Utils class is created for wrapping the results and base64-UIImage conversion.

class Utils {
    static public func convertBase64ToImage(_ imageStr:String) ->UIImage?{
        if let data: NSData = NSData(base64Encoded: imageStr, options:NSData.Base64DecodingOptions.ignoreUnknownCharacters)
        {
            if let image: UIImage = UIImage(data: data as Data)
            {
                return image
            }
        }
        return nil
    }
       
    static func getBase64FromImage(_ image:UIImage) -> String{
        let dataTmp = image.jpegData(compressionQuality: 100)
        if let data = dataTmp {
            return data.base64EncodedString()
        }
        return ""
    }
       
    static func removeDataURLHead(_ str: String) -> String {
        var finalStr = str
        do {
            let pattern = "data:.*?;base64,"
            let regex = try NSRegularExpression(pattern: pattern, options: NSRegularExpression.Options.caseInsensitive)
            finalStr = regex.stringByReplacingMatches(in: str, options: NSRegularExpression.MatchingOptions(rawValue: 0), range: NSMakeRange(0, str.count), withTemplate: "")
        }
        catch {
            print(error)
        }
        return finalStr
    }
       
    static func convertPoints(_ points:[[String:NSNumber]]) -> [CGPoint] {
        var CGPoints:[CGPoint] = [];
        for point in points {
            let x = point["x"]!
            let y = point["y"]!
            let intX = x.intValue
            let intY = y.intValue
            let cgPoint = CGPoint(x: intX, y: intY)
            CGPoints.append(cgPoint)
        }
        return CGPoints
    }
       
    static func wrapDetectionResult (result:iDetectedQuadResult) -> [String: Any] {
        var dict: [String: Any] = [:]
        dict["confidenceAsDocumentBoundary"] = result.confidenceAsDocumentBoundary
        dict["location"] = wrapLocation(location:result.location)
        return dict
    }
           
    static private func wrapLocation (location:iQuadrilateral?) -> [String: Any] {
        var dict: [String: Any] = [:]
        var points: [[String:CGFloat]] = []
        let CGPoints = location!.points as! [CGPoint]
        for point in CGPoints {
            var pointDict: [String:CGFloat] = [:]
            pointDict["x"] = point.x
            pointDict["y"] = point.y
            points.append(pointDict)
        }
        dict["points"] = points
        return dict
    }
}

Capacitor Document Scanner Online Demo

All right, we’ve now finished writing the Capacitor plugin. A document scanner demo app has been created in vanilla JS using it. You can try it out using this online demo.

Ionic Demo

Check out this article if you are using the Ionic framework.

Source Code

You can find the code of the plugin and the example in the following link:

https://github.com/tony-xlh/capacitor-plugin-dynamsoft-document-normalizer/

LANGUAGES

PLATFORMS

FEATURED

How to Build a Capacitor Plugin for Document Scanning

Build a Document Scanning Capacitor Plugin

New Plugin Project

Create an Example Project

Web Implementation

Add Dynamsoft Document Normalizer as a Dependency

Write Definitions

Implement the Interfaces

Android Implementation

Add Dynamsoft Document Normalizer as a Dependency

Implementation

iOS Implementation

Add Dynamsoft Document Normalizer as a Dependency

Write Definitions

Implementation

Capacitor Document Scanner Online Demo

Ionic Demo

Source Code