Labeling Multiple Barcodes with Augmented Reality and Dynamsoft Barcode Reader
The term “augmented reality (AR)” generally refers to the integration of digital information with the user’s environment in real time. AR can make user interfaces more immersive and intuitive, overlaying real-time information onto the real world to make tasks more efficient and less error-prone. For example, in warehouse management, AR can be used to track the inventory of items on shelves. In hospitals, AR can help identify tubes and bottles, and in retail, it can display product information. In this article, we will demonstrate how to use ARCore and Dynamsoft Barcode Reader to label multiple barcodes and avoid misreading.
Prerequisites
- An ARCore compatible device running Google Play Services for AR (ARCore) 1.24 or later
- Android Studio 4.1 or later
2D Barcode Overlay on Camera Preview
Usually, we display barcode results on the camera preview as follows:
The overlay is rendered on a 2D canvas that sits above the camera preview, displaying both the barcode information and its bounding box in real time.
Drawing an overlay on a camera view can be considered a basic form of augmented reality. A more advanced AR system often involves understanding the geometry of the environment and placing digital objects in a way that makes them appear to exist within the real world, responding appropriately to changes in the camera’s view or the environment itself.
In the following sections, we will combine ARCore and the Dynamsoft Barcode Reader SDK to enhance the barcode scanning experience.
About ARCore and Dynamsoft Barcode Reader
- ARCore is Google’s platform for building augmented reality (AR) experiences. By utilizing three key technologies — motion tracking, environmental understanding, and light estimation — ARCore enables your phone to sense its environment, understand the world, and interact with information in a more comprehensive and contextually aware manner.
- Dynamsoft Barcode Reader SDK is a commercial barcode scanning library for Windows, Linux, macOS, Android, iOS, and web. It supports various barcode types, including QR Code, DataMatrix, PDF417, Code 128, Code 39, EAN 13, UPC-A, etc. It also provides a free trial license for 30 days.
Combining Google’s ARCore with Dynamsoft Android Barcode SDK
Let’s get started with the sample project of ARCore: https://github.com/googlesamples/arcore-ml-sample.
The Android sample written in Kotlin demonstrates how to utilize ARCore and computer vision algorithms for object detection and tracking. For us, the primary goal is the detection and tracking of barcodes. Integrating the Dynamsoft Barcode Reader into the sample project involves a series of steps.
Add Dynamsoft Barcode Reader SDK to the Project
-
Add the Dynamsoft repository source
https://download2.dynamsoft.com/maven/aar
to the project’sbuild.gradle
file:allprojects { repositories { google() mavenLocal() mavenCentral() maven { url "https://download2.dynamsoft.com/maven/aar" } } }
-
Configure the dependency in the app’s
build.gradle
file:implementation 'com.dynamsoft:dynamsoftbarcodereader:9.6.20'
Detect Barcode, QR Code and DataMatrix in Kotlin
-
In
MainActivity.kt
, set the license key. You can get a free trial license from here.import com.dynamsoft.dbr.BarcodeReader BarcodeReader.initLicense("LICENSE-KEY") { isSuccessful, e -> runOnUiThread { if (!isSuccessful) { e.printStackTrace() Log.e(TAG, "Failed to verify the license: $e") } } }
-
Create a barcode reader object in
AppRenderer.kt
:import com.dynamsoft.dbr.BarcodeReader import com.dynamsoft.dbr.BarcodeReaderException import com.dynamsoft.dbr.EnumImagePixelFormat import com.dynamsoft.dbr.Point var reader: BarcodeReader? = null fun bindView(view: MainActivityView) { try { reader = BarcodeReader() val settings = reader!!.runtimeSettings settings.expectedBarcodesCount = 999 reader!!.updateRuntimeSettings(settings) } catch (e: BarcodeReaderException) { e.printStackTrace() } }
-
Find the code block that starts with
launch(Dispatchers.IO)
. You can utilize the acquired camera image to detect barcodes:if (reader != null) { var bytes = ByteArray(cameraImage.planes[0].buffer.remaining()) cameraImage.planes[0].buffer.get(bytes) var results = reader!!.decodeBuffer(bytes, cameraImage.width, cameraImage.height, cameraImage.planes[0].rowStride, EnumImagePixelFormat.IPF_NV21) objectResults = emptyList() if (results != null && results.isNotEmpty()) { val tmp: MutableList<DetectedObjectResult> = mutableListOf() for (result in results) { var points = result.localizationResult.resultPoints var confidence = 100 val (x1, y1) = points[0].x to points[0].y val (x2, y2) = points[1].x to points[1].y val (x3, y3) = points[2].x to points[2].y val (x4, y4) = points[3].x to points[3].y val centerX = (x1 + x2 + x3 + x4) / 4 val centerY = (y1 + y2 + y3 + y4) / 4 val content = result.barcodeText val label = "✓" val detectedObjectResult = DetectedObjectResult(confidence.toFloat(), label, centerX.toInt() to centerY.toInt(), content) tmp.add(detectedObjectResult) } objectResults = tmp } }
A barcode detection result contains four points, which are the four corners of the barcode. We can calculate the center point of the barcode and use it as the anchor point for the label.
val (x1, y1) = points[0].x to points[0].y
val (x2, y2) = points[1].x to points[1].y
val (x3, y3) = points[2].x to points[2].y
val (x4, y4) = points[3].x to points[3].y
val centerX = (x1 + x2 + x3 + x4) / 4
val centerY = (y1 + y2 + y3 + y4) / 4
We convert the barcode results to a DetectedObjectResult
list, which will be rendered on the screen later.
If we set the label
to result.barcodeText
, the content of the barcode will be displayed on the screen.
The effect works well for isolated barcodes. However, if multiple barcodes appear in the camera view, the display can become overcrowded. To address this issue, we can set the label
to ✓
and present the barcode content in a list view.
Enable Camera Auto Focus
By default, the ARCore camera is set to fixed focus, which can make the barcode appear blurry. To enhance barcode detection accuracy, enabling auto-focus is necessary.
A switch button is added to the UI to toggle the focus mode.
-
Create a switch button in the
activity_main.xml
layout file :<androidx.appcompat.widget.SwitchCompat android:id="@+id/switch_focus_mode" android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_alignParentEnd="true" android:checked="false" android:text="@string/switch_focus_mode" android:textColor="#ffffff" />
-
In
MainActivityView.kt
, obtain the reference to the switch button:var focusModeSwitch = root.findViewById<SwitchCompat>(R.id.switch_focus_mode)
-
Register the
OnCheckedChangeListener
of the switch button inAppRenderer.kt
:view.focusModeSwitch.setOnCheckedChangeListener { _, isChecked -> val session = activity.arCoreSessionHelper.sessionCache ?: return@setOnCheckedChangeListener val config = session.config config.focusMode = if (isChecked) Config.FocusMode.AUTO else Config.FocusMode.FIXED session.configure(config) }
Source Code
https://github.com/yushulx/android-camera-barcode-mrz-document-scanner/tree/main/examples/9.x/ARCore