Record a Video using CameraX and Read Barcodes from Video Files

Normally, we scan barcodes and QR codes from live camera streams or static images. But sometimes, we may need to read barcodes from video files, like scanning the barcodes on a long shelf and evaluating the performance of live scanning. A video contains rich information. A state-of-art mobile phone can shoot a 240 FPS high-speed video in 1920x1080 resolution. There are great potential reading barcodes and QR codes from videos.

In this article, we are going to create an Android app which can record videos using CameraX in Kotlin and read barcodes and QR codes from video files using Dynamsoft Barcode Reader.

Getting started with Dynamsoft Barcode Reader

Dynamsoft Barcode Reader  

Record Video using CameraX

Let’s create a new Android project with Android Studio and add video recording function to it. The following parts take some code from Google’s CameraXVideo sample.

Add CameraX Dependencies

Open the project’s build.gradle, add the following to include CameraX:

// CameraX dependencies (first release for video is: "1.1.0-alpha10")
def camerax_version = "1.1.0-beta01"
// The following line is optional, as the core library is included indirectly by camera-camera2
implementation "androidx.camera:camera-core:${camerax_version}"
implementation "androidx.camera:camera-camera2:${camerax_version}"
implementation "androidx.camera:camera-lifecycle:${camerax_version}"
implementation "androidx.camera:camera-video:${camerax_version}"
implementation "androidx.camera:camera-view:${camerax_version}"

Request Permissions

  1. Open AndroidManifest.xml, add the following permissions:

     <uses-permission android:name="android.permission.CAMERA" />
 <uses-permission android:name="android.permission.RECORD_AUDIO" />
 <uses-permission
     android:name="android.permission.WRITE_EXTERNAL_STORAGE"
     android:maxSdkVersion="28" />

  2. In MainActivity, request permissions.

     private var PERMISSIONS_REQUIRED = arrayOf(
         Manifest.permission.CAMERA,
         Manifest.permission.RECORD_AUDIO)
 override fun onCreate(savedInstanceState: Bundle?) {
     super.onCreate(savedInstanceState)
     // add the storage access permission request for Android 9 and below.
     if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.P) {
         val permissionList = PERMISSIONS_REQUIRED.toMutableList()
         permissionList.add(Manifest.permission.WRITE_EXTERNAL_STORAGE)
         PERMISSIONS_REQUIRED = permissionList.toTypedArray()
     }

     if (!hasPermissions(this)) {
         // Request camera-related permissions
         activityResultLauncher.launch(PERMISSIONS_REQUIRED)
     }
 }

 private fun hasPermissions(context: Context) = PERMISSIONS_REQUIRED.all {
     ContextCompat.checkSelfPermission(context, it) == PackageManager.PERMISSION_GRANTED
 }

 private val activityResultLauncher =
     registerForActivityResult(ActivityResultContracts.RequestMultiplePermissions())
     { permissions ->
         // Handle Permission granted/rejected
         var permissionGranted = true
         permissions.entries.forEach {
             if (it.key in PERMISSIONS_REQUIRED && it.value == false)
                 permissionGranted = false
         }
         if (!permissionGranted) {
             Toast.makeText(this, "Permission request denied", Toast.LENGTH_LONG).show()
         }
     }

Create a Fullscreen Activity for Video Recording

  1. Create a new empty activity named CameraActivity

  2. Add CameraX’s preview control in the XML file:

     <androidx.camera.view.PreviewView
     android:id="@+id/previewView"
     android:background="@color/purple_200"
     android:layout_width="0dp"
     android:layout_height="0dp"
     app:layout_constraintStart_toStartOf="parent"
     app:layout_constraintEnd_toEndOf="parent"
     app:layout_constraintTop_toTopOf="parent"
     app:layout_constraintBottom_toBottomOf="parent"
     app:layout_constraintDimensionRatio="V,9:16">

 </androidx.camera.view.PreviewView>

  3. Make the activity full screen:

     override fun onCreate(savedInstanceState: Bundle?) {
     super.onCreate(savedInstanceState)
     setContentView(R.layout.activity_camera)
     val decorView: View = window.decorView
     decorView.systemUiVisibility = (View.SYSTEM_UI_FLAG_LAYOUT_STABLE
             or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION
             or View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN
             or View.SYSTEM_UI_FLAG_HIDE_NAVIGATION
             or View.SYSTEM_UI_FLAG_FULLSCREEN
             or View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY)

 }

A button in MainActivity is used to launch the CameraActivity.

Start Camera Preview

Let’s display the camera preview in CameraActivity first.

@SuppressLint("UnsafeOptInUsageError")
private suspend fun bindCaptureUsecase() {
    val cameraProvider = ProcessCameraProvider.getInstance(this).await()

    val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA

    var previewView = findViewById<PreviewView>(R.id.previewView)
    previewView.updateLayoutParams<ConstraintLayout.LayoutParams> {
        val orientation = baseContext.resources.configuration.orientation
        if (orientation == Configuration.ORIENTATION_PORTRAIT) {
            dimensionRatio = "V,9:16"
        }else{
            dimensionRatio = "H,16:9"
        }
    }

    val previewBuilder = Preview.Builder()
    previewBuilder.setTargetAspectRatio(AspectRatio.RATIO_16_9)
    val preview = previewBuilder.build().apply {
        setSurfaceProvider(previewView.surfaceProvider)
    }
    try {
        cameraProvider.unbindAll()
        var camera = cameraProvider.bindToLifecycle(
            this,
            cameraSelector,
            preview
        )
    } catch (exc: Exception) {
        exc.printStackTrace()
        Toast.makeText(context,exc.localizedMessage,Toast.LENGTH_LONG).show()
        goBack()
    }
}

The PreviewView’s layout has to be adjusted according to the orientation.

Record Video with the VideoCapture Use Case

Let’s define a VideoCapture use case to add recording function.

  1. Create a Recorder and specifies the output video quality.

     var quality = Quality.HD
 val qualitySelector = QualitySelector.from(quality)
 var recorderBuilder = Recorder.Builder()
 recorderBuilder.setQualitySelector(qualitySelector)
 val recorder = recorderBuilder.build()

  2. Create a VideoCapture use case using the Recorder.

     videoCapture = VideoCapture.withOutput(recorder)

  3. Add the VideoCapture use case:

    var camera = cameraProvider.bindToLifecycle(
   this,
   cameraSelector,
+  videoCapture,
   preview
)

  4. Start recording and save the output as a file:

     @SuppressLint("MissingPermission")
 private fun startRecording() {
     // create MediaStoreOutputOptions for our recorder: resulting our recording!
     val name = "CameraX-recording-" +
             SimpleDateFormat(FILENAMEFORMAT, Locale.US)
                 .format(System.currentTimeMillis()) + ".mp4"
     val contentValues = ContentValues().apply {
         put(MediaStore.Video.Media.DISPLAY_NAME, name)
     }
     val mediaStoreOutput = MediaStoreOutputOptions.Builder(
         this.contentResolver,
         MediaStore.Video.Media.EXTERNAL_CONTENT_URI)
         .setContentValues(contentValues)
         .build()

     // configure Recorder and Start recording to the mediaStoreOutput.

     currentRecording = videoCapture.output
         .prepareRecording(this, mediaStoreOutput)
         .apply { if (audioEnabled) withAudioEnabled() }
         .start(mainThreadExecutor, captureListener)
 }

  5. If we need to stop recording, just stop it:

     currentRecording!!.stop()

Read Barcodes and QR Codes from Video Files

Next, we can record a video containing a QR code and read QR codes from it.

Decode while Playing to Emulate Live Scan

We can use VideoView to play the video and then use PixelCopy to take a snapshot of the current frame for decoding. In this way, we can emulate live scan.

  1. Play the video.

     videoView.setVideoURI(uri)
 videoView.start()

  2. Use PixelCopy to take a snapshot.

     /**
  * Pixel copy to copy SurfaceView/VideoView into BitMap
  * source: https://stackoverflow.com/questions/27434087/how-to-capture-screenshot-or-video-frame-of-videoview-in-android
  */
 @RequiresApi(Build.VERSION_CODES.N)
 fun usePixelCopy(videoView: SurfaceView, callback: (Bitmap?) -> Unit) {
     val bitmap: Bitmap = Bitmap.createBitmap(
         videoView.width,
         videoView.height,
         Bitmap.Config.ARGB_8888
     );
     try {
         // Create a handler thread to offload the processing of the image.
         val handlerThread = HandlerThread("PixelCopier");
         handlerThread.start();
         PixelCopy.request(
             videoView, bitmap,
             PixelCopy.OnPixelCopyFinishedListener { copyResult ->
                 if (copyResult == PixelCopy.SUCCESS) {
                     callback(bitmap)
                 }else{
                     decoding = false
                 }
                 handlerThread.quitSafely();
             },
             Handler(handlerThread.looper)
         )
     } catch (e: IllegalArgumentException) {
         callback(null)
         // PixelCopy may throw IllegalArgumentException, make sure to handle it
         e.printStackTrace()
     }
 }

  3. Start a timer to take snapshots and decode. A decoding property is used to decide whether to decode based on the status of the previous task.

     @RequiresApi(Build.VERSION_CODES.N)
 private fun decodeVideo(){
     val timer = Timer()
     timer.scheduleAtFixedRate(timerTask {
         try {
             if (videoView.isPlaying) {
                 if (decoding == false) {
                     decoding = true
                     usePixelCopy(videoView){ bitmap: Bitmap? ->

                         val bm = rotateBitmaptoFitScreen(bitmap!!)
                         val textResults = decodeBitmap(bm)
                         decoding = false
                            
                     }
                 }
             }
         }catch (exc:Exception) {
             exc.printStackTrace()
         }
     },0,2)
     videoView.start()
 }

  4. Dynamsoft Barcode Reader is used to decode the bitmap.

     private fun decodeBitmap(bm:Bitmap):ArrayList<String> {
     val results:ArrayList<String> = ArrayList<String>()
     val textResults = reader.decodeBufferedImage(bm)
     for (tr in textResults) {
         results.add(tr.barcodeText)
     }
     return results
 }

PS: How to add Dynamsoft Barcode Reader to the project and initialize a BarcodeReader instance.

  1. Add the following to the project’s build.gradle.

     allprojects {
     repositories {
         maven {
             url "https://download2.dynamsoft.com/maven/aar"
         }
     }
 }

  2. Add the following to the app’s build.gradle.

     implementation 'com.dynamsoft:dynamsoftbarcodereader:9.6.40@aar'

  3. In the VideoView’s activity, create an instance of BarcodeReader and set up its runtime settings for video decoding of EAN13 and QR codes.

     private lateinit var reader: BarcodeReader

 private fun initDBR(){
     reader = BarcodeReader()
     reader.updateRuntimeSettings(EnumPresetTemplate.VIDEO_SINGLE_BARCODE)
     val settings = reader.runtimeSettings
     settings.barcodeFormatIds = EnumBarcodeFormat.BF_EAN_13 or EnumBarcodeFormat.BF_QR_CODE
     reader.updateRuntimeSettings(settings)
 }

  4. Dynamsoft Barcode Reader requires a license to use. You can apply a license here and use it in the following way:

     BarcodeReader.initLicense("DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ=="
 ) { isSuccessful, e ->
     if (!isSuccessful) {
         e.printStackTrace()
     }
 }

Use FFmpeg to Grab Every Frame and Then Decode

We can also grab every video frame and then decode. Since Android does not provide a built-in API for this, we can use FFmpeg to do this.

  1. Install javacv which provides a FFMpeg library for Android.

    Add the following the the app’s build.gradle:

     implementation group: 'org.bytedeco', name: 'javacv', version: "1.5.7"
 implementation group: 'org.bytedeco', name: 'ffmpeg', version: '5.0-1.5.7'
 implementation group: 'org.bytedeco', name: 'ffmpeg', version: '5.0-1.5.7', classifier: 'android-arm64'
 implementation group: 'org.bytedeco', name: 'ffmpeg', version: '5.0-1.5.7', classifier: 'android-x86_64'

    Here, we only add the required architectures.

  2. Use FFMpegFrameGrabber to grab every frame from video and decode. The video frame may be horizontal, which needs rotation if the video is shot with the phone in portrait mode.

     val inputStream: InputStream? = contentResolver.openInputStream(uri)
 val frameGrabber = FFmpegFrameGrabber(inputStream)
 frameGrabber.start()
 val totalFrames = frameGrabber.lengthInVideoFrames
 //val totalFrames = 2
 val th = thread(start=true) {
     for (i in 0..totalFrames-1) {
         val frame = frameGrabber.grabFrame()
         var bm = AndroidFrameConverter().convert(frame)
         bm = rotateBitmaptoFitScreen(bm)
         decodeBitmap(bm)
     }
     frameGrabber.close()
 }

Reading Test

Let’s run a reading test on the following 5-second QR code video.

Result in video mode:

Item Statistics
First Video Position with Barcodes Found (ms) 143
Frames with Barcode Found/Frames Processed in Video Mode 48/49
First Barcode Result dynamsoft

Result in frame mode:

Item Statistics
First Video Frame Index with Barcodes Found 1
Frames with Barcode Found/Frames Processed in Frame Mode 139/140
First Barcode Result dynamsoft

We can see that Dynamsoft Barcode Reader has a good reading rate on this test video.

Source Code

https://github.com/xulihang/CameraXVideo