How to Get Camera Frames for Image Processing in .NET MAUI Windows App
Last week, we demonstrated how to create a Windows .NET MAUI app that can access the USB camera using Blazor web view. Although this solution provides a way to access the camera, for a Windows desktop application that requires optimal performance, it is recommended to prioritize native camera APIs over web APIs. In this article, we will guide you through the process of creating a .NET MAUI Windows application in C# that can capture camera frames for barcode and QR code scanning.
Required NuGet Packages
- OpenCvSharp4: A .NET wrapper for OpenCV 4.x. It is used to capture camera frames.
- OpenCvSharp4.runtime.win: Windows native binaries for OpenCvSharp4.
- BarcodeQRCodeSDK: A .NET wrapper for Dynamsoft Barcode Reader SDK. It is used to detect 1D and 2D barcodes in camera frames.
- SkiaSharp.Views.Maui.Controls: A set of views that can be used to draw on the screen.
Configure the above packages in your csproj
file:
<ItemGroup>
<PackageReference Include="BarcodeQRCodeSDK" Version="2.3.4" />
<PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.3" />
</ItemGroup>
<ItemGroup Condition="$([MSBuild]::IsOSPlatform('windows'))">
<PackageReference Include="OpenCvSharp4" Version="4.6.0.20220608" />
<PackageReference Include="OpenCvSharp4.runtime.win" Version="4.6.0.20220608" />
</ItemGroup>
The packages for Windows only are manually added to the ItemGroup
with the condition $([MSBuild]::IsOSPlatform('windows'))
.
Steps to Capture Camera Frames for Barcode Detection in .NET MAUI Windows
We create a new MAUI content page and add the SKCanvasView
to the page. The SKCanvasView
is used to draw the camera frames on the screen.
<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
xmlns:skia="clr-namespace:SkiaSharp.Views.Maui.Controls;assembly=SkiaSharp.Views.Maui.Controls"
x:Class="BarcodeQrScanner.DesktopCameraPage"
Title="DesktopCameraPage">
<Grid x:Name="scannerView" Margin="0">
<skia:SKCanvasView x:Name="canvasView"
Margin="0"
HorizontalOptions="FillAndExpand" VerticalOptions="FillAndExpand"
PaintSurface="OnCanvasViewPaintSurface" />
</Grid>
</ContentPage>
Afterwards, instantiate the barcode SDK and register the OnDisappearing
and OnAppearing
callback functions in the constructor:
public partial class DesktopCameraPage : ContentPage
{
private BarcodeQRCodeReader reader;
private Thread thread;
private volatile bool isCapturing;
private VideoCapture capture;
private ConcurrentQueue<SKBitmap> _bitmapQueue = new ConcurrentQueue<SKBitmap>();
private SKBitmap _bitmap;
private static object lockObject = new object();
public DesktopCameraPage()
{
InitializeComponent();
this.Disappearing += OnDisappearing;
this.Appearing += OnAppearing;
reader = BarcodeQRCodeReader.Create();
}
}
The two callback functions are used to start and stop the camera capture thread when the page is shown or hidden. For example, when you press the back button in the navigation bar, the OnDisappearing
function will be called to release the camera resources.
private void OnAppearing(object sender, EventArgs e)
{
Create();
}
private void OnDisappearing(object sender, EventArgs e)
{
Destroy();
}
private void Create()
{
lock (lockObject)
{
capture = new VideoCapture(0);
if (capture.IsOpened())
{
isCapturing = true;
thread = new Thread(new ThreadStart(FrameCallback));
thread.Start();
}
}
}
private void Destroy()
{
lock (lockObject)
{
if (thread != null)
{
isCapturing = false;
thread.Join();
thread = null;
}
if (capture != null && capture.IsOpened())
{
capture.Release();
capture = null;
}
ClearQueue();
if (_bitmap != null)
{
_bitmap.Dispose();
_bitmap = null;
}
}
}
As the camera capture thread is started, it will call the FrameCallback
function to capture camera frames in a loop.
private void FrameCallback()
{
while (isCapturing)
{
Decode();
}
}
What does the Decode()
function do:
-
Capture a camera frame to
Mat
using OpenCVVideoCapture.Read()
:Mat mat = new Mat(); capture.Read(mat);
-
Convert the
Mat
tobyte[]
:int length = mat.Cols * mat.Rows * mat.ElemSize(); if (length == 0) return; byte[] bytes = new byte[length]; Marshal.Copy(mat.Data, bytes, 0, length);
-
Decode barcodes from the
byte[]
using the barcode SDK:BarcodeQRCodeReader.Result[] results = reader.DecodeBuffer(bytes, mat.Cols, mat.Rows, (int)mat.Step(), BarcodeQRCodeReader.ImagePixelFormat.IPF_RGB_888);
-
Draw the barcode contours and text results on the camera frame using OpenCV
Cv2.DrawContours
andCv2.PutText
:if (results != null) { foreach (BarcodeQRCodeReader.Result result in results) { int[] points = result.Points; if (points != null) { OpenCvSharp.Point[] all = new OpenCvSharp.Point[4]; int xMin = points[0], yMax = points[1]; all[0] = new OpenCvSharp.Point(xMin, yMax); for (int i = 2; i < 7; i += 2) { int x = points[i]; int y = points[i + 1]; OpenCvSharp.Point p = new OpenCvSharp.Point(x, y); xMin = x < xMin ? x : xMin; yMax = y > yMax ? y : yMax; all[i / 2] = p; } OpenCvSharp.Point[][] contours = new OpenCvSharp.Point[][] { all }; Cv2.DrawContours(mat, contours, 0, new Scalar(0, 0, 255), 2); if (result.Text != null) Cv2.PutText(mat, result.Text, new OpenCvSharp.Point(xMin, yMax), HersheyFonts.HersheySimplex, 1, new Scalar(0, 0, 255), 2); } } }
-
Convert the color space of the frame from
BGR
toRGBA
:Mat newFrame = new Mat(); Cv2.CvtColor(mat, newFrame, ColorConversionCodes.BGR2RGBA);
- Convert
Mat
toSKBitmap
:SKBitmap bitmap = new SKBitmap(mat.Cols, mat.Rows, SKColorType.Rgba8888, SKAlphaType.Premul); bitmap.SetPixels(newFrame.Data);
-
Queue the
SKBitmap
to the_bitmapQueue
and invalidate theSKCanvasView
:if (_bitmapQueue.Count == 2) ClearQueue(); _bitmapQueue.Enqueue(bitmap); canvasView.InvalidateSurface();
After the InvalidateSurface()
is called, the OnCanvasViewPaintSurface
function will be triggered to draw the SKBitmap
on the screen:
void OnCanvasViewPaintSurface(object sender, SKPaintSurfaceEventArgs args)
{
lock (lockObject)
{
if (!isCapturing) return;
SKSurface surface = args.Surface;
SKCanvas canvas = surface.Canvas;
canvas.Clear();
_bitmapQueue.TryDequeue(out _bitmap);
if (_bitmap != null)
{
try
{
canvas.DrawBitmap(_bitmap, new SKPoint(0, 0));
}
catch(Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
_bitmap.Dispose();
_bitmap = null;
}
}
}
}
Source Code
https://github.com/yushulx/dotnet-barcode-qr-code-sdk/tree/main/example/maui