C# Camera API for Getting Video Frame

If you have a USB camera, how can you build a simple C# camera application on Windows 10? There are three options:  WIA (Windows Imaging Acquisition), DirectShow and MediaCapture. After trying some sample code that downloaded from CodeProject and GitHub, I got the conclusion: 1. WIA is not good because it does not support my webcam.  2. DirectShow can work well, but there is no C# API provided by Microsoft. You need to create a wrapper for C++ API. 3. MediaCapture class that designed for UWP apps provides C# APIs which provide low-level control over the capture pipeline and enable advanced capture scenarios. In this article, I want to share how to create a simple C# webcam app in which I can handle every preview frame myself.

UWP media capture camera preview

Getting Preview Frames via C# Camera API

Although the reference page of MediaCapture class is informative, to quickly learn APIs, I prefer to get started with UWP sample code that provided by Microsoft. Press Ctrl+F to search for camera, you will see following samples:

UWP camera samples

If you want to add custom effects or functionalities (OCR, barcode etc.) for the camera, a common way is to register a callback function for receiving every preview frame. After running all samples, I found the project CameraFrames is what I want.

How to build a UWP webcam app in C#

  1. To use webcam, double-click package.appxmanifest to check the corresponding capability: UWP webcam capability
  2. Create an image element for rendering preview frames: UWP webcam preview image

  3. Initialize FrameRenderer with the image element:

     _frameRenderer = new FrameRenderer(PreviewImage);

  4. Initialize the media capture object:

                     // Create a new media capture object.
                 _mediaCapture = new MediaCapture();
    
                 var settings = new MediaCaptureInitializationSettings()
                 {
                     // Select the source we will be reading from.
                     SourceGroup = groupModel.SourceGroup,
    
                     // This media capture has exclusive control of the source.
                     SharingMode = MediaCaptureSharingMode.ExclusiveControl,
    
                     // Set to CPU to ensure frames always contain CPU SoftwareBitmap images,
                     // instead of preferring GPU D3DSurface images.
                     MemoryPreference = MediaCaptureMemoryPreference.Cpu,
    
                     // Capture only video. Audio device will not be initialized.
                     StreamingCaptureMode = StreamingCaptureMode.Video,
                 };
    
                 try
                 {
                     // Initialize MediaCapture with the specified group.
                     // This can raise an exception if the source no longer exists,
                     // or if the source could not be initialized.
                     await _mediaCapture.InitializeAsync(settings);
                     _logger.Log($"Successfully initialized MediaCapture for {groupModel.DisplayName}");
                 }
                 catch (Exception exception)
                 {
                     _logger.Log(exception.Message);
                     DisposeMediaCapture();
                 }

  5. Use DeviceWatcher to list all connected devices:

             var deviceSelector = MediaFrameSourceGroup.GetDeviceSelector();
         _watcher = DeviceInformation.CreateWatcher(deviceSelector);
         _watcher.Added += Watcher_Added;
         _watcher.Removed += Watcher_Removed;
         _watcher.Updated += Watcher_Updated;
         _watcher.Start();
    
         private async void Watcher_Added(DeviceWatcher sender, DeviceInformation args)
         {
             await AddDeviceAsync(args.Id);
         }
    
         private async Task AddDeviceAsync(string id)
         {
             var group = await MediaFrameSourceGroup.FromIdAsync(id);
             if (group != null)
             {
                 await _dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                     {
                         _sourceCollection.Add(new FrameSourceGroupModel(group));
                     });
             }
         }

  6. Updates the current frame source to the one corresponding to the user’s selection:

     _mediaCapture.FrameSources.TryGetValue(info.SourceInfo.Id, out _source);

  7. Initialize MediaFrameReader and register a callback function:

                 if (_source != null)
             {
                 _reader = await _mediaCapture.CreateFrameReaderAsync(_source);
                 _reader.FrameArrived += Reader_FrameArrived;
             }

  8. Start reading webcam frames:

     MediaFrameReaderStartStatus result = await _reader.StartAsync();

  9. Receive all frames and render them on image element:

             private void Reader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
         {
             // TryAcquireLatestFrame will return the latest frame that has not yet been acquired.
             // This can return null if there is no such frame, or if the reader is not in the
             // "Started" state. The latter can occur if a FrameArrived event was in flight
             // when the reader was stopped.
             using (var frame = sender.TryAcquireLatestFrame())
             {
                 _frameRenderer.ProcessFrame(frame);
             }
         }
    
         public void ProcessFrame(MediaFrameReference frame)
         {
             var softwareBitmap = FrameRenderer.ConvertToDisplayableImage(frame?.VideoMediaFrame);
    
             if (softwareBitmap != null)
             {
                 // Swap the processed frame to _backBuffer and trigger UI thread to render it
                 softwareBitmap = Interlocked.Exchange(ref _backBuffer, softwareBitmap);
    
                 // UI thread always reset _backBuffer before using it.  Unused bitmap should be disposed.
                 softwareBitmap?.Dispose();
    
                 // Changes to xaml ImageElement must happen in UI thread through Dispatcher
                 var task = _imageElement.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
                     async () =>
                     {
                         // Don't let two copies of this task run at the same time.
                         if (_taskRunning)
                         {
                             return;
                         }
                         _taskRunning = true;
    
                         // Keep draining frames from the backbuffer until the backbuffer is empty.
                         SoftwareBitmap latestBitmap;
                         while ((latestBitmap = Interlocked.Exchange(ref _backBuffer, null)) != null)
                         {
                             var imageSource = (SoftwareBitmapSource)_imageElement.Source;
                             await imageSource.SetBitmapAsync(latestBitmap);
                             latestBitmap.Dispose();
                         }
    
                         _taskRunning = false;
                     });
             }
         }

Reference

Basic photo, video, and audio capture with MediaCapture

Source Code

https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraFrames