C# Camera API for Getting Video Frame

If you have a USB camera, how can you build a simple C# camera application on Windows 10? There are three options:  WIA (Windows Imaging Acquisition), DirectShow and MediaCapture. After trying some sample code that downloaded from CodeProject and GitHub, I got the conclusion: 1. WIA is not good because it does not support my webcam.  2. DirectShow can work well, but there is no C# API provided by Microsoft. You need to create a wrapper for C++ API. 3. MediaCapture class that designed for UWP apps provides C# APIs which provide low-level control over the capture pipeline and enable advanced capture scenarios. In this article, I want to share how to create a simple C# webcam app in which I can handle every preview frame myself.

UWP media capture camera preview

Getting Preview Frames via C# Camera API

Although the reference page of MediaCapture class is informative, to quickly learn APIs, I prefer to get started with UWP sample code that provided by Microsoft. Press Ctrl+F to search for camera, you will see following samples:

UWP camera samples

If you want to add custom effects or functionalities (OCR, barcode etc.) for the camera, a common way is to register a callback function for receiving every preview frame. After running all samples, I found the project CameraFrames is what I want.

How to build a UWP webcam app in C#

  1. To use webcam, double-click package.appxmanifest to check the corresponding capability: UWP webcam capability
  2. Create an image element for rendering preview frames: UWP webcam preview image
  3. Initialize FrameRenderer with the image element:

     _frameRenderer = new FrameRenderer(PreviewImage);
    
  4. Initialize the media capture object:

                     // Create a new media capture object.
                     _mediaCapture = new MediaCapture();
        
                     var settings = new MediaCaptureInitializationSettings()
                     {
                         // Select the source we will be reading from.
                         SourceGroup = groupModel.SourceGroup,
        
                         // This media capture has exclusive control of the source.
                         SharingMode = MediaCaptureSharingMode.ExclusiveControl,
        
                         // Set to CPU to ensure frames always contain CPU SoftwareBitmap images,
                         // instead of preferring GPU D3DSurface images.
                         MemoryPreference = MediaCaptureMemoryPreference.Cpu,
        
                         // Capture only video. Audio device will not be initialized.
                         StreamingCaptureMode = StreamingCaptureMode.Video,
                     };
        
                     try
                     {
                         // Initialize MediaCapture with the specified group.
                         // This can raise an exception if the source no longer exists,
                         // or if the source could not be initialized.
                         await _mediaCapture.InitializeAsync(settings);
                         _logger.Log($"Successfully initialized MediaCapture for {groupModel.DisplayName}");
                     }
                     catch (Exception exception)
                     {
                         _logger.Log(exception.Message);
                         DisposeMediaCapture();
                     }
    
  5. Use DeviceWatcher to list all connected devices:

             var deviceSelector = MediaFrameSourceGroup.GetDeviceSelector();
             _watcher = DeviceInformation.CreateWatcher(deviceSelector);
             _watcher.Added += Watcher_Added;
             _watcher.Removed += Watcher_Removed;
             _watcher.Updated += Watcher_Updated;
             _watcher.Start();
        
             private async void Watcher_Added(DeviceWatcher sender, DeviceInformation args)
             {
                 await AddDeviceAsync(args.Id);
             }
        
             private async Task AddDeviceAsync(string id)
             {
                 var group = await MediaFrameSourceGroup.FromIdAsync(id);
                 if (group != null)
                 {
                     await _dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                         {
                             _sourceCollection.Add(new FrameSourceGroupModel(group));
                         });
                 }
             }
        
    
  6. Updates the current frame source to the one corresponding to the user’s selection:

     _mediaCapture.FrameSources.TryGetValue(info.SourceInfo.Id, out _source);
    
  7. Initialize MediaFrameReader and register a callback function:

                 if (_source != null)
                 {
                     _reader = await _mediaCapture.CreateFrameReaderAsync(_source);
                     _reader.FrameArrived += Reader_FrameArrived;
                 }
    
  8. Start reading webcam frames:

     MediaFrameReaderStartStatus result = await _reader.StartAsync();
    
  9. Receive all frames and render them on image element:

             private void Reader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
             {
                 // TryAcquireLatestFrame will return the latest frame that has not yet been acquired.
                 // This can return null if there is no such frame, or if the reader is not in the
                 // "Started" state. The latter can occur if a FrameArrived event was in flight
                 // when the reader was stopped.
                 using (var frame = sender.TryAcquireLatestFrame())
                 {
                     _frameRenderer.ProcessFrame(frame);
                 }
             }
        
             public void ProcessFrame(MediaFrameReference frame)
             {
                 var softwareBitmap = FrameRenderer.ConvertToDisplayableImage(frame?.VideoMediaFrame);
        
                 if (softwareBitmap != null)
                 {
                     // Swap the processed frame to _backBuffer and trigger UI thread to render it
                     softwareBitmap = Interlocked.Exchange(ref _backBuffer, softwareBitmap);
        
                     // UI thread always reset _backBuffer before using it.  Unused bitmap should be disposed.
                     softwareBitmap?.Dispose();
        
                     // Changes to xaml ImageElement must happen in UI thread through Dispatcher
                     var task = _imageElement.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
                         async () =>
                         {
                             // Don't let two copies of this task run at the same time.
                             if (_taskRunning)
                             {
                                 return;
                             }
                             _taskRunning = true;
        
                             // Keep draining frames from the backbuffer until the backbuffer is empty.
                             SoftwareBitmap latestBitmap;
                             while ((latestBitmap = Interlocked.Exchange(ref _backBuffer, null)) != null)
                             {
                                 var imageSource = (SoftwareBitmapSource)_imageElement.Source;
                                 await imageSource.SetBitmapAsync(latestBitmap);
                                 latestBitmap.Dispose();
                             }
        
                             _taskRunning = false;
                         });
                 }
             }
    

Reference

Basic photo, video, and audio capture with MediaCapture

Source Code

https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraFrames

Search Blog Posts