Python Ctypes: Invoking C/C++ Shared Library and Native Threading

Python is renowned for its simplicity and ease of use, but when it comes to interacting with C-based shared libraries, Python’s high-level nature may present limitations. To overcome these limitations, developers often need to leverage C extensions or modules to access low-level functionalities. Dynamsoft’s Python Barcode SDK is an example of a CPython extension, specifically built around its C/C++ Barcode SDK, delivering high performance and precision. Python’s Ctypes module offers a different approach that enables direct interaction with low-level C APIs from Python. This article delves into how to use Ctypes to invoke the Dynamsoft C++ Barcode SDK, and further explores how to bypass Python’s Global Interpreter Lock (GIL) by employing native threading.

Prerequisites

Attempting to Call the C API of Dynamsoft Barcode SDK Directly via Ctypes

In this section, we will demonstrate how to write a simple barcode reading application by directly calling the C API of the Dynamsoft Barcode SDK using Ctypes. The goal is to decode a barcode image and print the barcode format and text to the console. To follow along, you‘ll need to reference DynamsoftBarcodeReader.h to understand the C API functions.

  1. First, load the shared library using windll for Windows and CDLL for Linux:

     import os
     import platform
     from ctypes import *
        
     dbr = None
     if 'Windows' in system:         
         dll_path = license_dll_path = os.path.join(os.path.abspath(
             '.'), r'..\..\lib\win\DynamsoftBarcodeReaderx64.dll')
         dbr = windll.LoadLibrary(dll_path)
     else:
         dbr = CDLL(os.path.join(os.path.abspath('.'),
                                 '../../lib/linux/libDynamsoftBarcodeReader.so'))
    
  2. Next, set the license key and initialize the Dynamsoft Barcode Reader:

     license_key = b"LICENSE-KEY"
     error_msg_buffer = create_string_buffer(256)
     error_msg_buffer_len = len(error_msg_buffer)
     ret = dbr.DBR_InitLicense(license_key, error_msg_buffer, error_msg_buffer_len)
     print('initLicense: {}'.format(ret))
    
     DBR_CreateInstance = dbr.DBR_CreateInstance
     DBR_CreateInstance.restype = c_void_p
     instance = dbr.DBR_CreateInstance()
    
  3. Decode a barcode image file:

     DBR_DecodeFile = dbr.DBR_DecodeFile
     DBR_DecodeFile.argtypes = [c_void_p, c_char_p, c_char_p]
     DBR_DecodeFile.restype = c_int
     ret = DBR_DecodeFile(instance, c_char_p(
         'test.png'.encode('utf-8')), c_char_p(''.encode('utf-8')))
     print('DBR_DecodeFile: {}'.format(ret))
    
  4. Define the necessary C structures in Python to handle the decoding results:

     class SamplingImageData(Structure):
     _fields_ = [
         ("bytes", POINTER(c_ubyte)),
         ("width", c_int),
         ("height", c_int)
     ]
    
    
     class LocalizationResult(Structure):
         _fields_ = [
             ("terminatePhase", c_int),
             ("barcodeFormat", c_int),
             ("barcodeFormatString", c_char_p),
             ("barcodeFormat_2", c_int),
             ("barcodeFormatString_2", c_char_p),
             ("x1", c_int),
             ("y1", c_int),
             ("x2", c_int),
             ("y2", c_int),
             ("x3", c_int),
             ("y3", c_int),
             ("x4", c_int),
             ("y4", c_int),
             ("angle", c_int),
             ("moduleSize", c_int),
             ("pageNumber", c_int),
             ("regionName", c_char_p),
             ("documentName", c_char_p),
             ("resultCoordinateType", c_int),
             ("accompanyingTextBytes", c_char_p),
             ("accompanyingTextBytesLength", c_int),
             ("confidence", c_int),
             ("transformationMatrix", c_double * 9),
             ("reserved", c_char * 52)
         ]
        
        
     class ExtendedResult(Structure):
         _fields_ = [
             ("resultType", c_int),
             ("barcodeFormat", c_int),
             ("barcodeFormatString", c_char_p),
             ("barcodeFormat_2", c_int),
             ("barcodeFormatString_2", c_char_p),
             ("confidence", c_int),
             ("bytes", POINTER(c_ubyte)),
             ("bytesLength", c_int),
             ("accompanyingTextBytes", POINTER(c_ubyte)),
             ("accompanyingTextBytesLength", c_int),
             ("deformation", c_int),
             ("detailedResult", c_void_p),
             ("samplingImage", SamplingImageData),
             ("clarity", c_int),
             ("reserved", c_char * 40)
         ]
        
        
     class TextResult(Structure):
         _fields_ = [
             ("barcodeFormat", c_int),
             ("barcodeFormatString", c_char_p),
             ("barcodeFormat_2", c_int),
             ("barcodeFormatString_2", c_char_p),
             ("barcodeText", c_char_p),
             ("barcodeBytes", POINTER(c_ubyte)),
             ("barcodeBytesLength", c_int),
             ("localizationResult", POINTER(LocalizationResult)),
             ("detailedResult", c_void_p),
             ("resultsCount", c_int),
             ("results", POINTER(POINTER(ExtendedResult))),
             ("exception", c_char_p),
             ("isDPM", c_int),
             ("isMirrored", c_int),
             ("reserved", c_char * 44)
         ]
        
        
     class TextResultArray(Structure):
         _fields_ = [
             ("resultsCount", c_int),
             ("results", POINTER(POINTER(TextResult)))
         ]
    
  5. Finally, retrieve and print the barcode results:

     pResults = POINTER(TextResultArray)()
     DBR_GetAllTextResults = dbr.DBR_GetAllTextResults
     DBR_GetAllTextResults.argtypes = [c_void_p, POINTER(POINTER(TextResultArray))]
     DBR_GetAllTextResults.restype = c_int
        
        
     ret = DBR_GetAllTextResults(instance, byref(pResults))
     print('DBR_GetAllTextResults: {}'.format(ret))
        
     if ret != 0 or pResults.contents.resultsCount == 0:
         print("No barcode found.")
     else:
         print(f"Total barcode(s) found: {pResults.contents.resultsCount}")
         for i in range(pResults.contents.resultsCount):
             result = pResults.contents.results[i]
             print(result)
             print(f"Barcode {i+1}:")
             # crash
             print(result.contents)
             print(f"  Type: {result.contents.barcodeFormatString.decode('utf-8')}")
             print(f"  Text: {result.contents.barcodeText.decode('utf-8')}")
    

    Unfortunately, the program might crash with a segmentation fault error when trying to print the barcode format and text.

    Python Ctypes segmentation fault

Resolving the Issue with a C Bridging Library

To resolve the issue, we can create a C bridging library to simplify the output data structures and memory management. Additionally, this approach allows us to use native threading to bypass Python’s Global Interpreter Lock (GIL) for improved performance.

Bridging Code for Ctypes and Dynamsoft C/C++ Barcode SDK

We create a CMake library project named bridge, which contains bridge.cpp, bridge.h and CMakeLists.txt. The CMakeLists.txt file is as follows:

cmake_minimum_required(VERSION 3.0.0)

project(bridge VERSION 0.1.0)

INCLUDE_DIRECTORIES("${CMAKE_CURRENT_SOURCE_DIR}" "${CMAKE_CURRENT_SOURCE_DIR}/../../../include")
if (CMAKE_HOST_WIN32)
    LINK_DIRECTORIES("${CMAKE_CURRENT_SOURCE_DIR}/../../../lib/Win")
else()
    LINK_DIRECTORIES("${CMAKE_CURRENT_SOURCE_DIR}/../../../lib/linux")
endif()
add_library(${PROJECT_NAME} SHARED bridge.cpp)

if(CMAKE_HOST_WIN32)
    target_link_libraries (${PROJECT_NAME} "DBRx64")
else()
    target_link_libraries (${PROJECT_NAME} "DynamsoftBarcodeReader")
endif()

  • bridge.cpp is the main source file for the bridging library. It includes the header file bridge.h and the Dynamsoft Barcode Reader header file DynamsoftBarcodeReader.h.
  • The bridge.h file defines the C structures and functions that will be used by Python Ctypes.

The bridge.h file is as follows:

# include "DynamsoftBarcodeReader.h"

#if !defined(_WIN32) && !defined(_WIN64)
#define EXPORT_API
#else
#define EXPORT_API __declspec(dllexport)
#endif

typedef struct {
    char* format;
    char* text;
} ResultInfo;

typedef struct {
    int size;
    ResultInfo** pResultInfo;
} ResultList;

typedef int (*callback_t)(ResultList*);

#ifdef __cplusplus
extern "C" {
#endif

EXPORT_API ResultList* dbr_get_results(void* barcodeReader);
EXPORT_API void dbr_free_results(ResultList* resultList);
EXPORT_API void thread_decode(void* barcodeReader, const char *fileName);
EXPORT_API int registerCallback(callback_t foo);

#ifdef __cplusplus
}
#endif

Explanation

  • ResultInfo: A structure that contains the barcode format and text.
  • ResultList: A structure that contains the size of the result list and a pointer to an array of ResultInfo structures.
  • dbr_get_results: A function that retrieves the barcode decoding results, replacing DBR_GetAllTextResults.
  • dbr_free_results: A function that frees the memory allocated for the barcode decoding results.
  • thread_decode: A function that decodes a barcode image file in a separate thread.
  • registerCallback: A function that registers a Python callback function for receiving the barcode decoding results.

In the bridge.cpp file, we add the implementation of the functions defined in bridge.h:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "bridge.h"
#include <thread>

using namespace std;

callback_t callback = NULL;
thread t;

ResultList *dbr_get_results(void *barcodeReader)
{
    TextResultArray *pResults;
    int ret = DBR_GetAllTextResults(barcodeReader, &pResults);
    int count = pResults->resultsCount;
    TextResult **results = pResults->results;

    ResultInfo **pResultInfo = (ResultInfo **)malloc(sizeof(ResultInfo *) * count);
    ResultList *resultList = (ResultList *)malloc(sizeof(ResultList));
    resultList->size = count;
    resultList->pResultInfo = pResultInfo;

    for (int i = 0; i < count; i++)
    {
        TextResult *pResult = results[i];
        ResultInfo *pInfo = (ResultInfo *)malloc(sizeof(ResultInfo));
        pInfo->format = NULL;
        pInfo->text = NULL;
        pResultInfo[i] = pInfo;
        // printf("Barcode format: %s, text: %s\n", pResult->barcodeFormatString, pResult->barcodeText);
        pInfo->format = (char *)calloc(strlen(pResult->barcodeFormatString) + 1, sizeof(char));
        strncpy(pInfo->format, pResult->barcodeFormatString, strlen(pResult->barcodeFormatString));
        pInfo->text = (char *)calloc(strlen(pResult->barcodeText) + 1, sizeof(char));
        strncpy(pInfo->text, pResult->barcodeText, strlen(pResult->barcodeText));
    }

    DBR_FreeTextResults(&pResults);

    return resultList;
}

void dbr_free_results(ResultList *resultList)
{
    int count = resultList->size;
    ResultInfo **pResultInfo = resultList->pResultInfo;

    for (int i = 0; i < count; i++)
    {
        ResultInfo *resultList = pResultInfo[i];
        if (resultList)
        {
            if (resultList->format != NULL)
                free(resultList->format);
            if (resultList->text != NULL)
                free(resultList->text);

            free(resultList);
        }
    }

    if (pResultInfo != NULL)
        free(pResultInfo);
}

void thread_func(void *barcodeReader, const char *fileName)
{
    DBR_DecodeFile(barcodeReader, fileName, "");

    TextResultArray *pResults;
    int ret = DBR_GetAllTextResults(barcodeReader, &pResults);
    int count = pResults->resultsCount;
    TextResult **results = pResults->results;

    ResultInfo **pResultInfo = (ResultInfo **)malloc(sizeof(ResultInfo *) * count);
    ResultList *resultList = (ResultList *)malloc(sizeof(ResultList));
    resultList->size = count;
    resultList->pResultInfo = pResultInfo;

    for (int i = 0; i < count; i++)
    {
        TextResult *pResult = results[i];
        ResultInfo *pInfo = (ResultInfo *)malloc(sizeof(ResultInfo));
        pInfo->format = NULL;
        pInfo->text = NULL;
        pResultInfo[i] = pInfo;
        // printf("Barcode format: %s, text: %s\n", pResult->barcodeFormatString, pResult->barcodeText);
        pInfo->format = (char *)calloc(strlen(pResult->barcodeFormatString) + 1, sizeof(char));
        strncpy(pInfo->format, pResult->barcodeFormatString, strlen(pResult->barcodeFormatString));
        pInfo->text = (char *)calloc(strlen(pResult->barcodeText) + 1, sizeof(char));
        strncpy(pInfo->text, pResult->barcodeText, strlen(pResult->barcodeText));
    }

    DBR_FreeTextResults(&pResults);

    if (callback)
    {
        int res = callback(resultList);
    }
}

void thread_decode(void *barcodeReader, const char *fileName)
{
    t = thread(thread_func, barcodeReader, fileName);
    t.join();
}

int registerCallback(callback_t foo)
{
    callback = foo;
    return 0;
}

Building the Bridge Library

To build the bridge library:

On Windows:

cd bridge && mkdir build && cd build
cmake -DCMAKE_GENERATOR_PLATFORM=x64 ..
cmake --build .

On Linux:

cd bridge && mkdir build && cd build
cmake ..
cmake --build .

Loading the Bridge Library in Python

Update the Python code to load the bridge library:

dbr = None
bridge = None
if 'Windows' in system:
    dll_path = license_dll_path = os.path.join(os.path.abspath(
        '.'), r'..\..\lib\win\DynamsoftBarcodeReaderx64.dll')

    dbr = windll.LoadLibrary(dll_path)

    bridge = windll.LoadLibrary(os.path.join(
        os.path.abspath('.'), r'bridge\build\Debug\bridge.dll'))
else:
    dbr = CDLL(os.path.join(os.path.abspath('.'),
                            '../../lib/linux/libDynamsoftBarcodeReader.so'))
    bridge = CDLL(os.path.join(os.path.abspath(
        '.'), 'bridge/build/libbridge.so'))

Note: The library loading sequence is vital for Linux: load libDynamsoftBarcodeReader.so first, followed by libbridge.so. Incorrect order may cause the Python code to fail on Linux.

Updated Python Code with Bridge Library

Here is the updated Python code to decode a barcode image file and print the barcode format and text using the bridge library:

import os
import platform
from ctypes import *
import threading


class ResultInfo(Structure):
    _fields_ = [("format", c_char_p), ("text", c_char_p)]

class ResultList(Structure):
    _fields_ = [("size", c_int), ("pResultInfo", POINTER(POINTER(ResultInfo)))]


system = platform.system()

dbr = None
bridge = None
if 'Windows' in system:
    dll_path = license_dll_path = os.path.join(os.path.abspath(
        '.'), r'..\..\lib\win\DynamsoftBarcodeReaderx64.dll')

    dbr = windll.LoadLibrary(dll_path)

    bridge = windll.LoadLibrary(os.path.join(
        os.path.abspath('.'), r'bridge\build\Debug\bridge.dll'))
else:
    dbr = CDLL(os.path.join(os.path.abspath('.'),
                            '../../lib/linux/libDynamsoftBarcodeReader.so'))
    bridge = CDLL(os.path.join(os.path.abspath(
        '.'), 'bridge/build/libbridge.so'))

# DBR_InitLicense
DBR_InitLicense = dbr.DBR_InitLicense
DBR_InitLicense.argtypes = [c_char_p, c_char_p, c_int]
DBR_InitLicense.restype = c_int

license_key = b"LICENSE-KEY"
error_msg_buffer = create_string_buffer(256)
error_msg_buffer_len = len(error_msg_buffer)
# https://www.dynamsoft.com/customer/license/trialLicense/?product=dcv&package=cross-platform
ret = DBR_InitLicense(license_key, error_msg_buffer, error_msg_buffer_len)

# DBR_CreateInstance
DBR_CreateInstance = dbr.DBR_CreateInstance
DBR_CreateInstance.restype = c_void_p
instance = dbr.DBR_CreateInstance()

######################################################################
# Call decoding method in native thread.


@CFUNCTYPE(None, POINTER(ResultList))
def callback(address):
    data = cast(address, POINTER(ResultList))
    size = data.contents.size
    results = data.contents.pResultInfo
    for i in range(size):
        result = results[i]
        print('Format: %s' % result.contents.format.decode('utf-8'))
        print('Text: %s' % result.contents.text.decode('utf-8'))

    dbr_free_results = bridge.dbr_free_results
    dbr_free_results.argtypes = [c_void_p]
    if bool(address):
        dbr_free_results(address)
        DBR_DestroyInstance = dbr.DBR_DestroyInstance
        DBR_DestroyInstance.argtypes = [c_void_p]
        DBR_DestroyInstance(instance)
    return 0


def run():
    print("Python thread" + str(threading.current_thread()))
    bridge.registerCallback(callback)
    thread_decode = bridge.thread_decode
    thread_decode.argtypes = [c_void_p, c_void_p]
    thread_decode(instance, c_char_p('test.png'.encode('utf-8')))


t = threading.Thread(target=run)
t.start()
t.join()

This code uses native threading via ctypes to bypass Python’s Global Interpreter Lock (GIL), allowing the barcode decoding to run in a separate thread for better performance. This approach can be particularly useful when dealing with CPU-intensive tasks or I/O-bound operations.

Run the Python code to decode a barcode image file and observe the results printed to the console:

Python Ctypes barcode detection

Drawbacks of Using Ctypes

While Ctypes offers flexibility and ease of access to C-based shared libraries, it also comes with several drawbacks:

  • Performance Overhead: Ctypes may introduce more overhead compared to C extensions like CPython, particularly for frequently called functions or intensive computations. This is because Ctypes operates through a higher level of abstraction and incurs additional processing costs when converting between Python and C types.
  • Manual Memory Management: Ctypes requires the developer to manage memory manually. Incorrect memory handling, such as failing to free allocated memory or incorrectly managing pointers, can lead to memory leaks or crashes.
  • Limited Error Handling: Error handling in Ctypes is less robust compared to using a native CPython extension. Errors in Ctypes often manifest as segmentation faults or crashes, which can be harder to debug than the exceptions provided by Python.
  • Complex API Usage: Ctypes can be complex and error-prone when dealing with more advanced C features like struct alignment, unions, or callback functions. Developers need to have a solid understanding of both the C API and how to map these to Python using Ctypes.

Conclusion

While ctypes offers a flexible and convenient way to interface with C libraries from Python, it is often better suited for specific scenarios where direct interaction with C functions is necessary. CPython extensions are optimized for performance and provide better integration with Python’s runtime, including more reliable memory management and error handling.

To develop a production-grade Python barcode reader application, it is recommended to use the official CPython extension provided by Dynamsoft.

pip install dbr

Source Code

https://github.com/yushulx/python-barcode-qrcode-sdk/tree/main/examples/ctypes