How to Use AppVeyor to Build and Deploy Python Wheels from C/C++ Code
AppVeyor is a continuous integration (CI) service used to automatically build code projects and deploy relevant artifacts. It provides build environments for Windows, Linux, and macOS. In this article, I will share how to use AppVeyor to build and deploy Python Wheels (Windows edition) from C/C++ code.
Creating Python Wheels
It takes two steps to build a Python wheel from CPython code:
- Build *.pyd extension file from C/C++ code.
- Pack *.pyd and dependent *.dll files into a wheel file.
For the past few years, I’ve been maintaining the source code of Python barcode extension based on Dynamsoft Barcode Reader C/C++ SDK. Let’s do some changes based on the repository.
The *.pyd Python module
Get the source code:
git clone https://github.com/yushulx/python.git
Install DynamsoftBarcodeReader7.2.1.exe. Copy *.dll files from Dynamsoft\Barcode Reader 7.2.1\Components\C_C++\Redist\x64 to bin folder and copy DBRx64.lib from Dynamsoft\Barcode Reader 7.2.1\Components\C_C++\Lib\DBRx64.lib to lib folder. Create an empty folder named wheel.
Open src\setup.py to set the link directories:
elif sys.platform == "win32":
# Windows
dbr_lib_name = 'DBRx64'
dbr_lib_dir = r'..\\lib'
dbr_dll = r'..\\bin'
Build the Python barcode module:
cd src
python setup.py build
The *.whl file
Create a folder named dbr under the wheel folder. Add a setup.py file:
from setuptools import setup, find_packages, Distribution
from codecs import open
from os import path
import sys
here = path.abspath(path.dirname(__file__))
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
class BinaryDistribution(Distribution):
"""Distribution which always forces a binary package with platform name"""
def has_ext_modules(foo):
return True
data_info = {'dbr':\['\*.pyd', 'vcomp110.dll', 'DynamicPdfx64.dll', 'DynamsoftBarcodeReaderx64.dll', 'DynamsoftLicClientx64.dll'\]}
setup(
name='dbr',
version='7.2.1',
description='Dynamsoft Barcode Reader Python project',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://www.dynamsoft.com/Products/Dynamic-Barcode-Reader.aspx',
license = 'https://www.dynamsoft.com/Products/barcode-reader-license-agreement.aspx',
author='Dynamsoft',
author_email='support@dynamsoft.com',
classifiers=\[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Topic :: Software Development :: Libraries',
'License :: Other/Proprietary License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: C++',
'Programming Language :: Python :: Implementation :: CPython',
'Operating System :: POSIX :: Linux ',
'Operating System :: Microsoft :: Windows :: Windows 10',
'Topic :: Scientific/Engineering',
'Topic :: Scientific/Engineering :: Image Recognition',
'Topic :: Software Development',
\],
keywords='barcode DataMatrix QRCode 1D PDF417 MaxiCode Aztec',
packages=find_packages(exclude=\['contrib', 'docs', 'tests'\]),
install_requires=\['numpy', 'opencv-python'\],
package_data=data_info,
distclass=BinaryDistribution,
platforms=\['Windows', 'Linux'\]
)
The data_info contains the *.dll and *.whl files.
Create __init__.py under the dbr folder:
from .dbr import \*
Copy *.pyd and *.dll files to the dbr folder from the src folder:
copy ..\\bin\\\*.\* ..\\wheel\\dbr\\
cd build\\lib.win-\*\\
copy \*.\* ..\\..\\..\\wheel\\dbr\\
Build the wheel package under the wheel folder:
python setup.py bdist_wheel
Configuring YAML file for AppVeyor Build Service
Login https://ci.appveyor.com/ with your GitHub account.
Click Projects to import the target repository:
To trigger the build, create an appveyor.yml file under the project root directory.
Set the branch to build:
branches:
only:
- wheel
Add the Python environments:
environment:
matrix:
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
PYTHON: "C:/Python27-x64"
- PYTHON: "C:/Python35-x64"
- PYTHON: "C:/Python36-x64"
- PYTHON: "C:/Python37-x64"
- PYTHON: "C:/Python38-x64"
Why do I use APPVEYOR_BUILD_WORKER_IMAGE for Python 2.7? When building the source code with the default environment, I got the following error message:
Traceback (most recent call last):
File "setup.py", line 63, in <module>
cmdclass={'install': CustomInstall}
File "C:\\Python27-x64\\lib\\distutils\\core.py", line 151, in setup
dist.run_commands()
File "C:\\Python27-x64\\lib\\distutils\\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\\Python27-x64\\lib\\distutils\\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\\Python27-x64\\lib\\distutils\\command\\build.py", line 127, in run
self.run_command(cmd_name)
File "C:\\Python27-x64\\lib\\distutils\\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\\Python27-x64\\lib\\distutils\\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\\Python27-x64\\lib\\distutils\\command\\build_ext.py", line 340, in run
self.build_extensions()
File "C:\\Python27-x64\\lib\\distutils\\command\\build_ext.py", line 449, in build_extensions
self.build_extension(ext)
File "C:\\Python27-x64\\lib\\distutils\\command\\build_ext.py", line 499, in build_extension
depends=ext.depends)
File "C:\\Python27-x64\\lib\\distutils\\msvc9compiler.py", line 473, in compile
self.initialize()
File "C:\\Python27-x64\\lib\\distutils\\msvc9compiler.py", line 383, in initialize
vc_env = query_vcvarsall(VERSION, plat_spec)
File "C:\\Python27-x64\\lib\\distutils\\msvc9compiler.py", line 299, in query_vcvarsall
raise ValueError(str(list(result.keys())))
ValueError: \[u'path'\]
Command exited with code 1
The issue seems caused by Visual Studio 2008. According to the official documentation – Windows images software, Visual C++ 2008 Express is not installed in the Visual Studio 2019 image. So I can use the Visual Studio 2019 image and set Visual Studio 2015 as the build tool to make it work:
install:
- SET VS90COMNTOOLS=%VS140COMNTOOLS%
Add the build script:
build_script:
- cmd: |
"%PYTHON%/python.exe" -m pip install --upgrade pip
"%PYTHON%/python.exe" -m pip install --upgrade setuptools wheel numpy
cd src
"%PYTHON%/python.exe" setup.py build
copy ..\\bin\\\*.\* ..\\wheel\\dbr\\
cd build\\lib.win-\*\\
copy \*.\* ..\\..\\..\\wheel\\dbr\\
cd ..\\..\\..\\wheel\\
"%PYTHON%/python.exe" setup.py bdist_wheel
Set the artifacts:
artifacts:
- path: wheel\\dist\\\*.whl
name: wheels
Add the scripts for PyPi deployment:
deploy_script:
- ps: |
if($env:APPVEYOR_REPO_TAG -eq 'true') {
Write-Output ("Deploying " + $env:APPVEYOR_REPO_TAG_NAME + " to PyPI...")
&"${Env:PYTHON}/python.exe" -m pip install twine
&"${Env:PYTHON}/python.exe" -m twine upload -u ${Env:USER} -p ${Env:PASS} --skip-existing dist/\*.whl
} else {
Write-Output "No tag for deployment"
}
When adding tags to the GitHub repository, the APPVEYOR_REPO_TAG value will be `true’. You can encrypt the username and password of the PyPi account by clicking Account > Encrypt YAML.
When building successfully, the *.whl files for Python 2.7, 3.5, 3.6, 3.7 and 3.8 will be uploaded to the PyPi website.
Reference
- https://www.appveyor.com/docs/deployment/
- https://github.com/skvark/opencv-python/blob/master/appveyor.yml