How to Benchmark Passport MRZ Recognition Accuracy on MIDV-500 with Python
Building an MRZ recognition pipeline is only half the battle — you also need to know how it performs across real-world conditions. This tutorial walks through a Python benchmark that evaluates the Dynamsoft Capture Vision SDK on 3,600 passport images from the MIDV-500 dataset, measuring exact accuracy, success rate, and per-image decode timing across 10 capture conditions.
What you’ll build: A Python command-line tool that benchmarks Dynamsoft Capture Vision MRZ recognition on the MIDV-500 passport dataset, producing a JSON report and an HTML dashboard with accuracy metrics and timing distribution charts.
Key Takeaways
- The Dynamsoft Capture Vision Python SDK achieves 100% exact MRZ accuracy on clean passport images, with an average decode time of 194 milliseconds on a 5-image smoke test.
- Across the full 3,600-image MIDV-500 passport benchmark, the SDK decoded MRZ text from 59.4% of images, including challenging captures with motion blur, extreme angles, and low light.
- The SDK’s P50 decode time is 283 milliseconds per image, with 75.9% of all images decoded in under 500 milliseconds — fast enough for real-time document processing pipelines.
- On the best capture condition subset (TA), the SDK achieved 92.2% success rate and 83.3% exact accuracy, demonstrating that performance scales with image quality.
- The benchmark uses the
ReadPassportAndIdtemplate, which handles TD1, TD2, and TD3 MRZ formats in a single API call viarouter.capture_multi_pages().
Common Developer Questions
How accurate is Dynamsoft Capture Vision for MRZ reading?
On the MIDV-500 passport dataset (3,600 images across 12 passport types and 10 capture conditions), the SDK achieved 44.5% exact MRZ accuracy and 59.4% success rate. On clean, well-lit images, it consistently achieves 100% exact accuracy. The gap between the two numbers reflects the dataset’s inclusion of motion-blurred, angled, and low-light captures that simulate real-world mobile scanning conditions.
What is the MIDV-500 dataset and why use it for benchmarking?
MIDV-500 is a publicly available dataset of 500 video clips covering 50 identity document types, published by Smart Engines. It is available at https://github.com/fcakyon/midv500. The passport subset used in this benchmark contains 3,600 images across 12 passport types (Azerbaijan, Brazil, Czech Republic, Germany, Algeria, Greece, Croatia, Hungary, Latvia, Moldova, Serbia) with 10 capture-condition variants per type, providing a rigorous test of MRZ recognition under varying image quality.
How fast is the Dynamsoft Capture Vision Python SDK for MRZ decoding?
The average decode time across all 3,600 images is 387 milliseconds, with a P50 of 283 ms, P95 of 900 ms, and P99 of 1,271 ms. On clean images, the average drops to 194 ms. The SDK processes images sequentially via router.capture_multi_pages(), and no parallelism is needed for sub-second per-image performance.
Prerequisites
- Python 3.8 or later
dynamsoft-capture-vision-bundle(Dynamsoft Capture Vision SDK for Python, version 3.2.1000+)Pillow(for .tif to .jpg conversion when downloading the raw MIDV-500 dataset)- A valid Dynamsoft license key. Get a 30-day free trial license.
Install the Python dependencies:
pip install -r requirements.txt
Step 1: Initialize the Dynamsoft Capture Vision Router
The first step is to initialize the SDK license and create a CaptureVisionRouter instance. The router is the central object that handles all image recognition tasks. The ReadPassportAndId template is a built-in preset that configures the SDK for passport and ID card MRZ recognition, supporting TD1, TD2, and TD3 document formats.
from dynamsoft_capture_vision_bundle import (
CaptureVisionRouter,
EnumErrorCode,
LicenseManager,
)
DEFAULT_LICENSE = "LICENSE-KEY"
DEFAULT_TEMPLATE = "ReadPassportAndId"
def init_dynamsoft_router(license_key: str) -> CaptureVisionRouter:
"""Initialise the Dynamsoft license and create a CaptureVisionRouter."""
code, message = LicenseManager.init_license(license_key)
if code not in (EnumErrorCode.EC_OK, EnumErrorCode.EC_LICENSE_WARNING):
raise RuntimeError(
f"License initialization failed. code={code}, message={message}"
)
return CaptureVisionRouter()
For production use, replace it with your own license obtained from the Dynamsoft portal.
Step 2: Load Ground-Truth MRZ Labels
The benchmark compares the SDK’s recognized MRZ text against ground-truth labels stored in a CSV file. Each row contains an image path and the corresponding MRZ text. The load_ground_truth function normalises image paths by taking the last two path components (e.g., ca/ca05_01.jpg), so the labels work regardless of where the dataset is stored on disk.
def load_ground_truth(csv_path: Path) -> Dict[str, str]:
"""Load ground-truth MRZ labels from a CSV file."""
if not csv_path.exists():
raise FileNotFoundError(f"CSV not found: {csv_path}")
mapping: Dict[str, str] = {}
with csv_path.open("r", encoding="utf-8", newline="") as fp:
reader = csv.reader(fp)
for row in reader:
if not row or len(row) < 2:
continue
image_path = row[0].strip()
mrz_label = row[1].strip()
if not image_path or not mrz_label:
continue
if image_path.lower().startswith("imagepath"):
continue
normalised = image_path.replace("\\", "/").strip("/").lower()
parts = normalised.split("/")
key = "/".join(parts[-2:]) if len(parts) >= 2 else normalised
mapping[key] = mrz_label
return mapping
A bundled labels.csv file ships with the benchmark script, containing ground-truth MRZ text for all 3,600 passport images in the MIDV-500 passport subset.
Step 3: Decode a Single Image and Extract MRZ Text
The decode_once function calls router.capture_multi_pages() with an image file path and the ReadPassportAndId template name. The SDK returns a CapturedResultArray containing one or more CapturedResult objects. The extract_mrz_from_captured_result function navigates the result object graph to pull MRZ text from the line1, line2, and line3 fields of the parsed result items.
def decode_once(
router: CaptureVisionRouter,
image_path: Path,
template_name: str,
) -> Tuple[Optional[str], Optional[str], int, float]:
"""Decode a single image and return (mrz_text, error, page_number, elapsed_ms)."""
start = time.perf_counter()
result_array = router.capture_multi_pages(str(image_path), template_name)
elapsed_ms = (time.perf_counter() - start) * 1000.0
results = result_array.get_results() if result_array is not None else None
if not results:
return None, "No captured results.", 1, elapsed_ms
for index, captured in enumerate(results, start=1):
text = extract_mrz_from_captured_result(captured)
if text:
return text, None, _page_number(captured, index), elapsed_ms
first = results[0]
try:
err_text = f"Capture error {first.get_error_code()}: {first.get_error_string()}"
except Exception:
err_text = "No parsable MRZ found in captured result."
return None, err_text, _page_number(first, 1), elapsed_ms
The timing measurement uses time.perf_counter(), which provides high-resolution monotonic timestamps suitable for benchmarking. The elapsed time is calculated immediately after capture_multi_pages() returns, before any result extraction — this ensures the measurement reflects only the SDK’s recognition work, not the application’s result parsing overhead.
Step 4: Run the Benchmark Across All Images
The benchmark function iterates over all image records, decodes each one, and computes a character-level match rate against the ground truth. When --runs is greater than 1, each image is decoded multiple times and the best (fastest successful) timing is kept.
def benchmark(
router: CaptureVisionRouter,
records: Sequence[ImageRecord],
runs: int,
template_name: str,
has_ground_truth: bool = True,
) -> List[ImageResult]:
output: List[ImageResult] = []
total = len(records)
for idx, record in enumerate(records, start=1):
best_text: Optional[str] = None
best_err: Optional[str] = None
best_page = 1
best_ms: Optional[float] = None
for _ in range(max(1, runs)):
recognized, err, page_number, elapsed_ms = decode_once(
router=router,
image_path=record.path,
template_name=template_name,
)
is_better = best_ms is None or elapsed_ms < best_ms
prefer_success = recognized is not None and best_text is None
if prefer_success or is_better:
best_text = recognized
best_err = err
best_page = page_number
best_ms = elapsed_ms
if has_ground_truth:
score = match_rate(best_text, record.truth_mrz)
exact = score >= 0.99999
else:
score = 0.0
exact = False
success = best_text is not None
output.append(ImageResult(
key=record.key,
success=success,
error=best_err if not success else None,
recognized_mrz=best_text,
truth_mrz=record.truth_mrz,
match_rate=score,
exact_match=exact,
run_ms=best_ms,
page_number=best_page,
))
tag = "PASS" if exact else ("MISS" if success else "ERR")
time_text = ms(best_ms) if best_ms is not None else "-"
print(f"[{idx:>4}/{total}] {record.key:<32} {tag:<4} {time_text}")
return output
An MRZ is considered an exact match when the normalized match rate reaches 99.999% — this threshold accounts for minor whitespace or padding differences while ensuring the MRZ content is functionally identical.
Step 5: Compute Summary Statistics and Generate Reports
After all images are processed, the build_summary function computes aggregate metrics: exact accuracy, success rate, average match rate, and timing percentiles (P50, P95, P99). The build_html function generates a visual dashboard with summary cards, an SVG timing histogram, and a per-image detail table.
def build_summary(results: Sequence[ImageResult], wall_ms: float) -> SummaryStats:
total = len(results)
success_count = sum(1 for row in results if row.success)
exact_count = sum(1 for row in results if row.exact_match)
avg_match = sum(row.match_rate for row in results) / total if total else 0.0
timings = [row.run_ms for row in results if row.run_ms is not None]
if timings:
avg_ms = float(statistics.fmean(timings))
min_ms = float(min(timings))
max_ms = float(max(timings))
p50 = quantile(timings, 0.50)
p95 = quantile(timings, 0.95)
p99 = quantile(timings, 0.99)
else:
avg_ms = min_ms = max_ms = p50 = p95 = p99 = 0.0
return SummaryStats(
total_images=total,
success_count=success_count,
exact_count=exact_count,
avg_match_rate=avg_match,
avg_ms=avg_ms,
min_ms=min_ms,
max_ms=max_ms,
p50_ms=p50,
p95_ms=p95,
p99_ms=p99,
total_wall_ms=wall_ms,
)
The main function ties everything together, writing both report.json (machine-readable) and report.html (visual dashboard) to the output directory.
Step 6: Configure the Dataset Path Dynamically
The benchmark supports three ways to specify the dataset path: the --dataset CLI argument, the MRZ_DATASET_DIR environment variable, or the --download flag that fetches the MIDV-500 dataset automatically. The full MIDV-500 dataset is large (~10 GB across 50 document types), so the download function fetches only the 12 passport types (~2 GB) by default, with a --download-limit option for quick smoke tests.
# Resolve default dataset path from env var or fall back to None.
env_dataset = os.environ.get("MRZ_DATASET_DIR")
default_dataset = Path(env_dataset) if env_dataset else None
parser.add_argument(
"--dataset",
type=Path,
default=default_dataset,
help=(
"Dataset root folder containing images and labels.csv. "
"Can also be set via the MRZ_DATASET_DIR environment variable."
),
)
parser.add_argument(
"--download",
action="store_true",
help="Download the MIDV-500 passport dataset before running the benchmark.",
)
parser.add_argument(
"--download-limit",
type=int,
default=0,
help="When using --download, only download the first N passport document types.",
)
When --download is used, the download_and_prepare_dataset function fetches passport archives from the MIDV-500 FTP server, extracts and converts .tif images to .jpg using Pillow. The raw downloaded data is kept in a _raw/ subdirectory so re-running the command skips already-downloaded types without re-fetching from the FTP server.
The bundled labels.csv uses a preprocessed directory naming scheme (e.g., CA/CA05_01.jpg) that does not match the raw MIDV-500 directory structure (e.g., 05_aze_passport/05_aze_passport_001.jpg). For downloaded data, use --no-ground-truth to run a speed-only benchmark:
# Download 1 passport type and run a speed-only benchmark
python benchmark_dcv_mrz.py --download --download-limit 1 --no-ground-truth
# Download all 12 passport types (~2 GB) and benchmark
python benchmark_dcv_mrz.py --download --no-ground-truth
For accuracy comparison with ground truth, use a preprocessed dataset with the CA/–TS/ directory naming scheme and --dataset /path/to/midv-500-passport.
Benchmark Results Analysis
Running the benchmark on the full 3,600-image MIDV-500 passport subset produces the following results:
| Metric | Value |
|---|---|
| Total images | 3,600 |
| Exact MRZ matches | 1,603 (44.5%) |
| Successful decodes | 2,139 (59.4%) |
| Partial matches (success, not exact) | 536 (14.9%) |
| Failed decodes (no MRZ) | 1,461 (40.6%) |
| Average decode time | 387 ms |
| P50 (median) | 283 ms |
| P95 | 900 ms |
| P99 | 1,271 ms |
| Fastest decode | 110 ms |
| Total elapsed | ~23 minutes |
The timing distribution shows that 75.9% of all images were decoded in under 500 milliseconds — 20.6% in under 200 ms and 55.3% between 200–500 ms. Only 3.3% of images exceeded 1,000 ms, and those were predominantly in the most challenging capture conditions.
Performance varied significantly across the 10 capture-condition subsets. The TA condition (ideal lighting, minimal motion) achieved 92.2% success rate and 83.3% exact accuracy at an average of 248 ms per image. The TS condition achieved 85.0% success and 67.5% exact accuracy at 236 ms. In contrast, the PS and PA conditions — which involve motion blur and extreme angles — saw success rates drop to 18.3% and 24.4% respectively.
On a 5-image smoke test with clean, well-aligned passport images, the SDK achieved 100% exact accuracy at an average of 194 ms per image, with a P50 of 170 ms. This confirms that when the input image quality is sufficient, the SDK delivers fast and accurate MRZ recognition.
Common Issues & Edge Cases
-
Motion blur and extreme angles: The MIDV-500 dataset intentionally includes challenging capture conditions. Images with significant motion blur or extreme angles may not contain a readable MRZ. The SDK returns a capture error code in these cases, which the benchmark records as a failed decode. This is expected behaviour, not a bug — the SDK correctly identifies that no MRZ can be extracted from an unreadable image.
-
Multi-page TIFF files: Some MIDV-500 images are stored as multi-page TIFF files. The SDK’s
capture_multi_pages()method handles these natively, returning oneCapturedResultper page. The benchmark iterates all pages and returns the first page where MRZ text is found. -
Field validation status: The SDK may return field values that failed validation (e.g., incorrect checksum digits in the MRZ). The
_safe_fieldfunction checksget_field_validation_status()and returnsNonefor failed fields, ensuring that only validated MRZ text is compared against ground truth. -
Downloaded data and labels.csv mismatch: The bundled
labels.csvwas built for a preprocessed version of the dataset with 2-letter capture-condition directories (CA/,CS/, etc.). Downloaded raw MIDV-500 data uses a different naming scheme (e.g.,05_aze_passport/). The benchmark detects this mismatch and prints an informational message suggesting--no-ground-truthfor speed-only benchmarking. To compare accuracy against ground truth, use a preprocessed dataset with--dataset.
Conclusion
This benchmark demonstrates that the Dynamsoft Capture Vision Python SDK delivers reliable MRZ recognition on real-world passport images, with sub-second decode times on the majority of the 3,600-image MIDV-500 passport dataset. The ReadPassportAndId template handles multiple MRZ formats in a single API call, and the SDK’s performance scales with image quality — achieving 100% accuracy on clean images and degrading gracefully on challenging captures. To integrate MRZ recognition into your own application, start with the Dynamsoft Capture Vision documentation and use the benchmark code as a reference for measuring SDK performance on your own dataset.
Source Code
https://github.com/yushulx/python-mrz-scanner-sdk/tree/main/examples/official/benchmark-midv500