How to Benchmark MRZ Recognition Accuracy on MIDV-500 with Python
Building an MRZ recognition pipeline is only half the battle. You also need to know how it performs across real-world capture conditions, and you need a dataset whose ground truth matches the pixels being scored. This tutorial walks through a Python benchmark that evaluates the Dynamsoft Capture Vision SDK on a curated 3,315-image MIDV-500 MRZ subset, measuring exact accuracy, success rate, character-level match rate, and per-image decode timing.
What you’ll build: A Python command-line tool that benchmarks Dynamsoft Capture Vision MRZ recognition on MIDV-500 images, uses audited ground truth, and exports a JSON report plus an HTML dashboard with accuracy metrics and timing distribution charts.
Key Takeaways
- The curated benchmark contains 3,315 scoreable MRZ images from an original 3,600 labeled MIDV-500 subset.
- The benchmark keeps all visible, scoreable MRZ pages, including passport MRZ images and the Algerian ID-card style MRZ images in document type
18. - The audit excludes 285 frames where the MRZ is cropped, incomplete, or not yet visible.
- Dynamsoft Capture Vision decoded MRZ text from 2,128 images, giving a 64.19% success rate.
- 1,602 images matched the ground truth exactly, giving 48.33% exact accuracy.
- The average decode time was 365.2 ms, with a P50 of 258.9 ms, P95 of 919.3 ms, and P99 of 1,283.7 ms.
- The benchmark uses the
ReadPassportAndIdtemplate, which handles passport and ID MRZ formats in one API call viarouter.capture_multi_pages().
Common Developer Questions
How accurate is Dynamsoft Capture Vision for MRZ reading?
On the curated 3,315-image MIDV-500 MRZ benchmark, the SDK achieved 48.33% exact MRZ accuracy and 64.19% success rate. The exact-match number is strict: the normalized recognized MRZ must match the ground-truth MRZ character for character. The success-rate number is higher because it counts images where the SDK returned MRZ text, even if one line or some characters did not match exactly.
Why is the curated benchmark smaller than the original 3,600 labels?
The original label file includes frames from moving capture sequences. Some early PA and PS frames show only part of the document, or the MRZ has not entered the frame yet. These images have a label row, but the MRZ cannot be fairly scored from the pixels. The audit removes 285 such frames and records every removal in assets/midv500_mrz_exclusions.csv.
What document types are included?
The source subset covers 12 MIDV-500 document groups: Azerbaijan passport (05), Brazil passport (06), Czech passport (11), German new passport (16), German old passport (17), Algerian document with an ID-card style MRZ (18), Greek passport (25), Croatian passport (27), Hungarian passport (28), Latvian passport (32), Moldovan passport (34), and Serbian passport (41). The benchmark is not limited to passport MRZ pages when another scoreable MRZ page is present.
How fast is the Dynamsoft Capture Vision Python SDK for MRZ decoding?
The average decode time across the curated benchmark is 365.2 ms per image. The median is 258.9 ms, with P95 at 919.3 ms and P99 at 1,283.7 ms. Timing starts immediately before the SDK call and stops immediately after it returns, excluding JSON serialization and report generation.
Prerequisites
- Python 3.8 or later
dynamsoft-capture-vision-bundlePillow, only needed when converting downloaded.tiffiles- A valid Dynamsoft license key. Get a 30-day free trial license.
Install the Python dependencies:
pip install -r requirements.txt
Step 1: Initialize the Dynamsoft Capture Vision Router
The first step is to initialize the SDK license and create a CaptureVisionRouter instance. The router is the central object that handles image recognition tasks. The ReadPassportAndId template is a built-in preset for passport and ID MRZ recognition, including TD1, TD2, and TD3 document formats.
from dynamsoft_capture_vision_bundle import (
CaptureVisionRouter,
EnumErrorCode,
LicenseManager,
)
DEFAULT_LICENSE = "LICENSE-KEY"
DEFAULT_TEMPLATE = "ReadPassportAndId"
def init_dynamsoft_router(license_key: str) -> CaptureVisionRouter:
"""Initialise the Dynamsoft license and create a CaptureVisionRouter."""
code, message = LicenseManager.init_license(license_key)
if code not in (EnumErrorCode.EC_OK, EnumErrorCode.EC_LICENSE_WARNING):
raise RuntimeError(
f"License initialization failed. code={code}, message={message}"
)
return CaptureVisionRouter()
Replace LICENSE-KEY with your own license before production use.
Step 2: Load Curated Ground-Truth MRZ Labels
The benchmark compares the SDK’s recognized MRZ text against ground-truth labels stored in a CSV file. The bundled labels.csv in this sample is curated: it contains 3,315 scoreable MRZ images and excludes 285 unscoreable frames.
By default, the script uses this bundled curated CSV:
BUNDLED_LABELS_CSV = Path(__file__).resolve().parent / "labels.csv"
if args.csv:
csv_path = args.csv
elif BUNDLED_LABELS_CSV.exists():
csv_path = BUNDLED_LABELS_CSV
else:
csv_path = dataset_dir / "labels.csv"
The loader normalizes image paths by taking the last two path components, for example CA/CA05_01.jpg becomes ca/ca05_01.jpg. This keeps the CSV independent of the absolute dataset location.
def load_ground_truth(csv_path: Path) -> Dict[str, str]:
"""Load ground-truth MRZ labels from a CSV file."""
mapping: Dict[str, str] = {}
with csv_path.open("r", encoding="utf-8", newline="") as fp:
reader = csv.reader(fp)
for row in reader:
if not row or len(row) < 2:
continue
image_path = row[0].strip()
mrz_label = row[1].strip()
if not image_path or not mrz_label:
continue
if image_path.lower().startswith("imagepath"):
continue
normalised = image_path.replace("\\", "/").strip("/").lower()
parts = normalised.split("/")
key = "/".join(parts[-2:]) if len(parts) >= 2 else normalised
mapping[key] = mrz_label
return mapping
Step 3: Decode a Single Image and Extract MRZ Text
The decode_once function calls router.capture_multi_pages() with an image file path and the ReadPassportAndId template name. The SDK returns one or more captured results. The benchmark extracts MRZ text from line1, line2, and line3 fields of the parsed result items.
def decode_once(
router: CaptureVisionRouter,
image_path: Path,
template_name: str,
) -> Tuple[Optional[str], Optional[str], int, float]:
"""Decode a single image and return (mrz_text, error, page_number, elapsed_ms)."""
start = time.perf_counter()
result_array = router.capture_multi_pages(str(image_path), template_name)
elapsed_ms = (time.perf_counter() - start) * 1000.0
results = result_array.get_results() if result_array is not None else None
if not results:
return None, "No captured results.", 1, elapsed_ms
for index, captured in enumerate(results, start=1):
text = extract_mrz_from_captured_result(captured)
if text:
return text, None, _page_number(captured, index), elapsed_ms
first = results[0]
try:
err_text = f"Capture error {first.get_error_code()}: {first.get_error_string()}"
except Exception:
err_text = "No parsable MRZ found in captured result."
return None, err_text, _page_number(first, 1), elapsed_ms
The timing measurement uses time.perf_counter(), which provides high-resolution monotonic timestamps suitable for benchmarking. The elapsed time is calculated immediately after capture_multi_pages() returns, before any result extraction — this ensures the measurement reflects only the SDK’s recognition work, not the application’s result parsing overhead.
Step 4: Run the Benchmark Across All Images
The benchmark iterates over matched image records, decodes each image, computes a character-level match rate, and records whether the result is an exact match. When --runs is greater than 1, each image is decoded multiple times and the fastest successful timing is kept.
def benchmark(
router: CaptureVisionRouter,
records: Sequence[ImageRecord],
runs: int,
template_name: str,
has_ground_truth: bool = True,
) -> List[ImageResult]:
output: List[ImageResult] = []
total = len(records)
for idx, record in enumerate(records, start=1):
best_text: Optional[str] = None
best_err: Optional[str] = None
best_page = 1
best_ms: Optional[float] = None
for _ in range(max(1, runs)):
recognized, err, page_number, elapsed_ms = decode_once(
router=router,
image_path=record.path,
template_name=template_name,
)
is_better = best_ms is None or elapsed_ms < best_ms
prefer_success = recognized is not None and best_text is None
if prefer_success or is_better:
best_text = recognized
best_err = err
best_page = page_number
best_ms = elapsed_ms
score = match_rate(best_text, record.truth_mrz) if has_ground_truth else 0.0
exact = score >= 0.99999 if has_ground_truth else False
output.append(ImageResult(
key=record.key,
success=best_text is not None,
error=best_err if best_text is None else None,
recognized_mrz=best_text,
truth_mrz=record.truth_mrz,
match_rate=score,
exact_match=exact,
run_ms=best_ms,
page_number=best_page,
))
Step 5: Generate JSON and HTML Reports
After all images are processed, the script computes aggregate metrics: exact accuracy, success rate, average match rate, and timing percentiles. It writes both a machine-readable JSON file and an HTML dashboard.
summary = build_summary(results, total_wall_ms)
report_obj = {
"sdk": "Dynamsoft Capture Vision",
"template": args.template,
"dataset": str(dataset_dir.resolve()),
"csv": str(csv_path.resolve()) if has_ground_truth else None,
"image_count": summary.total_images,
"summary": {
"exact_accuracy": round(summary.exact_count / summary.total_images, 6),
"success_rate": round(summary.success_count / summary.total_images, 6),
"average_match_rate": round(summary.avg_match_rate, 6),
"avg_ms": round(summary.avg_ms, 3),
"p50_ms": round(summary.p50_ms, 3),
"p95_ms": round(summary.p95_ms, 3),
"p99_ms": round(summary.p99_ms, 3),
},
"results": [result_to_dict(r) for r in results],
}
Step 6: Run the Benchmark
Use a preprocessed MIDV-500 image folder with CA/, CS/, HA/, HS/, KA/, KS/, PA/, PS/, TA/, and TS/ subfolders:
python benchmark_dcv_mrz.py --dataset /path/to/midv-500-mrz --output benchmark_report_dcv_python_full
Run a quick smoke test:
python benchmark_dcv_mrz.py --dataset /path/to/midv-500-mrz --limit 5 --output benchmark_report_dcv_python
Use --csv only when you intentionally want to override the bundled curated labels:
python benchmark_dcv_mrz.py --dataset /path/to/midv-500-mrz --csv /path/to/labels.csv
Benchmark Results Analysis
Running the latest full benchmark on the curated 3,315-image MRZ subset produced the following results:
| Metric | Value |
|---|---|
| Total benchmark images | 3,315 |
| Exact MRZ matches | 1,602 (48.33%) |
| Successful decodes | 2,128 (64.19%) |
| Partial matches (success, not exact) | 526 (15.87%) |
| Failed decodes (no MRZ text returned) | 1,187 (35.81%) |
| Average match rate | 58.09% |
| Average decode time | 365.2 ms |
| P50 decode time | 258.9 ms |
| P95 decode time | 919.3 ms |
| P99 decode time | 1,283.7 ms |
| Fastest decode | 81.4 ms |
| Slowest decode | 2,168.9 ms |
The exact-match score is intentionally strict. A result that returns only the first MRZ line, omits one line, or returns a second line with a wrong checksum character counts as a miss even when most characters are correct. This is why the success rate is higher than the exact accuracy: 2,128 images returned MRZ text, but only 1,602 matched the normalized ground truth exactly. The 526 partial matches are useful for debugging because the per-image report preserves the recognized MRZ, ground truth, match rate, and decode time.
The timing distribution shows that most successful and failed attempts still complete quickly. In this run, 955 images (28.8%) completed in under 200 ms, and another 1,667 images (50.3%) completed between 200 and 500 ms. That means 79.1% of the curated benchmark finished in under 500 ms. Only 118 images (3.6%) exceeded 1,000 ms, with the slowest image taking 2,168.9 ms.
Performance varies strongly by capture condition. TA was the best subset, with 92.2% success rate and 83.3% exact accuracy. TS followed with 85.0% success and 67.5% exact accuracy. KA and CA also performed above the overall average. The hardest subsets were PA and PS, with 32.4% and 33.5% success rates, because they contain motion, angle, partial entry into frame, and blur. The audit removes frames where a complete MRZ is not visible, but it intentionally keeps difficult images when the MRZ is visible and scoreable.
The curated subset changes the benchmark denominator, not the benchmark rules. It excludes 285 frames where the label exists but the image does not contain a complete scoreable MRZ. It still keeps all normal visible MRZ images, including passport MRZ pages and the ID-card style MRZ in document type 18. On a small clean smoke test (ca/ca05_01.jpg through ca/ca05_05.jpg), the SDK produced 5 out of 5 exact matches at an average of 192.3 ms, which confirms that the lower full-dataset score is driven mainly by image quality and capture-condition difficulty rather than by the ground-truth format.
Common Issues & Edge Cases
- Motion blur and extreme angles: MIDV-500 intentionally includes challenging capture conditions. Images with significant blur, glare, or perspective distortion may return no MRZ or only a partial MRZ.
- Incomplete capture frames: Some
PAandPSframes contain a label but no complete MRZ in the image. These are excluded from the curated benchmark and listed inassets/midv500_mrz_exclusions.csv. - Passport and ID MRZ formats: Most document groups are passports, while document type
18contains a visible ID-card style MRZ. It remains in the benchmark because the goal is MRZ recognition, not passport-only recognition. - Downloaded data and labels.csv mismatch: Downloaded MIDV-500 archives use original folder names. The curated labels expect the preprocessed
CA/toTS/directory layout. Use--no-ground-truthfor speed-only runs unless your images match the curated label paths. - Exact match vs. success rate:
success_ratecounts any returned MRZ text.exact_accuracyrequires the normalized MRZ to match ground truth exactly.
Conclusion
This benchmark demonstrates how to evaluate MRZ recognition with a reproducible Python workflow, audited ground truth, and per-image result records. The curated MIDV-500 MRZ subset keeps all visible, scoreable MRZ images, including passport and ID-card MRZ pages, while excluding incomplete or cropped frames. On the latest 3,315-image run, Dynamsoft Capture Vision achieved 48.33% exact accuracy, 64.19% success rate, and 365.2 ms average decode time.