How to Enhance Passport MRZ Detection in Python by Correcting Image Orientation

Updated Aug 27, 2026

Passport Machine Readable Zone (MRZ) detection is sensitive to the orientation of the passport. If the passport is rotated — especially 90 degrees — the MRZ can simply fail to be detected even though it is perfectly readable to a human. The fix is a two-stage correction pipeline built on a single SDK: use the Dynamsoft Capture Vision Python bundle to detect the passport edges and normalize the page with a perspective transformation (PT_DETECT_AND_NORMALIZE_DOCUMENT), then rotate the normalized image to upright using facial landmarks from RetinaFace before reading the MRZ (ReadPassportAndId).

What you’ll build: A Python script that reads and parses the MRZ of passports photographed at any rotation (0°, 90°, 180°, 270°) by combining document normalization and face-landmark orientation correction — all with one pip install and one SDK.

passport MRZ detection in any orientation

Key Takeaways

One SDK does both jobs: dynamsoft-capture-vision-bundle replaces the older separate mrz-scanner-sdk and document-scanner-sdk wrappers, and its bundle includes MRZ parsing, document edge detection, and perspective correction out of the box.
CaptureVisionRouter.capture() with PT_DETECT_AND_NORMALIZE_DOCUMENT returns a deskewed document image in one call — no manual quad extraction or normalize-buffer step.
CaptureVisionRouter.capture() with the ReadPassportAndId template detects and parses MRZ zones (TD1/TD2/TD3) into structured fields with check-digit validation.
After perspective transformation the page can still be upside-down or sideways; the portrait’s facial orientation matches the MRZ’s, so rotating by face landmarks fixes the remaining cases.
Among Dlib, MediaPipe, and RetinaFace, only RetinaFace reliably returns eye/nose landmarks on rotated faces, which makes it the right landmark source for the rotation step.

Common Developer Questions

Why does my passport MRZ detector fail on rotated photos?

MRZ localization models are trained on upright text lines. Edge detection and perspective transformation can fix skew, but they cannot tell 180° apart from 0°, nor 90° from 270°. The MRZ must additionally be rotated to a true-upright orientation, which is easiest to derive from a face-detection landmark step.

Can Dynamsoft Capture Vision do both document normalization and MRZ reading in Python?

Yes. The single dynamsoft-capture-vision-bundle package exposes both capabilities through CaptureVisionRouter: use the PT_DETECT_AND_NORMALIZE_DOCUMENT preset for edge detection and rectification, and the ReadPassportAndId template for MRZ text-line recognition and field parsing — no separate scanner SDKs needed.

Which face detector should I use to correct passport orientation?

RetinaFace. Dlib’s HOG/CNN frontal-face detector misses strongly rotated faces entirely, and MediaPipe’s BlazeFace detects faces but its landmarks are unreliable on sideways or upside-down portraits. RetinaFace correctly returns left-eye, right-eye, and nose landmarks in all four orientations, at the cost of the longest detection time of the three.

Prerequisites

Python 3.8+ on Windows, Linux, or macOS.
A valid Dynamsoft license key. A working MRZ + document pipeline requires commercial features; get a 30-day free trial license to follow along.

Install the required Python packages:

pip install dynamsoft-capture-vision-bundle dlib mediapipe retina-face opencv-python

dynamsoft-capture-vision-bundle: The Dynamsoft Capture Vision SDK for Python. It bundles MRZ recognition, document edge detection, and perspective correction (among other tasks) behind a single CaptureVisionRouter API.
dlib: An open-source software library that provides highly accurate and efficient face detection algorithm.
mediapipe: A Google-developed, open-source, cross-platform framework designed for rapid, real-time face detection.
retina-face: A deep learning based cutting-edge facial detector for Python coming with facial landmarks.
opencv-python: Used to display images and draw lines.

Step 1: Read MRZ from a Passport Image with Capture Vision

Let’s get started with a passport image taken in the correct orientation.

passport image

After activating the license with LicenseManager.init_license(), one CaptureVisionRouter.capture() call with the built-in ReadPassportAndId template detects the MRZ and parses its fields. The full sample script:

import argparse
import sys

import cv2
import numpy as np
from dynamsoft_capture_vision_bundle import *


def convertMat2ImageData(mat):
    """Convert an OpenCV matrix (BGR or gray) to a Dynamsoft ImageData object."""
    height, width = mat.shape[:2]
    if len(mat.shape) == 3:
        channels = 3
        pixel_format = EnumImagePixelFormat.IPF_RGB_888
        data = cv2.cvtColor(mat, cv2.COLOR_BGR2RGB).tobytes()
    else:
        channels = 1
        pixel_format = EnumImagePixelFormat.IPF_GRAYSCALED
        data = mat.tobytes()
    return ImageData(data, width, height, width * channels, pixel_format)


def print_parsed_result(parsed_result):
    for item in parsed_result.get_items():
        print("Document Type:", item.get_code_type())
        for i in range(1, 4):
            line = item.get_field_value(f"line{i}")
            if line is not None:
                print(f"  Line {i}: {line}")
        for field in ("passportNumber", "primaryIdentifier", "secondaryIdentifier",
                      "issuingState", "dateOfBirth", "dateOfExpiry"):
            value = item.get_field_value(field)
            if value is not None:
                print(f"  {field}: {value}")


def detect_mrz(cvr, image):
    result = cvr.capture(convertMat2ImageData(image), "ReadPassportAndId")
    parsed_result = result.get_parsed_result()
    if parsed_result is None or len(parsed_result.get_items()) == 0:
        print("No MRZ detected.")
        return

    print_parsed_result(parsed_result)

    # Draw detected MRZ text-line locations
    line_result = result.get_recognized_text_lines_result()
    if line_result is not None:
        for item in line_result.get_items():
            location = item.get_location()
            pts = [(p.x, p.y) for p in location.points]
            del location
            cv2.drawContours(image, [np.intp(pts)], 0, (0, 255, 0), 2)

    cv2.imshow("MRZ Detection", image)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Scan MRZ info from a given image")
    parser.add_argument("filename")
    parser.add_argument("-l", "--license", default="LICENSE-KEY", type=str)
    args = parser.parse_args()

    error_code, error_message = LicenseManager.init_license(args.license)
    if error_code != EnumErrorCode.EC_OK and error_code != EnumErrorCode.EC_LICENSE_WARNING:
        print("License initialization failed:", error_code, error_message)
        sys.exit(1)

    cvr = CaptureVisionRouter()
    image = cv2.imread(args.filename)
    detect_mrz(cvr, image)
    cv2.waitKey(0)

Explanation

LicenseManager.init_license() activates the SDK once per process with a Dynamsoft license key.
CaptureVisionRouter is the single entry point for every capture task. Its capture() method accepts an ImageData object (or a file path) plus a template name or preset.
The bundle works directly with ImageData, so an OpenCV matrix must be converted through convertMat2ImageData() (BGR → RGB, uint8 bytes, stride = width × channels). Passing a file path to capture() skips this conversion.
The MRZ text lines and their quadrilateral locations come from result.get_recognized_text_lines_result(), and the parsed identity fields (line1/line2, passportNumber, primaryIdentifier, dates, etc.) come from result.get_parsed_result(). Each field also carries a check-digit validation status via get_field_validation_status().

Run it on the upright passport:

python app.py passport.jpg

passport image

Step 2: Fix Skew with Edge Detection and Perspective Transformation

If the image is rotated at a significant angle, MRZ detection may fail.

rotated passport image

To address this issue, we use the same CaptureVisionRouter with the document preset PT_DETECT_AND_NORMALIZE_DOCUMENT, which detects the passport edges and applies the perspective transformation in one call:

result = cvr.capture(convertMat2ImageData(image),
                     EnumPresetTemplate.PT_DETECT_AND_NORMALIZE_DOCUMENT.value)
processed_document_result = result.get_processed_document_result()

# The four corners of the detected document quad
quad_items = processed_document_result.get_detected_quad_result_items()
location = quad_items[0].get_location()
x1, y1 = location.points[0].x, location.points[0].y
x2, y2 = location.points[1].x, location.points[1].y
x3, y3 = location.points[2].x, location.points[2].y
x4, y4 = location.points[3].x, location.points[3].y
del location

# The normalized (deskewed & cropped) passport image
enhanced_items = processed_document_result.get_enhanced_image_result_items()
rectified_document = convertImageData2Mat(enhanced_items[0].get_image_data())

The ImageData-to-NumPy helper used above:

def convertImageData2Mat(image_data):
    """Convert a Dynamsoft ImageData object to an OpenCV matrix (BGR)."""
    width = image_data.get_width()
    height = image_data.get_height()
    pixel_format = image_data.get_image_pixel_format()
    data = bytearray(image_data.get_bytes())
    if pixel_format == EnumImagePixelFormat.IPF_RGB_888:
        mat = np.array(data, dtype=np.uint8).reshape(height, width, 3)
        mat = cv2.cvtColor(mat, cv2.COLOR_RGB2BGR)
    elif pixel_format == EnumImagePixelFormat.IPF_BGR_888:
        mat = np.array(data, dtype=np.uint8).reshape(height, width, 3)
    else:  # gray / binary
        mat = np.array(data, dtype=np.uint8).reshape(height, width)
    return mat

Then run the MRZ detection from Step 1 on rectified_document:

rotated passport mrz detection

Step 3: Rotate Images Based on Facial Orientation

After perspective transformation, the image may be oriented in one of four directions: 0 degrees, 90 degrees, 180 degrees, or 270 degrees.

rotated passport

If you run the code above, you will find that MRZ detection after normalization is still unreliable — in our testing, some of the 90/180/270-degree variants decoded while others failed, and which ones pass can change with image content and SDK version. Only the 0-degree orientation is consistently readable. Thus, we aim to rotate the other three orientations to this correct angle. Considering that the orientation of the face on the passport is consistent with that of the Machine-Readable Zone, we can use face detection to rotate the image accordingly.

Numerous face detection algorithms exist, each with varying levels of performance. In this article, we will compare the effectiveness of three prominent algorithms: Dlib, MediaPipe, and RetinaFace.

Dlib

Download the pre-trained model from here.
Unzip the file and put it in the same folder as the Python script.

Create the Dlib face detector:

 import dlib
 import time

 detector = dlib.get_frontal_face_detector()
 predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

Detect the faces from the rectified passport image:

 img = cv2.imread(filename)  
 gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

 start_time = time.time()
 faces = detector(gray)
 end_time = time.time()
 print("Elapsed Time:", end_time - start_time)

dlib face detection

The dlib face detection algorithm is typically trained on datasets where faces are upright or near-upright. The features learned by the classifier assume that the faces in the images will be oriented in a specific way, usually right side up. When a face is rotated significantly (like upside-down or tilted at 90 degrees), the learned features may not match well, making it difficult for the algorithm to detect the face.

Mediapipe

Download the pre-trained model from here. At present, only BlazeFace (short-range) is available, which is a lightweight model for detecting single or multiple faces.
Put the model in the same folder as the Python script.

Create the MediaPipe face detector:

 import mediapipe as mp
 from mediapipe.tasks import python
 from mediapipe.tasks.python import vision
    
 mp_face_detection = mp.solutions.face_detection
 mp_drawing = mp.solutions.drawing_utils
    
 base_options = python.BaseOptions(model_asset_path='blaze_face_short_range.tflite')
 options = vision.FaceDetectorOptions(base_options=base_options)
 detector = vision.FaceDetector.create_from_options(options)

Detect the faces from the rectified passport image:

 img = cv2.imread(filename)
 img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
 image = mp.Image(image_format=mp.ImageFormat.SRGB, data=img)
    
 start_time = time.time()
 detection_result = detector.detect(image)
 end_time = time.time()
 print("Elapsed Time:", end_time - start_time)

mediapipe face detection

Compared to Dlib, Mediapipe is faster and more accurate. However, it still falls short of our requirements because it fails to detect some facial landmarks correctly.

RetinaFace

RetinaFace is a deep learning-based face detection model aimed at identifying faces in images with high accuracy. Let’s explore whether it meets our objectives.

from retinaface import RetinaFace
img = cv2.imread(filename)
obj = RetinaFace.detect_faces(img_path=img)

if type(obj) == dict:
    for key in obj:
        identity = obj[key]

        facial_area = identity["facial_area"]
        facial_img = img[facial_area[1]: facial_area[3],
                            facial_area[0]: facial_area[2]]

        landmarks = identity["landmarks"]
        left_eye = landmarks["left_eye"]
        right_eye = landmarks["right_eye"]
        nose = landmarks["nose"]
        mouth_right = landmarks["mouth_right"]
        mouth_left = landmarks["mouth_left"]

        cv2.rectangle(img, (facial_area[0], facial_area[1]),
                        (facial_area[2], facial_area[3]), (0, 255, 0), 2)
        cv2.circle(img, (int(left_eye[0]), int(
            left_eye[1])), 2, (255, 0, 0), 2)
        cv2.circle(img, (int(right_eye[0]), int(
            right_eye[1])), 2, (0, 0, 255), 2)
        cv2.circle(img, (int(nose[0]), int(nose[1])), 2, (0, 255, 0), 2)
        cv2.circle(img, (int(mouth_left[0]), int(
            mouth_left[1])), 2, (0, 155, 255), 2)
        cv2.circle(img, (int(mouth_right[0]), int(
            mouth_right[1])), 2, (0, 155, 255), 2)

cv2.imshow(filename, img)

retina face detection

RetinaFace takes the longest time for face detection, but it is the most accurate. It correctly identifies facial landmarks in all four directions, which we can use to rotate the image.

def rotate(img, left_eye, right_eye, nose):

    nose_x, nose_y = nose
    left_eye_x, left_eye_y = left_eye
    right_eye_x, right_eye_y = right_eye

    if (nose_y > left_eye_y) and (nose_y > right_eye_y):
        return img # no need to rotate
    elif (nose_y < left_eye_y) and (nose_y < right_eye_y):
        return cv2.flip(img, flipCode=-1) # 180 degrees
    elif (nose_x < left_eye_x) and (nose_x < right_eye_x):
        transposed = cv2.transpose(img)
        return cv2.flip(transposed, flipCode=0) # 90 degrees 
    else:
        transposed = cv2.transpose(img)
        return cv2.flip(transposed, flipCode=1) # 270 degrees 

Step 4: Combine Document Normalization and RetinaFace Rotation for MRZ Detection

We can now combine the above steps to detect the MRZ area in rotated passport images. The complete combine.py:

import argparse
import sys

import cv2
import numpy as np
from dynamsoft_capture_vision_bundle import *

import face_retina
from app import convertMat2ImageData, print_parsed_result


def convertImageData2Mat(image_data):
    """Convert a Dynamsoft ImageData object to an OpenCV matrix (BGR)."""
    width = image_data.get_width()
    height = image_data.get_height()
    pixel_format = image_data.get_image_pixel_format()
    data = bytearray(image_data.get_bytes())
    if pixel_format == EnumImagePixelFormat.IPF_RGB_888:
        mat = np.array(data, dtype=np.uint8).reshape(height, width, 3)
        mat = cv2.cvtColor(mat, cv2.COLOR_RGB2BGR)
    elif pixel_format == EnumImagePixelFormat.IPF_BGR_888:
        mat = np.array(data, dtype=np.uint8).reshape(height, width, 3)
    else:  # gray / binary
        mat = np.array(data, dtype=np.uint8).reshape(height, width)
    return mat


def detect_and_normalize_doc(cvr, image):
    """Detect the document quad and return the perspective-corrected image."""
    result = cvr.capture(convertMat2ImageData(image),
                         EnumPresetTemplate.PT_DETECT_AND_NORMALIZE_DOCUMENT.value)
    processed_document_result = result.get_processed_document_result()
    if processed_document_result is None:
        return image

    enhanced_items = processed_document_result.get_enhanced_image_result_items()
    if len(enhanced_items) == 0:
        return image
    return convertImageData2Mat(enhanced_items[0].get_image_data())


def detect_mrz(cvr, image):
    result = cvr.capture(convertMat2ImageData(image), "ReadPassportAndId")
    parsed_result = result.get_parsed_result()
    if parsed_result is None or len(parsed_result.get_items()) == 0:
        return None

    print_parsed_result(parsed_result)

    line_result = result.get_recognized_text_lines_result()
    if line_result is not None:
        for item in line_result.get_items():
            location = item.get_location()
            pts = [(p.x, p.y) for p in location.points]
            del location
            cv2.drawContours(image, [np.intp(pts)], 0, (0, 255, 0), 2)

    return parsed_result


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Scan MRZ info from a rotated image")
    parser.add_argument("filename")
    parser.add_argument("-l", "--license", default="LICENSE-KEY", type=str)
    args = parser.parse_args()

    error_code, error_message = LicenseManager.init_license(args.license)
    if error_code != EnumErrorCode.EC_OK and error_code != EnumErrorCode.EC_LICENSE_WARNING:
        print("License initialization failed:", error_code, error_message)
        sys.exit(1)

    cvr = CaptureVisionRouter()
    image = cv2.imread(args.filename)

    # 1. Edge detection + perspective transformation
    rectified = detect_and_normalize_doc(cvr, image)
    # 2. Rotate the rectified image based on facial orientation (RetinaFace)
    upright = face_retina.detect(rectified)
    # 3. MRZ detection on the orientation-corrected image
    mrz_image = upright.copy()
    parsed = detect_mrz(cvr, mrz_image)
    if parsed is None:
        print("No MRZ detected.")

    cv2.imshow("Original", image)
    cv2.imshow("MRZ Detection", mrz_image)
    cv2.waitKey(0)

Run it on each rotated passport (in this sample, all four orientations decode with the same MRZ content):

python combine.py passport.jpg
python combine.py passport_90.jpg
python combine.py passport_180.jpg
python combine.py passport_270.jpg

Document Type: MRTD_TD3_PASSPORT
  Line 1: P<CANAMAN<<RITA<TANIA<<<<<<<<<<<<<<<<<<<<<<<
  Line 2: ERE82721<9CAN8412070M2405252<<<<<<<<<<<<<<08
  passportNumber: ERE82721<9
  primaryIdentifier: MAN
  secondaryIdentifier: RITA TANIA
  issuingState: CAN
  dateOfBirth: 841207
  dateOfExpiry: 240525

passport mrz detection in any orientation

Common Issues & Edge Cases

ReadPassportAndId fails on a 90°-rotated page even though the MRZ is visible. Confirmed behavior: on this sample set, direct capture succeeds at 0°, 180°, and 270° but returns nothing at exactly 90°, which is what makes the RetinaFace rotation step necessary rather than optional.
EC_LICENSE_INVALID (-10003) when running the samples. Replace LICENSE-KEY with your own trial or commercial key, or pass it on the command line with -l <key>. The license is initialized once per process via LicenseManager.init_license().
RetinaFace downloads a ~119 MB model on first run. The retina-face package fetches retinaface.h5 into ~/.deepface/weights/ automatically; keep a stable network connection the first time you run the samples.
RetinaFace returns {"faces": "No face detected!"}. When no face is found — for example the portrait is cropped out — face_retina.detect() returns the image unchanged and depends on the MRZ template’s built-in tolerance. Passport cards where the portrait has been cut off may then still fail.
Memory-leak prevention with location objects. get_location() allocates memory in the C++ layer; call del location after extracting coordinates in long-running or camera loops.

Source Code

Get the complete sample project source code on GitHub