How to Benchmark Arabic OCR & Document Intelligence

Capability comparison vs performance benchmark

Two very different questions get confused in OCR marketing. The first is a capability comparison: does an engine support Arabic handwriting, table extraction, or key-value field extraction at all? The second is a performance benchmark: how accurately does it transcribe a specific, independently ground-truthed test set, measured with a defined metric and normalization policy?

A capability comparison answers "what is possible"; a performance benchmark answers "how well, on this data, under these rules." They must never be blended into a single score. A vendor can legitimately support a capability while still scoring poorly on it, and "not publicly documented" does not mean "not supported." This article focuses on the performance-benchmark side — how to measure Arabic OCR fairly — while keeping the distinction explicit.

To see which capabilities are actually supported across solutions, see the capability comparison.

Evaluation document categories

A single blended accuracy number hides a model's weaknesses. A meaningful Arabic benchmark breaks its test set into categories and reports each one separately, so a weak category is never buried inside a flattering average.

• English printed — clean (Latin baseline)
• English printed — degraded / old scans
• Arabic printed — clean (core Arabic)
• Arabic printed — degraded / old scans
• Arabic handwritten
• English handwritten
• Arabic printed + handwritten mixed
• English printed + handwritten mixed
• Arabic + English mixed script
• Very difficult — rotated, noisy, stamps, tables

Per-category page counts and totals must come from a frozen, published manifest. Until that manifest exists, counts are a design target — never a published fact.

Ground truth

Ground truth is the reference transcription every engine is scored against, so its quality sets the ceiling for the whole benchmark. It must be produced by human transcription with an independent second-pass verification. No engine output — including the engine under test or any competitor — is ever used as ground truth, because that would quietly bias the results toward whichever model wrote the reference.

For genuinely ambiguous or unreadable text, record uncertainty rather than forcing a guessed character. And ground truth is frozen: it is never edited to make any model look better.

Arabic normalization

Normalization is where Arabic benchmarks are won or lost. Arabic text can be written in several visually or semantically equivalent forms, and how you treat those forms before scoring can change a reported error rate dramatically. A credible benchmark discloses its normalization policy in full and reports results under two tracks that are never mixed in the same table.

CER & WER

Character Error Rate (CER) and Word Error Rate (WER) are the primary OCR metrics. Both are edit-distance ratios: they count the minimum number of edits needed to turn the engine's output into the reference transcription, divided by the length of the reference.

• Ssubstitutions — a character or word recognized as the wrong one
• Ddeletions — a character or word missing from the output
• Iinsertions — an extra character or word that is not in the reference

N is the number of characters (for CER) or words (for WER) in the reference. Edit distance is Levenshtein — character-level for CER, token-level for WER — with the same normalization track applied identically to every engine.

Structured extraction metrics

OCR recognition quality is not the same thing as structured field-extraction quality, and CER/WER must not be reused for extraction. Reading every character correctly tells you nothing about whether the right invoice total landed in the right field.

Structured extraction is evaluated with field-level exact-match precision, recall, and F1, plus a document-level success rate — the share of documents where every required field was extracted correctly. This is a separate benchmark from OCR and is reported separately.

Read more about Document Data Extraction.

Layout & table evaluation

Layout understanding — reading order, columns, headers, and tables — is evaluated independently of raw text accuracy. A table with perfect character recognition is still wrong if rows and columns are misaligned.

Table extraction is assessed by cell-level correctness and structural fidelity (correct number of rows and columns, correct cell-to-header association), and difficult-layout documents such as rotated pages, stamps, and multi-column scans are kept in their own category so their difficulty is visible rather than averaged away.

Arabic handwriting

Arabic handwriting recognition is substantially harder than printed OCR: letter forms vary between writers, ligatures are irregular, and baseline and spacing are inconsistent. It is always evaluated as its own category with its own ground truth and its own CER/WER, never folded into a printed-text average.

Mixed printed-plus-handwritten pages (for example a printed form completed by hand) form a further category, because an engine can be strong on one modality and weak on the other.

Learn about Arabic handwriting OCR and Arabic OCR.

Mixed Arabic / English documents

Real documents are rarely single-script. Arabic-English mixed pages test an engine's ability to handle script switching, correct reading order across directions, and digits that may appear in either Arabic-Indic or Western form.

These documents are scored in their own mixed-script category with the reading order documented in the ground truth, so right-to-left and left-to-right runs are evaluated on their true logical order rather than a guessed one.

Latency & throughput

Processing performance is a separate benchmark from accuracy and must never be blended into a quality score. Latency depends on region, network location, hardware, warm versus cold start, concurrency, page count, document size, and resolution — so all of these are documented, and latency is reported as P50, P90, and P95 rather than a single average.

A fair speed comparison requires environment fairness: local inference on one engine versus a remote API on another is not comparable unless the conditions are documented. When they cannot be equalized, latency is published separately with explicit caveats. Cost is likewise its own benchmark, recorded with currency, pricing date, and billing unit — never hard-coded from an undated source.

Reproducibility

A benchmark is only credible if someone else can reproduce it. Every run records a frozen test manifest identified by a content hash, so the exact same pages are used each time; new or changed pages mean a new benchmark version and a new hash.

Each run also records a full engine version manifest — model and API version, pipeline and routing versions, region, date tested, features enabled, and request configuration. "Azure OCR" or "the latest model" is not enough: services change over time, and a result is meaningless without the exact configuration that produced it. Results, tables, charts, and summary statistics all derive from a single canonical data source so published numbers cannot drift.

For programmatic integration, see the API documentation.

Fair competitor comparison

A fair comparison holds everything constant except the engine: the same documents, the same independent ground truth, the same normalization and metric implementation, the same retry policy, and the same failure accounting. No engine receives engine-specific manual cleanup unless that cleanup is an explicit, disclosed part of its production pipeline.

• Failures are accounted for, not silently dropped — attempted, successful, and failed pages are all published, with an explicit statement of how failures affect the aggregate.
• Vendor-reported benchmark results should not be treated as equivalent to independently measured results unless the evaluation conditions match.
• "Not publicly documented" does not mean "not supported" — absence of a published number is recorded as unknown, not as a weakness.

Reporting template

Every result table follows the same shape and always states whether it is the strict or normalized track — the two are never mixed in one table. The template below shows the columns; the values are placeholders, not measured results.

Table track: strict or normalized (stated once per table, never mixed).