PII Benchmark Board
How well do PII detectors actually work? We measure our own releases and the field on the same texts, with the same scoring, and publish everything — including where we are behind.
Three releases on one board
ShinrAI 1.1
The first open generation (2026). Still available as an archive.
ShinrAI 1.2
Larger inventories, 1024-token window. Archive.
ShinrAI 1.3
The current release: identifier formats in the model, 51-class origin attribute, Japanese at its best level yet.
Outlook: ShinrAI 1.4 is in training — long-document redaction, the paperwork register, Hebrew toward full support.
"ShinrAI 1.3" columns show the open model behind our default serving settings — the number that counts. The Enterprise product layers additional rule-based detection on top (checksum validators, deterministic identifiers); that layer is exact by construction and is not part of these scores.
Our 15-locale test
F1 score (0–100, higher is better · exact spans). 200 texts per locale. The median of 95.0 includes the Hebrew beta locale. Azure AI PII, measured on the same texts: 53–76.
| country | language | score | F1 | precision / recall |
|---|---|---|---|---|
| 🇯🇵 | JA | 98.1 | 98.3 / 97.8 | |
| 🇬🇧🇺🇸 | EN | 97.7 | 97.2 / 98.1 | |
| 🇧🇷 | PT-BR | 97.3 | 97.7 / 96.9 | |
| 🇪🇸 | ES | 96.9 | 97.2 / 96.6 | |
| 🇷🇺 | RU | 96.4 | 96.2 / 96.7 | |
| 🇫🇷 | FR | 96.0 | 96.3 / 95.8 | |
| 🇰🇷 | KO | 95.5 | 95.9 / 95.2 | |
| 🇮🇹 | IT | 95.0 | 96.3 / 93.8 | |
| 🇩🇪🇦🇹🇨🇭 | DE | 94.4 | 95.3 / 93.5 | |
| 🇵🇹 | PT-PT | 94.2 | 95.3 / 93.1 | |
| 🇵🇱 | PL | 90.9 | 91.4 / 90.3 | |
| 🇹🇷 | TR | 90.1 | 90.7 / 89.5 | |
| 🇸🇦 | AR | 87.9 | 90.3 / 85.6 | |
| 🇺🇦 | UA | 86.7 | 89.2 / 84.3 | |
| 🇮🇱 | HE beta | 59.5 | 68.8 / 52.3 |
Country matters: the model knows what a street name looks like in France versus Spain, and what a company name looks like in Japan. Detection and replacements follow each country's conventions.
Public test sets — everyone in one harness
Each competitor runs at its best threshold; ShinrAI runs at its default settings. Scores are F1 (0–100, higher is better) with generous matching — a find counts if it touches the right text with the right type (see the glossary). ★ marks the best system per row. ⚠ marks systems that trained on that very test set.
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | LFM2.5 PII (Liquid AI) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DE | 81.5 | 83.6 | 85.4 | 69.4 | 35.6 | 11.0 | 21.3 | 90.3 ★ ⚠ training overlap likely | 77.8 | — | — | — |
| EN | 80.5 | 75.5 | 80.8 | 76.7 | 38.3 | 10.9 | 21.1 | 86.9 ★ ⚠ training overlap likely | 70.2 | 68.4 | 50.0 | 69.1 |
| FR | 80.7 | 82.4 | 82.6 | 74.5 | 37.9 | 9.7 | 16.2 | 90.6 ★ ⚠ training overlap likely | 75.9 | — | — | — |
| IT | 76.5 | 77.6 | 83.3 | 70.9 | 45.2 | 13.1 | 25.3 | 89.0 ★ ⚠ training overlap likely | 73.8 | — | — | — |
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | LFM2.5 PII (Liquid AI) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DE | 72.8 | 78.1 | 92.1 ★ | 62.3 | 45.6 | 25.7 | 37.9 | 58.5 | 91.3 ⚠ training data undisclosed | — | — | — |
| EN | 79.3 | 83.2 | 94.9 | 60.9 | 45.8 | 17.3 | 28.5 | 58.7 | 96.2 ★ ⚠ training data undisclosed | 86.9 | 65.1 | 86.5 |
| ES | 85.6 | 73.5 | 97.7 ★ | 61.5 | 39.0 | 16.5 | 29.5 | 64.4 | 93.3 ⚠ training data undisclosed | — | — | — |
| FR | 84.1 | 72.8 | 93.3 ★ | 61.7 | 40.2 | 16.5 | 26.9 | 61.4 | 90.3 ⚠ training data undisclosed | — | — | — |
| IT | 88.5 | 67.8 | 96.8 ★ | 56.2 | 33.9 | 11.8 | 21.7 | 60.4 | 91.4 ⚠ training data undisclosed | — | — | — |
| PL | 74.7 | 73.5 | 81.2 | 65.9 | 32.3 | 20.1 | 23.7 | 45.7 | 87.8 ★ ⚠ training data undisclosed | — | — | — |
| PT | 81.0 | 73.3 | 94.7 | 62.7 | 38.4 | 15.4 | 28.8 | 61.0 | 96.1 ★ ⚠ training data undisclosed | — | — | — |
| RU | 71.1 | 70.5 | 82.5 ★ | 53.9 | 34.9 | 17.9 | 18.1 | 51.5 | 82.5 ★ ⚠ training data undisclosed | — | — | — |
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | Azure AI PII (Microsoft) | Presidio (open source) | LFM2.5 PII (Liquid AI) | Privacy Filter (OpenAI) | OpenMed PII E5 | OpenMed Privacy Filter | GLiNER PII (Urchade) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | Piiranha v1 (III) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EN | 24.1 | 25.4 | 52.2 | 30.7 | 44.2 | 84.5 ★ | 34.0 | 78.1 | 74.3 | 24.3 | 54.6 | 24.8 | 23.5 | 51.4 | 60.0 | 53.9 | 46.4 | 37.3 | 39.7 |
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | Azure AI PII (Microsoft) | Presidio (open source) | LFM2.5 PII (Liquid AI) | Privacy Filter (OpenAI) | OpenMed PII E5 | OpenMed Privacy Filter | GLiNER PII (Urchade) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | Piiranha v1 (III) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EN | 25.1 | 42.2 | 64.4 | 46.5 | 57.5 | 87.3 | 54.6 | 93.9 ★ ⚠ trained on this test set | 89.6 | 12.9 | 49.7 | 11.0 | 10.9 | 64.8 | 65.5 | 68.0 | 68.3 | 41.2 | 48.7 |
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | Azure AI PII (Microsoft) | Presidio (open source) | LFM2.5 PII (Liquid AI) | Privacy Filter (OpenAI) | OpenMed PII E5 | OpenMed Privacy Filter | GLiNER PII (Urchade) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | Piiranha v1 (III) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EN | 24.4 | 32.8 | 78.3 | 48.2 ⚠ API 5,120-char limit; long docs truncated | 72.8 | 60.0 | 36.1 | 49.1 | 50.4 | 22.4 | 53.0 | 15.5 | 17.8 | 30.3 | 23.5 | 83.6 ★ ⚠ training data undisclosed | 65.4 | 66.9 | 72.6 |
| language | ShinrAI 1.1 | ShinrAI 1.2 | ShinrAI 1.3 | Azure AI PII (Microsoft) | Presidio (open source) | LFM2.5 PII (Liquid AI) | Privacy Filter (OpenAI) | OpenMed PII E5 | OpenMed Privacy Filter | GLiNER PII (Urchade) | GLiNER2 PII (Fastino) | GLiNER Large (Knowledgator) | GLiNER Edge (Knowledgator) | Piiranha v1 (III) | EU PII Safeguard (Tabularis) | EU PII Multilang (Bards.ai) | PII BERT (Gravitee) | De-identifier (Stanford AIMI) | DeID RoBERTa (OBI) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DE | 22.6 | 43.4 | 60.5 ★ | 55.9 | 38.6 | 27.8 | 13.4 | 34.2 | 12.2 | 10.3 | 15.8 | 6.3 | 10.1 | 19.4 | 43.1 | 48.9 | — | — | — |
| EN | 36.8 | 22.9 | 36.2 | 23.4 | 19.3 | 31.7 | 28.6 | 53.4 ★ | 13.1 | 4.8 | 10.2 | 3.2 | 4.2 | 29.0 | 48.2 | 34.8 | 36.3 | 42.5 | 30.9 |
| ES | 53.7 | 29.5 | 39.0 | 28.2 | 3.5 | 43.6 | 32.5 | 73.0 ★ | 13.9 | 6.2 | 10.4 | 2.9 | 4.3 | 54.8 | 49.4 | 36.0 | — | — | — |
| FR | 22.4 | 41.5 | 48.6 | 42.9 | 25.1 | 32.5 | 6.1 | 33.0 | 11.2 | 8.0 | 17.3 | 5.2 | 7.6 | 18.1 | 31.7 | 44.4 | — | — | — |
| IT | 29.5 | 16.9 | 13.3 | 11.2 | 7.0 | 34.0 | 10.4 | 36.6 ★ | 10.0 | 1.7 | 2.8 | 1.2 | 1.3 | 30.1 | 30.3 | 11.9 | — | — | — |
| PT | 30.1 | 38.2 | 48.7 | 48.8 | 25.8 | 28.8 | 13.4 | 43.5 | 18.0 | 9.4 | 18.5 | 6.9 | 12.0 | 26.4 | 44.2 | 52.4 ★ | — | — | — |
Strict-boundary scores (exact character spans) for every cell above ship in the raw run files.
Attack resistance
F1 score (0–100, higher is better), generous matching, identical texts for every system.
| system | score |
|---|---|
| Azure AI PII preview (Microsoft) unreleased preview API — not their shipped product | 88.8 |
| ShinrAI 1.3 (Innovius.AI / EECC Labs) | 84.9 |
| Azure AI PII (Microsoft) | 81.1 |
| Presidio (open source) | 75.6 |
| GLiNER2 PII (Fastino) | 74.3 |
| GLiNER PII (Urchade) | 70.9 |
| OpenMed Privacy Filter | 67.1 |
| OpenMed PII E5 | 66.2 |
| Privacy Filter (OpenAI) | 55.0 |
| Piiranha v1 (III) | 27.2 |
Microsoft's shipped product (Azure AI PII GA) scores 81.1 here — below ShinrAI 1.3. Their preview API leads by 3.9; it is not generally available.
Japanese head-to-head 🇯🇵
ShinrAI 1.3
F1, generous matching · grades people, places, streets and companies here; detects 27 classes overall.
LFM2 PII Extract JP (Liquid AI)
Their Japanese-only extraction model. It outputs five categories: names, addresses, companies, e-mails, phone numbers.
Long documents — how ShinrAI reads them
How long text is processed
The model reads 1024 tokens at once and slides over longer text with overlapping windows, merging spans at the seams — a 30-page file processes as one document. The serving layer adds automatic segmentation for conversational input: short text decodes whole; longer text also gets a sentence-level pass, and the two agree before a span counts.
Measured on long documents
At 4× letter length the scores hold: EN 96.6, DE 93.8 — no quality cliff. The court-ruling test above (TAB, ~5,000 characters per document) runs through this exact path. Fairness note: competitor models limited to 512 tokens were given the same windowing on long documents, so the table measures their models, not their truncation defaults.
Areas of improvement
We publish these because you should choose tools on measured numbers — ours included. All items below are on the 1.4 programme; the first is in training now.
| area | measured today | status |
|---|---|---|
| Very long records (REDACT test) | 36.5 vs Azure 51.2 | A dedicated long-record training corpus is part of the 1.4 campaign. |
| Synthetic US-paperwork register (Gretel, Nemotron) | 52.2 / 64.4 partial vs Liquid AI 84.5 / 87.3 | A paperwork-register training track is scoped for 1.4. |
| Court-document span conventions (TAB) | detection 78.3 partial, but strict spans 34.4 | We find the entities; the span boundaries follow different conventions (honorifics, full institution names). A serving-side convention profile is in work — no retraining needed. |
| Chat-attack prompts | 84.9 vs Azure preview 88.8 | Already ahead of Presidio and GLiNER2; decoder work for conversational text continues in 1.4. |
| Encyclopedic register in 8 of 15 languages | improved for 7 languages in 1.3; the other 8 regressed on encyclopedia text | Business-document quality is unaffected. Rebalancing both is the headline 1.4 training goal — running now. |
How fast is ShinrAI on your hardware?
Time to detect PII in one request, measured with the open model innovius/shinrai-pii-m-v1.3 (full precision, ONNX Runtime). Four text sizes: a chat message (62 tokens ≈ 45 words), a paragraph (317 tokens ≈ 230 words — the size behind every ShinrAI latency figure), a page (1,024 tokens, the model's window) and a long document (10,000 tokens ≈ 15 pages). Plain numbers are measured on that machine; ≈ is an estimate from the model's compute cost (±40 %); ~ is scaled from a measured sibling machine.
| Machine | RAM / VRAM | Chat message | Paragraph | Page | Long document | What it is good for |
|---|---|---|---|---|---|---|
| Single-board computers | ||||||
| Raspberry Pi 4 Model B (4 GB) | 4 GB · too small | ≈706 ms | ≈3.7 s | ≈12.5 s | ≈2.6 min | Documents & data pipelines only — too slow for chats |
| Raspberry Pi 4 Model B (8 GB) | 8 GB · fits | ≈706 ms | ≈3.7 s | ≈12.5 s | ≈2.6 min | Documents & data pipelines only — too slow for chats |
| Raspberry Pi 5 (8 GB) | 8 GB · fits | ≈193 ms | ≈982 ms | ≈3.3 s | ≈40.4 s | Agentic AI & workflows (not real-time chat) |
| Raspberry Pi 5 (16 GB) | 16 GB · fits | ≈193 ms | ≈982 ms | ≈3.3 s | ≈40.4 s | Agentic AI & workflows (not real-time chat) |
| Mini PCs | ||||||
| Intel N100 mini PC (16 GB) | 16 GB · fits | ≈174 ms | ≈835 ms | ≈2.8 s | ≈33.9 s | Agentic AI & workflows (not real-time chat) |
| Laptops | ||||||
| Apple M1 (MacBook Air class, 4P+4E) | 8 GB · fits | ~87 ms | ~458 ms | ~1.8 s | ≈16.9 s | Agentic AI & workflows (not real-time chat) |
| Apple M4 (MacBook Air / iMac / Mac mini, 4P+6E) | 16 GB · fits | ~16 ms | ~78 ms | ~310 ms | ~4.0 s | Real-time: AI chats & assistants |
| x86 laptop, 8 cores (Ryzen 7 7840U / Core Ultra 7 class) | 16 GB · fits | ~48 ms | ~189 ms | ~676 ms | ~6.8 s | Agentic AI & workflows (not real-time chat) |
| Desktops & workstations | ||||||
| Apple M4 Pro (Mac mini 2024, 10P+4E) | 64 GB · fits | 13 ms | 56 ms | 233 ms | 3.1 s | Real-time: AI chats & assistants |
| AMD Threadripper 7960X workstation (24 cores, CPU only) | 64 GB · fits | ≈21 ms | ≈64 ms | ≈193 ms | ≈2.3 s | Real-time: AI chats & assistants |
| Edge GPUs (Jetson) | ||||||
| NVIDIA Jetson Orin Nano Super (8 GB) | 8 GB · fits | ≈24 ms | ≈106 ms | ≈344 ms | ≈4.2 s | Agentic AI & workflows (not real-time chat) |
| NVIDIA Jetson Orin NX (16 GB) | 16 GB · fits | ≈26 ms | ≈116 ms | ≈380 ms | ≈4.6 s | Agentic AI & workflows (not real-time chat) |
| NVIDIA Jetson AGX Orin (64 GB) | 64 GB · fits | ≈13 ms | ≈45 ms | ≈140 ms | ≈1.7 s | Real-time: AI chats & assistants |
| Desktop / workstation GPUs | ||||||
| NVIDIA DGX Spark (GB10) | 128 GB · fits | ≈6 ms | ≈9 ms | ≈19 ms | ≈175 ms | Real-time: AI chats & assistants |
| NVIDIA RTX 3090 (24 GB) | 24 GB · fits | ≈6 ms | ≈9 ms | ≈17 ms | ≈151 ms | Real-time: AI chats & assistants |
| NVIDIA RTX 4090 (24 GB) | 24 GB · fits | ≈6 ms | ≈8 ms | ≈16 ms | ≈131 ms | Real-time: AI chats & assistants |
| Servers & cloud VMs — CPU only | ||||||
| AMD EPYC 7002 'Rome' VM, 2 vCPU (cgroup limit) | 6 GB · fits | 131 ms | 589 ms | 2.2 s | 29.8 s | Agentic AI & workflows (not real-time chat) |
| AMD EPYC 7002 'Rome' VM, 4 vCPU | 6 GB · fits | 83 ms | 350 ms | 1.2 s | 15.5 s | Agentic AI & workflows (not real-time chat) |
| AMD EPYC 7002 'Rome' VM, 8 vCPU | 6 GB · fits | 59 ms | 237 ms | 845 ms | 8.5 s | Agentic AI & workflows (not real-time chat) |
| AMD EPYC 7002 'Rome' VM, 16 vCPU | 6 GB · fits | 76 ms | 215 ms | 668 ms | 7.2 s | Agentic AI & workflows (not real-time chat) |
| Cloud VM, 4 vCPU x86 (AMD EPYC 9V74 "Genoa", shared, GitHub-hosted runner) | 16 GB · fits | 105 ms | 514 ms | 1.9 s | ≈21.0 s | Agentic AI & workflows (not real-time chat) |
| Cloud VM, 4 vCPU Arm (Neoverse N2 / Cobalt 100, GitHub-hosted runner) | 16 GB · fits | 92 ms | 442 ms | 1.7 s | ≈18.0 s | Agentic AI & workflows (not real-time chat) |
| Cloud VM, 4 vCPU Intel Ice Lake / Sapphire Rapids (AVX-512 VNNI) | 16 GB · fits | ~70 ms | ~298 ms | ~1.0 s | ~13.1 s | Agentic AI & workflows (not real-time chat) |
| Hugging Face Space, cpu-upgrade (8 vCPU, 32 GB) | 32 GB · fits | ~83 ms | ~350 ms | ~1.2 s | ~15.5 s | Agentic AI & workflows (not real-time chat) |
| Servers — GPU | ||||||
| NVIDIA Tesla P40 (24 GB, Pascal 2016) | 24 GB · fits | 9 ms | 25 ms | 93 ms | 1.6 s | Real-time: AI chats & assistants |
| NVIDIA T4 (16 GB) | 16 GB · fits | ≈10 ms | ≈31 ms | ≈93 ms | ≈1.1 s | Real-time: AI chats & assistants |
| NVIDIA L4 (24 GB) | 24 GB · fits | ≈6 ms | ≈10 ms | ≈19 ms | ≈178 ms | Real-time: AI chats & assistants |
| NVIDIA A10 (24 GB) | 24 GB · fits | ≈6 ms | ≈9 ms | ≈19 ms | ≈171 ms | Real-time: AI chats & assistants |
| NVIDIA L40S (48 GB) | 48 GB · fits | ≈6 ms | ≈7 ms | ≈10 ms | ≈62 ms | Real-time: AI chats & assistants |
| NVIDIA A100 (80 GB) | 80 GB · fits | ≈6 ms | ≈7 ms | ≈11 ms | ≈72 ms | Real-time: AI chats & assistants |
| NVIDIA H100 SXM (80 GB) | 80 GB · fits | ≈5 ms | ≈6 ms | ≈7 ms | ≈26 ms | Real-time: AI chats & assistants |
Raspberry Pi: works for document and data pipelines (a paragraph takes about a second on a Pi 5, four on a Pi 4), not for chats. 4 GB RAM minimum with the compact model file layout, 8 GB comfortable, 64-bit OS.
Memory: one full-precision model session needs about 2.0 GB of RAM as shipped, or about 0.9 GB with the model file in ONNX external-data layout (same speed) — the layout we recommend for small devices.
Through the ShinrAI service the default long-input mode re-reads texts over 1,200 characters sentence by sentence for better recall; that costs about 3× on a paragraph. The rows below show both modes on the same GPU.
| ShinrAI service on a Tesla P40 (2016) | Chat message | Paragraph | Letter (2 pages) | Long document |
|---|---|---|---|---|
| model only (whole text in one pass) | 9 ms | 25 ms | 175 ms | 1.6 s |
| default: long-input mode (sentence pieces) | 9 ms | 69 ms | 348 ms | 4.9 s |
Generated 2026-09-02 from research/hardware-map/hardware-map.json (scripts/hardware/render-map.py).
What gets detected — 19 heads, 27 classes
| class | ShinrAI 1.3 | Azure PII | GLiNER2 | Presidio | OpenAI PF |
|---|---|---|---|---|---|
| PERSON common / uncommon / rare (name rank) | native head | Person | zero-shot | PERSON (spaCy) | private persons only |
| CITY major / medium / small (population) | native head | City/Location (preview) | zero-shot | LOCATION (spaCy) | — |
| STREET generic / specific / local | native head | Address | zero-shot | — | private address |
| ORG international / national / regional (sitelinks) | native head | Organization | zero-shot (P 15–33%) | ORGANIZATION (off by default) | — |
| EMAIL std | native head | zero-shot | EMAIL_ADDRESS | private email | |
| USERNAME std | native head | — | zero-shot | — | — |
| URL std | native head | URL | zero-shot | URL | private url |
| PHONE std | native head | PhoneNumber | zero-shot | PHONE_NUMBER | private phone |
| ACCOUNT std | native head | IBAN / BankAccount | zero-shot | IBAN_CODE, US_BANK_NUMBER | — |
| WALLET std | native head | — | zero-shot | CRYPTO | — |
| CARD std | native head | CreditCardNumber | zero-shot | CREDIT_CARD | — |
| NATIONAL_ID std | native head | ~70 country ID categories | zero-shot | ~76 country recognizers | — |
| PLATE std | native head | LicensePlate (preview) | zero-shot | — | — |
| DOB std | native head | DateOfBirth (preview) | zero-shot | DATE_TIME (any date) | — |
| NETADDR std | native head | IPAddress | zero-shot | IP_ADDRESS | — |
| POSTAL_CODE std | native head | ZipCode (preview) | zero-shot | — | — |
| CUSTOMER_ID std | native head | — | zero-shot | — | — |
| SECRET std | native head | Password (preview) | zero-shot | — | — |
| CODENAME std | native head | — | zero-shot (43% recall on the 27 prompts; 7 languages) | — | — |
Glossary
| F1 score | One number from 0 to 100 combining "found everything" (recall) and "no false alarms" (precision). Higher is better. |
| Strict | A find only counts with the exact character boundaries and the right type. |
| Generous / partial | A find counts if it touches the right text with the right type. Fairer across systems with different span conventions. |
| Test set × language | Each table row is one public test set in one language, typically 300 texts. |
| Best threshold | Competitors report their best score across sensitivity settings. ShinrAI always reports its default settings. |
| ⚠ trained on this test set | The system saw this test's data during training — its score there is inflated and not comparable. |
| Precision / recall | Precision: of everything flagged, how much was right. Recall: of everything present, how much was found. |
| Beta locale | Hebrew ships for integration and feedback; its quality is below release level and is shown, and counted, everywhere. |
Method
- Every number on this page comes from one measurement harness. Raw run files: dataset innovius/shinrai-pii-bench-results.
- Third-party test data is evaluated, never redistributed.
- ShinrAI results come from a sealed evaluation with acceptance gates frozen before measurement.
- Latency figures are CPU/GPU production measurements; GPU projections are marked (est.).
ShinrAI (信頼, Japanese for trust) · Innovius × EECC Research Labs · trained at the Jülich Supercomputing Centre (JURECA, WestAI) · open model weights · try ShinrAI live — PII playground