Innovius · ShinrAI 信頼 · semantic encryption

PII Benchmark Board

How well do PII detectors actually work? We measure our own releases and the field on the same texts, with the same scoring, and publish everything — including where we are behind.

🇩🇪 🇬🇧 🇺🇸 🇫🇷 🇪🇸 🇮🇹 🇵🇱 🇵🇹 🇧🇷 🇷🇺 🇺🇦 🇹🇷 🇸🇦 🇯🇵 🇰🇷 🇮🇱

15 locales · one model · 21 heads · 🇪🇺 EU-focus release, more coming soon · 🇮🇱 Hebrew is beta

The release on this board

ShinrAI 1.6.1 in the API today

languages15 locales

detection21 heads · dates and ages

Against ShinrAI 1.3: 15-locale median 95.0 → 95.5, Ukrainian 86.7 → 94.6, Hebrew (beta) 59.5 → 71.1, long-document redaction 36.5 → 57.6, encyclopedic recall on the eight rebuilt languages 34.5 → 81.9. On our field bench of real-world letters, forms and chats (the model alone), 1.6.1 protects 139 of 186 values; 1.3 protects 50. Exact-span means on public test sets: ai4privacy 75.2 → 79.2, MultiNERD 85.4 → 88.5. It passes our pre-release memorization checks. 1.6.1 runs in the managed API under the model ID v1.6; the open-weights model is ShinrAI 1.3 under the Innovius Open License.

Use 1.6.1 through the ShinrAI API →

ShinrAI 1.6.1 is measured and running in the managed API, hosted on STACKIT in the EU. Available today at getshinrai.com. The reference is ShinrAI 1.3: each table shows it next to 1.6.1 or 1.6, and the 1.4 and 1.5 values sit in a collapsed row under each table. The Enterprise product layers additional rule-based detection on top (checksum validators, deterministic identifiers); that layer is exact by construction and is not part of these scores.

Our 15-locale test

Business & clinic letters — our 15-locale test

Realistic letters, invoices and clinic notes per language. Find every name, place, company and identifier.

Sehr geehrte Frau Keller, Ihr Termin in Leipzig ist bestätigt — Kundennummer KD-2024-88371.

F1 score (0–100, higher is better · exact spans). 200 texts per locale. The 1.6.1 median of 95.5 (1.3: 95.0) includes the Hebrew beta locale. Azure AI PII, measured on the same texts: 53–76.

country	language	ShinrAI 1.6.1 F1	precision / recall	1.3 → 1.6.1
🇯🇵	JA	98.4	97.5 / 99.2	98.1 → 98.4
🇧🇷	PT-BR	98.0	97.4 / 98.6	97.3 → 98.0
🇬🇧🇺🇸	EN	97.4	96.2 / 98.7	97.7 → 97.4
🇪🇸	ES	97.4	96.5 / 98.2	96.9 → 97.4
🇷🇺	RU	96.9	95.9 / 97.8	96.4 → 96.9
🇫🇷	FR	96.6	95.9 / 97.3	96.0 → 96.6
🇮🇹	IT	95.7	94.7 / 96.7	95.0 → 95.7
🇩🇪🇦🇹🇨🇭	DE	95.5	94.7 / 96.4	94.4 → 95.5
🇰🇷	KO	95.2	93.2 / 97.2	95.5 → 95.2
🇺🇦	UA	94.6	92.0 / 97.4	86.7 → 94.6
🇵🇹	PT-PT	94.2	93.6 / 94.8	94.2 → 94.2
🇹🇷	TR	93.7	93.1 / 94.2	90.1 → 93.7
🇵🇱	PL	92.7	90.7 / 94.7	90.9 → 92.7
🇸🇦	AR	91.4	92.3 / 90.4	87.9 → 91.4
🇮🇱	HE beta	71.1	64.0 / 80.0	59.5 → 71.1

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
JA	98.5	98.5
PT-BR	98.2	98.3
EN	97.3	97.8
ES	97.6	97.7
RU	96.9	96.6
FR	96.4	96.2
IT	96.1	96.3
DE	95.5	94.5
KO	95.2	95.4
UA	95.0	95.1
PT-PT	94.6	94.3
TR	93.2	91.8
PL	93.5	91.7
AR	90.1	91.4
HE	71.2	70.5
median	95.5	95.4

Country matters: the model knows what a street name looks like in France versus Spain, and what a company name looks like in Japan. Detection and replacements follow each country's conventions.

Clinical and legal documents

ShinrAI detects dates and ages with two dedicated heads, DATE and AGE (since 1.5), trained on generated clinical and legal documents, French clinical reports, court judgments and EU legal sentences. The tables compare 1.6 with 1.3, which had neither head. Not yet measured with 1.6.1; these are the 1.6 values.

Court rulings

75.4

exact-span F1 on 127 ECHR judgments · 1.3: 34.4 · Azure preview: 47.6

Court rulings with dates

82.2

the same judgments, dates graded too · Azure preview: 55.5

EU legal text with dates

52.3

mean exact-span F1 over six MAPA languages · Azure preview: 48.4

Clinical cases

84.5

MEDDOCAN (es), exact-span F1 on people, places, organisations · 1.3: 51.0 · dates 94.9 · ages 95.2

Clinic letters

79.5

GraSCCo (de), exact-span F1 on people, places, organisations · 1.3: 50.2 · dates 80.0

Available today through the ShinrAI API, hosted on STACKIT in the EU.

Colors in the examples: person · place · organization · date / age

Clinic letters and clinical cases

Two public clinical corpora — MEDDOCAN (Spanish, 250 clinical cases) and GraSCCo (German, 63 clinic letters) — plus our own clinic-letter test sets. Find the people, places, organisations, dates and ages.

Sehr geehrte Frau Kollegin Dr. Weber, wir berichten über Herrn Lukas Brandt, 54 Jahre, der sich am 12.03.2026 in der Klinik St. Anna, Leipzig, vorstellte.

corpus	people · places · organisations ShinrAI 1.6	people · places · organisations 1.3	dates 1.6	ages 1.6
MEDDOCAN (es) — 250 clinical cases	84.5 / 93.1	51.0 / 70.0	94.9 / 98.2	95.2 / 96.3
GraSCCo (de) — 63 clinic letters	79.5 / 88.7	50.2 / 72.4	80.0 / 87.2	15.0 / 85.0
Our clinic letters (de · en · es · fr · it, 300 each)	86.8 · 88.9 · 86.2 · 86.5 · 88.7	86.5 · 89.5 · 87.9 · 86.1 · 90.5	—	—

Earlier releases: 1.4 and 1.5

corpus	people · places · organisations 1.5	dates 1.5	ages 1.5	people · places · organisations 1.4 build
MEDDOCAN (es) — 250 clinical cases	82.9 / 92.8	94.4 / 97.9	95.4 / 96.9	61.6 / 79.2
GraSCCo (de) — 63 clinic letters	80.0 / 90.0	79.4 / 88.0	15.0 / 85.0	57.2 / 75.9
Our clinic letters (de · en · es · fr · it, 300 each)	86.3 · 89.3 · 86.5 · 86.8 · 88.1	—	—	87.6 · 89.7 · 87.8 · 86.2 · 90.8

F1 (0–100), exact span / same-type overlap. People, places and organisations are graded together, dates and ages per type. 1.3 runs through the same scorer on the same documents; it had no date or age head. The 1.4 values in the collapsed table come from a training build of 1.4, not from its shipped weights.

Court rulings and EU law

Real European Court of Human Rights judgments (TAB, 127 long documents) and EUR-Lex sentences (MAPA, 300 per language). Find the parties, representatives, institutions and dates.

The applicant, Mr Henrik Hasslund, aged 57, lodged the application on 14 September 2016; the Aarhus District Court had dismissed the claim on 3 May 2019.

cell	dates	people	organisations
TAB — European Court of Human Rights judgments (en) 127 judgments	91.1	80.6	72.5
MAPA — EUR-Lex legal text (en) 300 sentences	96.9	43.6	24.4
MAPA (fr) 300 sentences	88.1	41.0	46.6
MAPA (de) 300 sentences	88.7	39.4	58.0

Earlier releases: 1.4 and 1.5

cell	dates 1.5	people 1.5	organisations 1.5
TAB — European Court of Human Rights judgments (en)	91.0	84.1	71.1
MAPA — EUR-Lex legal text (en)	97.5	45.1	24.3
MAPA (fr)	88.7	41.1	46.1
MAPA (de)	90.8	36.4	59.8

Exact-span F1 per type at the serving default. MAPA labels courts, parties and roles by conventions that differ from ours, and those cells stay low. 1.3 had no date head; on TAB it read 34.4 exact-span F1 on people and organisations together.

Public test sets — everyone in one harness

ShinrAI 1.6.1 against the field on the same test sets. The note under each ShinrAI score is ShinrAI 1.3 on the same cell; the 1.4 and 1.5 values sit in the collapsed row under each table. Gretel, Nemotron-PII, TAB and MAPA show ShinrAI 1.6: not yet measured with 1.6.1. The full per-corpus figures are on the model card.

Each competitor runs at its best threshold; ShinrAI runs at its default settings. Scores are F1 (0–100, higher is better) with generous matching — a find counts if it touches the right text with the right type (see the glossary). ★ marks the best system per row. ⚠ marks systems that trained on that very test set.

Colors in the examples: person · place · identifier · organization

ai4privacy — everyday messages

Short chat messages and form snippets with personal data planted in them. Four languages.

Hi Maria Ortega, your parcel to 8 Elm Road, Leeds arrives Friday.

language	ShinrAI 1.6.1	LFM2.5 PII (Liquid AI)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
DE	87.7 1.3: 85.4	69.4	35.6	11.0	21.3	90.3 ★ ⚠ training overlap likely	77.8	—	—	—
EN	81.3 1.3: 80.8	76.7	38.3	10.9	21.1	86.9 ★ ⚠ training overlap likely	70.2	68.4	50.0	69.1
FR	83.7 1.3: 82.6	74.5	37.9	9.7	16.2	90.6 ★ ⚠ training overlap likely	75.9	—	—	—
IT	83.6 1.3: 83.3	70.9	45.2	13.1	25.3	89.0 ★ ⚠ training overlap likely	73.8	—	—	—

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
DE	87.9	85.2
EN	81.5	81.7
FR	83.8	83.0
IT	80.4	79.0

MultiNERD — encyclopedia sentences

Wikipedia-style sentences in eight languages. Find the people and organizations.

Clara Immerwahr studied chemistry in Breslau before joining the university institute.

language	ShinrAI 1.6.1	LFM2.5 PII (Liquid AI)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
DE	92.9 ★ 1.3: 92.1	62.3	45.6	25.7	37.9	58.5	91.3 ⚠ training data undisclosed	—	—	—
EN	92.6 1.3: 94.9	60.9	45.8	17.3	28.5	58.7	96.2 ★ ⚠ training data undisclosed	86.9	65.1	86.5
ES	96.6 ★ 1.3: 97.7	61.5	39.0	16.5	29.5	64.4	93.3 ⚠ training data undisclosed	—	—	—
FR	92.3 ★ 1.3: 93.3	61.7	40.2	16.5	26.9	61.4	90.3 ⚠ training data undisclosed	—	—	—
IT	94.8 ★ 1.3: 96.8	56.2	33.9	11.8	21.7	60.4	91.4 ⚠ training data undisclosed	—	—	—
PL	88.2 ★ 1.3: 81.2	65.9	32.3	20.1	23.7	45.7	87.8 ⚠ training data undisclosed	—	—	—
PT	92.5 1.3: 94.7	62.7	38.4	15.4	28.8	61.0	96.1 ★ ⚠ training data undisclosed	—	—	—
RU	85.2 ★ 1.3: 82.5	53.9	34.9	17.9	18.1	51.5	82.5 ⚠ training data undisclosed	—	—	—

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
DE	91.5	91.8
EN	93.4	93.4
ES	96.0	95.0
FR	93.4	91.6
IT	95.1	94.1
PL	87.2	85.4
PT	92.4	92.2
RU	85.5	86.9

Gretel — synthetic business documents

Machine-generated certificates, applications and records — the paperwork register.

ADOPTION CERTIFICATE — confirms the adoption of Urvashi Jaggi, identifier UID-PRWBO4TB.

language	ShinrAI 1.6	Azure AI PII (Microsoft)	Presidio (open source)	LFM2.5 PII (Liquid AI)	Privacy Filter (OpenAI)	OpenMed PII E5	OpenMed Privacy Filter	GLiNER PII (Urchade)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	Piiranha v1 (III)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
EN	72.7 1.3: 52.2	30.7	44.2	84.5 ★	34.0	78.1	74.3	24.3	54.6	24.8	23.5	51.4	60.0	53.9	46.4	37.3	39.7

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
EN	70.9	—

Nemotron-PII — synthetic US forms

Machine-generated US application forms and letters (NVIDIA test set).

I, Jason, live at 87 Avenida De La Estrella. Date of birth 1987-05-22.

language	ShinrAI 1.6	Azure AI PII (Microsoft)	Presidio (open source)	LFM2.5 PII (Liquid AI)	Privacy Filter (OpenAI)	OpenMed PII E5	OpenMed Privacy Filter	GLiNER PII (Urchade)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	Piiranha v1 (III)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
EN	79.4 1.3: 64.4	46.5	57.5	87.3	54.6	93.9 ★ ⚠ trained on this test set	89.6	12.9	49.7	11.0	10.9	64.8	65.5	68.0	68.3	41.2	48.7

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
EN	76.7	—

TAB — real court rulings (long documents)

Real European Court of Human Rights decisions, about 5,000 characters each. Anonymize the people and organizations.

…lodged by Mr Henrik Hasslund, represented by Mr Tyge Trier, a lawyer in Copenhagen…

language	ShinrAI 1.6	Azure AI PII (Microsoft)	Presidio (open source)	LFM2.5 PII (Liquid AI)	Privacy Filter (OpenAI)	OpenMed PII E5	OpenMed Privacy Filter	GLiNER PII (Urchade)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	Piiranha v1 (III)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
EN	87.8 ★ 1.3: 78.3	48.2 ⚠ API 5,120-char limit; long docs truncated	72.8	60.0	36.1	49.1	50.4	22.4	53.0	15.5	17.8	30.3	23.5	83.6 ⚠ training data undisclosed	65.4	66.9	72.6

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
EN	87.3	—

MAPA — EU legal texts

Sentences from EU law (EUR-LEX) in six languages. Sparse — many sentences contain nothing to find, which makes it hard for every system.

Reference for a preliminary ruling from the Rechtbank Amsterdam concerning SF.

language	ShinrAI 1.6	Azure AI PII (Microsoft)	Presidio (open source)	LFM2.5 PII (Liquid AI)	Privacy Filter (OpenAI)	OpenMed PII E5	OpenMed Privacy Filter	GLiNER PII (Urchade)	GLiNER2 PII (Fastino)	GLiNER Large (Knowledgator)	GLiNER Edge (Knowledgator)	Piiranha v1 (III)	EU PII Safeguard (Tabularis)	EU PII Multilang (Bards.ai)	PII BERT (Gravitee)	De-identifier (Stanford AIMI)	DeID RoBERTa (OBI)
DE	62.7 ★ 1.3: 60.5	55.9	38.6	27.8	13.4	34.2	12.2	10.3	15.8	6.3	10.1	19.4	43.1	48.9	—	—	—
EN	36.6 1.3: 36.2	23.4	19.3	31.7	28.6	53.4 ★	13.1	4.8	10.2	3.2	4.2	29.0	48.2	34.8	36.3	42.5	30.9
ES	39.0 1.3: 39.0	28.2	3.5	43.6	32.5	73.0 ★	13.9	6.2	10.4	2.9	4.3	54.8	49.4	36.0	—	—	—
FR	49.9 ★ 1.3: 48.6	42.9	25.1	32.5	6.1	33.0	11.2	8.0	17.3	5.2	7.6	18.1	31.7	44.4	—	—	—
IT	14.2 1.3: 13.3	11.2	7.0	34.0	10.4	36.6 ★	10.0	1.7	2.8	1.2	1.3	30.1	30.3	11.9	—	—	—
PT	52.7 ★ 1.3: 48.7	48.8	25.8	28.8	13.4	43.5	18.0	9.4	18.5	6.9	12.0	26.4	44.2	52.4	—	—	—

Earlier releases: 1.4 and 1.5

language	ShinrAI 1.5	ShinrAI 1.4
DE	64.3	—
EN	36.4	—
ES	36.2	—
FR	50.6	—
IT	13.7	—
PT	53.7	—

Strict-boundary scores (exact character spans) for every cell above ship in the raw run files.

Head-to-head: ShinrAI 1.6.1 vs Azure AI PII preview

Microsoft's 2026-05 preview model, measured by us on 28 test-set cells in one harness, same scorer, two rules: exact spans, and spans that cover at least half of the gold span. ★ marks the leader per cell. Azure runs at its best threshold, ShinrAI at its default settings. Everyday messages (ai4privacy) and encyclopedia sentences (MultiNERD) show ShinrAI 1.6.1; the other rows show ShinrAI 1.6, not yet measured with 1.6.1.

Exact spans

15 / 28

cells where ShinrAI leads

Half-span overlap

15 / 28

cells where ShinrAI leads

language	ShinrAI · exact	Azure · exact	ShinrAI · ≥ 50 %	Azure · ≥ 50 %
Everyday messages (ai4privacy)
DE	81.6 1.3: 76.4	82.9 ★	84.4 ★ 1.3: 82.0	82.9
EN	74.9 ★ 1.3: 69.4	72.9	79.7 ★ 1.3: 77.7	74.1
FR	80.0 ★ 1.3: 75.8	78.3	82.1 ★ 1.3: 80.0	79.6
IT	80.3 ★ 1.3: 79.0	76.2	83.1 ★ 1.3: 82.9	77.3
Synthetic business documents (Gretel)
EN	56.3 ★ 1.3: 28.9	42.1	67.9 ★ 1.3: 43.1	52.4
Synthetic US forms (Nemotron-PII)
EN	72.9 ★ 1.3: 50.0	63.0	76.7 ★ 1.3: 56.7	69.9
Court rulings (TAB)
EN	75.4 ★ 1.3: 34.4	47.6	84.9 ★ 1.3: 69.0	79.5
EN + dates	82.2 ★	55.5	88.5 ★	81.9
EU legal text (MAPA)
DE	51.4 ★ 1.3: 44.4	44.0	59.4 ★ 1.3: 54.8	55.5
DE + dates	64.4 ★	33.2	70.2 ★	41.9
EN	30.3 ★ 1.3: 27.3	16.4	35.2 ★ 1.3: 33.7	21.5
EN + dates	54.2 ★	44.5	57.8 ★	48.8
ES	30.9 ★ 1.3: 30.9	28.6	30.9 ★ 1.3: 35.8	30.5
ES + dates	50.0	50.0	51.0	51.3 ★
FR	44.1 1.3: 41.0	49.4 ★	49.3 1.3: 47.3	51.9 ★
FR + dates	54.6	62.9 ★	59.4	64.7 ★
IT	3.8 ★ 1.3: 3.8	3.4	13.3 1.3: 11.4	13.4 ★
IT + dates	33.8	42.9 ★	40.8	50.2 ★
PT	47.1 ★ 1.3: 42.3	44.5	52.7 ★ 1.3: 48.3	46.9
PT + dates	57.0 ★	56.6	61.2 ★	58.4
Encyclopedia sentences (MultiNERD)
DE	90.0 1.3: 87.3	96.4 ★	92.9 1.3: 89.7	96.4 ★
EN	90.4 1.3: 90.4	95.7 ★	92.6 1.3: 94.4	96.3 ★
ES	93.2 1.3: 93.7	97.6 ★	96.6 1.3: 97.7	97.6 ★
FR	90.2 1.3: 89.6	96.1 ★	91.3 1.3: 91.2	96.1 ★
IT	93.4 1.3: 95.4	97.8 ★	94.8 1.3: 96.1	99.3 ★
PL	83.3 1.3: 66.9	88.0 ★	84.9 1.3: 71.1	88.9 ★
PT	88.4 1.3: 93.3	94.9 ★	91.8 1.3: 94.7	97.1 ★
RU	79.3 1.3: 66.9	83.3 ★	82.2 1.3: 75.8	85.6 ★

Earlier releases: 1.4 and 1.5

language	1.5 · exact	1.5 · ≥ 50 %	1.4 · exact	1.4 · ≥ 50 %
Everyday messages (ai4privacy)
DE	80.0	83.4	74.7	81.9
EN	73.3	78.5	70.7	77.7
FR	78.9	80.5	78.9	80.4
IT	76.6	79.9	73.4	78.5
Synthetic business documents (Gretel)
EN	53.4	65.3	49.0	61.5
Synthetic US forms (Nemotron-PII)
EN	69.6	73.1	62.5	66.2
Court rulings (TAB)
EN	75.8	84.4	41.3	74.7
EN + dates	82.4	88.4	29.8	53.9
EU legal text (MAPA)
DE	51.4	61.0	46.0	54.0
DE + dates	64.9	71.8	37.8	44.2
EN	30.8	35.0	25.2	31.1
EN + dates	54.7	57.4	20.5	25.3
ES	31.0	32.8	25.9	28.8
ES + dates	51.9	53.0	20.3	22.6
FR	44.0	50.6	43.1	46.3
FR + dates	54.8	60.6	37.7	40.5
IT	3.7	11.9	3.3	10.8
IT + dates	32.5	38.8	2.7	8.9
PT	46.9	53.3	46.6	52.5
PT + dates	55.8	60.8	40.2	45.3
Encyclopedia sentences (MultiNERD)
DE	88.7	91.5	87.2	91.4
EN	91.2	93.4	89.6	93.4
ES	92.1	95.5	91.1	95.0
FR	90.7	92.8	89.0	90.6
IT	93.7	95.1	92.0	94.1
PL	79.8	84.0	71.5	76.4
PT	91.0	92.4	88.7	92.2
RU	77.6	83.1	72.3	81.5

Everyday messages (ai4privacy): ShinrAI 1.6.1 leads on 3 of 4 cells with exact spans, Azure on 1. Synthetic business documents (Gretel): ShinrAI 1.6 leads on 1 of 1 cells with exact spans, Azure on 0. Synthetic US forms (Nemotron-PII): ShinrAI 1.6 leads on 1 of 1 cells with exact spans, Azure on 0. Court rulings (TAB): ShinrAI 1.6 leads on 2 of 2 cells with exact spans, Azure on 0. EU legal text (MAPA): ShinrAI 1.6 leads on 8 of 12 cells with exact spans, Azure on 3. Encyclopedia sentences (MultiNERD): ShinrAI 1.6.1 leads on 0 of 8 cells with exact spans, Azure on 8. "+ dates" rows also grade dates, which ShinrAI detects with its DATE head. These numbers are not comparable with the generous-matching tables above.

Attack resistance

Chat-attack prompts

27 chat messages written the way people actually leak data to an AI — nicknames, hints, spread-out details — designed to slip past filters (from a 2026 privacy study).

"my colleague — let's call him R. — just moved to the office on Karlsplatz for project Bluebird…"

F1 score (0–100, higher is better), generous matching, identical texts for every system.

system	score
Azure AI PII preview (Microsoft) unreleased preview API — not their shipped product	88.8
ShinrAI 1.6.1 1.3: 84.9 · on the updated gold that grades dates and ages: 87.2 (1.3: 82.3)	82.9
Azure AI PII (Microsoft)	81.1
Presidio (open source)	75.6
GLiNER2 PII (Fastino)	74.3
GLiNER PII (Urchade)	70.9
OpenMed Privacy Filter	67.1
OpenMed PII E5	66.2
Privacy Filter (OpenAI)	55.0
Piiranha v1 (III)	27.2

Earlier releases: 1.4 and 1.5

system	original gold	updated gold
ShinrAI 1.5	81.3	85.3
ShinrAI 1.4	85.1	82.8

Microsoft's shipped product (Azure AI PII GA) scores 81.1 here; ShinrAI 1.6.1 scores 82.9. Their preview API scores 88.8; it is not generally available. This suite's original gold has no date or age labels, so every date and age that ShinrAI finds counts against it in this table. The competitor rows stay on the original gold until their date and age mappings are validated on the updated gold.

Japanese head-to-head 🇯🇵

Our name is Japanese — so is the test

300 Japanese business and clinic letters. ShinrAI (信頼) versus Liquid AI's dedicated Japanese PII model.

担当者入迫かつなが様 — 川崎フロンターレ、川崎市中原区

ShinrAI 1.6

98.3

F1, generous matching · grades people, places, streets and companies here; detects 21 heads overall. Not yet measured with 1.6.1; this is the 1.6 value.

1.3: 98.1

LFM2 PII Extract JP (Liquid AI)

56.6

Their Japanese-only extraction model. It outputs five categories: names, addresses, companies, e-mails, phone numbers.

Earlier releases: 1.4 and 1.5

system	score
ShinrAI 1.5	98.4
ShinrAI 1.4	—

Long documents — how ShinrAI reads them

How long text is processed

The model reads 1024 tokens at once and slides over longer text with overlapping windows, merging spans at the seams — a 30-page file processes as one document. The serving layer adds automatic segmentation for conversational input: short text decodes whole; longer text also gets a sentence-level pass, and the two agree before a span counts.

Measured on long documents

At 4× letter length the scores hold: EN 97.5, DE 95.4 (1.6.1; 1.3: EN 97.7, DE 94.5) — no quality cliff. The court-ruling test above (TAB, ~5,000 characters per document) runs through this exact path. Fairness note: competitor models limited to 512 tokens were given the same windowing on long documents, so the table measures their models, not their truncation defaults.

Earlier releases: 1.4 and 1.5

system	EN 4×	DE 4×
ShinrAI 1.5	97.1	95.2
ShinrAI 1.4	97.8	94.9

Areas of improvement

We publish these because you should choose tools on measured numbers — ours included. Each row compares 1.6.1 with 1.3 and the strongest competitor, unless the row names 1.6 (not yet measured with 1.6.1); the open items stay on our programme.

area	measured today	status
Very long records (REDACT test)	57.6 vs Azure preview 51.2 (1.3: 36.5)	Recall on the 4-head view rose from 24.9 (1.3) to 51.8. Precision on the 14-head view, 54.5, is the next item.
Synthetic US-paperwork register (Gretel, Nemotron)	1.6: 72.7 / 79.4 generous vs Liquid AI 84.5 / 87.3 (1.3: 52.2 / 64.4)	The paperwork register stays an open item.
Court-document span conventions (TAB)	1.6: people 80.6, dates 91.1, organisations 72.5 exact-span (1.3: 34.4 on people and organisations together)	Institution names follow other conventions and stay open.
Chat-attack prompts	82.9 vs Azure preview 88.8 (original gold); 87.2 on the updated gold	Ahead of every shipped product. The unreleased Azure preview API leads on the original gold.
Encyclopedic register in 8 of 15 languages	recall 34.5 (1.3) → 81.9 (1.6.1)	Closed for the most part. The seven western languages read 92.3 recall (1.3: 86.0). In Portuguese, 221 of the 395 gold texts are also training texts of every version since 1.3; without them the seven languages read 92.2 recall (1.3: about 85.4).
Hebrew (beta)	71.1 on our 15-locale test (1.3: 59.5)	Below release level. Shown and counted in every median.

Earlier releases: 1.4 and 1.5

area	ShinrAI 1.5	ShinrAI 1.4
Very long records (REDACT test)	55.6	46.4
Synthetic US-paperwork register (Gretel, Nemotron)	70.9 / 76.7	—
Court-document span conventions (TAB)	84.1	—
Chat-attack prompts	81.3	85.1
Encyclopedic register in 8 of 15 languages	72.8	66.5
Hebrew (beta)	71.2	70.5

How fast is ShinrAI on your hardware?

Time to detect PII in one request, measured with the open model innovius/shinrai-pii-m-v1.3 (full precision, ONNX Runtime). Four text sizes: a chat message (62 tokens ≈ 45 words), a paragraph (317 tokens ≈ 230 words — the size behind every ShinrAI latency figure), a page (1,024 tokens, the model's window) and a long document (10,000 tokens ≈ 15 pages). Plain numbers are measured on that machine; ≈ is an estimate from the model's compute cost (±40 %); ~ is scaled from a measured sibling machine.

Machine	RAM / VRAM	Chat message	Paragraph	Page	Long document	What it is good for
Single-board computers
Raspberry Pi 4 Model B (4 GB)	4 GB · too small	≈706 ms	≈3.7 s	≈12.5 s	≈2.6 min	Documents & data pipelines only — too slow for chats
Raspberry Pi 4 Model B (8 GB)	8 GB · fits	≈706 ms	≈3.7 s	≈12.5 s	≈2.6 min	Documents & data pipelines only — too slow for chats
Raspberry Pi 5 (8 GB)	8 GB · fits	≈193 ms	≈982 ms	≈3.3 s	≈40.4 s	Agentic AI & workflows (not real-time chat)
Raspberry Pi 5 (16 GB)	16 GB · fits	≈193 ms	≈982 ms	≈3.3 s	≈40.4 s	Agentic AI & workflows (not real-time chat)
Mini PCs
Intel N100 mini PC (16 GB)	16 GB · fits	≈158 ms	≈774 ms	≈2.6 s	≈31.6 s	Agentic AI & workflows (not real-time chat)
Laptops
Apple M1 (MacBook Air class, 4P+4E)	8 GB · fits	~87 ms	~458 ms	~1.8 s	≈16.9 s	Agentic AI & workflows (not real-time chat)
Apple M4 (MacBook Air / iMac / Mac mini, 4P+6E)	16 GB · fits	~16 ms	~78 ms	~310 ms	~4.0 s	Real-time: AI chats & assistants
x86 laptop, 8 cores (Ryzen 7 7840U / Core Ultra 7 class)	16 GB · fits	~48 ms	~189 ms	~676 ms	~6.8 s	Agentic AI & workflows (not real-time chat)
Desktops & workstations
Apple M4 Pro (Mac mini 2024, 10P+4E)	64 GB · fits	13 ms	56 ms	233 ms	3.1 s	Real-time: AI chats & assistants
AMD Threadripper 7960X workstation (24 cores, CPU only)	64 GB · fits	≈21 ms	≈65 ms	≈193 ms	≈2.3 s	Real-time: AI chats & assistants
Edge GPUs (Jetson)
NVIDIA Jetson Orin Nano Super (8 GB)	8 GB · fits	≈24 ms	≈106 ms	≈344 ms	≈4.2 s	Agentic AI & workflows (not real-time chat)
NVIDIA Jetson Orin NX (16 GB)	16 GB · fits	≈26 ms	≈116 ms	≈380 ms	≈4.6 s	Agentic AI & workflows (not real-time chat)
NVIDIA Jetson AGX Orin (64 GB)	64 GB · fits	≈13 ms	≈45 ms	≈140 ms	≈1.7 s	Real-time: AI chats & assistants
Desktop / workstation GPUs
NVIDIA DGX Spark (GB10)	128 GB · fits	≈7 ms	≈12 ms	≈28 ms	≈278 ms	Real-time: AI chats & assistants
NVIDIA RTX 3090 (24 GB)	24 GB · fits	≈6 ms	≈11 ms	≈24 ms	≈239 ms	Real-time: AI chats & assistants
NVIDIA RTX 4090 (24 GB)	24 GB · fits	≈6 ms	≈10 ms	≈22 ms	≈206 ms	Real-time: AI chats & assistants
Servers & cloud VMs — CPU only
AMD EPYC 7002 'Rome' VM, 2 vCPU (cgroup limit)	6 GB · fits	131 ms	589 ms	2.2 s	29.8 s	Agentic AI & workflows (not real-time chat)
AMD EPYC 7002 'Rome' VM, 4 vCPU	6 GB · fits	83 ms	350 ms	1.2 s	15.5 s	Agentic AI & workflows (not real-time chat)
AMD EPYC 7002 'Rome' VM, 8 vCPU	6 GB · fits	59 ms	237 ms	845 ms	8.5 s	Agentic AI & workflows (not real-time chat)
AMD EPYC 7002 'Rome' VM, 16 vCPU	6 GB · fits	76 ms	215 ms	668 ms	7.2 s	Agentic AI & workflows (not real-time chat)
Cloud VM, 4 vCPU x86 (AMD EPYC 9V74 "Genoa", shared, GitHub-hosted runner)	16 GB · fits	105 ms	514 ms	1.9 s	≈21.0 s	Agentic AI & workflows (not real-time chat)
Cloud VM, 4 vCPU Arm (Neoverse N2 / Cobalt 100, GitHub-hosted runner)	16 GB · fits	92 ms	442 ms	1.7 s	≈18.0 s	Agentic AI & workflows (not real-time chat)
Cloud VM, 4 vCPU Intel Ice Lake / Sapphire Rapids (AVX-512 VNNI)	16 GB · fits	~70 ms	~298 ms	~1.0 s	~13.1 s	Agentic AI & workflows (not real-time chat)
Hugging Face Space, cpu-upgrade (8 vCPU, 32 GB)	32 GB · fits	~83 ms	~350 ms	~1.2 s	~15.5 s	Agentic AI & workflows (not real-time chat)
STACKIT c2a.4d (4 dedicated AMD EPYC Genoa vCPU, 8 GB)	8 GB · fits	76 ms	364 ms	1.2 s	13.4 s	Agentic AI & workflows (not real-time chat)
STACKIT n2.14d.g1 host CPU path (14 dedicated Intel Sapphire Rapids vCPU)	50 GB · fits	26 ms	91 ms	337 ms	≈3.4 s	Real-time: AI chats & assistants
STACKIT c2i.4 (4 shared Intel Ice Lake vCPU, overcommit 4, 8 GB)	8 GB · fits	57 ms	250 ms	1.2 s	16.3 s	Agentic AI & workflows (not real-time chat)
Servers — GPU
NVIDIA Tesla P40 (24 GB, Pascal 2016)	24 GB · fits	9 ms	25 ms	93 ms	1.6 s	Real-time: AI chats & assistants
NVIDIA T4 (16 GB)	16 GB · fits	≈10 ms	≈31 ms	≈93 ms	≈1.1 s	Real-time: AI chats & assistants
NVIDIA L4 (24 GB)	24 GB · fits	≈7 ms	≈12 ms	≈28 ms	≈282 ms	Real-time: AI chats & assistants
NVIDIA A10 (24 GB)	24 GB · fits	≈7 ms	≈12 ms	≈27 ms	≈271 ms	Real-time: AI chats & assistants
NVIDIA L40S (48 GB)	48 GB · fits	6 ms	8 ms	20 ms	305 ms	Real-time: AI chats & assistants
NVIDIA A100 (80 GB)	80 GB · fits	≈6 ms	≈8 ms	≈14 ms	≈112 ms	Real-time: AI chats & assistants
NVIDIA H100 SXM (80 GB)	80 GB · fits	≈6 ms	≈6 ms	≈8 ms	≈39 ms	Real-time: AI chats & assistants

Raspberry Pi: works for document and data pipelines (a paragraph takes about a second on a Pi 5, four on a Pi 4), not for chats. 4 GB RAM minimum with the compact model file layout, 8 GB comfortable, 64-bit OS.

Memory: one full-precision model session needs about 1.9 GB of RAM as shipped, or about 0.9 GB with the model file in ONNX external-data layout (same speed) — the layout we recommend for small devices.

Through the ShinrAI service the default long-input mode re-reads texts over 1,200 characters sentence by sentence for better recall; that costs about 3× on a paragraph. The rows below show both modes on the same GPU.

ShinrAI service on a Tesla P40 (2016)	Chat message	Paragraph	Letter (2 pages)	Long document
model only (whole text in one pass)	9 ms	25 ms	175 ms	1.6 s
default: long-input mode (sentence pieces)	9 ms	69 ms	348 ms	4.9 s

Generated 2026-09-03 from research/hardware-map/hardware-map.json (scripts/hardware/render-map.py).

What gets detected — 21 heads

class	ShinrAI 1.6.1	Azure PII	GLiNER2	Presidio	OpenAI PF
PERSON common / uncommon / rare (name rank)	native head	Person	zero-shot	PERSON (spaCy)	private persons only
CITY major / medium / small (population)	native head	City/Location (preview)	zero-shot	LOCATION (spaCy)	—
STREET generic / specific / local	native head	Address	zero-shot	—	private address
ORG international / national / regional (sitelinks)	native head	Organization	zero-shot (P 15–33%)	ORGANIZATION (off by default)	—
EMAIL std	native head	Email	zero-shot	EMAIL_ADDRESS	private email
USERNAME std	native head	—	zero-shot	—	—
URL std	native head	URL	zero-shot	URL	private url
PHONE std	native head	PhoneNumber	zero-shot	PHONE_NUMBER	private phone
ACCOUNT std	native head	IBAN / BankAccount	zero-shot	IBAN_CODE, US_BANK_NUMBER	—
WALLET std	native head	—	zero-shot	CRYPTO	—
CARD std	native head	CreditCardNumber	zero-shot	CREDIT_CARD	—
NATIONAL_ID std	native head	~70 country ID categories	zero-shot	~76 country recognizers	—
PLATE std	native head	LicensePlate (preview)	zero-shot	—	—
DOB std	native head	DateOfBirth (preview)	zero-shot	DATE_TIME (any date)	—
NETADDR std	native head	IPAddress	zero-shot	IP_ADDRESS	—
POSTAL_CODE std	native head	ZipCode (preview)	zero-shot	—	—
CUSTOMER_ID std	native head	—	zero-shot	—	—
SECRET std	native head	Password (preview)	zero-shot	—	—
CODENAME std	native head	—	zero-shot (43% recall on the 27 prompts; 7 languages)	—	—
DATE calendar dates and event years — new in 1.5; dates of birth stay DOB	native head	DateTime (any date)	zero-shot	DATE_TIME (any date)	—
AGE a stated age: number with unit, bare number in an age field, range, word form — new in 1.5	native head	Age	zero-shot	—	—

One record, everything at once

Dr. Anna Keller (Leipzig, Lindenweg 12) of Volksbank Nordpark — anna.k@example.de · +49 170 1234567 · IBAN DE89 3704 0044 0532 0130 00 · customer KD-448291 · project Bluebird · aged 54 · appointment 12 March 2026

Plus three attribute signals no other system ships — cultural origin (51 classes), gender expression, name part — the steering data for replacements that read naturally and restore losslessly.

Glossary

F1 score	One number from 0 to 100 combining "found everything" (recall) and "no false alarms" (precision). Higher is better.
Strict	A find only counts with the exact character boundaries and the right type.
Generous / partial	A find counts if it touches the right text with the right type. Fairer across systems with different span conventions.
Test set × language	Each table row is one public test set in one language, typically 300 texts.
Best threshold	Competitors report their best score across sensitivity settings. ShinrAI always reports its default settings.
⚠ trained on this test set	The system saw this test's data during training — its score there is inflated and not comparable.
Precision / recall	Precision: of everything flagged, how much was right. Recall: of everything present, how much was found.
Beta locale	Hebrew ships for integration and feedback; its quality is below release level and is shown, and counted, everywhere.

Method

Every number on this page comes from one measurement harness. Raw run files: dataset innovius/shinrai-pii-bench-results.
Third-party test data is evaluated, never redistributed.
ShinrAI results come from a sealed evaluation with acceptance gates frozen before measurement.
Latency figures are CPU/GPU production measurements; GPU projections are marked (est.).

ShinrAI (信頼, Japanese for trust) · Innovius × EECC Research Labs · trained at the Jülich Supercomputing Centre (JURECA, WestAI) · open model weights · try ShinrAI live — PII playground