PII model and benchmark report
Higher recall than any other PII detector.
Protegrity-PII is a 278M model that outperforms OpenAI, AWS and Microsoft in detection and classification of personal information. It scores 0.910 recall at 0.902 precision on the new PII-Scope benchmark, at a fraction of their time and cost.
Source text
Maya Patel asked us to update the delivery details for her order. Please use [email protected] for confirmations. Her callback number is 202-555-0147. The delivery address is 42 Willow Lane, Unit 3, Cedar Falls, IA 50613. Her date of birth is 14 May 1988, which was used to verify the request. Maya Patel confirmed that the updated details are correct.
Redacted text
[PERSON] asked us to update the delivery details for her order. Please use [EMAIL] for confirmations. Her callback number is [PHONE]. The delivery address is [ADDRESS]. Her date of birth is [DATE], which was used to verify the request. [PERSON] confirmed that the updated details are correct.
Overview
Detection, speed and cost
Protegrity-PII outperforms other models on detection, speed and cost. It has the highest typed recall on PII-Scope, and can process 1M characters (~2,000 documents) in half a second for a fraction of a cent. Frontier models come closest on detection, but they take far longer at thousands of times the cost.
Protegrity-PII
- Typed recall
- 0.915
- at 0.907 precision
- Seconds per million characters
- 0.49
- on one RTX 6000
- Cost per million characters
- $0.0005
- self-hosted on that GPU
Results
Recall that rivals frontier LLMs
Recall is the percentage of real personal information a model detects. In practice, every miss leaves personal information exposed. Typed recall is stricter, requiring correct detection and classification of personal information. F1 balances recall and precision.
Typed recall on PII-Scope · higher is better
- Protegrity-PII 0.910
- Claude Fable 5 0.848
- AWS Comprehend 0.460
- GLiNER2.5 Multi 0.422
- OpenAI Privacy Filter 0.389
- Microsoft Presidio 0.383
- Rampart 0.302
- Protegrity-PII
- Other detectors
- Frontier LLM reference (closed API)
Cost and speed
A fraction of a cent for a million characters
A compact token classifier does the work that frontier models do with prompts. Against Claude Fable 5, running 64 streams at once on a single RTX 6000 Blackwell:
Protegrity-PII vs Claude Fable 5
- ≈23,000× cheaper
- ≈18,000× faster
Cost per million characters
Time per million characters
Coverage
69 PII entity types, from names to neural data
A model cannot redact what it was never trained to label. Protegrity-PII learns a fine-grained taxonomy that is a superset of the native taxonomies of every model it was compared against.
Supported PII entity types (labels)
Supported by Protegrity-PII (69/69)
- ACCESS_SECURITY_CODE
- ACCOUNT_BALANCE
- ACCOUNTNAME
- ACCOUNTNUMBER
- AGE
- AMOUNT
- BIC
- BUILDING
- CARD_EXPIRATION_DATE
- CARD_SERVICE_CODE
- CARD_TRACK_DATA
- CCN
- CITY
- COMPANYNAME
- COUNTRY
- COUNTY
- CREDIT_SCORE
- CREDITCARDCVV
- CRIMINAL_RECORD
- CRYPTO_ADDRESS
- CURRENCY
- DATE
- DEVICE_IDENTIFIER
- DRIVERLICENSE
- FAX_NUMBER
- FINGERPRINT_DATA
- GENDER
- GENETIC_DATA
- GEOCOORD
- HEALTH_CONDITION
- HEALTH_PLAN_BENEFICIARY_NUMBER
- IBAN
- IDCARD
- IMMIGRATION_STATUS
- INCOME
- IP
- LOGIN_CREDENTIAL
- MAC
- MEDICAL_RECORD_NUMBER
- MOTHERS_MAIDEN_NAME
- NAME
- NEURAL_DATA
- PASSPORT
- PASSWORD
- PHONE
- PIN
- PLACE_OF_BIRTH
- POLITICAL_OPINION
- PROFESSIONAL_LICENSE_NUMBER
- RACE_ETHNICITY
- RELIGIOUS_BELIEF
- RETINA_SCAN_DATA
- SECADDRESS
- SEX_LIFE_DATA
- SEXUAL_ORIENTATION
- SSN
- STATE
- STREET
- TIME
- TITLE
- TRACKING_IDENTIFIER
- TRADE_UNION_MEMBERSHIP
- TRANSACTION_HISTORY
- URL
- USERNAME
- VEHICLE_IDENTIFICATION_NUMBER
- VEHICLE_REGISTRATION
- ZIPCODE
Benchmark
PII-Scope: the densest PII benchmark
The public PII datasets used were generated by LLMs, which leave some PII unlabeled. A model that correctly detects one of those entities is scored as wrong, penalizing the right answer. PII-Scope was built to avoid this: it is dense, varied and carefully labeled.
Entities per row · richer rows leave more room for error
- 8,030 rows across several domains
- 1.8× the entity density of the next densest benchmark
- Lowest overgeneration of the datasets we measured
Composition
Other includes Other formats 22.3%, Support ticket 3.9%, Social media post 3.5%, Forum post 3.3%, Form data 2.5%, Meeting notes 2.0%, Markdown 1.0%.
Other includes Social media 4.9%, Legal 4.7%, Telecom 4.3%, Education 2.0%, Hospitality 1.6%, General 1.5%, Energy and utilities 1.4%, Travel 1.4%, Manufacturing 1.3%, Real estate 1.2%, Logistics 1.2%.
The model
A token classifier, calibrated for recall
Protegrity-PII is not a prompted LLM. It tags every token in one forward pass, fast enough for logs and agent workflows and small enough to run on premises.
- Source text
-
Preprocessor
- Tokenized sliding windows
-
PII model
- XLM-RoBERTa encoder
- Classification head
-
BIO decoder
- Recall calibration
- Viterbi decoding
- Span generation
- Redacted text
How it was built
-
Built on XLM-RoBERTa
A pretrained, multi-language encoder with a linear head that tags each token as outside, the beginning or inside of one of 69 entity types. It is trained on the full taxonomy directly.
-
Trained on 1.2M samples
A synthetic dataset generated using the same technique as PII-Scope. It produced models with the highest scores on benchmarks they were not trained on.
-
Pruned and distilled
After layers were pruned, the smaller model was trained further by distillation from the original. The result is about 25% smaller and faster, with ~97% of the recall.
Get started
Read the report. Run the model.
Protegrity-PII Technical Report
Protegrity · Menlo Park, CA · Stamford, CT · Lisbon