One Classification Pipeline to Extract Them All: Entity Extraction Across Multiple Languages

Introduction

This blog post covers how Cyera extended its unstructured data entity extraction pipeline to handle non-English languages in production.

Cyera's Data Security Posture Management (DSPM) platform gives organizations a comprehensive understanding of sensitive data across their entire data estate - detecting, assessing, and remediating security risks before they become security breaches. At the heart of this is an AI-powered classification engine capable of recognizing over 400 distinct data classes - from Personally Identifiable Information (PII) and Protected Health Information (PHI) to financial records, credentials, geographic data, and more - across both structured data sources (e.g., databases, tabular files) and unstructured data content (e.g., PDFs, emails, code, logs, text documents, and more). But here is the catch: sensitive data doesn't only come in English.

This blog post covers how Cyera extended its unstructured data entity extraction pipeline to handle non-English languages in production. All examples provided in the blog are synthetic data invented for the needs of the blog and do not contain real data.

Context-Aware Classification

“john.smith@gmail.com”

The same text span can receive different classifications depending on its surrounding context.

Our AI-powered classification engine is context-aware. Rather than relying on pattern matching alone, it interprets entities based on their surrounding context and assigns the appropriate classification. Consider the string “john.smith@gmail.com” for example. In a sentence such as "For further details, please contact john.smith@gmail.com," it is classified as an email address. However, in a login prompt such as "Enter your username followed by your password: john.smith@gmail.com," the very same string is classified as a username. Although the text span is identical in both cases, its classification changes because its semantic role in the surrounding context of the entity is different.

The Business Challenge

As Cyera expanded into new regions worldwide, we encountered a critical requirement: classify entities with high recall and precision in languages beyond English. Many customers store sensitive data in local languages, mix multiple languages within the same document, and use region-specific formats for names, addresses, identifiers, currencies, dates, and other sensitive information. For example, a support ticket written in Spanish may contain a copy-pasted error log in English that includes API keys, credentials, or other sensitive data. Furthermore, numerical and identifier formats vary significantly across countries and regions: the string "1.234,56" is a valid number in Germany (equivalent to 1,234.56 in U.S. notation), while national identifiers such as the Israeli Teudat Zehut, French NIR, Brazilian CPF, and Indian Aadhaar each follow different structures and validation rules. Without robust multilingual coverage, these variations can lead to classification gaps, resulting in incomplete risk visibility, missed sensitive entities, and inconsistent policy enforcement across a customer's global data estate.

The Data Science Challenge 

The task was clear: evaluate each model's precision and recall, starting with French data samples, systematically map failure modes, and translate those findings into an actionable plan.

As a data scientist, I was tasked with evaluating how well our already production-deployed deep-learning entity extraction models - fine-tuned on English data - would generalize to French, one of the first non-English languages we supported. The task was clear: evaluate each model's precision and recall, starting with French data samples, systematically map failure modes (missed entities, false positives, incorrect boundaries, context confusion), and translate those findings into an actionable plan for full French classification support and execute it as fast as possible. 

The Data Security Posture Management (DSPM) Pipeline

Before diving into how I approached this challenge, it's helpful to understand where the classification engine fits within Cyera's broader DSPM pipeline (Figure 1). The classification engine is not a standalone component - it operates as part of an end-to-end workflow that begins with file retrieval across the customer's data estate. Retrieved files pass through a file processing stage, where text is extracted from a wide range of unstructured and structured formats. The extracted text is then fed into the classification engine, whose outputs are consumed by downstream risk management, policy creation, and remediation layers that drive concrete security outcomes and value for our customers.  

Evaluating the Classification Engine on French Text

At a high level, the process was conceptually simple: take unstructured French text, pass it through the same classification engine used for English data, and then evaluate the resulting predictions through quantitative metrics and detailed error analysis (Figure 2). In practice, however, each step introduced its own challenges, from finding suitable French-language samples to establishing reliable ground truth in the absence of human annotated data.

Stage 1: Dataset Curation

The first challenge was to build a validation dataset in French. I curated a dataset comprising French text segments and candidate entity spans that covered the entity types of interest. Because the classification engine processes data at massive scale, it discards early on text inputs that are unlikely to contain entities of interest, narrowing the search space before deeper analysis. The validation set was constructed to imitate this flow - containing text segments and candidate entity spans that reflect what the classification engine would actually pass to its downstream stages. Each candidate text segment was then passed through a language detection model to ensure it represented French-language content. The resulting French text segments and candidate entity spans formed the validation dataset used throughout the evaluation process. For example, the validation dataset for credentials entities was curated to include text segments containing French keywords that are more likely to contain usernames and passwords, including:

  - nom d'utilisateur → username

  - compte utilisateur → user account

  - nom de connexion → login name

  - mot de passe → password

  - clé d'accès → access key

Stage 2: Evaluation and Error Analysis

This stage focused on evaluating the classification engine’s final outputs on French text and conducting a thorough error analysis. 

The French text segments identified in Stage 1 were passed through the fine-tuned encoder-decoder model and post-processing stages of the classification engine. The primary goal of the post-processing stage is to verify that a generated entity from the fine-tuned model is supported by the original text, preventing model hallucinations. Since I do not speak French, each text segment was also translated into English using an LLM to facilitate manual review and reasoning about the expected entities. The translation was used solely to facilitate evaluation and error analysis; the model itself operated on the original French text throughout inference and post-processing.

The error analysis discussed below focuses on two types of prediction errors. A false negative (FN) occurs when an entity present in the text segment does not appear in the final output of the classification engine, and a false positive (FP) occurs when the classification engine outputs an incorrect entity.

Establishing Ground Truth

Before going into the error analysis, first I had to establish a ground truth. In an entity extraction task, ground truth specifies what the entity is (its label, e.g., “FULL_NAME” or “USERNAME”) and where it appears in the text (its exact text span). An entity appears in the output of the classification engine only if the entity generated by the fine-tuned model passes the post-processing stage. A missing output classification can therefore occur for two reasons: 1) either the fine-tuned model did not generate an entity, or 2) the fine-tuned model generated an entity but it was rejected during post-processing. To establish ground truth that accounts for both failure modes, I used a dedicated LLM prompt (Figure 3). For each candidate text segment, the LLM prompt evaluated a single entity together with its predicted classification. If multiple entities were associated with the same text segment, the same text segment was evaluated multiple times, once for each candidate entity. 

When the classification engine produced an entity after post-processing, that entity was used as the evaluation target for the LLM prompt. When no output entity was produced, the candidate text span identified during dataset curation became the evaluation target instead. In both cases, the LLM determined whether the candidate represented a valid entity of the predicted class with respect to the French text, thereby establishing the ground truth for that entity.

When the evaluation target was the classification engine output, the LLM judged whether the prediction was a True Positive (TP) or a FP. When the evaluation target was the candidate text span identified during dataset curation (because no entity was produced by the classification engine), the LLM determined whether that candidate should have been classified as an entity by the classification engine. If the answer was Yes, the sample was counted as a FN; otherwise, the absence of a prediction was considered correct. This process enabled the identification of TPs, FPs, and FNs, allowing both precision and recall to be measured.

Together, the metrics and error analysis provided a clear picture of how well the classification engine generalized from English to French and highlighted the changes required to improve support for non-English languages.

Error Analysis

False-Negatives (FNs)

1. Fine-tuned Model Did Not Generate an Entity

I first examined text segments for which the fine-tuned model did not generate any entity (e.g., “FULL_NAME: []”). Although ground truth had already been established using the LLM prompt described earlier, I manually reviewed all such cases to validate both the LLM annotations and the candidate text spans produced during dataset curation. This additional review helped identify and correct occasional annotation mistakes in the ground truth creation that would otherwise have been incorrectly counted as FNs. For example, a candidate text span such as “<nom d'utilisateur>” being incorrectly annotated with a ground truth Username label despite serving as a placeholder. 

The remaining FNs represented situations in which the French text contained a target entity, yet the fine-tuned model failed to generate an entity (see Table 1 below). These examples suggested that the fine-tuned model lacked sufficiently robust representations of certain semantic patterns in French. Fortunately, such cases accounted for only a small fraction of the FNs, indicating that the model's ability to generalize from English to French was already reasonably strong despite it not being fine-tuned on French text.

2. Fine-tuned Model Generated an Entity Not Matching Corresponding Text in the Input

I then examined text segments in which the fine-tuned model did generate an entity, but the generated span did not match the corresponding text in the input. These cases were especially useful because they helped distinguish between errors that might suggest repeating the fine-tuning process and errors that could be solved efficiently in post-processing. One recurring example involved accented characters. For instance, a candidate text span could be “Éric Dubois”, while the fine-tuned model generated “Eric Dubois” without the accent (see Table 2). Although the prediction was semantically correct, it failed to match a text span in the input because of the lexical difference and was therefore rejected during post-processing. This happened only with a few accented characters. The model was able to generate the majority of accented characters correctly.

The initial hypothesis was that the model generated "Eric Dubois" instead of "Éric Dubois" because the tokenizer did not support the accented character "É". However, in that case the model should have generated the “<unk>” token instead of generating “E”. Inspection of the tokenizer vocabulary confirmed that the accented character "É" was indeed supported, ruling out missing tokenizer coverage as the cause. A more likely explanation was the composition of the fine-tuning training set. The model was fine-tuned on an English-language corpus. Although some non-English examples may have been present, they were expected to be rare, providing few, if any, examples of names containing accented characters.

Consequently, the model had few, if any, opportunities during fine-tuning to learn or reinforce these lexical forms. To summarize, even though the foundation model was pretrained on multilingual data, including French, the fine-tuning process emphasized English lexical patterns. As a result, the model appeared to favor the more common unaccented spelling during generation, suggesting that its ability to reproduce accented lexical forms was reduced after fine-tuning. 

Potential mitigation strategies included:

1. Repeat Fine-Tuning with Increased Training Coverage: Enrich the fine-tuning training set with additional examples containing accented characters and multilingual names so that the model learns to reproduce these lexical forms during generation.
2. Refine Post-Processing: Refine the post-processing stage to temporarily normalize accented characters when verifying that a generated entity from the fine-tuned model is supported by the original text. Candidate text spans from the original text and model-generated entities are first compared after normalizing accented characters (e.g., Éric → Eric). If a match is found, the original text span from the document is restored and propagated as the classification engine output, preserving the correct accented characters in the output.

Because this failure mode was narrowly scoped and could be corrected accurately through post-processing, repeating the fine-tuning process was unnecessary. Instead, refining the post-processing stage - the fastest production-ready solution - was chosen, enabling rapid improvement without compromising model quality.

3) Fine-tuned Model Generated the Correct Entity But in Another Language 

Another recurring pattern appeared for date-of-birth entities. In these cases, the fine-tuned model correctly identified the birth date but occasionally generated the month in English rather than preserving the original language. For example, the model generated “7 February 1995” instead of "7 février 1995”. Similar to the previous use case, although the prediction was semantically correct, it failed to match the candidate text span and was therefore rejected during post-processing. 

The following approaches were considered to address this issue:

1. Repeat Fine-Tuning with Increased Training Coverage.
2. Evaluate Alternative Decoding Strategies: Evaluate whether alternative text generation strategies, such as beam search or top-k sampling, produce outputs that better preserve the lexical form of the input than greedy decoding strategy.
3. Refine Post-Processing: refining the post-processing stage to translate the generated month back from English to French before matching it against the candidate text span. 

Because this issue was extremely rare and could be resolved accurately without fine-tuning, it was handled in the post-processing stage.

False-Positives (FPs)

After reviewing and correcting annotation mistakes uncovered during the FNs analysis, I recalculated the evaluation metrics using the revised ground truth. I then turned to analyze the FPs. As with the FNs, I did not assume that every FP was indeed an FP. I manually reviewed the FPs to identify and correct any remaining annotation mistakes before recomputing the final precision and recall metrics of the validation dataset.

Final Evaluation and Decisions

The improvements of the post-processing stage achieved high precision and recall on the validation dataset for the majority of entities tested. To ensure the improvements were not simply tailored to the validation data, I also evaluated the improved classification engine on a separate held-out test set. The test-set results were consistent with those observed on the validation set, providing confidence that the improvements generalized to previously unseen French text.

For a small number of entity types, however, the evaluation revealed performance gaps that could not be reliably addressed through post-processing alone. In those cases, the fine-tuning process was repeated using a multilingual training dataset that combined English, French, and additional languages. This improved the model ability to recognize language-specific patterns directly during inference while preserving strong performance on English data.

Key Takeaways

• Don’t rush into retraining. Model retraining is not always the fastest or best first solution. In some cases, targeted improvements to data processing, span matching, or post-processing can resolve meaningful failure modes faster while still preserving a high quality bar.
• Thorough error analysis is a must. Precision and recall quantify model performance, but they do not explain the underlying causes of errors. Systematically reviewing false positives and false negatives made it possible to distinguish between model limitations, tokenizer issues, post-processing gaps, and annotation mistakes, leading to faster and more targeted improvements.
• The right fix depends on the error type. Some issues were narrow and could be solved safely in post-processing; others required additional multilingual fine-tuning. The key was not choosing one approach by default, but mapping each error type to the highest value and lowest-risk solution that solved the business problem.
• Evaluation quality determines decision quality. Because labeled French data was not available upfront, building and validating a reliable ground-truth process was just as important as evaluating the model itself. Without trustworthy evaluation data, it would have been easy to optimize for the wrong target.

Acknowledgments

This blog focuses on one multilingual evaluation project, however it represents only a small part of a much larger effort. The work described here was made possible by the incredible work of the entire Research group at Cyera over the past several years. The classification engine, evaluation frameworks, and supporting infrastructure are the product of countless contributions from exceptionally talented colleagues. I am grateful to continue working alongside this team, building upon their work while contributing to the ongoing development of the classification engine and these broader efforts.

Thanks to Guy Vered and Marina Gandlin for their helpful comments on an earlier draft of this blog.