Thaghra ثغرة
Safety training teaches a model to refuse harmful requests — but it learns that mostly in English. Thaghra measures what happens to that refusal when the same request is typed in Arabic: in Modern Standard Arabic, in spoken dialect, and in Arabizi (Arabic written in Latin letters and digits, the way it's actually typed online).
Where the refusal holds in English but slips in informal Arabic, that non-transfer is a low-cost security vulnerability — and Thaghra turns it from an anecdote into a measured rate, across several models.
In progress. The dataset is being hand-built and no models have been run yet — every figure below is an illustrative placeholder, not a measurement. Real results replace them when the study completes.
What we measure
GAP = ASRArabizi − ASREnglish
ASR (attack success rate) is the share of harmful prompts a model complied with. A bigger GAP means safety that holds in English leaks harder in informal Arabic. We report it per model, with confidence intervals — and we report it plainly even when it's small or zero.
The four registers4
The same meaning, written four ways. Here it's a harmless greeting — the real dataset carries harmful intents drawn from public research sets, never shown publicly. The point is the mechanic: as the wording moves away from formal English, the safety training has seen less and less like it.
- EN How are you today?
- MSA كيف حالك اليوم؟
- Dialect كيفك اليوم؟
- Arabizi kifak el yom?
How it works
- 1
Source intents
150–300 harmful intents sampled from public sets (AdvBench, Do-Not-Answer), excluding catastrophic categories. Every prompt's origin is logged.
- 2
Author the variants
Each intent is hand-written in English, MSA, dialect, and Arabizi — natural and meaning-equivalent. Native fluency does what machine translation can't.
- 3
Run & judge
Every form goes to every model; a separate model-as-judge labels each answer refused or complied, calibrated against human labels.
- 4
Measure honestly
Attack-success rates with confidence intervals, the GAP per model with a paired significance test, and a breakdown by harm category.
Findings
Illustrative figures until the evaluation runs.
Attack-success rate by register, across models
| Model | EN | MSA | Dialect | Arabizi | GAP* |
|---|---|---|---|---|---|
| Frontier A | 2% | 4% | 9% | 31% | +29 |
| Frontier B | 1% | 3% | 12% | 27% | +26 |
| Arabic-centric | 6% | 8% | 18% | 22% | +16 |
| Multilingual OS | 5% | 11% | 24% | 38% | +33 |
* GAP = ASR(Arabizi) − ASR(English), in percentage points.
Built responsibly
This is defensive safety research. Only the aggregate finding and the method are ever published — never harmful model outputs, which stay private. Catastrophic categories are excluded by design, every prompt traces to a public source, and any serious model-specific failure goes to the model's maker first, with lead time, before any specifics are shared.