Multilingual scam detection for Indian SMS and WhatsApp, measured against a cheap baseline

The problem

Indian scam messages mix English with Hindi, Tamil, Telugu or Bengali, often in Latin script, which English-trained filters handle poorly. Built for people who receive those messages and the services that screen them.

Team: Personal project. Timeline: June–October 2026.

What changed

Scam F1, validation

0.998

Baseline: TF-IDF + logistic regression 0.908

XLM-R base with adversarial augmentation on a 1,554-message validation split of a largely synthetic corpus, threshold 0.5, 95% CI 0.995–1.000 (results/evaluation_with_ci_aug.json). Evidence about the method, not real traffic

F1 under a character-swap attack

0.993

Baseline: TF-IDF + logistic regression 0.863

Six deterministic perturbation families applied to text only; augmented copies used in training, the validation split kept clean (results/robustness.json)

CPU latency p50, one message

4.44 ms

Baseline: 11.71 ms in PyTorch fp32

int8 ONNX encoder, 278 MB against 1,129 MB, with no measured F1 loss (results/latency.json); the TF-IDF baseline runs in 0.51 ms

Architecture

Decision 1 of 4

How should the data be split?

Options

Random rows · Whole near-duplicate groups

Chose

Group near-duplicates (cosine ≥ 0.82) and split whole groups, stratified by language, script and label

Trade-off

The split is harder to balance and smaller cells get noisier.

Evidence

A random row split over a template-built corpus puts the same template in train and test and reports 1.0 F1; 0.82 is the threshold at which an independent check finds no cross-split pair above 0.90.

Decision 2 of 4

A fine-tuned encoder or a prompted LLM?

Options

Prompted 7B LLM · LoRA-tuned small decoder · Fine-tuned XLM-R

Chose

XLM-R base with augmentation, served as int8 ONNX

Trade-off

The encoder can't quote evidence for its tactics the way the baseline can.

Evidence

Prompted Qwen2.5-7B over-flagged and scored 0.693–0.726 F1 on a 354-message subsample, and a 135M LoRA model beat it by 14 points.

Decision 3 of 4

How is it made robust to obfuscated text?

Options

Normalise input before the model · Train on augmented copies

Chose

Augmentation in training

Trade-off

Training takes longer, and only six static attack families are covered.

Evidence

F1 under a character-swap attack is 0.993 for the augmented encoder against 0.863 for the baseline; normalisation would be a fixed function an attacker can invert.

Decision 4 of 4

Which model does the free demo serve?

Options

The encoder · The TF-IDF baseline

Chose

The baseline, with the encoder loaded by default wherever memory allows

Trade-off

Demo visitors see the cheaper, weaker model; GET /api/health says which one is serving.

Evidence

A process holding the int8 encoder uses about 1.1 GB, and the free host allows 512 MB; the baseline is 10.2 MB and 0.09 F1 lower on validation.

Splitting whole near-duplicate groups, never rows

def assign_splits(frame: pd.DataFrame, groups: np.ndarray, val_fraction: float, seed: int) -> np.ndarray:
    """Assign whole near-duplicate groups to train/val, stratified per cell."""
    splits = np.full(len(frame), "train", dtype=object)
    frame = frame.reset_index(drop=True)
    strata = frame["language"] + "|" + frame["script"] + "|" + frame["label"]
    group_frame = pd.DataFrame({"group": groups, "stratum": strata}).drop_duplicates()
    for stratum, members in group_frame.groupby("stratum"):
        member_groups = list(members["group"])
        sizes = {group: int((groups == group).sum()) for group in member_groups}
        target = val_fraction * sum(sizes.values())
        rng = random.Random(f"{seed}:{stratum}")
        rng.shuffle(member_groups)
        member_groups.sort(key=lambda group: -sizes[group])
        assigned = 0.0
        for group in member_groups:
            if assigned + sizes[group] / 2 <= target:
                mask = groups == group
                splits[mask] = "val"
                assigned += sizes[group]
    return splits

A template family lands entirely in training or entirely in validation, which is what turns the reported F1 into an estimate rather than a memory test.

Questions?

Read the full write-up