Pavan Dangeti

Multilingual scam detection for Indian SMS and WhatsApp, measured against a cheap baseline

Classifies messages as payment scams or legitimate, names the manipulation tactics, and works in code-mixed Indian languages

AI/MLLLM

Problem
Indian scam messages mix English with Hindi, Tamil, Telugu or Bengali, often in Latin script, which English-trained filters handle poorly. Built for people who receive those messages and the services that screen them.
My role
Sole developer
Team
Personal project
Status
Shipped
Timeline
June–October 2026
Stack
Python, PyTorch, Transformers (XLM-R), scikit-learn, ONNX Runtime, FastAPI, SQLite
Next
Freeze a hand-labelled test set of real messages and measure on it.

What shipped

  • Raised scam-detection F1 from 0.908 to 0.998 on a largely synthetic 1,554-message validation set by fine-tuning XLM-R with adversarial augmentation.
  • Held F1 at 0.993 under a character-swap attack, against 0.863 for the TF-IDF baseline, by training on augmented copies instead of normalising input.
  • Cut CPU latency to 4.44 ms p50 per message, from 11.71 ms, with no measured F1 loss, by exporting the encoder to int8 ONNX.
Scam F1, validation
0.998
Baseline: TF-IDF + logistic regression 0.908
XLM-R base with adversarial augmentation on a 1,554-message validation split of a largely synthetic corpus, threshold 0.5, 95% CI 0.995–1.000 (results/evaluation_with_ci_aug.json). Evidence about the method, not real traffic
F1 under a character-swap attack
0.993
Baseline: TF-IDF + logistic regression 0.863
Six deterministic perturbation families applied to text only; augmented copies used in training, the validation split kept clean (results/robustness.json)
CPU latency p50, one message
4.44 ms
Baseline: 11.71 ms in PyTorch fp32
int8 ONNX encoder, 278 MB against 1,129 MB, with no measured F1 loss (results/latency.json); the TF-IDF baseline runs in 0.51 ms
On this page
Source: docs/demo/scamshield_demo.mp4 in the ScamShield repository

Context and constraints

Payment scams in India arrive by SMS and WhatsApp in mixed languages, such as “Unga account block aagidum, ippo click pannunga”, where English-trained filters do poorly. The product decision is scam or legitimate, plus which manipulation tactics a scam uses, explained in plain language. The constraint that shapes everything is data: the only real data with a known collection method is English messages from the UK, 2004–2011, so most of the corpus is synthetic or of unverified provenance.

Architecture

Sources are declared in code with a licence, provenance and SHA-256 hash, then scrubbed of personal data, deduplicated, grouped by near-duplicate and split by group. Every model, from TF-IDF to the fine-tuned encoder and the prompted LLMs, implements one predictor interface, so one evaluation harness scores them all on the same split, threshold and metric code. The encoder predicts the scam decision and the tactic set from one backbone. The FastAPI service serves the int8 ONNX encoder behind a model registry, falls back to TF-IDF where memory is short, logs to SQLite for feedback and drift monitoring, and answers a WhatsApp webhook. See the design document.

Architecture: ScamShieldWeb app and WhatsApp webhook, FastAPI, Model registry, SQLite feedback and drift log, XLM-R int8 ONNX (scam + tactic heads), TF-IDF + logistic regression fallback. Connections: Web app and WhatsApp webhook to FastAPI; FastAPI to Model registry; Model registry to XLM-R int8 ONNX (scam + tactic heads); Model registry to TF-IDF + logistic regression fallback; FastAPI to SQLite feedback and drift log.defaultwhen memory is shortWeb app and WhatsApp webhookclientFastAPIgatewayModel registryserviceSQLite feedback and drift logdata storeXLM-R int8 ONNX (scam + tactic heads)modelTF-IDF + logistic regression fallbackmodel

Request flows

  1. Classify a message: Web app and WhatsApp webhook → FastAPI → Model registry → XLM-R int8 ONNX (scam + tactic heads)
  2. Fall back on a small host: FastAPI → Model registry → TF-IDF + logistic regression fallback

Architecture: ScamShield

Components

Request flows

Choose a component to focus it, or a flow to watch its requests. In the view, drag or use the arrow keys to turn it and + or − to zoom. Escape closes.

Engineering decisions

How should the data be split?
Options
  • Random rows
  • Whole near-duplicate groups
Chose
Group near-duplicates (cosine ≥ 0.82) and split whole groups, stratified by language, script and label
Trade-off
The split is harder to balance and smaller cells get noisier.
Evidence
A random row split over a template-built corpus puts the same template in train and test and reports 1.0 F1; 0.82 is the threshold at which an independent check finds no cross-split pair above 0.90.
A fine-tuned encoder or a prompted LLM?
Options
  • Prompted 7B LLM
  • LoRA-tuned small decoder
  • Fine-tuned XLM-R
Chose
XLM-R base with augmentation, served as int8 ONNX
Trade-off
The encoder can't quote evidence for its tactics the way the baseline can.
Evidence
Prompted Qwen2.5-7B over-flagged and scored 0.693–0.726 F1 on a 354-message subsample, and a 135M LoRA model beat it by 14 points.
How is it made robust to obfuscated text?
Options
  • Normalise input before the model
  • Train on augmented copies
Chose
Augmentation in training
Trade-off
Training takes longer, and only six static attack families are covered.
Evidence
F1 under a character-swap attack is 0.993 for the augmented encoder against 0.863 for the baseline; normalisation would be a fixed function an attacker can invert.
Which model does the free demo serve?
Options
  • The encoder
  • The TF-IDF baseline
Chose
The baseline, with the encoder loaded by default wherever memory allows
Trade-off
Demo visitors see the cheaper, weaker model; GET /api/health says which one is serving.
Evidence
A process holding the int8 encoder uses about 1.1 GB, and the free host allows 512 MB; the baseline is 10.2 MB and 0.09 F1 lower on validation.

Evaluation

All models are scored on the same 1,554-message validation split at threshold 0.5 (README evaluation).

Model Precision Recall F1 Tactic macro-F1 CPU p50 Size
TF-IDF + LR 0.852 0.972 0.908 0.825 0.51 ms 10.2 MB
TF-IDF + LR, augmented 0.859 0.990 0.920 0.889 0.52 ms 16.6 MB
IndicBERTv2, fine-tuned 0.998 0.988 0.993 0.962 11.73 ms 1,120 MB
XLM-R base, fine-tuned 0.994 1.000 0.997 0.927 11.71 ms 1,129 MB
XLM-R base, augmented 0.996 1.000 0.998 0.987 4.44 ms (int8) 278 MB

The aggregate hides a failure: on native-script Bengali (192 messages) the baseline scores F1 0.652 at precision 0.48, while the encoders score 1.0. Near-perfect encoder scores on a partly template-built corpus mean the corpus was learned, not that the problem is solved. A CI test fails if the README drifts from the committed result files.

Deployment and operations

The live demo runs the baseline on a free host that sleeps when idle, so the first request can take a minute. The fine-tuned int8 model is published on Hugging Face with a model card and a data card. On CPU, batching doesn’t help: one process scores 178 messages a second one at a time against 91 in batches of 32, because every message in a batch is padded to the longest one. Throughput comes from more processes or a GPU.

Results

The method works on the data available: the encoder beats the baseline on every measure, holds up under character-swap attacks, and runs in under 5 ms on a CPU. What the numbers don’t show is performance on real traffic. The repository states the limits: no real-traffic measurement yet; Bengali is 37% of the corpus from an unverified source; Tamil and Telugu cells are too small to read individually; tactic labels are template-derived; explanations are English only; language detection is a word list; and labels come from one person.

Code highlight

Splitting whole near-duplicate groups, never rows

A template family lands entirely in training or entirely in validation, which is what turns the reported F1 into an estimate rather than a memory test.

def assign_splits(frame: pd.DataFrame, groups: np.ndarray, val_fraction: float, seed: int) -> np.ndarray:
    """Assign whole near-duplicate groups to train/val, stratified per cell."""
    splits = np.full(len(frame), "train", dtype=object)
    frame = frame.reset_index(drop=True)
    strata = frame["language"] + "|" + frame["script"] + "|" + frame["label"]
    group_frame = pd.DataFrame({"group": groups, "stratum": strata}).drop_duplicates()
    for stratum, members in group_frame.groupby("stratum"):
        member_groups = list(members["group"])
        sizes = {group: int((groups == group).sum()) for group in member_groups}
        target = val_fraction * sum(sizes.values())
        rng = random.Random(f"{seed}:{stratum}")
        rng.shuffle(member_groups)
        member_groups.sort(key=lambda group: -sizes[group])
        assigned = 0.0
        for group in member_groups:
            if assigned + sizes[group] / 2 <= target:
                mask = groups == group
                splits[mask] = "val"
                assigned += sizes[group]
    return splits

What I'd do next

Freeze the hand-labelled test set of real messages, which is in progress, and report every figure on it before anything else.

Talk about this project

I'm happy to walk through any decision here in more depth.

pssaikumardangeti@gmail.com