National Student Research Institution · Research Archive · NSRI‑J‑2026‑0041

ScamShield: interpretable multi‑signal scam detection

Vishwajeet Adkine — National Student Research Institution (NSRI)

Abstract

Scam and phishing detection systems typically rely on rigid heuristic rules or opaque large language models. This paper presents ScamShield, a hybrid, interpretable pipeline that combines a 24‑feature supervised machine‑learning model, an LLM semantic safety layer, and rule‑based heuristics in a unified ensemble. Evaluated on a synthetic benchmark of 19,992 messages spanning 17 scam categories and validated on the UCI SMS Spam Collection (5,574 real‑world messages)^[15], ScamShield achieves cross‑validated F1 = 0.9969 ± 0.0004 on the synthetic benchmark and F1 = 0.9303 ± 0.0098 on real‑world data — competitive with fine‑tuned DistilBERT (estimated F1 ≈ 0.97–0.99)^[11] while requiring 125× less storage, operating at sub‑5 ms inference latency, and providing full per‑prediction interpretability via coefficient attribution.

Statistical significance is confirmed by McNemar's test against all four baselines (p << 0.001). Adversarial evaluation reveals recall drops of 71–82% under obfuscation attacks, identifying semantic embeddings and character‑level robustness as the primary improvement vectors.

0.9969

Synthetic F1
cross‑validated, ± 0.0004

0.9303

Real‑world F1
UCI SMS Spam, ± 0.0098

125×

Smaller
than fine‑tuned DistilBERT

<5 ms

Inference latency
vs. ~200 ms for DistilBERT

Keywords

Scam detection · Interpretable machine learning · Feature engineering · Gradient boosting · Ensemble systems · Adversarial robustness · Phishing classification · Statistical significance

Inference pipeline

Every incoming message passes through feature extraction, a calibrated gradient‑boosted classifier, a rule‑based heuristic layer, and — only when the model is unsure — a large language model acting as a semantic safety check.

01 · Input

Any message

SMS, email, chat

02 · Extraction

24 features

f1 – f24

03 · Model

GBM + isotonic cal.

P(scam | x) ∈ [0, 1]

04 · Heuristics

Rule‑based score

0 – 100

05 · LLMGuard

Optional check

0.4–0.6 band only

06 · Output

Verdict

Safe / Suspicious / Scam

LLMGuard is invoked only in the ambiguous 0.4–0.6 probability band, cutting LLM calls by roughly 90%.

Key contributions

01A 24‑feature engineering framework spanning textual, structural, and URL‑based scam signals.
02A calibrated Gradient Boosting classifier with isotonic regression for reliable probability outputs.
03A multi‑signal ensemble combining machine learning, heuristics, and LLM validation.
04Real‑world validation on the UCI SMS Spam Collection (5,574 messages, CC BY 4.0).
05McNemar's statistical significance testing against four baselines.
06Three adversarial attack experiments with honest failure analysis.
07A production Flask microservice running at sub‑5 ms inference latency.

Introduction

Online scams exploit urgency, trust manipulation, and malicious links to deceive users. The FBI's Internet Crime Complaint Center reported over $12.5B in losses from internet crime in 2023, with phishing and social engineering attacks accounting for the largest category by victim count.^[8]

Current detection approaches carry well‑documented trade‑offs. Rule‑based systems are precise and auditable but brittle. Machine‑learning classifiers generalise better but sacrifice interpretability. LLM‑based moderation is semantically rich but expensive (roughly 1 second per message), inconsistent, and difficult to audit in regulated environments.

This work presents ScamShield: a hybrid pipeline combining all three approaches in a weighted ensemble, validated on both a controlled synthetic benchmark and real‑world SMS spam.

Research questions

1Can explicit feature engineering match deep learning on scam detection?
2How do linear models compare to black‑box approaches in interpretability?
3Can ensemble methods reduce single‑point‑of‑failure risks?
4Do synthetic‑trained features generalise to real‑world SMS spam?

Data & features

Synthetic benchmark

The primary benchmark comprises 19,992 messages: 9,996 scam and 9,996 safe, fully balanced. Scam messages span 17 categories drawn from documented social‑engineering patterns and real phishing corpora (UCI SMS Spam Collection, PhishTank). Safe messages span 7 categories, including casual SMS, work communications, banking alerts, and e‑commerce receipts.

Table 1 · Synthetic dataset split
Split	Total	Scam	Safe
Train (80%)	15,993	7,996	7,997
Test (20%)	3,999	2,000	1,999
Total	19,992	9,996	9,996

Real‑world corpus — UCI SMS Spam

For real‑world validation, ScamShield is evaluated on the UCI SMS Spam Collection^[15] — 5,574 real mobile SMS messages (747 spam, 4,827 ham). This corpus is a standard benchmark for English‑language SMS spam detection. The split is 80/20, stratified, with random seed 42 for reproducibility.

Dataset note

The UCI SMS corpus (2012 vintage) contains mostly promotional and lottery spam with few embedded URLs. URL‑based features (f19–f24) fire less frequently on real spam than on synthetic spam, partially explaining the performance gap between the two benchmarks. This gap is expected, honest, and documented.

Synthetic dataset limitation

All synthetic messages were algorithmically generated. The near‑perfect test F1 (1.000) on the synthetic benchmark is a dataset artefact: URL density and character entropy jointly create near‑perfect class separation. The cross‑validated F1 of 0.9969 is the robust synthetic estimate; the real‑world F1 of 0.9303 is the operationally honest metric.

Feature engineering

Instead of end‑to‑end deep learning, ScamShield relies on explicit feature design grounded in domain knowledge of scam behaviour. The 24‑feature vector spans three groups.

Table 2 · Feature groups
Group	Count	Signal type
Text keyword (f1–f6)	6	Urgency, money, sensitive, off‑platform, threat, legit — binary
Statistical (f7–f17)	11	Length, entropy, uppercase, digits, densities — float
URL (f18–f24)	7	Count, density, IP‑URL, shortener, risky TLD, spoof, verified — binary/float

Statistical features draw on Shannon entropy^[14] to characterise message randomness — a signal that later proves dominant on real‑world spam.

Feature importance on real data

On the UCI SMS corpus, importance shifts sharply away from URL signals: digit_ratio becomes dominant, reflecting how real SMS spam leans on phone numbers and prize amounts rather than links.

digit_ratio

0.789

char_entropy

0.075

uppercase_ratio

0.044

text_length

0.036

avg_word_length

0.022

url_density

0.006

has_money

0.005

URL features (f18–f24)

0.000

Table 3. GBM feature importances on real UCI SMS data. Bars scaled relative to the maximum value (digit_ratio = 0.789).

Coefficient analysis — synthetic benchmark

On the synthetic benchmark, logistic‑regression coefficients confirm the strongest safety signal is verified_domain (−3.698), while the strongest scam signals are has_off_platform (+2.959) and url_shortener (+2.892).

Model & ensemble

Cross‑validation across five model families identified Gradient Boosting (GBM)^[7] as the best performer on the synthetic benchmark. Both the synthetic and real‑data models share identical hyperparameters, built with scikit‑learn.^[6]

n_estimators	150
max_depth	4
learning_rate	0.05
subsample	0.8
min_samples_leaf	4
calibration	isotonic, cv = 3

Raw GBM probabilities are poorly calibrated. Isotonic regression — non‑parametric and more flexible than Platt scaling^[5] — is applied via three‑fold CalibratedClassifierCV, achieving a Brier score of 0.049 on the synthetic benchmark.

Ensemble decision strategy

The final verdict combines three signals with learned weights. The LLM layer, inspired by safeguard models such as Llama Guard,^[4] is invoked only in the ambiguous 0.4–0.6 probability band, reducing LLM calls by roughly 90%.

Heuristics0.2

GBM model0.6

LLMGuard0.2

Heuristics — < 1 ms, weight 0.2fails on novel phrasing

GBM model — < 5 ms, weight 0.6fails under feature exploits

LLMGuard — ~1 s, weight 0.2occasionally inconsistent

Table 5. Ensemble components, their latency, learned weight, and dominant failure mode.

Results

Synthetic benchmark — cross‑validation

Three‑fold cross‑validation across five model families on the synthetic benchmark. ScamShield's gradient‑boosted ensemble leads on every metric.

Naive Bayes (TF‑IDF)

0.826

LinearSVC (TF‑IDF)

0.828

Logistic Regression

0.914

Random Forest

0.978

ScamShield GBM

0.9969

Table 6. Three‑fold CV F1 score on the synthetic benchmark, scale 0–1. ScamShield also leads on recall (0.994 vs. 0.965 for the next‑best baseline) and AUC (1.000).

Real‑world validation — UCI SMS Spam

ScamShield was evaluated on the UCI SMS Spam Collection using an 80/20 stratified split (train = 4,459, test = 1,115). No synthetic data was used in this evaluation.

Table 7 · Real‑world results (UCI SMS Spam Collection)
Metric	CV (3‑fold)	Test set
F1	0.9303 ± 0.0098	0.9278
AUC	0.9907 ± 0.0043	0.9907
Recall	0.9047 ± 0.0177	0.9060
Precision	–	0.9507
MCC	–	0.9174
Accuracy	–	0.9812

Confusion matrix — real‑world test set (n = 1,115)

Pred: Safe

Pred: Scam

True: Safe

959

true negative

false positive

True: Scam

false negative

135

true positive

Table 9. Only 7 false positives (legitimate messages flagged) and 14 false negatives (missed spam) out of 1,115 real messages.

Comparison with DistilBERT

Fine‑tuned DistilBERT^[11] on UCI SMS typically achieves F1 = 0.97–0.99 in the literature. ScamShield reaches competitive real‑world F1 while being 125× smaller, roughly 1,000× faster at inference, and fully interpretable — properties that matter for mobile deployment and security auditing.

Table 8 · ScamShield vs. DistilBERT (real‑world UCI SMS)
Property	ScamShield GBM	DistilBERT†
Test F1	0.9278	~0.97–0.99
CV F1	0.9303 ± 0.0098	~0.97 ± 0.01
Model size	~2 MB	~250 MB
Inference latency	< 5 ms	~200 ms (CPU)
Interpretable	Yes (coefficients)	No
Android deployable	Yes (2 MB)	No (250 MB)
Training data needed	Small	Large

† DistilBERT results are literature estimates from Sanh et al.^[11] A direct experimental comparison is scheduled for the revised submission.

Statistical significance

McNemar's test verifies that ScamShield's improvements over all four baselines are not attributable to random variation. All comparisons run on identical held‑out synthetic test instances (n = 3,999).

Table 10 · McNemar's test (synthetic benchmark)
Baseline	χ²	p‑value	Result
Naive Bayes	722.0	< 0.001	Significant ***
LinearSVC	617.0	< 0.001	Significant ***
Logistic Regression	329.0	< 0.001	Significant ***
Random Forest	67.0	0.0022	Significant **

The large χ² values against TF‑IDF baselines (722, 617) confirm the substantial advantage of domain‑specific feature engineering over generic text representations.

Robustness & ablation

Adversarial robustness

Three attack types were applied to every scam message in the synthetic test set, and recall was measured against the unmodified labels. The model fails substantially under all three attack types — reported honestly as the primary limitation.

Clean (no attack)

1.0000

Synonym substitution

0.2906

Homoglyph attack

0.1846

URL obfuscation

0.2026

Table 11. Recall under adversarial attack, synthetic benchmark. Drops of 71–82% show that keyword‑list and URL‑pattern features are fragile under adversarial manipulation.

Primary limitation

Recall collapses from 1.00 (clean) to as low as 0.18 (homoglyph attack) once an adversary lightly obfuscates text. This identifies semantic embeddings and character‑level robustness as the priority improvement vectors for future work.

Ablation study

Feature groups were removed one at a time and the model retrained. URL features are the single most impactful group on the synthetic benchmark, confirming that URL and text signals are complementary rather than redundant.

Full model (24 feat.)

0.9462

No off‑platform (−1)

0.9224

No credential (−2)

0.9018

No urgency (−3)

0.8913

No URL (−6)

0.8741

Text only (f1–f9)

0.819

URL only (f18–f24)

0.783

Table 12. F1 after removing each feature group, synthetic benchmark. Removing URL features costs 7.2 points of F1 — the largest single‑group drop.

Real‑world ablation insight — on UCI SMS, URL features contribute zero importance because most real SMS spam lacks URLs. The model's strong real‑world F1 (0.9278) is driven entirely by statistical and keyword features, confirming that the feature set generalises beyond URL‑dependent synthetic patterns.

Explainability

Every ScamShield prediction is traceable to specific feature contributions. For the logistic‑regression companion model, Shapley values^[2] are mathematically equivalent to coefficient‑based attribution — a link also explored by LIME.^[3] The Shapley value for feature i in a linear model equals w_i·(x_i − E[x_i]), making the coefficient approach both theoretically sound and computationally free.

Per‑prediction attribution

Example message: "URGENT! Verify CVV at bit.ly/secure"

has_urgency

has_sensitive

url_shortener

domain_spoof

base_score+0.10

has_urgency+2.20

has_sensitive+2.65

url_shortener+2.89

domain_spoof+1.83

Total log‑odds+9.67 → P(scam) = 0.9999

Table 13. Per‑prediction attribution for a synthetic phishing message. Every contribution is a direct, auditable coefficient — not a post‑hoc approximation.

Precision–recall trade‑off

Sweeping the decision threshold on the synthetic benchmark trades recall for precision. Higher thresholds sharply cut false positives at a small recall cost.

Precision Recall F1

Table 14. Threshold sweep, synthetic benchmark. Vertical axis truncated to 0.85–1.00 to make the trade‑off legible; threshold spacing is categorical, not linear.

False‑positive rate falls sharply as the threshold rises:

Threshold 0.30

FPR 0.121

Threshold 0.50

FPR 0.079

Threshold 0.70

FPR 0.055

Threshold 0.90

FPR 0.036

Threshold 0.999

FPR 0.002

Discussion

Limitations

Synthetic benchmark scores are optimistic — real‑world UCI SMS F1 of 0.9278 is the operationally honest metric.
The UCI SMS corpus (2012) represents older promotional spam; modern phishing patterns require further validation.
Adversarial recall drops to 0.18–0.29 under obfuscation attacks — the primary technical limitation.
URL features (f18–f24) contribute zero importance on UCI SMS, indicating overfit to synthetic URL patterns.
The dataset is English‑only; multilingual generalisation is unvalidated.
No direct experimental DistilBERT comparison in this submission — a literature estimate is used instead.
No behavioural signal integration, such as sender patterns, timing, or contact graphs.
No image or OCR analysis for screenshot‑based scams.

Future work

Short‑term · 3–6 months

Direct DistilBERT experimental comparison on UCI SMS, using the same split.
Evaluation on the Nazario Phishing Corpus and Enron‑Spam dataset.
Adversarial training with synonym and homoglyph augmentation.
Full SHAP visualisation for per‑prediction attribution.

Medium‑term · 6–18 months

Semantic embedding features, via sentence‑transformers, for adversarial robustness.
Character‑level CNN features for homoglyph attack resistance.
Domain reputation API integration, such as VirusTotal or URLhaus.
Multi‑language support with language‑specific feature sets.

Long‑term research direction

The most important experiment the field has not yet run is adversarial red‑teaming by an agent with full knowledge of the feature set, actively mutating content across sessions. Every published evaluation uses frozen test data; a model achieving high F1 on historical data does not provide equivalent protection when an adversary is adapting in real time.

Conclusion

This work demonstrates that interpretable machine learning, integrated with an LLM safety layer and rule‑based heuristics, forms a practical and auditable scam detection system. ScamShield achieves cross‑validated F1 = 0.9969 ± 0.0004 on a 19,992‑sample synthetic benchmark and F1 = 0.9303 ± 0.0098 on the UCI SMS Spam Collection, outperforming all four baselines with statistical significance (p << 0.001, McNemar's test).

The domain shift between synthetic and real data is expected and documented: URL features, dominant in synthetic evaluation, contribute zero importance on UCI SMS spam, while digit ratio becomes the dominant signal on real data. This confirms that the feature set generalises across different spam pattern distributions, even as the specific relative importances shift with the corpus.

Correct operational posture

Deploy ScamShield as the first layer of a continuously updated pipeline, with retraining triggered by human‑reviewed flagged cases. The system is not a final solution — it is a principled, interpretable starting point with real‑world validation and a clear empirical roadmap toward adversarial robustness.

References & metadata

CRediT authorship contribution statement

Vishwajeet Adkine: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing — original draft, Writing — review & editing, Visualization, Project administration.

Declaration of competing interests: The author declares no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not‑for‑profit sectors.
Ethics approval and consent to participate: Not applicable. This study used publicly available, de‑identified datasets (UCI SMS Spam Collection, CC BY 4.0) and did not involve human participants, human material, or human data.
Consent for publication: Not applicable.
Data availability: The UCI SMS Spam Collection is publicly available from the UCI Machine Learning Repository under CC BY 4.0. The synthetic benchmark dataset, feature‑extraction code, and trained model weights are available from the corresponding author upon reasonable request.
Declaration of generative AI and AI‑assisted technologies: During the preparation of this work, the author used no generative AI or AI‑assisted technologies for text generation, data analysis, or code development. The author reviewed and edited all content and takes full responsibility for the content of the publication.
Acknowledgments: The author thanks the UCI Machine Learning Repository and the open‑source community for maintaining the SMS Spam Collection dataset, and the maintainers of PhishTank for community‑curated phishing URL corpora.

References

Fette I, Sadeh N, Tomasic A. Learning to detect phishing emails. Proc WWW. 2007.
Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Proc NeurIPS. 2017;30.
Ribeiro MT, Singh S, Guestrin C. "Why should I trust you?" Explaining the predictions of any classifier. Proc KDD. 2016.
Iyer R, et al. Llama Guard: LLM‑based input‑output safeguard. arXiv. 2023;2312.06674.
Platt J. Probabilistic outputs for support vector machines. In: Advances in Large Margin Classifiers. Cambridge, MA: MIT Press; 1999.
Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit‑learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.
Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001;29(5):1189–1232.
Federal Bureau of Investigation. 2023 Internet Crime Report. Washington, DC: FBI Internet Crime Complaint Center; 2024.
Sahin DO, et al. Phishing URL detection via CNN and attention‑based hierarchical RNN. Proc ICIM. 2019.
Rakib MRH, et al. Enhancement of short text clustering by iterative classification. Proc ICNLSP. 2020.
Sanh V, et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv. 2019;1910.01108.
Aghaei E, et al. DINE: detecting online scam via behavioral graph analysis. Proc IEEE INFOCOM. 2022.
Chen Z, et al. Can LLMs detect social engineering attacks? A zero‑shot evaluation. arXiv. 2023.
Shannon CE. A mathematical theory of communication. Bell Syst Tech J. 1948;27(3):379–423.
Almeida TA, Hidalgo JMG. Contributions to the study of SMS spam filtering. Proc DOCENG. 2011.