Beyond Tool Accuracy: End-to-End Reliability of On-Device SMS Agents Under Homoglyph Attacks

Cheng-Che Lee, Chia-Pin Cheng, Hsu-Sheng Chang, Chia-Lin Chou
Digital Innovation Laboratory, Chunghwa Telecom Laboratories, Taoyuan, Taiwan
NeurIPS 2026 Workshop on Agents in the Wild: Safety, Security, and Beyond
Paper OpenReview
TL;DR: We fix a homoglyph-attack vulnerability in lightweight URL classifiers via CHT loss (homoglyph detection ~9% → 96%), then show that whether the fix reaches the agent's decision depends on the host LLM: one host follows the tool, two override it.

Tool-level problem: classifiers miss homoglyph URLs. System-level problem: host LLMs differ in how they use the tool's verdict.

Left (tool level): lightweight classifiers and guardrail models misclassify homoglyph URLs because the feature representation and the decision boundary disagree; our model aligns the two. Right (system level): even when the tool correctly clears a benign link, host models differ systematically in whether they act on that verdict.

Abstract

On-device SMS security agents must operate under strict latency constraints to intercept phishing links in real time, and therefore rely on lightweight classifiers—rather than large foundation models—as their decision-making tool. However, existing lightweight classifiers remain highly vulnerable to homoglyph attacks, constituting a known bottleneck on the tool side. We trace this vulnerability to a Representation-Decision Paradox: discriminative signals induced by homoglyph perturbations are detectable at the embedding layer, yet dissipate at the decision layer due to architecture-specific inductive biases (over-smoothing in CNNs, decision inertia in RNNs). To address this, we propose Contrastive Homoglyph Tolerance (CHT) loss, which substantially improves classifier sensitivity to homoglyph attacks on sequential architectures while preserving tolerance to random perturbations; relative to a 2B-parameter general-purpose guardrail model, our lightweight tool also reduces end-to-end inference latency by up to 22%. Whether mitigating this tool-side bottleneck suffices to ensure reliable behavior of the overall agent system, however, has not been systematically examined. We integrate the repaired tool into a real agent-tool pipeline, evaluating three small language models from distinct model families (Llama-3.2, Gemma-4, Qwen2.5) as hosts, and cross-validate against three off-the-shelf industry guardrail models. Results show that improving tool quality alone is insufficient to guarantee end-to-end agent reliability: on Qwen2.5, agent safety improves markedly with tool quality, whereas on Llama-3.2 and Gemma-4, the agent escalates tool-cleared messages as fraud irrespective of the tool backend in use. This finding reveals substantial, systematic differences in how host models incorporate tool outputs into their final decisions, exposing a blind spot in existing agent evaluation practices concerning end-to-end reliability.

The Representation-Decision Paradox

Lightweight URL classifiers reach ~98% accuracy on standard phishing detection, yet catch only 7–10% of homoglyph URLs (e.g., apple.com → аpple.com with a Cyrillic “а”). To find where this fails, we track the ℓ2 distance between a benign URL and (I) its random variant, (II) its homoglyph variant, and (III) a malicious URL, at the embedding, hidden, and logit stages.

  • Embedding: (II) ≈ (III). The model does perceive the homoglyph.
  • Logits: (II) collapses toward (I). The signal is discarded at the decision boundary.

The failure is in decision, not perception. The paradox appears in every architecture we tested, through two different mechanisms: CNN pooling over-smooths the perturbation as spatial noise, while RNNs show decision inertia that locks the output to “benign”.

Layer-wise feature distances for five architectures, without and with CHT loss.

Distances are comparable only within a panel; what matters is the ratio of (II) to (III). Top: standard training, where (II) collapses at the logits. Bottom: with CHT loss, sequential architectures keep (II) at or above (III) through the decision layer.

Method: Contrastive Homoglyph Tolerance (CHT) Loss

CHT casts two seemingly opposing goals into a training-time regularizer on the classifier's predicted distribution Pθ(y|x). For a benign URL, we generate a homoglyph variant xhomo and a random, non-homoglyph variant xrand:

Sensitivity: push homoglyphs away

Lhomo = max(0, m − DKL(Pθ(y|xbenign) || Pθ(y|xhomo)))

A hinge on the KL divergence, with margin m to keep the divergence bounded and training stable.

Invariance: keep random typos close

Lrand = DKL(Pθ(y|xbenign) || Pθ(y|xrand))

Prevents the model from treating every character substitution as an attack.

Ltotal = LBCE + α Lhomo + β Lrand

No extra inference-time cost. Perturbed URLs receive no hard label; CHT only constrains the output distribution relative to the unperturbed URL.

Tool Level: CHT Resolves the Paradox

  • Homoglyph detection: 7–10% → 96%+ on all three sequential architectures.
  • Random-perturbation tolerance: 89–90% → ~99%.
  • Base accuracy: unchanged at ~98%.
  • Ablation: Lhomo alone causes hypersensitivity (CNN-BiLSTM Random Acc −2.93); adding Lrand restores it. Both terms are needed.
Model Lhomo Lrand Base Dataset Homoglyph Random
AccPrec.Rec.F1AccAcc
DEPHIDE-LSTM 98.1797.2597.9697.60 9.6488.83
✓ 98.0897.4697.4997.48 96.51 (+86.87)93.46 (+4.63)
✓✓ 98.0297.6897.0897.38 96.45 (+86.81)98.66 (+9.83)
DEPHIDE-BiLSTM 97.9996.3098.5097.38 9.1889.53
✓ 98.1597.1997.9497.57 96.61 (+87.43)91.50 (+1.97)
✓✓ 98.1898.1497.0497.59 96.30 (+87.12)98.97 (+9.44)
CNN-BiLSTM 98.1696.5898.6597.60 7.4090.47
✓ 98.1596.8898.2897.57 96.67 (+89.27)87.54 (−2.93)
✓✓ 98.3597.8297.8297.82 96.53 (+89.13)98.87 (+8.40)

Baselines (standard BCE training) in gray; full CHT loss highlighted. Changes relative to the baseline in parentheses. Pure CNNs (URLNet, DEPHIDE-CNN) do not benefit, since pooling has already erased the signal.

System Level: Does a Better Tool Make a Better Agent?

We plug the classifier into an SMS safety agent. The host LLM must call check_link_safety, which returns {is_malicious, confidence, raw_logit}, and then choose OPEN_LINK, IGNORE, or MARK_SUSPICIOUS. We test three hosts (Llama-3.2-3B, Gemma-4-E4B, Qwen2.5-3B) × three classifiers with and without CHT, plus three off-the-shelf guardrails as alternative backends, on 400 SMS episodes per configuration.

Metrics (lower is better): ASR, homoglyph → OPEN_LINK (attack succeeds); FBR, random variant → blocked; Benign FPR, benign → blocked.

For each host, attack success and benign block rates with no tool, with a classifier, and with a classifier plus CHT. Qwen2.5 moves with the tool; Llama-3.2 and Gemma-4 barely move.

Agent outcomes, averaged over the three classifiers. Attacks succeed = ASR; Benign blocked = Benign FPR. Without a tool, every host blocks nearly everything, so ASR is 0 by default. Qwen2.5 follows the tool: it stops over-blocking, and CHT then cuts ASR from 30% to 2%. Llama-3.2 and Gemma-4 stay near their no-tool behavior.


HostArchitectureCHT ASR ↓FBR ↓Benign FPR ↓Compliance
Llama-3.2 3B No tool–0.001.001.00–
DEPHIDE-LSTM0.010.970.901.000
✓0.010.950.891.000
DEPHIDE-BiLSTM0.010.920.891.000
✓0.010.960.901.000
CNN-BiLSTM0.020.950.851.000
✓0.010.950.870.998
Gemma-4 E4B No tool–0.000.990.93–
DEPHIDE-LSTM0.000.990.951.000
✓0.000.980.931.000
DEPHIDE-BiLSTM0.001.000.930.998
✓0.001.000.950.998
CNN-BiLSTM0.001.000.961.000
✓0.000.990.971.000
Qwen2.5 3B No tool–0.000.990.97–
DEPHIDE-LSTM0.310.520.521.000
✓0.030.500.520.998
DEPHIDE-BiLSTM0.310.590.590.998
✓0.010.470.490.995
CNN-BiLSTM0.270.680.630.998
✓0.010.570.630.998

tool-following host tool-overriding hosts. Without a tool, all three hosts block nearly every message (Benign FPR 0.93–1.00), even though benign and random episodes use neutral-toned templates. Message tone alone does not make any host open a link, so any movement away from this baseline comes from the tool.


⚠️ Llama-3.2 & Gemma-4 override the tool

  • FBR / Benign FPR stay ≥ 0.85 with every backend, barely moving from the no-tool baseline.
  • Tool-Override Rate > 0.9: the tool says safe, the agent still blocks.

This is not an artifact:

  • Not tool formatting: LlamaGuard-3, ShieldGemma, and Qwen3Guard as backends show the same pattern.
  • Not non-compliance: the tool is called in more than 99% of episodes, so the verdict reaches the agent and is discarded at the decision step.
  • Not caution: the blocks are mostly MARK_SUSPICIOUS (escalated as fraud), not IGNORE.

Efficiency: End-to-End Latency

Relative to a 2B-parameter guardrail (ShieldGemma), our lightweight CHT classifier cuts end-to-end latency by 22% on Llama-3.2 and 18% on Qwen2.5.

HostNo-ToolShieldGemma (2B)Ours (CHT)Δ vs. ShieldGemma
Llama-3.2 3B1,9415,8084,531−1,277 (−22%)
Qwen2.5 3B2,0455,8324,803−1,029 (−18%)
Gemma-4 E4B32,65240,92842,526+1,598 (+4%, within noise)

Mean latency in ms over 100 episodes, CPU-only inference on a server (not a phone). Only relative differences should be read from this table.

BibTeX

@inproceedings{lee2026beyond,
  author    = {Lee, Cheng-Che and Cheng, Chia-Pin and Chang, Hsu-Sheng and Chou, Chia-Lin},
  title     = {Beyond Tool Accuracy: End-to-End Reliability of On-Device {SMS} Agents Under Homoglyph Attacks},
  booktitle = {Third Workshop on Agents in the Wild: Safety, Security, and Beyond},
  year      = {2026},
  url       = {https://openreview.net/forum?id=k0tisYvu30}
}