Adversarial Robustness of Vision Transformers Under Black-Box Query-Limited Attacks: A Comparative Study with Proposed Defense—PRISM: Patch Randomization with Inference-time Smoothing and query Monitoring

Abstract

Vision Transformers (ViTs) have rapidly become a dominant architecture for image classification, with several studies claiming they are inherently more robust to adversarial perturbations than convolutional neural networks (CNNs). However, prior comparisons have been confounded by inconsistent training recipes, model scales, and evaluation protocols, and almost none have focused specifically on the practical and increasingly important black-box, query-limited threat model. Under this threat model, an adversary has no access to model weights or gradients but may submit a bounded number of input queries and observe model outputs. This paper conducts a systematic comparison of ViT and CNN architectures under two representative query-limited attack families-score-based (Square Attack; Andriushchenko et al., 2020) and decision-based (HopSkipJumpAttack; Chen et al., 2020)-using the controlled training framework established by Bai et al. (2021) to isolate architectural effects from training confounders. We then propose PRISM (Patch Randomization with Inference-time Smoothing and query Monitoring), a training-free, architecture-agnostic inference-time defense that combines patch-level randomization, query-budget detection, and ensemble label smoothing to reduce the signal available to query-limited attackers without requiring adversarial training. We describe PRISM’s design principles, present a concrete evaluation protocol on ImageNet-1K [7] under the RobustBench threat model (Croce et al., 2021), and discuss expected behavior, limitations, and open questions.