A Comparative Study of XGBoost and Transformer for E-commerce Purchase Prediction under Class Imbalance

Abstract

Class imbalance poses a persistent challenge in e-commerce purchase prediction, where positive samples rarely exceed 10% of observations. This study compares XGBoost and Transformer architectures under controlled imbalance conditions using the Tmall Repeat Purchase Prediction dataset (6.12% positive rate). A 2 × 2 factorial design crossed model type (XGBoost vs. Transformer) with balancing strategy (original imbalanced vs. SMOTE-balanced), drawing on 22 tabular features and 20-length behavioral sequences derived from user interaction logs. On the original data, XGBoost outperformed Transformer (AUC 0.657 vs. 0.631; F1 0.194 vs. 0.169). SMOTE, counter to expectations, degraded both models. Ablation experiments confirmed this was not an artifact of the sampling ratio. At extreme imbalance (1% positive rate), the pattern reversed: Transformer proved more robust, its AUC falling by only 1.7% against XGBoost’s 9.1%, and surpassing XGBoost on AUC at this level. These results challenge the common assumption that tree-based models are always preferable under class imbalance and offer practical guidance for model selection in real-world e-commerce settings.