A Study on the Differences in the Integration of Different Types of Gestures and Language Based on the Modality-Specific Simulation

Abstract

The integration of gestures and language is a core topic in the study of multimodal communication. From the theoretical perspective of modality-specific simulation, this paper classifies gestures into four types: indicative, metaphorical, rhythmic, and pictographic. It systematically compares the differences between gesture and language integration in terms of semantic integration depth, temporal process, neural mechanism, and cognitive load, and provides a unified explanation using the modality compatibility hypothesis, the extended dual encoding theory, and the predictive coding framework. Finally, it explores the practical value of this in education and clinical rehabilitation. The research shows that the differences in the integration of different types of gestures are rooted in the different attributes and processing pathways of the modality-specific simulations they activate.