Real-time Gender-neutral Content Filtering System: AI Architecture Based on Distributed Stream Processing

Abstract

This paper proposes a real-time gender-neutral content filtering system based on distributed stream processing, which aims to reduce gender bias in generative AI output and enhance the Chinese public's trust in AI-generated content. The system uses Apache Flink to build a distributed architecture and combines the RoBERTa model to achieve efficient gender bias detection. In high-concurrency scenarios, the throughput reaches 5500 texts/s, the latency is as low as 6.7 milliseconds, the F1 score is 0.92, and the AUC-ROC reaches 0.96, which is better than the BERT and DistilBERT models. The public trust evaluation shows that the trust score of the filtered AI output has increased from 3.22 to 4.08, which is close to the 4.62 of traditional media, and young high-frequency users have higher trust. The study shows that the system effectively balances detection accuracy and real-time performance, provides a feasible solution for gender-neutral content generation of generative AI, and enhances the public's trust in AI technology, laying the foundation for its responsible application in Chinese society.