StepFun Releases StepAudio 2.5 Realtime: An End-to-End Voice Model with Roleplay-Specific RLHF and Paralinguistic Comprehension

StepFun, an AI laboratory based in Shanghai, has unveiled StepAudio 2.5 Realtime, an end-to-end real-time speech large language model. This innovative model features fully customizable persona capabilities and supports both Chinese and English. It represents a significant advancement by unifying speech recognition, reasoning, and synthesis into a single system, departing from traditional pipeline-based approaches.

The model's architecture incorporates several key innovations. StepFun utilized algorithmic augmentation to create a vast persona feature matrix from over 10,000 high-quality natively authored personas, supplemented by millions of real-world conversational samples. This approach aims to achieve stable performance across diverse and challenging conversational topics.

One notable innovation is the application of Reinforcement Learning from Human Feedback (RLHF) specifically optimized for persona consistency in roleplay scenarios. This addresses the common issue of models exhibiting "out-of-character" behavior during conversations, ensuring the model maintains its defined persona throughout interactions.

StepAudio 2.5 Realtime also excels in paralinguistic perception, which involves understanding non-verbal acoustic information such as tone, speaking rate, pauses, and laughter. By analyzing these elements, the model can discern a user's mood and intentions. This capability allows for "global scene-level tonal setting" and "intra-sentence detail sculpting," enabling the model to set an overall emotional register while fine-tuning acoustic details within sentences.

The model demonstrated strong performance in paralinguistic comprehension, scoring 82.18 on a benchmark that assesses perception of vocal speed, emotion, and age. Comprehensive subjective and objective evaluations were conducted against leading real-time voice models across various dimensions.