Abstract Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation. However, challenges remain in achieving effective cross-modal alignment and high-quality speech generation. In this work, we systematically investigate the impact of key components (i.e., speech tokenizers, speech heads, and speaker modeling) on the performance of LLM-centric SLMs. We compare coupled, semi-decoupled, and fully decoupled speech tokenizers under a fair SLM framework and find that decoupled tokenization significantly improves alignment and synthesis quality. To address the information density mismatch between speech and text, we introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens. This leads to up to 12× faster decoding and a substantial drop in Word Error Rate (from 6.07 to 3.01). Furthermore, we propose a speaker-aware generation paradigm and introduce RoleTriviaQA, a large-scale role-playing knowledge QA benchmark with diverse speaker identities. Experiments demonstrate that our methods enhance both knowledge understanding and speaker consistency.

Contents

• Overview
• SLMs Trained with Different Speech Tokenizers
• NTP vs. MTP & Speaker-Aware TTS
• Role-Playing Knowledge QA

This page is for research demonstration purposes only.

Overview

Figure 1: Left: Overview of a Speech Language Model (SLM) trained with decoupled speech tokenizer (Section 2.1) and Speaker-Aware TTS; Right: The architecture of a possible decoupled speech tokenizer, featuring speech quantization, reconstruction in a decoupled manner and speaker-specified embedding extraction.

Figure 2: Illustration of our NTP and MTP architecture. (a) NTP: single vocabulary and single prediction head; (b) MTP: multiple vocabularies and multiple prediction heads, generating multiple tokens in parallel.

Figure 3: Illustration of Role-Playing Knowledge QA.

SLMs Trained with Different Speech Tokenizers

Comparison of SLMs trained with different kinds of speech tokenizers (Coupled, Semi-decoupled, Decoupled).

NTP vs. MTP & Speaker-Aware TTS

Comparing Next-Token Prediction (NTP) and Multi-Token Prediction (MTP) for TTS on SLMs using decoupled tokenizers*, w/ and w/o Speaker-Aware paradigm.

Role-Playing Knowledge QA

Comparison of SLMs with Top-3 Coupled, Semi-Decoupled and Decoupled speech tokenizers on Role-Playing Knowledge QA task.

Roles are selected from Genshin Impact.