MusicTwins: Teaching Audio Language Models to Listen to Music Like a Specific Persona

Everyone perceives music differently, forms different associations, has different preferences, and reacts differently. If computational models are to simulate human behavior in music-related environments, these individual differences must be taken into account. While large language models (LLMs) have been shown to reproduce certain aspects of human behavior, whether audio language models (ALMs) can reproduce a specific individual's responses to text and music stimuli remains an underexplored question. A recent paper proposes the MusicTwins framework, which attempts to teach audio language models to "listen like their persona."

The Core Problem: The Lack of Persona Awareness

Current audio language models typically pursue generic, averaged responses when handling music-related tasks, ignoring a fundamental fact: different people can have vastly different feelings and understandings of the same piece of music. For example, a cheerful pop song might remind one person of youthful memories, while another might find it noisy. This difference stems from various factors such as personal background, personality, values, and writing style.

The paper points out that when simulating human behavior in music-related environments, these individual differences must be considered. However, whether existing ALMs can reproduce a specific individual's responses lacks systematic study. MusicTwins is designed precisely to fill this gap.

The MusicTwins Framework: From Persona Representation to Post-Training

MusicTwins is a framework that combines persona summaries, personal memory, and post-training to generate responses targeting a specific individual. Its core idea is to first construct a persona representation from human text, then post-train the audio language model via supervised fine-tuning and reinforcement learning, so that its outputs more closely match the target persona.

Constructing the Persona Representation

The research team constructs persona representations from human writing, specifically covering five dimensions: background, views, personality, writing style, and musical profile. Together, these dimensions characterize a person's unique profile, providing the model with rich personalized context.

Audio Language Model (ALM) is a class of multimodal models capable of processing both audio and text inputs and generating text outputs. Unlike pure-text LLMs, ALMs map sound signals into the language model's representation space via an audio encoder, thereby understanding content such as music, environmental sounds, and speech. Audio Flamingo Next is one representative of such models, with capabilities including music description and audio question answering. MusicTwins post-trains it to inject persona specificity, transforming a general-purpose ALM into a personalized listener.

Post-Training Strategy

Based on the constructed persona representations, the researchers post-trained the Audio Flamingo Next model using a combination of supervised fine-tuning and reinforcement learning. Supervised fine-tuning enables the model to learn persona-related response patterns, while reinforcement learning further optimizes the model's outputs to better match the target persona's characteristics.

Supervised Fine-Tuning (SFT) refers to further training a pre-trained model using labeled input-output pairs so that the model learns response patterns for a specific task. In MusicTwins, SFT enables Audio Flamingo Next to learn the target persona's language style and musical reactions from persona summaries and example dialogues. Reinforcement learning (here likely RLHF or a similar method based on human feedback) optimizes the model's outputs through reward signals, where rewards typically come from similarity to the target persona's responses or human preference ratings. Combined, SFT provides initial alignment, and reinforcement learning further fine-tunes the generation policy, thereby improving persona fidelity.

Evaluation and Results: Improved Persona Fidelity

Evaluated on ten held-out personas, the results show that Audio Flamingo Next achieved significant improvements in persona fidelity, including personality, writing style, and reactions to music. This means the model's generated responses not only more closely resemble the target individual's way of expression but also better simulate their musical preferences and emotional reactions.

This result supports the effectiveness of MusicTwins as a foundational framework for training audio language models to "listen like their persona." The authors believe that this framework provides a viable path for the development of personalized audio language models.

Persona fidelity measures the degree to which a model's generated responses match the target person's real responses, typically evaluated across multiple dimensions: language style (e.g., word choice, sentence structure), personality expression (e.g., extraversion, openness), consistency of views, and musical preference reactions. In MusicTwins, the researchers likely combined automatic metrics (e.g., text similarity to real responses, style classifier accuracy) with human evaluation (e.g., raters judging whether a response resembles the target person). High fidelity means the model not only imitates surface language but also closely approximates a specific individual in musical emotion and associations.

Significance and Outlook

The proposal of MusicTwins marks an important step toward personalization in audio language models. Traditional ALMs often pursue generality, whereas MusicTwins demonstrates how persona representation and post-training techniques can enable models to capture and reproduce individual differences. This is not to be overlooked for building more natural and credible human-computer interaction systems, especially in scenarios such as music recommendation, mental health companionship, and education that require understanding individual musical reactions.

Of course, this research is still in its early stages, with evaluation based on only ten held-out personas. Future work needs larger-scale, more diverse persona data to verify the framework's generalization ability. In addition, how to balance persona fidelity with model safety and avoid generating harmful or biased content are issues that subsequent research needs to address.

Overall, MusicTwins opens a new direction for personalization in audio language models, and its "persona summary + personal memory + post-training" framework approach is worth further exploration.

One-line Summary

MusicTwins enables audio language models to simulate a specific individual's musical reactions through persona representation and post-training.

Paragraph Summary

MusicTwins is a framework aimed at enabling audio language models to reproduce a specific individual's music perception and reactions. Addressing the problem that existing models ignore individual differences, the study constructs persona representations from human writing across five dimensions—background, views, personality, writing style, and musical profile—and uses them to post-train the Audio Flamingo Next model via supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows significant improvements in persona fidelity regarding personality, writing style, and musical reactions. This work opens a new direction for personalized audio language models, but the evaluation scale is limited, and generalization ability and safety still require further verification.