Explorative and Evaluative Method to Study Users' Personalized Music Experience with Spotify

MY ROLE:

Growth Product Analyst

Video Observation Plan & Analysis
Business Strategy
Personas (Cluster Analysis)

Stakeholders:

I sought guidance and insights from these research experts from various areas (VR game Research & Customer Insights)

Dr. Jan Plass
Dr. David Bosch
Ms. Amalba Kola

Client:

Spotify (Passion Project)

Duration:

Jan - May 2023

Research Type:

Foundational Research
Evaluative Research
Descriptive Research

Research Methods:

Video Observation
User Interview
Survey
Contextual Inquiry

Toolkit:

Datavyu
Dovetail
R Programming

Context

Spotify and Its Personalization

Spotify is a music streaming titan with over 515 active users spread across 180 markets worldwide.

What sets Spotify apart? Its secret sauce is a blend of AI technologies including natural language processing, collaborative filtering, and audio models, delivering an incredibly personalized listening experience to each user.

From Discover Weekly to Your Time Capsule, Spotify's diverse range of tailored playlists has transformed music consumption. No matter what you're in the mood for - be it working out, commuting, or simply unwinding, Spotify has a playlist for you. The platform even creates unique Blend playlists, merging the musical tastes of two users into one, while also introducing users to new songs and artists based on their listening trends and streaming history.

New AI DJ Feature

On February 22, 2023, Spotify introduced a new feature: the AI-powered DJ. This innovative feature was designed to enhance users' radio listening experience by combining AI technology with elements of traditional radio

Just like a human DJ, the AI DJ shares interesting facts about songs, seamlessly transitions between tracks, and adjusts the music selection according to the user's mood and preferences. This new feature was developed with the aim of delivering a more personalized and engaging listening experience on Spotify.

Behind AI DJ Feature

Voice technology: powered by Sonantic AI (acquired by Spotify in 2022)

Generative AI technology: from OpenAI, combined with expertise from Spotify's culture experts, music experts, data curators, and scriptwriters
Debate: strengths and potential weaknesses of AI technology, humanizing algorithms in response to generative AI trends

Problem Discovery

Business Strategy Problem: Spotify is struggling with their business

At the heart of Spotify's business challenge is a significant question: Is the heavy investment in AI technology for personalization, such as the new AI DJ feature, worth the cost?

After a vast investment in AI technology for personalization, Spotify lost $401 million in 2022
They have tried to adjust by adding more human curation in their personalization service
Despite operating at a loss, Spotify continues to invest in personalization to improve the user experience and stay ahead in a competitive market. Most recently partnering with Open AI and Sonantic AI to launch the AI DJ feature. However, it's worth questioning whether these investments might deepen their financial difficulties
One consideration is the potential cost-effectiveness of human curation. While AI technology enables hyper-personalization, it's uncertain whether consumers can distinguish between music recommendations made by AI or human curators

Design Problem: Unsatisfactory User Experience Derived from AI Algorithm's Music Recommendations

Even with Spotify's advanced algorithms crafting hyper-personalized playlists, there remains user dissatisfaction concerning the accuracy and diversity of these recommendations, a concern that directly ties into the company's broader business challenges

Research Goals

Below is the thinking process to get a concise research goals:

AI Algorithms vs. Human Curation

Originally, I planned to use A/B Testing to see if the LABEL “human” or “AI” by itself makes a difference in how people perceive the experience without regard to whether it is really AI or human-generated

Narrow Down Research Scope to AI DJ Feature

However, as I delved deeper, I realized that this approach might not fully address the business challenge at hand

While the results could provide insights into user perceptions, they would not directly assess the value or efficacy of the recent, substantial investments in AI technology

I addressed two research issues in this project: the AI DJ feature's user experience and the comparative effectiveness of AI versus human curation

Additionally, obtaining internal data on Spotify's human-curated playlists was a challenge.

Therefore, I decided to narrow down the scope of my research and focus on the AI DJ feature

Final Research Objective

The primary objective of this project was to assess the user experience of Spotify's AI DJ feature, and how it contributes to the broader personalization strategy.

Problem Statement

"How can Spotify optimize the personalization and emotional resonance of its music streaming platform through AI-powered algorithms and a human-touch approach?”

Research Questions

In the process of formulating the research questions, I identified the necessity of pinpointing what constitutes a satisfying personalized music listening experience.

I did a comprehensive literature review and dive into user stories to gather contextual insights. Then, I created a structured list of independent variables (IV) and Dependent variables (DV), which helped me write research questions:

Figure 1. Video Observation Research Tasks [JP1] for Spotify's AI DJ Feature

- What is the emotional context of users' music listening preferences, and how can Spotify better understand and incorporate this into their personalization algorithms (i.e., AI DJ)?
- What are users' expectations and needs for music personalization, and how can Spotify meet those needs through its AI-powered personalization features?
- How well do users perceive the accuracy and diversity of the personalized playlists generated by Spotify's AI DJ feature?
- What factors do users find frustrating or challenging when using Spotify's AI DJ personalized playlists, and how can they be addressed?
- What role does the AI DJ feature play in Spotify's broader AI-powered personalization strategy, and how effective is the AI DJ feature in personalizing the real-time emotion and needs of users, such as relaxation, focus, and sleep?

Research Approach

To address research questions, I employed mixed-methods strategy
- Each method was chosen to specifically correspond with each individual research question.

Research Challenges

Limited usage data

AI DJ feature is new, lacking sufficient usage data for meaningful conclusions

Figure 2. Flow chart of Research Process

Adoption barrier

Some users may be unaware of the AI DJ feature, complicating the collection of diverse experiences.

Remote Testing Problem

Remote-controlled testing for context research may not show users' natrual behaviors and settings, which can cause bias or data distoritions.

Lack Generalizability

The sample of users from the US and Canada, where the AI DJ feature is available, may not be representative of Spotify's global user base.

Primary Method: Video Observation

Why Video Observation as the Main Method?

Usability Emphasis

Video observation highlights emotional reactions and non-verbal cues, aiming more at functionality.

Rich Data

Video observation captures detailed behavioral data, including speech, facial expressions, gaze, emotions, postures, gestures, social interactions, and object interactions.

How I Coded Videos?

Step 1: Design Tasks for Video Observation

Designing tasks for video observation during usability testing of Spotify's AI DJ feature can be challenging but also interesting to use psychological principles to categorize how people feel and behave

To ensure that the tasks accurately reflect the user's experience, I realized the importance of considering the user's level of familiarity with the feature and design tasks that test various aspects of usability and user experience
Also, after testing my originally designed tasks, I found that tasks should be open-ended and avoid being too prescriptive or leading to ensure the validity of the observations.

Step 2: Designing Code Scheme

Studying user interactions with AI DJ features was challenging due to the subjective nature of emotions and behaviors.

To tackle this, I developed a multi-dimensional observation framework that captured both explicit actions and subtle cues.

After pilot testing with a small sample, I refined the approach to improve accuracy and reliability of outcomes.

Key Variables Measured

Analyzed verbal and non-verbal reactions to Spotify's AI DJ, focusing on:

Emotional & Cognitive Responses: Positive engagement, frustration, or emotional shifts.
Behavioral Indicators: Adjusting AI DJ settings, adding songs, or speeding up playback.
Perception of Accuracy & Diversity: How users rated playlist recommendations and acknowledged musical variety.
Pain Points: Navigation struggles or frequent playlist adjustments.

Step 3: Video Coding

This was my first time using Datavyu code videos — I spent 1 more week to learn the tool with the help of Dr. Jan Plass.

Being unfamiliar with this tool, I sought guidance from my supervisor, who generously provided me with resources and tutorials to help me navigate the software.

Dedicating a week to master Datavyu, I experimented with different functionalities.

Through this intensive learning and practice, I was able to tailor my coding scheme more efficiently and accurately, paving the way for comprehensive analysis of user interactions with the AI DJ feature.

Versatility

Helps analyze complex interactions and changes more effectively than direct observation.

Table 3. Video Observation Research Tasks [JP1] for Spotify's AI DJ Feature

Table 4. Rubrics for coding of observations (Examples)

Figure 5. Rubrics for coding of observations (Examples)

Findings

- 81% liked AI DJ’s music selection, but 19% found its interruptions annoying, leading to boredom after 20 minutes.
- Users preferred curated playlists over AI DJ when feeling down since filtering songs felt like extra effort.
- The need for better emotional alignment in recommendations was evident.
- Users favored daily mixes as they better suited their moods without requiring adjustments.
- "Made For You" playlists were particularly appreciated for introducing new music while blending with existing favorites.
- Both are valuable but need clearer differentiation and improved personalization.

Growth Strategies

Combined with feedback was synthesized through affinity diagramming, I proposed these recommendations for redesigning the AI DJ feature to Spotify

☺ Enhance Personalization

Balance familiar and new songs, making AI DJ feel less robotic. Let users filter by artist, genre, or mood

☺ Differentiate AI DJ from General AI Algorithms

Clarify AI DJ’s unique purpose

☺ Enable Voice Interaction

Allow users to talk to AI DJ, making it feel more engaging

☺ Add Interactive Feedback

Introduce like/dislike options to refine recommendations

☺ Give Users a Toggle for DJ Commentary

Some found the interruptions disruptive, so adding an option to disable them would improve experience

Expected Impact

35%

Higher session durations

20%

Active users could adopt AI DJ as a primary tool for discovering new music after iterative improvements

30%

Improved user satisfaction by setting a new standard for AI-driven music personalization

How I Estimated the Impact

Engagement (+35%)
- Based on AI-generated playlists (Daily Mix, Discover Weekly) increasing listening time
- AI DJ refinements reduce skips and improve engagement
  - Further Testing: A/B test session durations before and after AI DJ improvements
Adoption (+20%)
- Benchmarked against curated playlist adoption (~30-40%)
  - Further Testing: Track repeat usage rates and compare AI DJ adoption vs. traditional playlists.
User Satisfaction (+30%)
- Similar AI-driven features show ~30% satisfaction boosts with better personalization
  - Further Testing: Conduct pre/post-user surveys and sentiment analysis on AI DJ interactions.

My Learnings

⇧Define Key Variables – Defining clear independent variables was key to making the video analysis meaningful. I adjusted them to better capture user behavior

⇧ Improve Coding Rubrics – Codes should be specific and relevant to the context of video coding, with examples of observable user behavior that accurately reflect the user's engagement and satisfaction with the product. Perform a pilot testing with a small sample is a necessary step.

⇧ Iterate & Practice – Regular hands-on practice helped refine video observation methods with stakeholders

What I will do differently next time?

Corporate with a UX Designer and Data Scientist to realize the redesign and test solutions, and solve the constraints of this research project

What I will do differently next time?

Conduct Longitudinal Study: Next time, I will conduct a longitudinal study to track user behavior and satisfaction with the AI DJ feature to capture changes over time

Expand User Sampling: I will expand user sampling to include a diverse group of Spotify users globally once the AI DJ feature is available
Include AI DJ Feature Non-users: I will investigate reasons some users do not use the AI DJ feature to identify potential improvements and overlooked needs

🎥 Check out this video showcasing my exploration of AI-driven music personalization: YouTube Link

(This is an independent passion project and is not affiliated with Spotify.)