MeloChron: Adoption Prediction

The Question

When someone hears a song for the first time, do they return to it? That is the question this project aims to predict.

I chose that target on purpose. The obvious thing to predict is what a listener plays next, but most of that is shuffle and autoplay, where the queue advances on its own and no real choice is made. Coming back to a track is a choice, so it is a cleaner signal of taste, and a better thing to predict.

For every first time a listener meets a track, I wanted to know: will they come back to it within their next 200 plays? Across all listeners and all tracks, 31% come back to the track. Before building the model, I built the baselines, and the surprising part was what turned out to be hardest to beat.

The Model

The model reads a listener's recent plays in order and, for a track they just met, predicts the chance they come back. It has three parts:

• an encoder that turns the ordered play history into one summary of the listener's taste right now,
• a representation of the track they just met, placed in that same space,
• a small predictor that compares the two and gives a probability of return.

The encoder is a Transformer, the self-attention architecture behind language models, reading listening histories instead of text. It has 2 attention blocks and 2 heads at a width of 128, and reads up to the 200 most recent plays. It goes through them in order, learns which ones matter, and folds in the time gaps between plays, bucketed into a learned time encoding, since how recently and how densely someone listens says a lot about whether they come back.

A few details:

• The encoder is strictly causal, looking only backward in time, so an encounter is scored using only earlier plays, never the future.
• The predictor is a two-layer MLP. It concatenates three 128-value vectors, the history summary, the candidate track, and their elementwise product, into 384 numbers, maps them to 128 with a GELU and 0.2 dropout, then down to a single return probability. About 49,000 parameters.
• How a track is represented is the one thing the iteration changes: a learned ID (one 128-value vector per track) or its content. For the content model, each track's 50-value audio embedding is frozen and projected to 128 by a learned layer, so the model never memorizes individual tracks.
• Trained end to end with binary cross-entropy and the Adam optimizer, early-stopped on a held-out slice of time.

By the numbers:

• About 468,000 trained parameters: the 2-block Transformer (roughly 419,000) plus the head (about 49,000).
• Underneath sits a frozen table of precomputed audio embeddings, 56,513 tracks by 50 values, 2.8 million in all. The model reads it as an input rather than learning it.
• Width 128, 2 blocks, 2 heads, history length 200, dropout 0.2.
• Developed on a single 6 GB laptop GPU at a reduced config (15,000 listeners, history 100), then retrained at full scale on a cloud GPU: the whole pool of about 100,000 listeners with a history of 200. The full-scale run is the headline below.

Baselines

I score with PR-AUC, a standard 0-to-1 measure for yes or no predictions. Blind guessing that the user always comes back scores the base rate, here 0.3079, and a perfect predictor scores 1.0. So anything well above 0.31 is doing real work.

The first baselines were rate combinations, learned weightings of two simple numbers. The user × item baseline combines how often this listener comes back to tracks in general with how often people come back to this particular track. The user × artist baseline swaps the track for the artist: the listener's own return rate combined with the artist's.

The strongest baseline, a running per-listener rate, I only built partway through the project, when I got suspicious of a result. It belongs here with the others though: it needs no training, and it turned out to be the bar. How it came about is in the iteration below.

The running rate at 0.4212 is the number to beat. The bars run from the base rate up to it, so their widths show the real distances between approaches.

Iteration

Beating that first bar took several tries, and along the way the bar itself moved.

I started with learned ID embeddings, a separate vector per track. They beat the rate baselines, and on cold users, people with no history for those baselines to fit, the ID model looked especially strong.

That cold-user win is the first result I got wrong. The fitted baselines are blind on a listener they have never seen, so the model might just be rediscovering each listener's own return rate from their history rather than understanding anything new. To check, I built exactly that: a running rate, computed live from each listener's prior returns. It beat the model. The edge had been rate-rediscovery, so I withdrew the cold-user claim. That running rate was now the real bar, higher than every baseline and every model so far, at 0.4212.

The ID model had a second problem: it overfit, memorizing which tracks did well in the training years, so once habits drifted that memory misled it. I switched to content, the track's audio. My first attempt measured content badly and I wrote it off as weak, the second result I got wrong; measured cleanly, audio content scored 0.414, better than IDs but still under the running rate on its own.

The move that worked was to stop competing with the rate and use it. I fed the running rate into the model as a fixed base and let the sequence predict only a correction on top. That made the question precise: does the listening sequence add anything the rate does not already have? I ran it with both track representations, and two ways of combining, on the same set of encounters.

The answer split on the representation. With IDs the correction hurt, since it carried the same overfitting. With content it helped, and beat the rate both ways of combining. The best version, audio content with the rate fed in, scored 0.4520 at the reduced config I could fit on the laptop. Scaling that same model to the full listener pool on a cloud GPU pushed it up to 0.4820.

One more try: could richer content do better? I added lyrics and release era to the audio. It scored 0.4486, no improvement, so the audio embedding had already captured what content offers here. That is where the gains stopped.

Results

These are the full-scale numbers. The model wins in every group, by a wider margin than it managed on the laptop, and all four gaps pass a paired user-bootstrap test with the 95% interval clear of zero.

Precision-Recall (PR AUC)

Prediction vs Actual, per Listener

Ranking Within a Listener

Demo

Pick a listener and step through the tracks they met for the first time. The model starts from the listener's habit so far, their running return rate, and adds a nudge for how the track sounds. The card turns green when that nudge moved the call closer to what actually happened. The question is whether the sound of a track improves on the listener's habit alone.

Roadblocks

The two results I withdrew, on cold users and on content, came from the same habit: when a number looks good, build the test that could break it. The cold-user win fell to the running rate I built to check it, and my early call that content was weak fell once I measured it properly. The running rate itself started as one of those tests, and it turned out to be the hardest thing in the project to beat.

Discussion

The model was identical in every run, so the whole difference is the track description. An ID lets the model memorize the training years, and once habits drift that memory misleads it. A description by sound cannot memorize a single track, so it carries over to new data. That is why content wins where IDs do not, and why it holds up on cold users, people with no history, where the sound is all there is to go on.

A few things make the numbers trustworthy:

• I resampled whole listeners, not rows, and checked the 95% range stayed clear of zero. Rows would look more certain than they are, since one person's plays are not independent.
• The running rate never looks at the future: a past encounter counts only once its outcome is known.
• The baselines were built first and built hard, so the model had to win a fair fight.

Limitations

• One seed and one train/test split, so the smaller per-group margins are not yet firm.
• Adoption drifts down over time (0.37 to 0.31), which miscalibrates any rate fit on older data.
• Developed at a reduced config on a 6 GB laptop; retrained at full scale on a cloud GPU, which lifted the headline to 0.4820 and roughly doubled the margin over the rate.
• Content only pulls ahead with the running rate alongside it; on its own it scores 0.414.
• Not deployed: trained on 2005 to 2020 listening, so a live account today would be out of distribution.

Conclusion

The hardest baseline to beat was a two-line running count of how often each listener comes back. The winning model feeds that count into a sequence model that describes tracks by their sound, lifting the score from 0.4212 to 0.4820 at full scale. The margin, +0.061, is real, and larger than the laptop config had suggested.

Why is that running count so strong? I think adoption depends mostly on the listener. Some people are picky and come back to almost nothing; others are open and come back to a lot. The running rate reads that openness straight from a person's own history, no model required. Taste and the sound of a track add to it, and they matter: audio content on its own scored 0.414, a statistical tie with the rate's 0.421. Even so, most of the signal looks like how open a listener is, and content is what adds to it.