On Collaborative Filtering

A recent set of technologies have been devised to help websites learn about their users and take intelligent actions accordingly. These technologies, called “recommendation engines” or “collaborative filtering,” examine a user’s past viewing habits and compare them with other users who have similar interests. If your interests were found to parallel another group of users, then the system could start making suggestions: suppose you normally never listened to any country music, but you liked bands X, Y, and Z, a whole lot. Now if a whole bunch of other people who don’t normally like country and also like X, Y, and Z suddenly all are listening to (and loving) this one country band, the system might suggest it to you and be relatively confident that you’ll like it.

This technology is neither brand new or obscure: Amazon.com uses it extensively on their website to recommend books to buyers. Indeed, Firefly applied recommendation engine technology to CD purchases on the web many years ago. Unfortunately for them, they took too long to license out their technology (they wanted to be the only people on the ‘Net with that technology) and were subsequently steamrolled over as companies like Net Perceptions came to market swiftly with sophisticated engines. Microsoft quietly picked Firefly off with what was rumored to be a humiliatingly cheap acquisition.

But the fact that no online digital music providers have yet to openly embrace this technology seems surprising to me: this technology is absolutely key to the success of online audio. Why? Because new publishing and distribution infrastructures will make it very easy for artists to publish profusely on the web. Like the 500-channel television, the diversity of content is appreciated, but the sheer quantity of music on the Net could prove so overwhelming as to discourage listeners (and potential buyers) from seeking out the music they would enjoy. Techie geeks refer to this as “the signal-to-noise ratio problem:” if you only hear one band you like (signal) for every twenty you don’t (noise), you won’t want to spend your time poking around for that one band.

The record industry had a fairly effective technique for increasing the signal-to-noise ratio for music: the original point of those practicing A&R, or “Artists and Repertoire,” at record labels was to seek out the good bands that the majority of the population would enjoy. But the Internet offers us what no A&R man could – the potential for individuals to have access to the bands they love both big and small, from all around the world. Recommendation engines make this possible, and reduce the signal-to-noise ratio by presenting music that, based on your prior listening tastes, you’re likely to enjoy.

Ultimately, this obviates A&R: once somebody has heard a band that she deems pleasant to listen to, it will be recommended to those of similar taste – if they like it, it may get recommended to their friends, etc. In this way, the popularity of music is decided on more by the taste of the people than the marketing push of a major label. A small folk artist in Oklahoma could become vastly popular in Northern India; who knows? Everybody benefits from this technology: artists, who get better exposure; consumers, who hear more music that they enjoy; and sites, which have more satisfied customers than before.

To date, people have argued that online audio sites have not yet adopted this technology because of a paucity of content: when there are only 15 artists on a site, a recommendation engine is hardly appropriate. But with the rising tide of acceptance of online distribution, floods of artists have been flocking to centralized music portals like eMusic, MP3.Com, and Audio Explosion. This newfound influx has left the sites unable to provide tasteful experiences for their users, leaving them instead awash in a flow of “exotic” (to phrase it kindly) amateur music from around the world. They have reached the size and maturity to move to collaborative filtering. And move they must: the stakes are large in this brave new world and the listeners plentiful. The winners will quickly adopt and manage these new technologies, and those less nimble will be left wondering why more people didn’t come listen to their 15,000 artists.