As we move towards a world of data-driven optimization, the optimal path will become the obvious, uninteresting one. This will temper our hunger for the best outcome, and instead make us crave the more interesting, unexpected, serendipitous one.

At last week’s Strata conference, all the focus was on real-time data, and on connecting the Internet of Things to big data better. But this uncertainty about the long-term effects of optimization was an undertone. I thought I’d write some of it up here.

Instrument everything

We’re in the middle of an unprecedented wave of instrumentation. Having tasted the online world, and—from a data perspective—found it deliciously easy to analyze, we’re putting sensors on everything from trucks to golfers to patients to thermostats.

All the data these sensors create is grist for the mill, giving us insight into what was previously unmeasurable. Once, the best indicator of a song’s popularity was asking radio DJs what tracks people were calling in to request; since 2008, Arbitron’s Portable People Meter has tracked exactly which songs make listeners stick around, and which songs make them change channels; in recent years, it’s been expanded to track the wearer’s location as well.

But data alone isn’t a goal. To be useful, data needs to be part of a bigger system that tries to produce a goal, such as weight loss, faster manufacturing, or a more energy-efficient home. Lyft, for example, is a system that includes a smartphone app, data, drivers, and road conditions, all aimed at efficiently connecting an available driver to a waiting fare. Salesforce’s Lisa Hammitt offers the example of a smart toothbrush: to be useful, it needs to be part of a community that includes the family, dentists, and reminders.

Just as sensors beget data, so data begets optimization. “You can’t manage what you can’t measure” is a quote often attributed to Peter Drucker (except that he never said it; his views were far more nuanced.)

But in an era of data-driven optimization, we might consider the non-Drucker corollary:

That which can be measured will get optimized.

If the data is available, then we can find out what the waste is; and once we identify waste, we can extract profit from it by making the system more efficient. Applications squeeze every drop of inefficiency from parking meters, empty apartments, snow cleaning, conference space, house cleaners, travel discounts, idle drivers, and more.

When everything’s optimized, is that still optimal?

When everything’s optimized, however, the optimal path is no longer optimal. That’s because optimization is part of a system, too, and it produces changes in behavior. An 1882 London magazine appropriately titled The Nonconformist and Independent ran the following joke:

“I’m afraid you’ll be late to the party” said an old lady to her stylish granddaughter, who replied, “Oh, you dear grandma, don’t you know that in our fashionable set, nobody ever goes to a party ‘till everybody gets there?”

Self-fulfilling prophecies

One reason for this is that optimization can become a self-fulfilling prophecy. One of the companies I’ve been looking into for a report on music and data is Shazam, which lets you identify a song simply by pressing a blue button on your smartphone (Soundhound is a similar application.)

Shazam’s head of product, Cait O’Riordan, says that this means the company can predict with great confidence which new songs will be popular within a few hours of their release, simply based on how many people ask their application to identify the song.

Armed with this information, the label’s marketing departments can decide where to spend their money, doubling down on promising tracks and starving those that don’t fare as well in the first few hours. But as this practice becomes commonplace, how much of a song’s eventual success or failure will be due to the marketing? And will artists eager for an extra boost tailor their songs to provoke listeners to wonder what it is?

Remember: Shazam isn’t a sensor—it’s part of a system that includes artists and labels and listeners, and it’s constantly optimizing.

The local maximum

Another problem with optimization is that it focuses on the local maximum. Optimization is really good at improving the current model; it’s not as good at finding new models. That’s why incumbent businesses can often maximize their existing revenues and margins, but are frequently eclipsed by new competitors.

When it comes to tastemaking, optimal may not be what you want. Apple has hired tastemaker DJ Zane Lowe to help them, presumably with music curation. And at a recent Data and Music meetup run by The Hive, Google Play’s Dr. Douglas Eck says that when their service correctly recommends an artist to a listener who already has all that artist’s music, the listener actually gets angry—they don’t want to be told what they already like. The local maxima isn’t interesting, because it’s obvious.

An interview with a system

Whenever I take an Uber, I ask the driver questions about their experience. I do the same with AirBnB hosts, and any other person who’s part of a blended human/machine optimization platform. It’s an unusual opportunity to have a conversation with a component of a service.

Initially, Uber drivers told me they liked that the service suggested where they should idle for the best chance of picking up the next passenger. But in DC one day, as the service became more common and ubiquitous, a driver told me he now ignores those suggestions:

“I don’t wait where Uber tells me to, because there are already a bunch of other drivers there, waiting. I have my own system,” he assured me, “and I like to park where I can see two or three hotels at once.”

To paraphrase Yogi Berra, nobody parks there any more—it’s too busy.

This tension between optimization and inspiration, between the efficiency of listening to the machine and the serendipity of ignoring it—or outright contradicting it—will keep the over-optimization of taste at bay. When we forget that optimization cycles are part of a system, or that optimization can be the enemy of novelty, we treat data as a panacea rather than a tool.

Finding serendipity

A few years ago, Localmind/AirBnB’s Lenny Rachitsky talked about how computers that recommend what’s best might cost us the serendipity of the unexpected. As we optimize more and more of our worlds, inspiration and leaps of faith become increasingly unusual, unique, and ultimately, attractive.

We’re wired for novelty; like the constant tension between science and art, many data-driven systems that focus on taste won’t stabilize, because they’re full of messy humans and misunderstood feedback loops.