The Custom Internet & Recommendation Engines
Almost as long as the internet has been around, there have been recommendation engines. I remember my mom telling how cool she thought it was that Amazon would tell her what other people had bought, and how if she bought a spatula, Amazon might recommend a whisk.
Over the years, recommendation engines have evolved and gotten more intelligent. Everything from Google’s results pages to the watch lists Netflix offers you are informed by algorithms, and what these apps think you’ll like. My favorite example is the Spotify feature that will offer a continuation of any given playlist or album after you’ve listened to the last song. They use data to sift through millions of playlists and determine the 17th or 18th song that also tends to go with your initial playlist. They are able to intuit what you might find interesting based on what other people that are similar to you have found. To me, that's a great success story and a great user experience.
Therefore, the question comes up as to why this isn’t the case in more areas of our lives, whether it's food delivery, job applications, or anything else. The internet could certainly benefit from it: When it first came out, it was this amazing thing because it made information easily accessible, and we're now in an evolution of the internet where there's too much information. I believe it’s helpful when all of that information is parsed down to the one percent of data that is relevant to us and what we actually want, need, or like. But I also believe that doing so needs to respect our ability to choose — without it, the technology can be terrifying.
Why Aren’t All Recommendation Engines as Good as Google?
I believe deeply in free markets and that in the analog world, there's more or less a meritocracy where the best businesses will get the most work. It’s sort of a capitalist notion that the better you are, the further you'll rise, whether you're an institution or an individual.
When it comes to who or what is determining the meritocracy of the internet, I guess the answer to that question would be Google. The better your website, the higher it comes up in Google and the more discoverable it is and the more it's used. Your company’s growth can easily compound in the way. These powers that be determine what lives and what dies and what is surfaced and what is not, which I think is good, but I'm fearful that sometimes the things that you want are buried below that first page of your search. If that happens, you might never find something that you would have really wanted.
And while Google is pretty good at getting their first page to be good, based on what you've typed in based on your past search history, few other services give you that caliber of results. And because people so frequently rely on Google to serve them good options, it’s easy for them to expect the same kind of results from apps that don’t function in the same way. Tinder and Postmates don’t show you the best options based on your search history; they just show you everything. And that's important, especially if you live in a big city. Those random offerings aren’t catering to your specific tastes in romantic partners or food.
Most people that I know are admittedly in a certain socioeconomic strata, but they buy a lot of things that they don't need. It's just stuff that they want. And to an extent that a recommendation engine just makes it easier for them to do that, I don't know that that's a good thing. But I also think that a recommendation engine, if built correctly, is just perpetuating what you would do anyway.
How the Biases of Recommendation Engines Can Work For or Against You
All recommendation engines are built by people, who inherently have their own biases. But I believe that if you democratize the process and collect enough data, enough diverging opinions will override the biases that may have been coded into the engine and give you what is objectively true.
Because I'm really interested in objectivity and truth and in removing bias, I built a recommendation engine at The Hub maybe four years ago that is basically a machine learning engine. I selected a panel of 15 creators in our community and I tried to pick people who are as different as humanly possible: completely different geographies, completely different levels of experience, completely different styles of photography, completely different genders, completely different races. These people log into a certain portal on my website and are shown flashcards containing work by an anonymized photographer. They don't know if the artist is a man or a woman, or their race or experience level. It just shows the recent images taken by a photographer and they rate the photography and categorize it by style. Every photographer on my platform has therefore been rated by this panel of 15 people that are all very different, and I believe the cumulative rating is the objective truth. By putting 15 opinions in a blender, it accounts for all biases and because they don't know who they're reviewing, the panel couldn't possibly be biased towards another human being. It is simply the work. It is as pure a meritocracy as I could muster.
Because recommendation engines will typically expedite the decisions that other people make, I believe that their use actually helps the meritocracy system. Take résumés for example: They're an incredibly static way of presenting information. It’s impossible to take the enormity of your career and how you’ve done all the things you’ve done, and consolidate it into a single page. So if machine learning is going through and picking out certain keywords, and removing 20% of the applicants and flaging 20% of the applicants with a certain GPA threshold so that HR has an easier time, then that HR is necessarily going to miss a lot of qualified candidates who will ultimately go elsewhere. Then, I believe that the other institutions will become more talented and better, and will beat out the prejudiced institution that created those arbitrary thresholds. I see nothing but good in the idea that if you are biased and you're leaving talent on the table, then you will be weaker as an institution and natural selection will root you out.
Let’s say there are two law firms and one law firm uses machine learning to filter through candidates and the other law firm doesn't and they give everyone a chance. Studies have been done about IQ across gender and there's no correlation between IQ and any other identity factor. So if an institution is inherently biased or prejudiced and they're not hiring women or people of color, all of those smart people will go to another establishment. If the law firm that gives everyone a chance gets talent that was overlooked by the first law firm, that law firm will be better and it will ultimately book more revenue than the biased law firm. And then the biased law firm, over time, will lose.
Do Recommendation Engines Create a Thought Bubble?
The danger of recommendation engines is that you tend to get further entrenched into your ways of thinking. We’re seeing this happening in real time on Facebook, where people with conservative views are adopting right-wing views given their news diet. Or take the Spotify example: If the algorithm plays a certain type of music again and again, and you are deeply embedded into that type of music, are you not discovering other options and broadening your horizons? Not only do recommendation engines run the risk of keeping you sequestered, they can actually deepen the myopia of your existence. You go deeper and deeper into your own wormhole.
There's this old expression: Walk a mile in another man's shoes right before judging him. And I believe it would be really interesting if you were able to swap algorithms with another person for a week so that you could view the internet as them. I predict I would be able to learn about a person’s biases and opinions, and I might have a lot more empathy and respect for why they are the way they are and why they think the way they think.
I think I’m OK with technology knowing so much about me but at the same time, I'm very scared of it. I am more alarmist and concerned than most but I think a lot about where we're going to be in 10 or 15 years and what technology will have done to us. I think it's causing massive ripple effects in our psyche, our anxiety, and our sense of self. Our autonomic nervous system is a mess as a function of how connected and stimulated we are. I will raise my children with probably less technology than I have access to right now, because I don't like how technology is so integrated into our existence and way of being.
But on the other hand, I very much participate in technology. Having this bionic proposition where what I want and what I'd like to do and accomplish are intuited by the technologies that I use so that I can do those things faster and get more done in the day — all of that is exciting to me. The fact that I'm as connected as I am has allowed me so much time, but the question becomes: For what, exactly? How do we use that time that technology gifts us? Through my software, I have access to a network of 40,000 photographers — all from a humble little laptop in the middle of nowhere. And that in and of itself is amazing. But if we don’t ask broader questions like what we’ll do with that time and how it can benefit us, perhaps the utility is wasted.
Does the Power of Choice Override an Algorithm?
I fundamentally believe in the power of choice, but at the same time, we as humans tend to overvalue the differences between outcomes. Gaining a job, losing a job, getting into a college, not getting into a college, gaining a romantic partner, losing a romantic partner — these are all things that we think will have outsized effects on our happiness but ultimately don’t. If I were to lose my job today, three months from now, I'd be the same level of happiness that I was before losing my job. I have days where I'm depressed and down, and I have days where I'm really happy and elated, and with few exceptions, that would be the case irrespective of what I did for a living, where I went to college, or who I'm dating. So what we choose is actually far less significant to our happiness than we think.
Attainment does not predict happiness. Progression along a continuum does. For me, having aspirations and striving towards them is what makes me happy. Accomplishing the goal that I set forth does not. So as recommendation engines help us to get the things that we want more quickly, there's less toiling, less work, and less progression along a continuum to get what you want. There’s less work — it’s just clicking a button.
I can imagine a future in which an algorithm correctly intuits that because it’s Tuesday night and raining, you’re going to want sushi from a specific restaurant, so it sends you that delivery without you having to press anything. That would be a world in which you want for nothing at all, and everything is just predetermined and provided for you. You would have nothing to strive for, and no means of attaining happiness.
It doesn't matter what your continuum is — you can want to be a billionaire and own a jet, or you could want your kid to go to college because you didn't. Whatever your goal is, It just matters that you're getting better and striving forward and feeling like you're actualizing and accomplishing and bettering yourself. If you’re doing those things, then you will be happier. And if you simply just achieved every goal without any effort, you would not be happy. So the ideal recommendation engine still allows you to work through and for what you actually want while still giving shortcuts and pathways to get there. Think of it as the light that guides the way, rather than a teleportation device that places you there.
The Paradox of Options
It is behavioral science that the more choices you have, the less happy you are with your choice. Think back to a life before Amazon and even malls, when the store on Main Street stocked maybe two kinds of milk and those were the only choices. And while there are now dozens of kinds of milk, now recommendation engines synthesize that same process and show you two or three things. It’s a digital bottleneck, but it quiets out the choices that aren’t quite right for you.
There are two ways of limiting choices right now: One is a recommendation engine, which is an algorithm, and the other one is old-school tastemakers making the decisions for us. The example that always comes to mind for me is a DJ choosing what is worth playing, or a food critic deciding which restaurant is good and which one is bad. You could have a human being be that filtration system, or you can have an algorithm be that filtration system.
I would argue that using a human being to filter out the noise can be helpful, but it's actually much more biased than an algorithm. Meanwhile, if you have 10,000 Yelp reviews, that final average score is how good the restaurant is. A certain critic has their own personal opinions, and they might like one restaurant more than average or less than average. But crowdsourced reviews are also very helpful because they are the truth without bias.
We’re barreling down a path in which the algorithms may one day supersede the human critics, whom we’ve trusted forever. I could tell you that driverless vehicles get in 1/100th, the accidents as vehicles driven by humans, but the first 10 times that you get in a car and don't touch anything, it would still be a little scary to give that governance over to a machine — even if, statistically speaking, that machine is way more accurate in terms of its driving decisions than a human is.
Giving up power to an algorithm that curates our options for us is an odd evolution, but it is yielding the same thing that we’ve always known, which is fewer choices. As long as we retain our free will and feel as though we still have a choice, I think we’ll be OK. It’s when music could come on your Amazon Echo without you even asking for it, just because it knows what you like at a certain time of day that things could get a little scary. But that is a possible future we’ll have to revisit in perhaps five years. It will be interesting to see if we are more or less just as happy then as we are now.