Monday, July 10, 2023

the bias spotlight

Imagine walking home at night, down the long and darkened street leading up to your door. Repetition has fostered familiarity but you're never completely comfortable on this road - narrow or poorly maintained sidewalks that occasionally blend into a dirt shoulder, cars zipping past both too fast and too close, streetlights spaced far enough apart such that they emphasize rather than reduce the darkness. It's always a relief to make it home.

As you reach for your keys, you sense something isn't right. You try one pocket, then another - no phone. You look into your backpack even though you know your phone always goes into your front right pants pocket. Automatically you turn around and head back to the road, literally retracing steps while your mind replays the night's events. You remember putting your phone back into the pocket after dinner, then you went to the bar for a last drink. Did you have it at the bar? You think so. It's likely that the phone fell onto the road during the walk. Plus, if the phone was at the restaurant or bar then someone else is going to find it. The best option here might be to ask a neighbor to call your phone but it seems too late to bother acquaintances. You consider asking a friend to help, but you can't reach them without a phone and none live within walking distance. You decide to spend a few minutes looking for it on the road just in case you can spot it right away.

But back on the road, the issue is immediately obvious - visibility. The streetlights offer pitiful cones of light which are swallowed up into the black hole of night. You aren't sure you have a flashlight at home so you decide to try your luck around the streetlights. The visibility is good in these areas so you'll find the phone if it fell under one of these lights. But in the end, it's a futile exercise. The reality is that the darkness on this road, even just alongside the street, covers much more ground than the light. You make one last attempt, standing deliberately in the dark hoping that your eyes will adjust enough to pick out fallen rectangular shapes, but you give up after a couple of minutes. The only remaining option is to crawl, so that's what you do - you crawl into bed, hoping it doesn't rain before the dawn.

******

I'm not the biggest fan of using carefully curated parables to make a point. But I've found the above coming to mind recently as I've worked through a number of TOA posts. It's based on an anecdote that I believe is known as the "Streetlight effect", which must be its official name since there is a Wikipedia entry for it [citation needed]. It's sometimes told as an anecdote about a drunkard looking for lost keys. In this version, though he knows they were lost on the darkened side of a street he ends up looking on the other side. When asked to explain why he's on the wrong side, he remarks - this side has lighting.

I originally heard the example in the context of economics, perhaps as much as a decade ago. My understanding is that the economists are trying to make a point about the availability of information, which often creates a form of bias by dictating that they only look for answers where there is available data (economics is known as a social science but you'd be forgiven for thinking it was a data science, especially if you are unaware of the definition for data science). To put it another way, the fact that some information is more visible than others is an especially dangerous form of bias because you'll assign greater weight to the available information.

These are all reasonable points. And yet, embedding such in these kinds of stories tend to create their own problem. When a lesson is contained in such an easily digestible form, I fear it creates a false sense of understanding. The specific issue is how stories sometimes sand away the subtleties in order to establish universality. The problem is that since the drunkard's error is presented in a somewhat ridiculous context, most of us hear the story and feel that we already know not to make such a foolish error. For most, I think the story serves more as a reinforcement or a reminder than a new lesson - we'd have to be drunk to do that! It never occurs to us that with some further thinking we can ask more important questions about the story. In real life, is it always so obvious which side of the metaphorical street is lit and which side is dark? Curious students who wish to maximize their learning from the story would consider how to make this determination within the areas of investigation they encounter in the future.

And yet, there is still something else that is missing. Is the only important thing to know which side of the steet is lit? I've had this feeling with me while working on another post, which I'm hoping to finish in the coming days. It's about Jeremy Lin and the way his career intersected with the stereotype of Asian athletes being unathletic. Here's one detail I plan to include in the final version. Speaking about the scouting reports ahead of the 2010 NBA draft, Lin once said - me and John Wall were the fastest people in the draft, but he was 'athletic' and I was 'deceptively athletic.' This isn't another example of an athlete making his own case, which of course has a long history of being slightly favorable to their cause. Daryl Morey, then general manager of the Houston Rockets, said a few years later that the team's pre-draft scouting model suggested Lin was the 15th-best player in the draft. On draft night, the athletic Wall went with the first overall pick and the deceptively athletic Lin, after sixty total selections, went undrafted.

We'll have to recalibrate the anecdote a small bit here - this is like if the drunkard got himself a powerful flashlight, then decided that the suddenly visible keys represented something else, like a soda can, maybe, or someone else's keys. And who could rule out the possibility of a hopeful mirage at the end of a tough night? Regardless, the outcome is the same, the drunkard doesn't find his keys, but I suppose it imparts a different lesson to us. What can I say about those who, despite seeing the "unathletic" player zip past their opponents, fail to relabel the player? What can we learn when a general manager, in possession of a scouting tool which offered to illuminate the darkened street, decided he would rather stand under the streetlights with his twenty-nine peers? We say what we need is a perfectly lit street on a cloudless, sunny day, but we should remember that we'll need something more for those who just refuse to see.