leftovers - the bias spotlight on the dumb smart trade (qualitative and quantitative evaluation methods)

A final thought here that spans the previous two posts. It occurred to me that one issue in evaluating someone like Smart is how his best qualities as a basketball player are not easily measured by the sport's conventional data analysis methods. This means comparisons against players with more easily measured skills will never quite reach the apples-to-apples standard. In a sense, when you evaluate a player like Marcus Smart you actually need to evaluate your evaluation methods before you can reach a meaningful conclusion about the player.

This obviously makes little sense so let's try a metaphor of measuring students. A common method is using a letter grade for each course, which is then translated to a cumulative GPA. If you have one student with a 3.6 GPA and another with a 3.3 GPA, the scores offer a strong suggestion of the better student. But what if the 3.3 GPA student also took one class per semester pass/fail? All of a sudden, you need to evaluate the evaluation method - how does the pass/fail detail influence the overall evaluation of the student? I see players like Smart as the equivalent of the 3.3 GPA student with a handful of "pass" grades where we know "pass" puts him at the top of the cohort. How do you compare him to a player whose top attributes are supported with clear data (essentially, the equivalent of a 3.6 GPA) yet we know that despite having identical "pass" grades those should carry less weight than Smart's?

The right answer here is probably that you can't make a meaningful comparison, at least with data alone. The wisest course of action would be to reduce the importance of data in the decision. However, I think the data analytics mentality encourages pushing on rather than stepping back, and this insistence is where the method approaches one of its shortcomings. What if we came up with a way to measure hustle, or dedication, or leadership, with more precision than pass/fail? Why can't we translate pass/fail grades into a GPA-type score? These are reasonable questions and their implied directions deserve some consideration. However, there is a point where some factors should be acknowledged as inherently qualitative, and it's a major mistake to treat these as quantitative just because you've devised a method for assigning a numeric value. In other words, what I consider the biggest failure in the data analytics mentality is the inability to recognize situations where quantitative data is essentially meaningless because a qualitative method was used to create that data (1).

If you always push forward and try to refine the data such that the numbers will have the last word then you become a bit like the drunkard refusing to search outside the visibility of the street light. What gets lost in the absurdity of the drunkard's tale is a subtle commentary on ideology - by standing under the light, the drunkard demonstrates his conviction that without lighting he will never locate the keys. This means other approaches that do not fit within the ideology - such as a thorough retracing of steps - are dismissed out of hand. Likewise, being data-driven has its benefits as long as the data itself remains a reliable source of unbiased information. But if you don't have a way to include other evaluation styles in your decisions, then you allow a certain bias to infiltrate your evaluations - you will tend to prefer options whose strongest qualities just so happen to be the most easily measurable. If this is the way you make data-driven decisions then what you have is an ideology, not a methodology, and you will find that rather than eliminating bias you are simply trading one form of it for another.

Footnotes

1. TOA was well-reviewed, 4 stars out of 5 

I notice this problem during your average corporate performance review, where a manager will be asked to evaluate a team member using a numerical scale of 1-5 for essentially subjective skills such as "communication" or "empathy" or "wastes company time writing TOA". There are some aspects of these (or any) skills where a numerical measurement would be appropriate, but I don't think anyone will convince me that an overall evaluation distilled into a single number would have any meaningful value whatsoever, particularly if that number is used to make comparisons against peers in discussions regarding recognition, bonuses, or promotions.

the bias spotlight

Imagine walking home at night, down the long and darkened street leading up to your door. Repetition has fostered familiarity but you're never completely comfortable on this road - narrow or poorly maintained sidewalks that occasionally blend into a dirt shoulder, cars zipping past both too fast and too close, streetlights spaced far enough apart such that they emphasize rather than reduce the darkness. It's always a relief to make it home.

As you reach for your keys, you sense something isn't right. You try one pocket, then another - no phone. You look into your backpack even though you know your phone always goes into your front right pants pocket. Automatically you turn around and head back to the road, literally retracing steps while your mind replays the night's events. You remember putting your phone back into the pocket after dinner, then you went to the bar for a last drink. Did you have it at the bar? You think so. It's likely that the phone fell onto the road during the walk. Plus, if the phone was at the restaurant or bar then someone else is going to find it. The best option here might be to ask a neighbor to call your phone but it seems too late to bother acquaintances. You consider asking a friend to help, but you can't reach them without a phone and none live within walking distance. You decide to spend a few minutes looking for it on the road just in case you can spot it right away.

But back on the road, the issue is immediately obvious - visibility. The streetlights offer pitiful cones of light which are swallowed up into the black hole of night. You aren't sure you have a flashlight at home so you decide to try your luck around the streetlights. The visibility is good in these areas so you'll find the phone if it fell under one of these lights. But in the end, it's a futile exercise. The reality is that the darkness on this road, even just alongside the street, covers much more ground than the light. You make one last attempt, standing deliberately in the dark hoping that your eyes will adjust enough to pick out fallen rectangular shapes, but you give up after a couple of minutes. The only remaining option is to crawl, so that's what you do - you crawl into bed, hoping it doesn't rain before the dawn.

******

I'm not the biggest fan of using carefully curated parables to make a point. But I've found the above coming to mind recently as I've worked through a number of TOA posts. It's based on an anecdote that I believe is known as the "Streetlight effect", which must be its official name since there is a Wikipedia entry for it [citation needed]. It's sometimes told as an anecdote about a drunkard looking for lost keys. In this version, though he knows they were lost on the darkened side of a street he ends up looking on the other side. When asked to explain why he's on the wrong side, he remarks - this side has lighting.

I originally heard the example in the context of economics, perhaps as much as a decade ago. My understanding is that the economists are trying to make a point about the availability of information, which often creates a form of bias by dictating that they only look for answers where there is available data (economics is known as a social science but you'd be forgiven for thinking it was a data science, especially if you are unaware of the definition for data science). To put it another way, the fact that some information is more visible than others is an especially dangerous form of bias because you'll assign greater weight to the available information.

These are all reasonable points. And yet, embedding such in these kinds of stories tend to create their own problem. When a lesson is contained in such an easily digestible form, I fear it creates a false sense of understanding. The specific issue is how stories sometimes sand away the subtleties in order to establish universality. The problem is that since the drunkard's error is presented in a somewhat ridiculous context, most of us hear the story and feel that we already know not to make such a foolish error. For most, I think the story serves more as a reinforcement or a reminder than a new lesson - we'd have to be drunk to do that! It never occurs to us that with some further thinking we can ask more important questions about the story. In real life, is it always so obvious which side of the metaphorical street is lit and which side is dark? Curious students who wish to maximize their learning from the story would consider how to make this determination within the areas of investigation they encounter in the future.

And yet, there is still something else that is missing. Is the only important thing to know which side of the steet is lit? I've had this feeling with me while working on another post, which I'm hoping to finish in the coming days. It's about Jeremy Lin and the way his career intersected with the stereotype of Asian athletes being unathletic. Here's one detail I plan to include in the final version. Speaking about the scouting reports ahead of the 2010 NBA draft, Lin once said - me and John Wall were the fastest people in the draft, but he was 'athletic' and I was 'deceptively athletic.' This isn't another example of an athlete making his own case, which of course has a long history of being slightly favorable to their cause. Daryl Morey, then general manager of the Houston Rockets, said a few years later that the team's pre-draft scouting model suggested Lin was the 15th-best player in the draft. On draft night, the athletic Wall went with the first overall pick and the deceptively athletic Lin, after sixty total selections, went undrafted.

We'll have to recalibrate the anecdote a small bit here - this is like if the drunkard got himself a powerful flashlight, then decided that the suddenly visible keys represented something else, like a soda can, maybe, or someone else's keys. And who could rule out the possibility of a hopeful mirage at the end of a tough night? Regardless, the outcome is the same, the drunkard doesn't find his keys, but I suppose it imparts a different lesson to us. What can I say about those who, despite seeing the "unathletic" player zip past their opponents, fail to relabel the player? What can we learn when a general manager, in possession of a scouting tool which offered to illuminate the darkened street, decided he would rather stand under the streetlights with his twenty-nine peers? We say what we need is a perfectly lit street on a cloudless, sunny day, but we should remember that we'll need something more for those who just refuse to see.

the dumb smart trade

A few days ago I sat down with a plan to write about the Marcus Smart trade. It seemed like a straightforward task. I had spent the prior couple of days digesting the deal, which sent Smart to Memphis in exchange for Kristaps Porzingis, and I had reached my conclusions. Sometimes with TOA I've found that writing is more like transcribing, particularly in cases like this one where the topic had been on my mind, so I thought I'd have something written and posted within an hour. Naturally, I got stuck about as soon as I started writing.

Where to begin? The logical starting point felt like examining the trade from a tactical perspective. I'm admittedly not the city's most dedicated Celtics fan, tuning in pretty much exclusively during the playoffs, but combining my experience as a college basketball player with the forty or so games I've seen over the past couple of years gives me enough of a foundation to offer reasonable insights. It struck me that last season the Celtics seemed to have three main areas for improvement - offensive consistency, passing, and late-game defense against elite perimeter players. This trade doesn't seem to help the team in any of those areas. But the problem with this approach is that such analysis is, for the most part, nonsense. There's no real way to prove your point. The best you can do is put your bets down now, so to speak, on what you think will happen, then see the results during the season. But even then, whatever happens next spring will just be one outcome out of many possibilities, and luck often plays enough of a factor to render most analysis inconclusive. There is also the more basic reality that so many things could still change between now and then (such as another trade). In other words, to write about this trade means I'll have to write knowingly about what I don't know (a definition I'm sure will be familiar to longtime readers). I'll leave this esteemed task to the paid professionals.

I guess this means I should write about what I do know. I know that when you are leading a team there are certain qualities you just can't teach, and that the only way to build them into the team is to have team members set the example for others. Smart's toughness, hustle, and intelligence are hardly unique qualities but the combination of these in one player might be a different story. These are also qualities that represent areas for improvement at a team level. Crucially, losing these qualities in a team is hard to notice until they deteriorate beyond the point of repair, so I worry that the team may now be on a trajectory for these qualities to become significant enough weaknesses such that they offset any technical improvement from making the trade. I also know that teams require a certain unity to overcome obstacles, and although this is an intangible quality I think from observation the Celtics had the right level of togetherness in these past two seasons. Due to their extended playoff runs, the Celtics have basically played an extra half-season of basketball over the past two years (and sacrificed two months of off-season rest to do so). Therefore, one obvious consideration for this year is burnout. Does this trade strengthen the team's ability to stick together throughout the long grind of making another attempt at a championship? Something tells me that Smart's leadership as a proven veteran leader in the team will be missed in the exact moment when it's needed the most.

But the thing I can't get past is how back when I used to play basketball it was a nightmare to play against a guy like Smart. When I lined up against an opponent like Smart, I knew he was going to make the game as difficult for me as possible, and that it would take everything I had to compete against him. I don't know enough about Porzingis to say if he's the same type of opponent but my suspicion is that rival players would prefer to play against Porzingis rather than Smart. What I am talking about here regards NBA players so I understand a comparison to my modest experience is a bit of a stretch. It's possible that being a professional basketball player means these kinds of considerations are left for those like me, the obnoxious combination of Division 3 never-was and internet "writer". And yet, if I'm reminded of anything during the NBA playoffs it's that elite performance is as much mental as it is physical. In the playoffs, athletes who might hit one hundred shots in a row during practice suddenly find themselves unable to string together consecutive baskets. Others who are comfortable taking the initiative during the regular season start passing it to the next guy when the consequence of defeat is elimination. There is a certain mental edge that a player needs to have in order to thrive under pressure and you often don't know if they have it until you put them into these games. If my goal was to win the championship, I would collect as many guys as possible who had the mental edge, then do everything in my power to keep those players. Smart, for all his shortcomings, had the mental edge, so it worries me that the Celtics traded him for a player with no history of the same. 

The title of this post hints at how I feel but it's actually not a great representation - the puns write themselves on TOA and as usual I couldn't help myself. The people who made this decision have forgotten more about basketball than I'll ever know so I don't raise my objections with full conviction. I mostly wanted to write them down just so I have something to look back on a year from now. But the reality is that mistakes are made all the time. In the case of this trade there is the obvious point that Memphis, a championship contender just like the Celtics, identified Smart as a way to help them reach their goal. Around here the talk is about how the trade impacts the Celtics but in terms of evaluating the trade it might help to think about what Memphis got in the deal. What I think is that Memphis prioritized certain qualities at the expense of others, then identified Smart as one player who fit their priorities. On the other hand, the Celtics prioritized different qualities and acquired them in the trade. My hunch is that Memphis acquired the qualities that matter far more in determining a championship team. These are the qualities that can override the technical advantages of basketball skill or team tactics that the Celtics gained from the trade. Time will offer its own verdict on the trade so perhaps the only thing left to do now is to wait and see, but as far as my reaction is concerned I'm less optimistic today than I was two weeks ago regarding the Celtics winning next year's championship.