I have tried very hard to keep my website/blog/twitter focused solely on my professional interests – using open source data to forecast socially driven outcomes. Now, I’m going to bend the rules a bit and bring in my love of combat sports in order to apply lessons from mixed martial arts (MMA, or “cage fighting” or “ultimate fighting” to the layperson) to forecasting models.
Last Saturday, Fox aired “UFC on Fox 5”, which was undoubtedly the greatest collection of MMA talent ever aired on a free TV. The main event – Nate Diaz vs. Benson Henderson – garnered 5.7 million views. Not bad, but consider this: Kimbo Slice has fought 3 times on free TV, each time surpassing 6 million views (6.1, 6.45, and 7.1 million to be exact). Until a string of losses that exposed him as a D level fighter, Kimbo was among the biggest draws in all of combat sports. But why? I believe that it derives from our obsession with the mythical. In the context of fighting, we seem to be drawn to someone who, for untraditional reasons, seems to be invincible. In terms of MMA, the fighters who achieve mythical status share two things in common: 1) untraditional or secret training methods and 2) a string of dominant victories over easy opponents. As I argue a bit later, the same is true of predictive models
In the early stages of MMA, the Gracie family dominated. For many years, their brand of jiu-jitsu (submission fighting on the ground) allowed physically inferior men like Royce to appear invincible in actual no holds barred fighting. Their highly technical method of ground fighting seemed like magic to the untrained eye, and since virtually no one in the United States was familiar with jiu-jitsu at the time, it just seemed like magic. Eventually, the Gracie’s began to fight better competition; Kazushi Sakuraba tarnished the Gracie legacy by beating Royce, Renzo, Royler, and Ryan, and Matt Hughes drove the final nail in the coffin when he mauled Royce.
The Russian fighter Fedor Emelianenko regained and arguably surpassed a Gracie-level of mystique. Like the Gracie’s, the specific details of his training regimen were not know outside of Russia, and the only glimpses of his training U.S. audiences saw were Youtube clips of minimalist workouts on Russian playgrounds. He was (and likely still is) a secretive man who seemingly cared more about his religious devotion than fighting (his Orthodox priest often accompanied him to fights). Also, like the Gracies, Emelianenko gained mythical status by beating a mix of legitimate, semi-legitimate, and absurdly illegitimate competition.
Kimbo Slice achieved event greater U.S. fame than the Gracie’s or Emelianenko through a series of backyard street fights in Miami that circulated on YouTube. He destroyed lesser competition (i.e. people with no idea how to fight) in dramatic fashion (i.e. he once popped out someone’s eyeball. Seriously.). No one really knew his fighting background or training, but people seemed to assume that he was so innately tough that it didn’t much matter. By the time he fought James Thomson live on CBS in 2008, he had reached mythical status. Of course, this came crashing down when he was knocked out in 14 seconds by unranked journeyman Seth Petruzelli.
So what does this all have to do with forecasting models? As with MMA fighters, we seem to be intuitively drawn to black box, secret models. Additionally, like in the fight game, models can artificially inflate their reputation by “beating up” on easy problems.
Like with out fighters, we seem to be intuitively drawn towards secret, mythical approaches to prediction. Just consider the absurdity of the history of human forecasting: Oracles interpreting hallucinogenic dreams, astronomers staring at the heavens, monks pouring over ancient texts, etc.. My sense is that many still seem to think that a secret formula exists that can predict the world. Many practitioners, like Bruce Bueno de Mesquita or the CIA, certainly help to propagate this myth (see the History Channel special that pitted Nostradamus vs. BDM). I suspect if you surveyed 1,000 random people about whether academics using open source data and open-source statistical packages could build better forecasts of civil unrest than the CIA using only classified data and proprietary algorithms, most would pick the CIA. I’d bet anything on the open source team. In predictive models as in fighting, there is no secret approach to success.
Additionally, similar to the way in which fighters achieve mythical status by often padding their records fighting easy competition, predictive models have a few tricks to “pad” their records as well. First, and a favorite of the game theorists, is to use a model to make non-falsifiable predictions. Thus, no matter what happens, the game theorists can make a strong case that his or her model correctly predicted it after the fact. Second, the primary tool of empirically driven prediction models is to inflate results by predicting that tomorrow will look a lot like today. For outcomes like whether or not a country will be at civil war tomorrow, this approach is correct 90+% of the time.
In reality, neither mythical fighters nor mythical predictive models exist. Everyone knows exactly how the best fighters — Jon Jones, Anderson Silva, Georges St. Pierre — train. These fighters get to that level by having slightly more genetic gifts, work slightly harder, and having slightly better coaches, all of which contribute to a minor advantage over their opponents come fight time. This is almost identical to the predictive model world. In actual, objective tests of predictive accuracy (such as Kaggle competitions), the wining models tend to only narrowly beat the competition and the methodological approaches tend to be highly similar. The reality of predictive models is just like that of fighting: there are no mythical approaches. The winning teams tend to put in slightly more time on slightly more powerful machines with teams comprised of slightly more experienced modelers.
With fighters as with predictive models, if it seems to good to be true, it almost certainly is.