The MITRE Corporation’s defense advisory group JASON has released a report (h/t Secrecy News) on the feasibility of predicting and mitigating extremely rare events, e.g, a terrorist attack on the scale of 9/11. The report is titled simply, “Rare Events,” and it investigates the effectiveness of a wide range of statistical and formal models for predicting the likelihood of such an event. Regular readers may note that the subject of rare events is of particular interest to me, and as such I was intrigued by how a pseudo-government agency that MITRE might address the problem. After a tertiary examination of the 100+ page document there are a few things worth noting.
In their section reviewing predictive models broadl the authors state at the outset the difficulty in predicting human behavior, even going so far as to say that current social science models are ill-equipped to answer such questions. While it is clearly the case that there does not exist a black box (magic ball?) to predict the exact time, location and magnitude of a given terrorist attack, the authors begin by saying these models are useless, but then go on to suggest as a solution exactly the types of social science models used in practice as a solution. For example, the authors state:
There may be one way forward for predictive modeling of rare events. Suppose we assume rare events are drawn from a distribution that includes other events that are sufficiently common that we can observe many of them, enough to evaluate a model. We could, for instance, assume that small observable events and large rare events are sufficiently related in their causes that we are willing to assume large rare events are just the high-magnitude tail of some underlying distribution of events.
The notion of underlying distributions and prior belief are at the foundation of all Bayesian statistical analysis, of which there are several examples examining terrorist events. I find it a bit intellectually dishonest to propose this framework as a novel “way forward” given the growing literature on the subject. In addition, a running theme through the paper is the importance of the power-laws in examining the distribution of terrorist events. This recycled observation has become a bit of a nagging trend in terrorism studies; therefore, I will not go any further but to say in both of these cases a cursory review of the literature would have benefitted this work greatly.
To their credit, in their section on game theory the authors propose an interesting model for terrorist target selection; unfortunately, their conclusions seem to run in contradiction to observed selection behavior. That is, rather than follow an equilibrium path to a target that provides a best response to the counter-terrorism tactics of the target country, the MITRE model predicts a terrorist will chose randomly among a finite set “defended targets.” The game (pg. 67-69) is very straightforward, and I recommend those interested review it; however, for comparison I also suggest the model of target selection proposed here for a somewhat more intuitive model.
Finally, the authors make one final and fatal mistake at the conclusion of the paper, wherein they actually attempt to predict the probability of a high magnitude terrorist event (pg. 89). What is troubling is that for nearly 90 pages the authors point out the difficulty of these predictions, then in a single calculation of expectation using the power-law distribution and a mean of 2,400 terrorist events worldwide peg the probability at ~7%. To the econometrically inclined this is somewhat laughable, given the numerous issues: over/under reporting of terrorist events based on geography (how often are terrorist attacks in China or DPRK reported?), autocorrelation of attack success over time (how are terrorists becoming more effective over time?), country fixed effects (variation in CT policies in different countries), the list could continue. I understand that JASON was most likely directed by DoD to provide an actual number, but I point these issues out only to highlight to difficulty and danger of doing this.
Despite my misgivings, the paper will likely be influential in the U.S. defense community, and as such should be at the top of everyone’s reading list for this week.
Photo: IBM
Automatically Generated Related posts:
I admit this math is way over my head, but terrorist attacks are black swans, unpredicatable by nature. I have to side with Nassim Taleb on this one who wrote an entire book on rare, unpredictable events.
To me it seems similar to trying to bet where the stock market is headed. Hundreds of thousands of financial analysts get paid millions to use their technical models to predict which stocks are going to rise and which will fall, but few if any can do so with much accuracy for a prolonged period of time.
Again, I realize the guys working on this project are probably way smarter than I am, but to me this study is completely useless. The sooner we accept that we cannot predict when the next terrorist attack will happen, the sooner we can use our brain bytes on more productive endeavors. 7% huh, I’m not betting on it.
[Reply]
Ok, so after a little more time to read I may have been a bit quick to call JASON “useless.” Appears that they have some quality material in various sections so I stand corrected.
As you mentioned, they cautioned that predicting the occurrence of terrorist events should be discouraged – “it is simply not possible to validate (evaluate) predictive models of rare events that have not occurred, and unvalidated models cannot be relied upon.”
However, just as you mentioned it makes me chuckle that they follow this up by making the 7% prediction. It’s like saying, “hey guys, there’s no way we can actually predict this, but, what the heck…7%!”
[Reply]
Interesting! Coming from a completely different field, this reminded me of a recent study on stranger homicide by people with schizophrenia in Schizophrenia Bulletin. It’s also so rare that it’s hard to design any type of risk assessment in psychosis to meaningfully distinguish the group that would be at most risk. But the fear of people with schizophrenia is so damaging and stigmatizing nonetheless. http://schizophreniabulletin.oxfordjournals.org/cgi/content/abstract/sbp112v1
[Reply]
Their game theory gets a little more interesting when they get into asymmetric value preferences for the two sides, but I’m not sure I understand their logic in how preferences change when defense on Targets is incorporated. They create a scenario over four targets and two sets of preferences (I recoded targets[1:4] into [W:Z] because too many numbers was too confusing).
Targets Us Terr.
W 1 2
X 2 1
Y 3 4
Z 4 3
Figure 13 (pg76 in pdf)shows defense deployed which “defend[s] targets so as to order the terrorists preferences in the reverse of ours.” However, Figure 13 is showing us that when Us uses defense on a target, it decreases the targets value. When Us’ payoff function is setup that way you get weird things happening like changing the Terrorist’s preferences so that Target X, which held the second most value to Us originally, is now the most preferred Terrorist target.
The result of payoff fcts that looks like:
Us: target.value = x1+x2+…+xn-defense | x[1:n]being loss of life, infa damage,etc
Terr.: target.value = x1+x2+…+xn-defense
is this (according to Figure 13):
Tar Us Us.df Terr Terr.df
W 1-> 1 2-> 4
X 2-> 4 1-> 1
Y 3-> 2 4-> 3
Z 4-> 3 3-> 2
.df being after defense is deployed
If deploying defensive resources reduces a target’s value to Us then it also makes things complicated when moving to the next iteration (Us would have to switch up all their defense based on their new Target prefs)
I would think you would want to deploy defense that would change terrorist values/preference order like this:
http://docs.google.com/View?id=dfbjkhrq_450g4bmpbc2
Just curious if anyone else saw that as confusing and strange modeling. It’s probably too nitpicky(or I’m overlooking something in their model), but it completely changes the practical advise as to how to deploy your defense.
Thoughts? Reactions?
Too long of post? (yes…)
[Reply]
Drew Conway Reply:
November 4th, 2009 at 8:31 pm
This is a very interesting point, and I especially like the plot you provide in the link.
You make a good point about the consequences of their model vis-a-vis how to interpret from a policy level. For a moment, however, abstract further and consider what the interpretation of their assumption about changing value means. In their model, as you rightly note, the value changes given some level of defense, but does this accurately reflect how we believe targeting happens?
It seems more intuitive to me to model this as a cost rather than a valuing. That is, ceteris paribus a terrorist would always rather attack a high-value target, but cannot because of the risk or cost associated with that decision. Valuation never changes, but the budget constrain does.
[Reply]
=/
Sorry those pref. tables got all smashed. My tabbing didn’t transfer. Check out the google doc link for a proper example if trying to decipher the ones here gets too confusing.
[Reply]
Rare events such as terrorist attacks belong to the class of entities that Karl Popper has characterized as “clouds”, rather than “clocks”. Clocks can be successfully addressed by probabilistic modelling because the sample space of their states is known a priori, prescribed and well-behaved. Clocks may be complicated, but operate by coherent and decidable principles. Such is not the case in the “cloud” domain. Anyone interested in understanding the fundamental reasons for its unpredictability would do well to ponder over a recent and remarkable essay by Stuart Kauffman, “Towards a Post Reductionist Science: The Open Universe”, http://arxiv.org/pdf/0907.2492v1. Kauffman is concerned with quite a different context – the dynamics of Darwinian evolution in Nature – but his observations are generic and relevant in the fullest. Terrorist attacks – or financial crisis of recent times – emerge from a dense web of (nonlinear) interactions. These evolve over some general pre-enabling conditions for which a statistical framework may be justified but their specifics are triggered by (often miniscule) “causes” that operates through an opportunistic combination of circumstances. The state-space of such events spans a universe of unknown unknowns and no sound probability statement can be made about it. It is not just that statistics cannot be reliably estimated because of rarity of occurrence; it is more the fact that the realm of the very nature of the emerging dynamics is unknow a priori, hence statistics are, in a sense, meaningless. The RAND document hints at this conundrum in its Appendix A where Taleb’s Black Swan is discussed but Kauffman’s paper goes much deeper into the essentials. This has unescapable consequences on predictability.
[Reply]
Interesting piece, saw it just now.
I think their game theory chapter was heavily influenced by Robert Powell models, thought they do not cite him.
I agree with DMT that the asymmetric values of targets are not very clear. I did not read the entire document, but from what I read it’s not clear why there would be asymmetric values. Since the terrorists are trying to impose the highest expected loss on the US, why would they ever prefer to attack something that the US does not really value?
Also, they assume that all sites are equally defendable, but in reality this is definitely not the case. But still, very interesting, and thanks for posting it!
[Reply]