Yesterday, Andrew Exum—a person who admits his own ignorance of the current state-of-the-art in political science literature—presented his “manifesto” on the quantitative analysis of conflict. While Exum’s bonafides in counterinsurgency and military strategy go without saying, given that he knows almost nothing about quantitative analysis I found this manifest rather disingenuous. Furthermore, since he has referred to me as a quantitative “hired assassin,” I felt an additional duty to respond.
To be fair, Exum has recently praised the work of contemporary quantitative analysis of conflict by scholars such as Lyall, Berman and Shapiro, all three of which whom are well deserving of praise. As such, it is peculiar that Exum would feel the need to present this manifesto after having to first be told by others about this work (due presumedly to his own admitted ignorance); and second, actually liking it. To be sure, such an endeavor is useful, as it is clear that both subtle formal models and sophisticated statistical analyses can be manipulated and misinterpreted to present dangerous falsehoods; however, Exum’s attempt to undervalue the contributions of this work with respect to policy is equally dangerous.
Exum’s manifesto has six points, and I would like to respond to each one.
War is a human endeavor. I recognize that it is a phenomenon that does not conform to neat mathematical equations.
This statement is much more a platitude than meaningful manifesto item. There are several quotations that any quantitatively minded person could more easily insert here that summarize this sentiment more effectively. Of those, my favorite are “Essentially, all models are wrong, but some are useful,” by the venerable George E.P. Box, or “No one but the modeler believes in his results; everyone except the observer believes in his data.” Any modeler that is to be taken seriously will be guided by these words. The danger is that those who are willfully ignorant of the tools and techniques employed in this research cannot differentiate between those that are humbled by the complexity of human behavior, and those that are not. As such, policy minded people would do well to attempt a tertiary understanding of the methods, rather to wholesale discredit them.
I will use quantitative analysis in conjunction with theory and qualitative analysis to describe what I see as phenomena in war and peace. I will be honest about the limits of both my theory and my analysis.
This is critical, and I applaud Exum for highlighting this point. In fact, it has been my own experience that many people who make their careers in this field have a debilitating arrogance of the power of quantitive models, which leads to research that has little or no bearing on real world events. A better way to address this issue is to require that any quantitative research be well founded in the observation of some meaningful qualitative phenomenon. This would prevent the oft observed research conundrum of “theory seeking data”, or “data seeking a theory.”
In war and peace, the variables are infinite, and not everything can be measured or assigned a numerical value.
Indeed, but many things can be measured, especially with respect to conflict. David Kilcullen, the master to Exum’s COIN apprenticeship, has discussed at length various considerations for metrics in conflict. As with any analysis, the assumptions about each variable must be made clear at the start and parsimony is paramount. Rather than collapse in the presence of the infinite scale of human complexity, considerable thought must be given to what variables are most important, and which models provide the best approximation of conflict dynamics. Of course, such work would only be valuable if experienced practitioners—such as Exum—were willing to see the value in this work and offer their expertise to the problems.
I will not use numbers to signify what are fundamentally qualitative assessments without acknowledging to my reader that I have done so in order to satisfy a departmental requirement, gain tenure, or get published in the APSR. Or because I have been in graduate school for so long that I have forgotten how to effectively write in prose.
This, along with the remaining three points, are all quite cynical, but given Exum’s position in policy making these are clearly sentiments that need to be addressed. As Exum himself has never been published in an academic journal, his knowledge of publication and or tenure requirements is anecdotal at best, or dubious. While the value of quality case study work, which I am assuming Exum is referring to by “fundamentally qualitative assessments,” are valuable, their fundamental flaw is in external validity. Large N statistical studies allow researcher to observe macroscopic behavior, or at least begin to understand their correlates.
I recognize there are no mathematical equations in Vom Kriege and that it is nonetheless unlikely that my legacy will transcend that of Clausewitz.
To be sure, any academic doing research in this field would benefit greatly from reading these classic contributions to the literature on military strategy; however, to say that no work in this field will ever rise to the level of these works is both subjective and likely the result of a limited knowledge of the literature. Works such as The Evolution of Cooperation, An Economic Theory of Democracy, and the work of Fearon, Powell and Bueno de Mesquita (to name a very few) will all have a lasting affect on how we study both war and peace.
I recognize that very few squad leaders in the 10th Mountain Division have ever taken a course in statistics yet probably know more about the conduct and realities of war than I do.
Though I will expose my own ignorance to the training of military leaders, I fail to understand why squad leaders would not be given a basic class in statistics, or furthermore why this lack of training is seen as efficient. Withholding all of one’s pretense for the value of quantitative research in conflict, would these squad leaders not benefit from a basic understanding of probability theory, the law of large numbers, or the fundamentals of regression analysis? Perhaps Exum is pointing this out in an attempt to motivate policy makers to provide this type of training, but from the text it seems he presents this ignorance as a badge of honor when surly it should be viewed as a critical deficiency.
Finally, I thank Exum for engaging in this debate, as I think at its core the notion of a manifesto for quantitative analysis of conflict is a good idea. As with any analytical endeavor there will be those that will seek to use these tools to promote an agenda (be it political, their own career, or otherwise), and Exum is right to highlight the danger is such motivation. That said, it is also extremely important to not loose sight of the tremendous potential this field has in contributing to our understanding of conflict and war making.
Photo: Geographic Travels
Automatically Generated Related posts:
Imagine this manifesto recast to another profession. Say transport analysis. Would these points say something like
1. Traffic is caused by humans and so won’t fit mathematical equations. True to a point but more accurately false.
6. Bus drivers know more about traffic than you do. They certainly might and if I was designing a transport system I would try get as much information form them as possible. Should they be the ones who design the system? Probably not.
On the other points
2. seems a useful promise
3. Is true of any mathematical analysis, and is worth remembering.
4. I should not use numbers because they are scary. People only use numbers to impress other propeller heads.
5. You are an intellectual midget standing on the shoulders of giants. This is almost definitely true for most people.
While I am commenting. What do you think of the work of Philip Tetlock on the prediction abilities of experts? He has a long now podcast on it
http://www.longnow.org/seminars/02007/jan/26/why-foxes-are-better-forecasters-than-hedgehogs/
And what do you think of Bruce Bueno de Mesquita’s work? I read the popsci predictioneer book and the use of Bayesian nets to predict political outcomes seems interesting.
[Reply]
Drew Conway Reply:
February 27th, 2010 at 7:17 pm
I am highly biased on Bruce’s work since I work under him (at least to some degree) at NYU. If you want to read his core contribution read the Logic of Political Survival, to understand the Selectorate Theory.
[Reply]
Ummm, Drew? Ex called Josh Foust et al at Registan a “gang of hired assassins.” Wasn’t directed at you at all. Setting that aside…
I think you raise some pretty valid points – particularly your note re. basic understanding of academic conventions and protocols regarding research and publication.
My preference is for qualitative methods, and that’s what I use, but they can be just as useless or dangerous as quantitative approaches. The issue isn’t either/or, but how each is practiced – the rigor with which they’re applied, the meticulousness with which adherents investigate their subjects, etc.
Anyway, my two cents…
[Reply]
Drew Conway Reply:
February 27th, 2010 at 7:15 pm
Mike, tisk tisk, who was one of Josh’s co-author’s on that piece at Registan?
[Reply]
Mike Innes Reply:
February 27th, 2010 at 7:26 pm
My mistake: I read AM’s comment, but not the original. Tisk tisk right back at you, though. Josh is infamous for not playing well with others, and equally infamous for taking swipes at Ex; my assumption had been that the comment was something to do with the usual enmity.
Still, my shame rests entirely on the fact that I read Ex’s comment in full, but not yours. A travesty of priorities, to be sure.
[Reply]
So after reading his piece, Andrew Exum comes off as a rather snarky social scientist. Frankly, I could care less about his status as a leading thinker on counterinsurgency. When you willfully flaunt your ignorance of a methodology–all the while denigrating it–you don’t exactly pain a flattering picture of yourself.
Methods are like tools–not every tool is appropriate for every job, but lacking fluency in all tools severely limits one’s ability to tackle different jobs as they come along (or to tackle certain jobs well). Carpenters don’t use lathes for every project, nor do they use a sawzall. But they need how to use both and understand when it is appropriate to use either or both. It is the same for social scientists. Understanding which methods are most appropriate for separating relevant information from noisy observations is key. If you willfully ignore an entire set of these methods you risk reaching erroneous conclusions in your own work, and will fail in your attempt to legitimately criticize the work of others.
[Reply]
Drew Conway Reply:
March 1st, 2010 at 12:56 pm
Well said!
[Reply]
Hi Drew–I saw the works you cite for Exum before I wrote the post. I omitted to mention them because with the (partial) exception of BdM, although all are excellent works none offers a primary contribution based on statistical analysis of data generated by real world processes. So, they don’t satisfy the minimum requirement of my request: statistical research on the real world. Sorry to cross post this comment but I wanted to be sure you saw it.
Thanks,
Thomas
[Reply]
[...] Drew Conway at Zero Intelligence Agents: Yesterday, Andrew Exum—a person who admits his own ignorance of the current state-of-the-art in political science literature—presented his “manifesto” on the quantitative analysis of conflict. While Exum’s bonafides in counterinsurgency and military strategy go without saying, given that he knows almost nothing about quantitative analysis I found this manifest rather disingenuous. Furthermore, since he has referred to me as a quantitative “hired assassin,” I felt an additional duty to respond. [...]