Failing to distinguish technically sound studies from poor ones

[jazzhound]

"ALL STUDIES ARE CREATED EQUAL": FAILING TO DISTINGUISH TECHNICALLY SOUND STUDIES FROM POOR ONES


Reviews of large bodies of research studies can be misleading if the reviewers implicitly give equal weight to all studies. Most of the research done in the guns-violence field, especially that published in medical journals, is technically primitive, relying on research methods that most social scientists would regard as reflective of the technical standards of the mid-1960’s or earlier. More specifically, the research commonly (1) uses simple univariate or bivariate analysis procedures rather than multivariate procedures that control for variables that may confound the relationship between violence and guns or gun control, (2) ignores the possible two-way relationship between guns and violence (gun levels may increase violence rates, but higher violence rates may also increase gun acquisition for defensive purposes), (3) uses primitive, invalid measures of gun availability (or none at all), and (4) relies on small local samples that are not representative of any larger population.

If the strong studies yielded the same findings as the weak ones, this would not be a problem. Unfortunately, in general the research supporting the ideas that guns cause violence and that gun laws reduce violence is nearly all of the technically primitive variety, while technically competent studies tend to support the null hypotheses that gun levels and gun laws have no significant net effect on violence rates. For example, among studies of the relationship between gun levels and homicide rates, technically inferior studies ignore the effects of violence rates on gun levels, find positive associations, and erroneously interpret them as reflecting the effect of gun levels on violence rates (e.g., Brearley 1932; Newton and Zimring 1969; Seitz 1972; Fisher 1976; Phillips, Votey, and Howell 1976; Brill 1977; Cook 1979; Lester 1988b). The technically better studies that use complex statistical procedures to take account of the possible two-way relationship generally find no evidence of net positive effect of gun levels on violence rates .

Consequently, it can be misleading when reviewers of the research literature engage in “research democracy,” acting as if “all studies are created equal” (Kleck 1985). Whenever scholars summarize evidence on a topic by simply listing studies, without comment of the relative methodological adequacy of each study, they are practicing research democracy. In drawing conclusions, serious scholars are supposed to weight evidence by the soundness of the methods used to generate it. To merely count up studies favoring a particular conclusion would generally lead to an outcome dominated by the technically inferior studies, since these tend to be more numerous. Probably in most fields poor research is more common that good research, but this is especially likely to be true in fields that generate intense emotions and ideologically based conflict, and it is certainly the case with work on guns and violence.

Dr. Gary Kleck – “Targeting Guns – Firearms and Their Control” pp. 32 & 33
(emphasis added, reprinted with permission)