Statisticians not wanted

David Traver · Posted: Wed Sep 20, 2006 09:07 pm Post subject: Statisticians not wanted

It appears Social Security's administrative courts are not the only ones who like to use bogus statistical analyses.

Craig Jarvis · Posted: Thu Sep 21, 2006 07:05 pm Post subject:

After reading Devlin and the Johnson case, I find it hard to take Devlin's concerns seriously. Devlin suggests that there is a statistical controversy as to how to calculate the probability of "cold hit" DNA searches, i.e. searches in which the police don't have a suspect, and that because the court in Johnson let one method in as evidence, it was somehow deciding science instead of leaving that to statisticians. I don't buy it.

In the Johnson case the defendant tried to exclude testimony from an expert concerning the probability of a match using the method that she employed. That method (known as random match probability (RMP)) produced a probability of a correct matching of approximately 1/4x10^15. Devlin admits that this is a method accepted by many statisticians. Johnson tried to exclude this evidence because there is a controversy as to what the true statistical method for calculating probability should be. The case doesn't say whether Johnson tried to admit evidence based on the other statistical method. Apparently he did not. The court says nothing about whether they would have admitted this other method had Johnson tried to admit it as evidence.

Devlin suggests in his piece that the court and not statisticians are deciding the correct method, but I don't see that the court has done that at all. It is simply permitting one method, which has been accepted by statisticians, to be used.

I also think that I know why Johnson never tried to admit evidence based on the other method. According to Delvin the better formula to use in cold hit cases is N x RMP where N is the size of the database used. When I applied this formula to the facts in Johnson I get the number 1/2x10^10, still an astronomically high number, many times the population of the earth. (and these are just statistic for hispanics)

As a postcript, I am no statistician, but I have a hard time understanding why the size of the database would have any bearing on the probability of a random match. For instance, suppose that you had a database that contained every person on earth. If you have one match in that database, then there would be a 100% probability that that is the right match. However, N x RMP suggests that as N gets larger the probability of a match being accurate deminishes. Intuitively, this does not seem right.

Craig Jarvis · Posted: Mon Sep 25, 2006 08:02 pm Post subject:

Keith Devlin's Response:

PeterNordberg · Guest

I think the "Devlin's Angle" piece is marred by misunderstandings and/or distortions of the legalities that distract seriously from the substantive message of the piece. I also think the substantive message is suspect, though I don't understand the technical aspects well enough to be certain it's wrong.

First, the legal misunderstandings and/or distortions. The California Supreme Court did nothing except exercise its discretion to refuse to entertain an appeal. That decision is without precedential force, and it doesn't constitute an adoption of the lower appellate court's opinion. Any gripe is therefore really with the Court of Appeal -- not with the Supreme Court, as Devlin suggests. In addition, no court excluded any statistician's testimony in this case, nor ruled that contrary testimony from a different statistician would be inadmissible. Moreover, the courts apparently entertained an amicus brief from statisticians. That does not constitute a refusal to listen to statisticians' views, as Devlin contends. The worst of which the courts can be accused is hearing and then rejecting some some statisticians' views, including Devlin's -- with the result that the jury did hear from one statistician, rather than hearing from none. All the professional huffiness on this score, therefore, seems as unwarranted as it does graceless. It is, in addition, the sort of argument-from-authority that any rational judicial system rightly discounts.

Moreover, one legal view espoused by Devlin, on which his arguments against admissibility of the RMP test seem largely to be based, happens to be incorrect -- i.e., his belief that under California's Kelly / Frye test, the existence of debate within some discipline is sufficient to preclude the admissibility of the evidence. As a matter of doctrine, the "general acceptance" test applies, as the Court of Appeal's opinion explains, only to novel scientific methods or techniques. Moreover, even if that test did apply to all expert evidence, whether novel or not, "general acceptance," though not a sharply defined term, has never been understood to demand unanimity. As a matter of policy, a unanimity standard, or anything resembling it, would effectively bar all expert evidence from the courtroom. (On the policy front, it might also be felt that research on lay understanding of technical concepts, such as Devlin describes, should focus on how to communicate those concepts in a way that will enable a jury or other lay audience to follow the concepts and spot their misuse, as opposed to locating occasions to censor what the lay audience hears -- the latter approach being one that suggests a distrust of rational discourse, to my ears.)

As for the substantive question: Although people in the statistical community seem very convinced that the DMP statistic is the more "correct" number to present, and that presentation of the RMP statistic is less correct or altogether "incorrect," the pertinence of various statistics to the relevant inquiry is not a question exclusively within the province of statisticians. Some lawyers and judges may have thought very hard about this problem too, and they certainly have intellectual and institutional standing to do so.

In that vein, I will admit that I may have failed to follow Devlin's explanation, because the intuition persists that the method by which the authorities first identify their suspect simply are not germane to the question whether the RMP statistic should be admissible at trial. As a lawyer, I approach the latter problem by first noting the legal definition of relevance: evidence is relevant if it makes the existence of a fact more likely or less likely than it would be in the absence of the evidence. That is not how the word "relevance" is usually used in ordinary discourse, nor perhaps in academic dialogue, but it is the definition that the law has adopted. Under it, the RMP statistic seems plainly relevant to the question of guilt or innocence. This is especially so if the statistic is evaluated by the jury as one piece of evidence in a larger set of evidentiary facts, including nonstatistical facts, whose collective weight must be assessed as a matter of juror judgment by essentially nonstatistical means. In the California case, those included matching tattoos, the defendant's having a daughter of the same age as the one the perp mentioned to the victim, and a truck with a missing stereo. To me, for what it's worth, those facts seem analogous to the television reporter in Devlin's lottery example. The question here, for me, isn't about the odds that somebody would win the lottery; it's about the odds that this guy with the tattoos would.

Indeed, there were enough corroborative facts in the California case that a world may easily be imagined in which the suspect was first identified not through a DNA database, but through (say) his tattoos. Largely for that reason, I remain unable to follow why the admissibility of an RMP number should depend on how the suspect was first hauled in. To suppose that it should so depend seems to me an example of the genetic fallacy. The procedure followed after the California suspect's identification as a suspect was the same one that would have been followed had the suspect been fingered by Ouija board or a psychic. A blood sample was drawn, a DNA comparison with a sample from the crime scene was made, and a match was found. Why should the RMP number be admissible in the psychic situation but not when the suspect came to the authorities' attention via database? What if, unbeknownst to the authorities, the psychic cheated and consulted the database? The analytical problem confronting the jury in both cases seems the same.

(Or should the RMP number never be used in court? Should we always present the DMP number? To say the latter would almost be to say that we were under some sort of epistemological obligation to have a DNA database in the first place -- whereas, I take it, we are not.)

To me, the real mystery is why the DMP number should ever be thought admissible at all. (Here I leave to one side Donnelly's point that the database run has actually added information by excluding some potential suspects.) For one thing, the Court of Appeal seems correct that the question confronting the jury isn't the probability that there would be some match in the database. That might be the pertinent question for a statistician to whom we had delegated the entire task of determining guilt or innocence by exclusively statistical means and measures, based entirely on the intelligence that we had found the suspect through a database match. But that is not how our system has divided the intellectual labor, nor how it structures the substantive inquiry. In our system, a suspect is first identified, on grounds that may or may not afford a good epistemic or evidentiary basis to believe the suspect guilty. Material of evidentiary quality is then separately presented to an independent trier of fact (usually including evidence above and beyond a DNA match), and the jury weighs that evidence as a whole.

There are, meanwhile, independent reasons to be concerned about presenting the database number. Not all of our evidentiary rules spring from a truth-seeking animus. Some are positively truth-repelling. One such rule concerns the inadmissibility of prior arrests or convictions to prove guilt. To be sure, there are exceptions to that rule. In federal court and some state jurisdictions, the rule doesn't apply in ... well, the site's spam filter won't let me use the phrase that identifies the relevant class of cases, but perhaps the reader will be able to deduce it. Details involving prior offenses may also be admissible, sometimes, to show a common modus operandi, or for similar reasons. The general rule of exclusion remains just that, however. We follow this exclusionary rule not because a previous conviction has no logical bearing on the likelihood that the defendant committed this new crime. It is for other reasons, perhaps involving the notion that we should convict people of crimes one by one, because some people can overcome their criminal past and lead noncriminal leaves. Or at least we want to give them that chance.

The rule against offering prior convictions seems starkly implicated, if the jury hears about DNA match probabilities based on criminal databases. I'm not a criminal defense lawyer, but if I were, I wouldn't think my clients would want the jury to hear that they had turned up in such a database. I've heard it suggested that we could simply keep the jury in the dark about who populates the database. But it would take only a few CSI episodes before jurors had figured it out. I gather that defense lawyers are nevertheless keen to tout the DMP statistics. But at a time when those DMP numbers still leave the odds against a chance match at stratospheric levels, that keenness baffles.