with No Comments

Post No.: 0818fingerprint


Furrywisepuppy says:


An example of the ‘prosecutor’s fallacy’ is when – given a 1-in-100 chance that the defendant’s blood sample would match the sample found at the scene of a crime by mere coincidence alone – a prosecutor claims that there’s therefore only a 1% chance that the defendant is innocent.


But this is a fallacious interpretation of the statistic because in a community of millions – tens of thousands of innocent people will have blood samples that’d match the sample found at the scene by mere coincidence alone!


Probably a simpler illustration of this fallacy is when a hypochondriac reckons he/she has early stage kidney disease, but when the doctor queries how he/she knows because there’d be no pain or outward symptoms – he/she says, “Those are my exact symptoms!” What the hypochondriac has failed to consider is how many people will experience those exact symptoms but won’t have early stage kidney disease. In this case, the overwhelming majority of people in the population with those same ‘no pain or outward symptoms’ won’t have early stage kidney disease, thus the probability that he/she has it is incredibly low too… unless there’s other evidence to increase that probability – such as what can be revealed from a relevant blood or urine test.


The prosecutor has confused the probability of the sample matching given the defendant is guilty, with the probability that the defendant is guilty given the sample matches. We’re asking ‘if the defendant were guilty, is this the evidence we would see?’ when we need to more crucially ask ‘given the evidence we see, is the defendant guilty?’


The evidence still has value in that it drastically narrows down the possible number of suspects. But it doesn’t narrow it down to the point where we can say there’s a 99% chance that the defendant is the guilty party. We cannot actually tell what the probability is that this particular suspect is/isn’t the source based on the blood sample match alone. One has to consider the strength of all the other evidence and whether this particular defendant is more/less likely to be guilty than anybody else who could also coincidentally match the blood sample. So it adds confidence to the verdict but it needs to be viewed in light of all other evidence in a factual manner – no isolated piece of evidence alone is enough. Not even DNA samples.


The prosecutor’s fallacy is a fundamental misunderstanding of probabilities and misapplying them to individuals when they only apply to the population as a whole. The rarity of something happening alone isn’t enough to tell anyone whether a particular person did it.


A variation of the prosecutor’s fallacy is the ‘defence attorney’s fallacy’. Using the same example with the 1-in-100 chance that the defendant’s blood sample would match the one found at the scene by mere coincidence alone, the defence would commit this error if, in a community of 1,000 people, they reason that 10 matches are expected, thus the defendant is no more likely than any of the other 9 matches to be guilty, thus the evidence suggests a 90% chance that the defendant is innocent. This would be fallacious if there’s other evidence suggesting that this particular defendant is more likely to be the guilty party than the other 9 people. And this evidence is still highly relevant because it drastically narrows the group of individuals who are or could’ve been suspects, while not excluding the defendant in question.


A similar kind of fallacy emerges when we’re only interested in reading the biographies and hearing the views of those who worked hard, believed in themselves, had the talent and drive to succeed and succeeded – but we fail to also be interested in hearing all the (possibly many more) views of those who worked hard, believed in themselves, had the talent and drive to succeed yet didn’t succeed despite all that because of other factors like luck wasn’t on their side.


A ‘black male who’s wearing a hoodie’ may ‘fit the description’ of a suspect at large – but so would hundreds of others in the area who are innocent. Thus ‘an individual who sacrificed going to parties in order to go training’ may ‘fit the description’ of a gold medal winner – but it’d also fit the description of hundreds of others who did the same thing yet didn’t win (and it’s not simply a case of training for longer if you want to win because over-training is deleterious for performance).


There are high odds that a successful person is also hardworking, talented and driven – but there are also high odds that many unsuccessful people are too. It’s the bias of confirming our hypothesis whilst neglecting all evidence that would disconfirm it. We’re focusing on a few trees whilst ignoring the wider forest of information that would give us the fuller, truer picture. To also actively seek evidence that disconfirms our hypotheses would be a grand demonstration of critical thinking.


Whenever a large amount of data is available, there’s going to probably be some clustering of a subset of data at least somewhere. But that may be due to pure chance or a confounding reason – thus to only form a hypothesis from this data after observing this cluster would be to commit a potential fallacy.


This is the ‘Texas sharpshooter fallacy’ where we, metaphorically, paint the target around the bullet hole after it has been shot! This makes our predictions seem more accurate than they really were.


For example, when comparing a DNA sample with a suspect’s DNA sample, one may use the suspect’s profile to help interpret the evidence profile in a way that causes them to seem closer-fitting with each other than they really are. One may then find a match and say one has ‘hit the bullseye’. And one may compute from that the probability of the DNA match occurring by chance, but this would be a vast underestimate because it doesn’t take into account that one was able to move the bullseye around according to one’s will – the target was effectively painted on after one knew what the defendant’s DNA profile was, and the sample was interpreted in such a way to make a match more likely.


This effect can also happen if one has found a fingerprint and scours a database to find how closely it matches a particular person’s fingerprint without any other supporting reason to suspect that particular individual. It relates to confirmation bias again. With an ambiguous or imperfect source, one doesn’t interpret what’s signal and what’s noise until one sees parts that match the defendant’s DNA profile or fingerprint and calls those parts the signal, and the parts that don’t match the noise. It then enters a circular logic i.e. these parts match because these parts match.


Somewhat similarly in post-hoc analyses, the data is used twice – first to inspire a hypothesis, then to test that hypothesis. Although tempting – such double-use of the data invalidates statistical inference.


…Some general notes about forensic science – even with DNA profiling (with the risk of old, degraded, contaminated or mixed samples) and fingerprint matching (with the risk of partial or blurry fingerprints, and different fingerprints can look quite similar by eye) – there can be a fair degree of ambiguity and therefore subjectivity when judging whether something is ‘sufficiently’ a match or regarding what the results ultimately mean. Whenever ambiguity and human judgements are involved, there’s going to be an element of subjectivity. We must also realise that it’s a human being who decides what organic matter or prints to analyse in the first place, and different experts can reach different conclusions too. Post No.: 0720 dissected expert testimonies.


Two individually half-sure or even faulty pieces of evidence can erroneously allay the uncertainties of each other, and fallaciously become two sure pieces when put together.


In the medical profession, there’s a slight pressure and ‘response bias’ towards committing false positives if there’s a greater cost for committing a false negative. For instance, taking a healthy appendix out is bad, but not as bad as not operating on an inflamed appendix. Nevertheless, all false positives and false negatives generate a wasteful and potentially harmful cost. And in the context of criminal investigations, we could incriminate the wrong suspect. Woof.


Blinding the forensic scientist and minimising the amount of contextual or extraneous information given is a possible solution so that they don’t know whether some sample is supposed to match another or not, and so that they don’t know how the other pieces of evidence stack up (don’t even let them know whether there’s any other evidence at all), and so that they don’t hear any suspicions or theories at all, and therefore what the answer is ‘hoped’ to be.


The same technique can be used when marking students’ papers – mark them without knowing who wrote them. A major problem we’re trying to avoid is that we cannot subsequently un-know what we’re told (like someone is the prime suspect) and we therefore cannot help but then be influenced by that information. Therefore it requires rigorous procedures to ensure that extraneous information isn’t passed on that may influence a fluffy forensic scientist’s decision. People may claim that they won’t be biased by extraneous information but they’re unaware that they inevitably will be to some degree on an unconscious level. Sometimes less information therefore leads to better, fairer decisions.


A DNA profile having only a ‘one-in-a-billion’ chance of matching another tells us how rare that profile is, but it doesn’t tell us about the probability that the two profiles could match by mistakes or by errors in the process – such as by cross-contamination in the laboratory. And how did the ‘one-in-a-billion’ statistic come about? Did they actually test over a billion people? Therefore some assumptions in making this inference must’ve been made. These assumptions can be fair if the sample sizes were large enough and randomised – but one therefore needs to check if the sample sizes were indeed large enough and suitably randomised, and that the science conducted passed the gold standard. We cannot assume that everything under the banner of ‘science’ was well-conducted.


DNA found on an item also doesn’t tell us whether it got there directly or indirectly, or when it got there. A strand of fur found on a chair in a house could’ve gotten there when somebody picked it up on their jacket after sitting on a bus then sitting on that chair later.


A useful thing is that even identical twins don’t have identical fingerprints (or vein patterns on the backs of their hands). But fingerprints also aren’t 100% clear either because they can be fragmented, mixed with other fingerprints, and again finding a fingerprint won’t tell us how it got there and when. Forensic examiners might be swayed to apply confirmation bias as a result of the confidence of a police officer’s belief of whom the perpetrator is; albeit procedures around the world should’ve hopefully updated by now so that examiners aren’t given information that might prejudice an examination.


In short, DNA and fingerprint analysis conclusions aren’t always a matter of clear black-or-white fact, hence why different analysts can come to different conclusions regarding the same evidence. Different analysts can come up with different figures for their level of confidence that a particular match is true. It’s science, but even scientific predictions are never 100% certain.


In criminal forensic science – as with many other fields – the physical evidence is what it is, but it still requires human interpretation. For example there’s blood on the scene and it matches the suspect’s blood but, on its own, its incredibly ambiguous how it could’ve gotten there. And where there’s human interpretation involved, there’s subjectivity, as well as potential basic human error.


Woof. You could use the Twitter comment button below to tell us how much you think the reliability of forensic evidence needs to be better understood by all?


Comment on this post by replying to this tweet:


Share this post