comparing bluejeans seams is relatively useless
February 26, 2020 1:19 PM   Subscribe

A Key FBI Photo Analysis Method Has Serious Flaws, Study Says After ProPublica’s reporting last year, scientists at UC Berkeley tested one of the FBI Lab’s photo analysis techniques, identifying bluejeans by the pattern on their seams, and found flaws that challenge the method’s reliability. [ProPublica]
posted by readinghippo (41 comments total) 14 users marked this as a favorite
 
It fails at the first assumption of these sorts of techniques: that there is sufficient random variation there in the first place.
posted by sfred at 1:33 PM on February 26, 2020 [1 favorite]


Mod note: One deleted; let's aim to engage with the actual linked study rather than jumping straight to "here's a different bad thing the FBI did".
posted by LobsterMitten (staff) at 1:34 PM on February 26, 2020 [2 favorites]


As forensic tests go, jeanetic fingerprinting still has some work to go.
posted by They sucked his brains out! at 1:45 PM on February 26, 2020 [24 favorites]


Interesting! I've noticed over these last several decades that every pair of jeans I wear develops 2 diagonal wear-lines on my left leg: starting from my mid inner-thigh pointing to the middle of my kneecap. I sometimes develop one crease like this on my right leg, but it's never as apparent. I bet I could identify my own jeans out of a pile of others just from that one area.

My guess is that I'm fairly slim, and that I often fold my left leg up underneath myself when sitting at a desk chair. And lemme tell you—I do a lot of sitting at desk chairs. Never thought the FBI would try using wear patterns on seams as identification.
posted by SoberHighland at 1:45 PM on February 26, 2020 [2 favorites]


bluejeans

This isn’t an actual compound word that I’ve missed, right? BlueJeans is a teleconference company, blue jeans are what you wear.
posted by zamboni at 2:01 PM on February 26, 2020 [3 favorites]


every pair of jeans I wear develops 2 diagonal wear-lines on my left leg: starting from my mid inner-thigh pointing to the middle of my kneecap

I have had noticed this on my jeans and other trousers over decades. I am not so slim so not sure that's a factor for me. I have wondered whether its to do with sitting with my left leg crossed over my right when i am sitting, and rarely the other way around.
posted by biffa at 2:18 PM on February 26, 2020


Am I understanding correctly that the sum total of evidence that this could ever work is a single article by an FBI employee?
posted by PMdixon at 2:20 PM on February 26, 2020


Novel forensic evidence analysis is often made up bullshit. Basically as long as you can convince a judge that your improvised test which more often than not are never tested in a double blind fashion you can get expert testimony in. The standard relies on the idea that the expert can be cross examined and challenged in that arena but given the massive disparity in resources for the defense vs the prosection it's rare that happens effectively.

Often analysis is done by someone trying to get evidence to fit a theory rather than testing something to create a theory.

It's also a thing that years of forensic files and csi have convinced jury pools that science is magic and if an well dressed expert witness claims that an their analysis proves that a single piece of grass found on a suspects boot could *only* have come from the victims yard because they say so it must be true.
posted by Ferreous at 2:48 PM on February 26, 2020 [22 favorites]


Blood splatter analysis? Not something that's actually backed by evidence. Same with lots of other things like bite mark analysis or handwriting analysis. Then actually forced to do double blind testing many of these techniques fail spectacularly.

Expert witness in these fields are hired guns with zero fucking morals and a willingness to send people to death for a paycheck.
posted by Ferreous at 2:55 PM on February 26, 2020 [12 favorites]


They would be comparing the wear patterns on jeans? So you could frame someone else by, say, borrowing your roommate’s clothing? I see no potential flaws in this method.
posted by ricochet biscuit at 2:55 PM on February 26, 2020 [3 favorites]


As forensic tests go, jeanetic fingerprinting still has some work to go.

I will thank you to be a little more open minded about my startup, 30x33andMe.

Often analysis is done by someone trying to get evidence to fit a theory

And 99.99999% of the time that theory is, "the person you have is better to convict than the person you have to look for."

Blood splatter analysis?

Bite marks, too. I have half a mind to start discreditpedia.
posted by rhizome at 3:00 PM on February 26, 2020 [18 favorites]


Confirmation bias is a hell of law enforcement methodology.
posted by srboisvert at 3:21 PM on February 26, 2020 [3 favorites]


Fingerprints are pretty weak too, and eyewitness testimony is super unreliable. Basically the whole prosecution apparatus is a con.
pun intended
posted by rodlymight at 3:22 PM on February 26, 2020 [8 favorites]


Fashion policing might still work. Corduroy could at least rule out suspects younger than 68.
posted by They sucked his brains out! at 3:26 PM on February 26, 2020 [7 favorites]


There’s gonna release all the people rotting in prison due to denim phrenology immediately, right?
posted by Vulgar Euphemism at 3:38 PM on February 26, 2020 [2 favorites]


Time to pay ProPublbica again.
posted by crush at 3:57 PM on February 26, 2020 [6 favorites]


Speaking of, the Brennan Center just released a detailed report showing how recent administrations (yes, both parties) have eroded the "clear working consensus. . . that government science and research should be unbiased and accessible." and offering some steps for the next administration to take to restore integrity to government science.
posted by crush at 4:00 PM on February 26, 2020 [4 favorites]


Forensic science is almost entirely bullshit, and the bits that aren't bullshit are often lied about by prosecutors- presenting a fragment of a fingerprint and telling the jury the odds of a match are the same as the odds of a match for a full set. (Also the odds of both are pure conjecture, but that's another story...)
posted by Pope Guilty at 4:38 PM on February 26, 2020 [5 favorites]


Arson investigations also bullshit.
posted by Mavri at 5:38 PM on February 26, 2020 [2 favorites]


stonewashed
posted by clavdivs at 8:14 PM on February 26, 2020


Having taken a seminar on such topics run by the Innocence Project, back when I was in law school? The National Academy of Sciences did a pretty lengthy study on every available method of forensic analysis back in the 2000s. Every single one of them that isn't nuclear DNA analysis is unscientific bullshit at best and deliberate deception at worst, and even nuclear DNA analysis is statistically bullshit. It's all make-believe. Yes, even fingerprinting.
posted by kafziel at 9:04 PM on February 26, 2020 [7 favorites]


I’ve seen them all and, man, they’re all the same.
posted by mr_roboto at 9:16 PM on February 26, 2020 [2 favorites]


Having taken a seminar on such topics run by the Innocence Project, back when I was in law school? The National Academy of Sciences did a pretty lengthy study on every available method of forensic analysis back in the 2000s. Every single one of them that isn't nuclear DNA analysis is unscientific bullshit at best and deliberate deception at worst, and even nuclear DNA analysis is statistically bullshit. It's all make-believe. Yes, even fingerprinting.

Do you happen to have a link handy to the report? It sounds fascinating.
posted by codacorolla at 9:24 PM on February 26, 2020 [1 favorite]


Do you happen to have a link handy to the report? It sounds fascinating.

Right here.
posted by kafziel at 10:03 PM on February 26, 2020 [12 favorites]


even nuclear DNA analysis is statistically bullshit

Can you point readers to the relevant part of your link?
posted by They sucked his brains out! at 12:39 AM on February 27, 2020


If you read that NRC report it does not say that all non-DNA forensic techniques are unscientific bullshit. It, quite rightly, identifies some (bite mark analysis) as having little basis, others (fingerprints) as basically robust but doubtful where applied on poor or partial evidence, and others (firearm witness marks) as having a credible basis but suffering from a lack of robust statistical support.

I should explain that I am near completion of a Master's degree in forensic engineering and science, which included study of basic forensic techniques, statistics and evaluation of scientific reports and research, and in my case an introduction to firearms analysis and forensic ballistics, and an introduction to fire investigation. I'm a qualified and practising lawyer of nearly a decade's experience, and although my work is mainly in civil and family litigation it has included cases with a forensic science element. (If anything, I think the NRC report is too positive about handwriting analysis, which from my experience I treat with great care, although document forensics, which is based on analysis rather than opinion, is much more robust.)

Forensic science as a whole is not junk. Most well-established techniques, if applied properly, are robust. But there are often serious problems with forensic science as used in court. For many techniques there is a deplorable lack of research on the false positive rate, or the factors that can compromise otherwise reliable tests. DNA analysis, especially with high-STR techniques, is extremely accurate, but the problem now is that it has become so sensitive that secondary transfer (contamination via intermediaries) is a real problem, and much more research is needed on this.

The biggest issue though is the standard of forensic practice. Forensic examination must be carried out by properly trained and qualified examiners, using documented and audited processes with careful control to preserve the chain of evidence and avoid contamination. There has to be a robust body of peer-reviewed science behind techniques, including evidence as to how they can fail.

You do not become a forensic specialist by attending a one-day course on a technique, especially if that technique is new, controversial or has been carried out by only a few specialists. I am certainly not a forensic scientist in any sensible meaning of the term; rather, I am a lawyer with just enough training (an MSc!) to be able to carry out a basic evaluation of a forensic expert report such as to identify shortcomings within it, or be able to know how to research the background science to assess whether the techniques used are based on good science and widely accepted.

We should not throw out forensic science, and the NRC report did not recommend doing so. Like other such reports, such as the UK Forensic Science Regulator's recent report on forensic science and the criminal justice system, it identifies the need for more research and better practice.
posted by Major Clanger at 2:02 AM on February 27, 2020 [13 favorites]


Most well-established techniques, if applied properly, are robust. But there are often serious problems with forensic science as used in court.

Can you help me understand why I should care about anything besides how it's used in court?
posted by PMdixon at 5:19 AM on February 27, 2020 [7 favorites]


Cops really do be just making shit up and calling it evidence. The Justice system is a sham
posted by dis_integration at 5:55 AM on February 27, 2020 [3 favorites]


A lot of the problem with how it's used in court is that the adversarial system is a terrible way to present scientific evidence. Explaining DNA analysis to a lay jury (some of whom haven't thought about these concepts since 10th grade biology) in terms that are simplified without being wrong takes training and practice, and it's worse when you can only answer the questions asked, not the question you think the attorney should have asked or what you think they're trying to get at.

In my perfect world scientific testimony would look like

a) voir dire (who are you and why should we trust you?)
b) short presentation from witness on the principles and methods of their field of expertise and what their conclusions and statistics do and do not mean
c) prosecutor questions
d) defense questions
e) jury questions

where e) is the most important. The grand jury system has its own flaws, but jurors being able to ask clarifying questions until they understand is a definite feature. I'm always disappointed when they don't have any - this is your chance! I'm here to educate! When attorneys ask the same question three times it's annoying, but if you Jane Juror need to hear the same concept repeatedly explained in slightly different terms till you get it I will happily do that! The CSI effect and its reverse (this method was discredited, therefore it's all crap) do everyone a disservice.
posted by Flannery Culp at 6:16 AM on February 27, 2020 [5 favorites]


A lot of the problem with how it's used in court is that the adversarial system is a terrible way to present scientific evidence

Yes. Given that the US uses the adversarial system and will for the foreseeable future, it seems to me that any rigorous notion of forensic science is literally of purely academic interest. The actually existing courts actually exist. Courts that aren't going to use these types of claims to provide some seemingly authoritative justification for decisions they have already made don't, as far as I can tell.
posted by PMdixon at 6:28 AM on February 27, 2020


Well, no. You certainly want "rigorous notion[s] of forensic science" underpinning any methods that are used at trial. Assuming that's true, there are still better and worse ways to present evidence within the adversarial system. I'd also like to see more rigorous training for attorneys so they don't grossly misrepresent the scientific conclusions in opening and (especially) closing statements.
posted by Flannery Culp at 6:33 AM on February 27, 2020


I want a desire for accurate information and not just a railroading underpinning any methods that are used at trial. That is in absence and there is currently no seeming evidence that it will appear.
posted by PMdixon at 6:50 AM on February 27, 2020


More research and better practice would be a hell of a good recommendation if the whole system weren't built in order to convict the person sitting in the defendant's chair, and little else.
posted by allthinky at 7:23 AM on February 27, 2020 [3 favorites]


Oh good grief, fully 75% of my clothes are secondhand, gods forbid they wind up as evidence for anything.

(I am not actually worried about my jeans getting me nabbed for bank robbery. I'm frustrated by how the US criminal justice system at every level uses ludicrously circumstantial non-evidence fluffed up with jargon and circular arguments. )
posted by desuetude at 8:33 AM on February 27, 2020


As a scientist with a statistics background, I'd like to know how nuclear DNA analysis is "statistical bullshit". I get that there are ways to fudge samples, lab error, and so forth, which are meant to railroad defendants, usually people of color. But I'd be curious to know more about the scientific arguments against DNA fingerprinting, as applied to forensics analysis.
posted by They sucked his brains out! at 11:11 AM on February 27, 2020 [1 favorite]


Like I said I think the single biggest issue after reliability of evidence analysis is that the people analyzing evidence work for cops. How the hell are you going to get impartial analysis when you're actively searching for evidence to fit a scenario and implicate a target suspect? Forensics labs should be black boxes getting anonymous evidence in and pushing results out. It's real easy to find identifying points on a muddy tire track if you are looking for things to match the car of your suspect.
posted by Ferreous at 1:19 PM on February 27, 2020


My understanding about DNA testing is that every hit is 1:1 billion. 7 billion people in the world? 6 others will match your DNA.
posted by rhizome at 1:59 PM on February 27, 2020


Not gonna go look it up but there are lots of ways that crime labs and DAs fuck up the statistics so that their one in a gazillionty-jillion statements are routinely wildly false. ISTR some of it is the usual Bayesian problems with testing a very large number of people for a rare condition.
posted by GCU Sweet and Full of Grace at 2:14 PM on February 27, 2020


They sucked his brains out!, the PCAST report (a follow-up to the report that kafziel linked) discusses DNA analysis in sections 5.1-5.2. They say that it's foundationally valid for single-source samples and simple-mixture samples where one source is known, but human error, contamination, and close relatives mean that probabilities like 1 in a billion are overstated ("for first degree relatives, the random match probability may be on the order of 1 in 100,000").

They're a lot harsher on DNA analysis of samples involving multiple unknown individuals. For example, they mention a case in Texas where a change in some formulas increased the calculated probability of a random match from 1 in 1.4 billion to 1 in 36, which sounds like statistical bullshit to me. The report says that there are promising attempts to calculate probabilities using software rather than relying on subjective judgments, but the reliability and the limits of the software would have to be established.
posted by ectabo at 3:08 PM on February 27, 2020


As a scientist with a statistics background, I'd like to know how nuclear DNA analysis is "statistical bullshit". I get that there are ways to fudge samples, lab error, and so forth, which are meant to railroad defendants, usually people of color. But I'd be curious to know more about the scientific arguments against DNA fingerprinting, as applied to forensics analysis.

And let us not forget the False Positive Paradox. If a sample will accurately match 1 person in a population of ten million, on a test with 99.9999% reliability, you still have a 90% chance that any given positive is a false one.
posted by kafziel at 3:44 PM on February 27, 2020 [1 favorite]


I have worked in garment manufacturing since 1991, and I have probably gazed upon half a million pairs of jeans in production.
Some factories use lasers to "burn" a wear-pattern. Other factories spray the pants with starch and then bake them on bent-legged forms. (Imagine a person in wide-legged squat , but made out of
metal tubes.) Some factories use sandpaper and scratch a pattern by hand.
Ridiculous to think they can trace wear patterns back to owner!!
posted by ohshenandoah at 4:37 PM on March 3, 2020 [1 favorite]


« Older Cinematic, emotional piano from neoclassical...   |   ⬆️⬆️⬇️⬇️⬅️➡️⬅️➡️🅱️🅰️ RIP Newer »


This thread has been archived and is closed to new comments