The mystery of the phantom reference
October 13, 2020 12:38 PM Subscribe
To cut a long story short, the article appeared to be completely made up and did not in fact exist. It was a "phantom reference" that had been created merely to illustrate Elsevier's desired reference format. Even so, Pieter found that in the Web of Science there were nearly 400 articles citing this non-existing reference and many more citing articles appeared in the more comprehensive Google Scholar.
By Anne-Wil Harzing.
Also: The phantom reference and the propagation of error by Rhodri Ivor Leng.
By Anne-Wil Harzing.
Also: The phantom reference and the propagation of error by Rhodri Ivor Leng.
Another way they know academics don’t read the papers they cite is that some papers get misspelled once in a citation & then the misspelling becomes more common in later citations of the paper than the original correct spelling. It’s difficult to see how this will change as long as the incentives for an academic is to increase your citation count, regardless of whether the paper is read or not.
posted by jonp72 at 1:18 PM on October 13, 2020 [6 favorites]
posted by jonp72 at 1:18 PM on October 13, 2020 [6 favorites]
Related: The Case of the Creeping Fox Terrier Clone discusses how data gets copied from one paper to the next, from one textbook to the next, without regard to updates in the field. And of course, it gets confirmed even when checked, because all the older works contain the same info.
posted by ErisLordFreedom at 1:33 PM on October 13, 2020 [3 favorites]
posted by ErisLordFreedom at 1:33 PM on October 13, 2020 [3 favorites]
I got funding for a grad to go on to PhD study a few years ago, on a specific research topic, basically applying known tech to develop a smaller scale application for a practical system. Once he started, his literature review required getting some numbers straight, doing so revealed that there was an assumption about optimal system performance that had no basis in experimental data. Someone had got it from someone, who'd got it from someone else, and so on, back to the 1950s or 60s where it seemed to have been plucked from the air.
We ended up shifting the topic just to focus on the accuracy of the assumed figure. Our evidence suggested it was an incorrect assumption.
The assumption has been used in deployed technology with implications for cost and efficiency of the process, it may even have helped slow down uptake of the technology.
posted by biffa at 1:51 PM on October 13, 2020 [10 favorites]
We ended up shifting the topic just to focus on the accuracy of the assumed figure. Our evidence suggested it was an incorrect assumption.
The assumption has been used in deployed technology with implications for cost and efficiency of the process, it may even have helped slow down uptake of the technology.
posted by biffa at 1:51 PM on October 13, 2020 [10 favorites]
Previously.
(Hopefully this joke doesn't break the site.)
posted by clawsoon at 2:18 PM on October 13, 2020 [4 favorites]
(Hopefully this joke doesn't break the site.)
posted by clawsoon at 2:18 PM on October 13, 2020 [4 favorites]
The big publishers are such a fucking joke of course they failed to properly mark a made-up example as such.
I am finalizing a paper for a top journal in my field (published by Wiley). They can't even manage to correctly communicate whether et al. should be italicized when citing, using a bizarre mix of Roman and italic styles when attempting to talk about it.
posted by SaltySalticid at 2:22 PM on October 13, 2020
I am finalizing a paper for a top journal in my field (published by Wiley). They can't even manage to correctly communicate whether et al. should be italicized when citing, using a bizarre mix of Roman and italic styles when attempting to talk about it.
posted by SaltySalticid at 2:22 PM on October 13, 2020
Kutsuwamushi: It seemed like one person made a mistaken citation (either out of misunderstanding or laziness), and then that citation was copied to make the same point in other papers.
I seem to recall this coming up in Rebecca Jordan-Young's excellent Brain Storm: The Flaws in the Science of Sex Differences as one of the ways that a scientific consensus could get established in a field without the data actually supporting the consensus.
posted by clawsoon at 2:22 PM on October 13, 2020 [6 favorites]
I seem to recall this coming up in Rebecca Jordan-Young's excellent Brain Storm: The Flaws in the Science of Sex Differences as one of the ways that a scientific consensus could get established in a field without the data actually supporting the consensus.
posted by clawsoon at 2:22 PM on October 13, 2020 [6 favorites]
On the other hand, I'll happily admit to not fully reading every paper I cite, it's just a shitty part of the shitty game. I do at least look at a copy though.
posted by SaltySalticid at 2:23 PM on October 13, 2020 [2 favorites]
posted by SaltySalticid at 2:23 PM on October 13, 2020 [2 favorites]
Citogenesis is at least as bad in the nonacademic world, of course.
posted by ivan ivanych samovar at 2:46 PM on October 13, 2020 [3 favorites]
posted by ivan ivanych samovar at 2:46 PM on October 13, 2020 [3 favorites]
This seems like a problem that can be fixed with Machine Learning or RPA. Maybe not fixed, but at least those tools could help researchers cull the scientific record of these types of references...
posted by joecacti at 3:47 PM on October 13, 2020 [1 favorite]
posted by joecacti at 3:47 PM on October 13, 2020 [1 favorite]
I'll happily admit to not fully reading every paper I cite
Who does? You'll typically ref say 40 papers per each of your papers. Even at three papers a year, that's 120 papers, before even throwing out the rubbish, reading stuff for teaching and generally to keep up. Its just not credible. The only papers I ever read start-to-end now are ones I'm writing or ones I'm reviewing.
posted by biffa at 4:17 PM on October 13, 2020 [1 favorite]
Who does? You'll typically ref say 40 papers per each of your papers. Even at three papers a year, that's 120 papers, before even throwing out the rubbish, reading stuff for teaching and generally to keep up. Its just not credible. The only papers I ever read start-to-end now are ones I'm writing or ones I'm reviewing.
posted by biffa at 4:17 PM on October 13, 2020 [1 favorite]
Whoa, when templates attack!
One of my more gleeful moments from just a few years back was figuring out the source of an ongoing citation error in Protestant apologetics (in this case, two people with the same last name getting scrambled together, meaning that the citation goes to the wrong book). Very ongoing. As in, it originated in a quotation from an article published in 1838.
I'll happily admit to not fully reading every paper I cite
Who does?
Papers, yes, but like everyone else in the humanities, I was trained very early on to figure out the glories of the index when it comes to monographs. This causes its own citation errors--e.g., when the index is itself in error, and then someone gives a page range clearly based on the faulty index...
There is also a bit of a "rolling stone" problem, in which a source slowly accrues legitimacy as it is repeated from citation to citation. After a certain point, people neglect to ask relevant questions.
posted by thomas j wise at 6:10 PM on October 13, 2020 [1 favorite]
One of my more gleeful moments from just a few years back was figuring out the source of an ongoing citation error in Protestant apologetics (in this case, two people with the same last name getting scrambled together, meaning that the citation goes to the wrong book). Very ongoing. As in, it originated in a quotation from an article published in 1838.
I'll happily admit to not fully reading every paper I cite
Who does?
Papers, yes, but like everyone else in the humanities, I was trained very early on to figure out the glories of the index when it comes to monographs. This causes its own citation errors--e.g., when the index is itself in error, and then someone gives a page range clearly based on the faulty index...
There is also a bit of a "rolling stone" problem, in which a source slowly accrues legitimacy as it is repeated from citation to citation. After a certain point, people neglect to ask relevant questions.
posted by thomas j wise at 6:10 PM on October 13, 2020 [1 favorite]
Ironically, I went down a rabbit hole looking for this exactly paper a few years ago. Saw it cited, thought it looked interesting. Never could find it. Now I know why. /0\
posted by skye.dancer at 6:36 PM on October 13, 2020 [3 favorites]
posted by skye.dancer at 6:36 PM on October 13, 2020 [3 favorites]
Who does? You'll typically ref say 40 papers per each of your papers. Even at three papers a year, that's 120 papers, before even throwing out the rubbish, reading stuff for teaching and generally to keep up. Its just not credible. The only papers I ever read start-to-end now are ones I'm writing or ones I'm reviewing.
I do. But then, I suppose that may be one of the reasons my research career hasn't been stellar. Blase statements about how nobody fully reads the papers they cite or review then make me no small amount of annoyed at the people with looser ethics who cut corners and apparently don't care about the quality of their work, yet reap greater career benefits. It's behavior like this that creates the grinding publish or perish pressure mill for everyone, especially given that we academics are largely responsible for evaluating ourselves and setting our fields' own standards. Why yes I am also opposed to capitalism in general. And yes, I am more than implying that you, personally, should probably feel ashamed of your behavior in this respect.
posted by eviemath at 8:00 PM on October 13, 2020 [8 favorites]
I do. But then, I suppose that may be one of the reasons my research career hasn't been stellar. Blase statements about how nobody fully reads the papers they cite or review then make me no small amount of annoyed at the people with looser ethics who cut corners and apparently don't care about the quality of their work, yet reap greater career benefits. It's behavior like this that creates the grinding publish or perish pressure mill for everyone, especially given that we academics are largely responsible for evaluating ourselves and setting our fields' own standards. Why yes I am also opposed to capitalism in general. And yes, I am more than implying that you, personally, should probably feel ashamed of your behavior in this respect.
posted by eviemath at 8:00 PM on October 13, 2020 [8 favorites]
Turns out academia is just Twitter for people with a tolerance for longer dopamine loops.
posted by turbowombat at 8:11 PM on October 13, 2020 [3 favorites]
posted by turbowombat at 8:11 PM on October 13, 2020 [3 favorites]
I have at least downloaded every paper cited into endnote, so it would go automatically into my reference list. I never even thought of just copying the reference because that would have been more work than relying on endnote.
posted by Tandem Affinity at 8:26 PM on October 13, 2020
posted by Tandem Affinity at 8:26 PM on October 13, 2020
I am more than implying that you, personally, should probably feel ashamed of your behavior in this respect.
I can live with that. I don't feel at all ashamed, I just don't think its credible to operate like that. Looking at my last paper: 97 refs, 7 authors. Your paradigm suggests all 7 should read all 97 papers? Spending maybe 2 months each? Alongside teaching, doing the actual research, writing the paper, getting the paper through one or two review/rewrites. Its in an interdisciplinary space, and that sort of work would be impossible since it implies reading papers that require substantive background knowledge to comprehend.
posted by biffa at 3:01 AM on October 14, 2020
I can live with that. I don't feel at all ashamed, I just don't think its credible to operate like that. Looking at my last paper: 97 refs, 7 authors. Your paradigm suggests all 7 should read all 97 papers? Spending maybe 2 months each? Alongside teaching, doing the actual research, writing the paper, getting the paper through one or two review/rewrites. Its in an interdisciplinary space, and that sort of work would be impossible since it implies reading papers that require substantive background knowledge to comprehend.
posted by biffa at 3:01 AM on October 14, 2020
It would be entirely possible. Just slower. There's been an inflation in academic publishing expectations/requirements over the past several decades. There's no good reason for why publishing expectations should be as high as they currently are. (Kind of ironic given the complaints some faculty express about grade inflation.)
Meanwhile, for every current academic who is shirking doing the background reading they cite, or just writing poorly written and hard to read/understand papers (a contributing factor to the situation, partially stemming from the same publication pressures), there is someone just as talented but from an racialized background or working class background or first generation in their family to obtain a PhD who has always had to do a more complete job in order to be judged as highly, and who likely has a bad case of imposter syndrome on top of that, who didn't get your job.
At the least, you could be aware of the publications inflation issue as a problem for both quality of academic research and equity, and actively work against it from your current position, rather than blithely supporting it.
posted by eviemath at 4:11 AM on October 14, 2020 [4 favorites]
Meanwhile, for every current academic who is shirking doing the background reading they cite, or just writing poorly written and hard to read/understand papers (a contributing factor to the situation, partially stemming from the same publication pressures), there is someone just as talented but from an racialized background or working class background or first generation in their family to obtain a PhD who has always had to do a more complete job in order to be judged as highly, and who likely has a bad case of imposter syndrome on top of that, who didn't get your job.
At the least, you could be aware of the publications inflation issue as a problem for both quality of academic research and equity, and actively work against it from your current position, rather than blithely supporting it.
posted by eviemath at 4:11 AM on October 14, 2020 [4 favorites]
One can feel bad about participating in a bad system at the same time as not knowing how to solve it. I buy clothes and fruit; I know about, and am uncomfortable with, the poor wages and working conditions which get them to me at such cheap prices. If someone tells me, "You should really seek out more ethical sources," I'm not going to respond by justifying the current system and my role in it.
posted by clawsoon at 4:15 AM on October 14, 2020
posted by clawsoon at 4:15 AM on October 14, 2020
As it happens I am the first gen person from a working class background. I have impostor syndrome aplenty. I did get my job and this is how I keep it, by setting realisable goals and meeting targets for research outputs. Thanks for your support.
posted by biffa at 4:33 AM on October 14, 2020 [4 favorites]
posted by biffa at 4:33 AM on October 14, 2020 [4 favorites]
One can feel bad about participating in a bad system at the same time as not knowing how to solve it.
For sure. And then one can express feeling bad about that, rather than express direct support for the system.
posted by eviemath at 4:40 AM on October 14, 2020
For sure. And then one can express feeling bad about that, rather than express direct support for the system.
posted by eviemath at 4:40 AM on October 14, 2020
As it happens I am the first gen person from a working class background. I have impostor syndrome aplenty. I did get my job and this is how I keep it, by setting realisable goals and meeting targets for research outputs. Thanks for your support.
Ah, speaking as a peer then, maybe help the rest of us by not working to support and maintain this system? We will also appreciate your support.
posted by eviemath at 4:41 AM on October 14, 2020
Ah, speaking as a peer then, maybe help the rest of us by not working to support and maintain this system? We will also appreciate your support.
posted by eviemath at 4:41 AM on October 14, 2020
I read all the papers I cite- possibly not in full (I only read the methods section on a lot of papers if I need to replicate the experiment, like most chemists), and I make sure that if I'm copying a citation I have it right and it, as best as I can, says what I think it says.
I put out papers with 100 citations, so yeah. It doesn't take that long, and you reuse most of them in your next paper anyway since your introduction is probably mostly the same (same five papers saying coordination polymers are useful for X,Y and Z, plus the one cool new one that came out since the last paper)
The only ones I don't bother reading are the software citations (Since what I'm actually citing is the software) and papers from my own group (Since I was likely at all the group meetings, saw you doing it in the lab, etc).
Oh and papers in languages I don't speak, then I just get someone who does to tell me if it says what I think it says or not, but I'm usually just putting those in as historical background as pretty much all good chemistry is published my area of chemistry in English these days (except for that one German guy who keeps publishing critical papers in German only journals just to piss off non-German speakers, but luckily I don't work in his area)
posted by Canageek at 10:20 AM on October 14, 2020 [3 favorites]
I put out papers with 100 citations, so yeah. It doesn't take that long, and you reuse most of them in your next paper anyway since your introduction is probably mostly the same (same five papers saying coordination polymers are useful for X,Y and Z, plus the one cool new one that came out since the last paper)
The only ones I don't bother reading are the software citations (Since what I'm actually citing is the software) and papers from my own group (Since I was likely at all the group meetings, saw you doing it in the lab, etc).
Oh and papers in languages I don't speak, then I just get someone who does to tell me if it says what I think it says or not, but I'm usually just putting those in as historical background as pretty much all good chemistry is published my area of chemistry in English these days (except for that one German guy who keeps publishing critical papers in German only journals just to piss off non-German speakers, but luckily I don't work in his area)
posted by Canageek at 10:20 AM on October 14, 2020 [3 favorites]
I wonder whether people are using the same definition of “reading” in this thread. Does “reading” always mean a deep dive, or does it just mean that at one point you looked at the paper and verified that it in fact provided a certain result and/or supported the point you were trying to make?
There are even very different degrees of rigor in that second option. I’ve been “drive-by” cited in a way that suggests that the reader barely looked at the paper, and it’s certainly somewhat disappointing, but in most cases (by no means all!) they are indeed at least referring to something I really said, although there might be a “better” reference to support their point.
(I definitely wouldn’t expect that every author on a paper in my field would have read every reference — for one thing subfields are so specialized and teams are getting so big that I don’t think this is a really fair expectation to put on people. If there were a paper where, say, an organic chemist synthesizes a molecule that’s used in one analysis and a computer scientist does some machine learning for another, it would seem like a pretty big ask to have everyone be jointly responsible for every citation. They wouldn’t have the background to evaluate it properly anyway.)
posted by en forme de poire at 7:46 AM on October 15, 2020 [1 favorite]
There are even very different degrees of rigor in that second option. I’ve been “drive-by” cited in a way that suggests that the reader barely looked at the paper, and it’s certainly somewhat disappointing, but in most cases (by no means all!) they are indeed at least referring to something I really said, although there might be a “better” reference to support their point.
(I definitely wouldn’t expect that every author on a paper in my field would have read every reference — for one thing subfields are so specialized and teams are getting so big that I don’t think this is a really fair expectation to put on people. If there were a paper where, say, an organic chemist synthesizes a molecule that’s used in one analysis and a computer scientist does some machine learning for another, it would seem like a pretty big ask to have everyone be jointly responsible for every citation. They wouldn’t have the background to evaluate it properly anyway.)
posted by en forme de poire at 7:46 AM on October 15, 2020 [1 favorite]
I’ll also just say that from lurking around places like Academia Stack Exchange, it’s become clear to me that norms around publication vary really dramatically across fields. Math and computer science, for example, tend to not take it as a given that a PhD student’s advisor will appear on all of their papers, in some cases to the point of considering it unethical or fraudulent to add an advisor if all they have done is provide funding, materials, feedback on writing, and high-level guidance. Experimental biologists on the other hand might actually consider it unethical to omit their advisor from the author list under the same circumstances. I’ve seen discussions about this get really heated, where people were accused of “lying” by following the standards of their field, and I don’t think that’s particularly helpful or illuminating.
posted by en forme de poire at 8:19 AM on October 15, 2020 [3 favorites]
posted by en forme de poire at 8:19 AM on October 15, 2020 [3 favorites]
When I was a younger grad student I read every paper end to end. Now I read the main points, but might skim the justification for the research (since nine out of ten times you make that up afterwards with the real reason being "we wanted to do cool chemistry no one had done before"
posted by Canageek at 7:42 PM on October 15, 2020 [2 favorites]
posted by Canageek at 7:42 PM on October 15, 2020 [2 favorites]
« Older Can your mind bend the yellow dots to do your will... | Did You Ever Really Look At Your Hand? Newer »
This thread has been archived and is closed to new comments
It seemed like one person made a mistaken citation (either out of misunderstanding or laziness), and then that citation was copied to make the same point in other papers.
Made me feel a bit frustrated that I was doing what I should, by doing my best to understand what I was citing, when others were writing sloppy literature reviews with citations to papers they hadn't even read.
posted by Kutsuwamushi at 1:08 PM on October 13, 2020 [6 favorites]