Join 3,562 readers in helping fund MetaFilter (Hide)


Does it matter whether you think this is Best of the Web?
September 26, 2006 10:49 PM   Subscribe

Reviewing peer review.
posted by Gyan (33 comments total) 4 users marked this as a favorite

 
Of course, the "revolutionary" journal Philica is located at its namesake address.
posted by Gyan at 10:59 PM on September 26, 2006


Does this make any comments on this thread meta^3? Too much meta these days. Damn kids!
posted by neckro23 at 11:18 PM on September 26, 2006


A great read! Peer review is overly trusted, and remains a barrier to the publication of legitimate research. We all know journals that will reject your paper if it goes against the editor's pet hypothesis, and reviews who read more into the paper's authors and institution than the actual research conducted.
posted by Jimbob at 12:24 AM on September 27, 2006


We do?
posted by jmhodges at 12:54 AM on September 27, 2006


Yes. We do.
posted by Jimbob at 1:11 AM on September 27, 2006


After the disappearance of peer reviewing, the next logical step in scientific publishing should be the random insertion of hAWKINZ IS TEH GAY!!!LOLZ in any physics paper. After that, user 193.29.23.234 aka DragonLove28 will finally be able to publish his paper titled DNA sequencing of the Zakuro Fujiwara character from the Tokyo Mew Mew anime series in the Journal of Human Genomics.
posted by elgilito at 1:45 AM on September 27, 2006 [1 favorite]


You do who do.

Great article. elgilito, I think we will both gain and loose something with the new processes.
posted by stbalbach at 6:39 AM on September 27, 2006


Jimbob, your exactly right. We need a lot of crappy non-peer reviewed papers to be published on the internet so that we can swamp out publications with merit.

Taking off the sarcasm hat - I do not know about your field Jimbob, but in biology I do not think this is a good idea. Generally biologists do not trust non-peer review publications. When reading papers from a field in which you are not an expert, it is difficult to critically assess the work. Reviewers generally work in the field of the publication, and therefore if a paper passes peer review, one can have more confidence in the results. Peer review has and can be abused, but it is uncommon.

If you cannot get your paper published in a journal because it goes against the editor's pet hypothesis, publish elsewhere. There are plenty of journals around, some of which are very easy to get published in. And if one cannot pass peer review at any journal, they should consider the possibility that the problem lies with the research. And if they still want to publish - post it on the net.
posted by batou_ at 6:49 AM on September 27, 2006


In a word, reviewers are often not really “conversant with the published literature”; they are “biased toward papers that affirm their prior convictions”; and they “are biased against innovation and/or are poor judges of quality.” Reviewers also seem biased in favor of authors from prestigious institutions.

Makes sense. A thought-provoking article: thanks, Gyan.

*waits for Time Cube Guy to appear in Philica*
posted by languagehat at 6:50 AM on September 27, 2006


thanks.
The problem with editors of journals such as Nature and Science is that they are often unfamiliar, let alone misinformed, on emerging or highly debated fields (e.g. climate change) and they therefore make wrong decisions. Unfortunately not only regarding the fate of a manuscript but even before that, by picking the wrong set (i.e. not diverse enough) group of reviewers. Someone I know (and who has published in both) calls them "the tabloids of science".

Ultimately, the peer-review of any paper is given in the citations and is reflected in the general impact on the community.
posted by carmina at 6:57 AM on September 27, 2006


We all know journals that will reject your paper if it goes against the editor's pet hypothesis

Gonna need some evidence to back up that assertion, chief.
posted by grubi at 6:57 AM on September 27, 2006


grubi,
how do you want the evidence served?

An editor we used recently chose to rely on the review of one and only one scientist whose method was falsified in our paper (although that was not the purpose of the paper of course). She refused to even acknowledge this fact. Talking about objectivity...
posted by carmina at 7:04 AM on September 27, 2006


oh and guys, keep it straight: we are talking about journals such as Nature, Science and perhaps some Proceedings etc whatever each field's most pre-eminent publication journals are. Of course, if you try a little, you get published somewhere. This is not the point. The point is about journals that disseminate research results fast and to a greater,wider audience.
posted by carmina at 7:08 AM on September 27, 2006


I have a Bachelor's degree in a science, and I can tell you that peer review beats not having peer review. The Internet is full of wackos who just hate the concept of peer review, because it is a barrier to entry from millions of people who just have a strongly held opinion and not a lot else. No math, no falsifiability, no review of the literature, no examination of past relevant experiments. Get on IRC, log on to a #physics channel, and learn to love peer review. Even before The Tubes became clogged with geocities pages hosting someone's pet idea (called a "theory," and deserving of "not even wrong"), we'd get weird papers sent to the department, bearing drawings of electrons shaped like pyramids, etc.

It's not a perfect system, but until someone has a better approach ... and, no, the Wikipedia version of science is not going to be it, either.
posted by adipocere at 7:19 AM on September 27, 2006


because it is a barrier to entry from millions of people who just have a strongly held opinion and not a lot else

That seems to be the biggest motivator for those who say how much they dislike peer review.
posted by grubi at 7:30 AM on September 27, 2006


Have you people SEEN my peers? I wouldn't want to be in the same room with them, let alone have them reviewing anything for scientifc accuracy!
posted by blue_beetle at 8:00 AM on September 27, 2006


Ooh, I like this Philica. I hope it does well.
posted by sonofsamiam at 8:01 AM on September 27, 2006


Interesting article.

Shatz describes a study in which "papers that had been published in journals by authors from prestigious institutions were retyped and resubmitted with a non-prestigious affiliation indicated for the author. Not only did referees mostly fail to recognize these previously published papers in their field, they recommended rejection."

(Course, we can wonder if that study was peer reviewed...)
posted by gimonca at 8:03 AM on September 27, 2006


The real problem with peer review is that if I tell you I did an experiment to prove or disprove something and got these results, you can look at what I say I did and decide whether or not it would really shed any light on what I was looking at, but when it comes to my results, you either have to take me at my word, or repeat all my work (which could take years).

My boss once had to write a retraction because he included work done by a second author from another institution. After he began recieving letters questioning the phenomenon described, he went to contact his co-author only to find out that he was no longer with that institution (with overtones of being kicked out for because someone else caught him faking work).

But, while peer review could stand improvement, if we just did away with it that other guy wins. He can crank out a paper a week without being bothered with chemicals and instruments and statistics and stuff, leaving the rest of us to do whatever we're doing and repeat his non-work.
posted by Kid Charlemagne at 8:05 AM on September 27, 2006


This means that the opinion of somebody whose work is highly regarded carries more weight than the opinion of somebody whose work is rated poorly.

At first blush, that seems like a great idea, but there's a threat it could in fact be a mechanism to inhibit new ideas...for instance, if someone tried to publish a paper refuting a Nobel Prize winner's work (not the prize-winning work, let's say something after) and that particular winner is not willing to concede and gets nasty (yeah, that happens) then the challenger will get poor reviews by persons of high rank and continue to be considered a poor scientist, regardless of quality.

Which is not to say this is an awful idea, just that it doesn't solve ALL the problems of peer review.
posted by solotoro at 8:10 AM on September 27, 2006


What's right about peer review can solve what's wrong with peer review.

(apologies to Pres. Clinton)
posted by grubi at 8:15 AM on September 27, 2006


Not only did referees mostly fail to recognize these previously published papers in their field, they recommended rejection.

I'm not sure this is indicitive of a problem. To use a macro scale example, would you publish a paper that explained the structure of DNA? The reviewer might have scimmed the paper and said, "The Journal of Obscure Enzymes published a paper just like this last summer.

Patrick Nielsen Hayden, an editor at Tor Books, described flaws in the retyped manuscript experiment once upon a time. There were a number of reasons that something of quality might get rejected. Ultimatly he said that even if you immediatly recognize it as a retype of someone elses paper, do you want to deal with some fruitcake who has enough free time to type up and resubmit someone elses work, or just dash off a quick form letter and get it the hell off your desk.
posted by Kid Charlemagne at 8:19 AM on September 27, 2006


Wikipedia is peer review. Peer review just means peers review your work. Processes differs and the technology of the web has opened up new possibilities that could solve some of the problems with the 1957-style peer review. Even if you had a Wikipedia-style peer review, you can still have it moderated to keep out the "barbarians". IMO what we need is a way to reward the reviewers so to increase the quality and competition among reviewers. The Atlantic article suggests one way to do this is giving reviewers added weight and "scores" based on previous reviews, degrees, etc.. kinda of a sophisticated Amazon book review system, a reputation based system.
posted by stbalbach at 8:54 AM on September 27, 2006


Science is all about inhibiting new ideas: not their expression but their acceptance. Ideas are not judged by their elegance, nor their uniqueness, nor their poetry. Only those judged to be paying their way are incorporated. In other words, only those ideas that can be falsified and fail to be will be used and the others discarded.

As someone once said about democracy, peer review is a flawed system that is used because all the others are worse. From my point of view, about 90% of the peer-reviewed literature is garbage, but about 99% of the rejected literature is. It's bad enough having to look through 10 papers to find a good one, so I don't want to have to look through 100 instead.
posted by Mental Wimp at 1:55 PM on September 27, 2006


The problem with the test of falsification as a necessary attribute of scientific usefulness, of course, is that it fails the test of falsification.
posted by jamjam at 2:24 PM on September 27, 2006


The problem with the test of falsification as a necessary attribute of scientific usefulness, of course, is that it fails the test of falsification.

I'm not sure that that statement parses: falsification is part of the definition of scientific theory. Usefulness is a commercial concern, unless you mean useful to predict things. And, yes, one can construct ones own parallel reality where prediction is not a fundamental concept in science. However, I'm not sure you find many people playing along.
posted by Mental Wimp at 2:49 PM on September 27, 2006


An important point to remember is that peer reviewedness does not make something true or part of the scientific canon - it merely makes it open for discussion on a certain level, and possibly worth looking into further to establish reproducibility etc. Barriers to entry are what makes it a worthwhile excercise, and editors with hobby horses are a flaw, but a surmountable one as no one journal is pre-emininent in its field (though one might be more famous, eg Nature)
posted by Sparx at 2:56 PM on September 27, 2006


This isn't really news...Sokel showed us back in 1996 that Peer Review was busted (although for reasons that weren't exactly wholesome). There are tons and tons of examples that peer review is faulty at best, and just a bad way of evaluating information at worst...at least in the current academic form.

As an academic librarian, I certainly DO believe that the wikipedia model will be the model for the future of information dissemination. Perhaps with slight modification, but academia is about open dialogue with others of a dissenting opinion. Yes, there are nuts...no one is saying they should be taken seriously. It's the process that's the issue, not the content.
posted by griffey at 2:57 PM on September 27, 2006


In other words, only those ideas that can be falsified and fail to be will be used and the others discarded.

Usefulness is a commercial concern, unless you mean useful to predict...

As the quoted text I've bolded makes clear, MW, you introduced the criterion of use of a scientific idea; I confess I am unsure how I ought to proceed in order to explain to you what you meant by it.

My point was, and remains, that Popper's assertion that all scientific truths must be falsifiable is not itself falsifiable, and if true, is therefore forever beyond scientific investigation or enquiry-- an extremely broad and very highly doubtful claim. Or it is simply wrong, and we can safely haul it away to the crowded junkyard of philosophers' silly ideas about science, as ought to have been done three generations ago.
posted by jamjam at 3:54 PM on September 27, 2006


I do not know about your field Jimbob, but in biology I do not think this is a good idea.

Well, I'm in ecology. I don't mean to say we should scrap peer review, just that it needs to be challenged and improved and experimented with, like every other process. Because for every bogus study it catches out, it probably rejects a decent one out of bias or misunderstanding, and lets through another crappy one out of nepotism. Your mileage may vary. And I think it's fairly easy for an editor to spot clearly bogus, fruity "research", so I doubt any decision to modify the peer review process will result in the Timecube guy getting published in Physics Review Letters.

Gonna need some evidence to back up that assertion, chief.
posted by grubi at 12:27 AM ACST on September 28 [+] [!]


You're kidding, right? You really believe every journal in your field is fair and balanced and unbiased, pure as the driven snow, and carrying no ideological baggage? Suffice to say that, in the field of Ecology, there are a number of schools of thought, driven by charismatic figures for many years, and you've got no hope of being accepted in the journal they sit on the editorial board for if your results contradict theirs. It's true that you can probably get your paper published somewhere, but it just serves to create little isolated pools of researchers who publish within themselves, instead of debating widely.
posted by Jimbob at 4:29 PM on September 27, 2006


jamjam

Nice work with the deconstruction and all. What do propose to put in its place?
posted by Mental Wimp at 10:41 PM on September 27, 2006


Mental Wimp, I was sort of hoping you could tell me.
posted by jamjam at 10:05 AM on September 28, 2006


Bias? Of course individual referees cleave to their preferred schools of thought. This does not mean that submissions that do not adhere to their school of thought are ignored or heavily discounted. Good scientists, those that get asked to be reviewers for top journals, keep an open mind -- they weigh theories in terms of "more important" and "less important", "more likely" vs. "less likely." Rarely do they think of theories as "right" or "wrong" (unless subjected to the crazed ravings of internet wackjobs...)

That said, the editors of journals to which we submit do certainly act in a fair and balanced manner, and they do their best to send submissions to those who are best qualified to review them.

The very top journals publish new and interesting papers that challenge existing dogma and open areas of understanding -- Nature, Cell, and Science did not garner Impact Factors of 30 by printing prosaic confirmations of previously published theories, or plodding extensions of existing work.

Thus, I do not buy the idea that peer review stifles new theories and ideas that challenge existing dogma. Certainly, if dogma is well entrenched, your data had better be damn good, your statistics meticulous, your experiments repeated multiple times, your methods well-accepted, but those are things that ought to be expected. Extraordinary claims require extraordinary proof, and when the two come together, you have a paper in a top journal.
posted by NucleophilicAttack at 4:15 PM on September 28, 2006


« Older R.I.P. Sir Malcom Arnold...  |  Arty hotel rooms.... Newer »


This thread has been archived and is closed to new comments