Can social networks substitute for peer review in science?
November 16, 2015 1:54 PM Subscribe
This year, the scientific workshop ADAPT shakes the status quo in the scientific process by using reddit as its primary review system.
Traditionally, the only quality control for scientific results prior to publication in journals and conferences is performed by increasingly overworked volunteers hand-picked by editors/organizers. This process is long due for an overhaul.
Lately, G. Fursin and C. Dubach, from the cTuning foundation, have been proposing to use social networks as a complement or substitute to outsource/distribute scientific quality control.
Last year, they tried Slashdot to discuss results post-publication. One lesson learned was that Slashdot makes moderation difficult, so perhaps Slashdot is not the right tool, but the main lesson learned was that the quality of the feedback can be good despite the lack of credentials of the internet mob. This year, as organizers of the ADAPT workshop, they will experiment by gather discussion on Reddit first and use this to evaluate submissions prior to publication. See also their Twitter stream.
Traditionally, the only quality control for scientific results prior to publication in journals and conferences is performed by increasingly overworked volunteers hand-picked by editors/organizers. This process is long due for an overhaul.
Lately, G. Fursin and C. Dubach, from the cTuning foundation, have been proposing to use social networks as a complement or substitute to outsource/distribute scientific quality control.
Last year, they tried Slashdot to discuss results post-publication. One lesson learned was that Slashdot makes moderation difficult, so perhaps Slashdot is not the right tool, but the main lesson learned was that the quality of the feedback can be good despite the lack of credentials of the internet mob. This year, as organizers of the ADAPT workshop, they will experiment by gather discussion on Reddit first and use this to evaluate submissions prior to publication. See also their Twitter stream.
Please note, this is "computer science" and not "science" as a whole. The title/FPP should reflect that!
For a more generalizable model, see PubPeer, although this is falls under publication peer review.
posted by lalochezia at 2:14 PM on November 16, 2015 [4 favorites]
For a more generalizable model, see PubPeer, although this is falls under publication peer review.
posted by lalochezia at 2:14 PM on November 16, 2015 [4 favorites]
Kinda depends on the subreddit now doesn't it. Is there an automated way to check if a reviewer has a 4chan account? :-)
posted by sammyo at 2:23 PM on November 16, 2015 [4 favorites]
posted by sammyo at 2:23 PM on November 16, 2015 [4 favorites]
Reddit in general is terrible, but it is possible to maintain a quality subreddit. AskHistorians is routinely hailed as a sterling example of what a subreddit can be. However, it achieves this by ignoring the general reddit ethos of "MUH FREEDOMS" and enacting a very strong moderation regime with clear rules about topics, discussion, and harassment, and has dedicated mods who constantly and consistently strictly enforce them.
If this ADAPT subreddit could do the same it could work, but it takes a lot dedication and vigilance.
posted by Sangermaine at 2:26 PM on November 16, 2015 [2 favorites]
If this ADAPT subreddit could do the same it could work, but it takes a lot dedication and vigilance.
posted by Sangermaine at 2:26 PM on November 16, 2015 [2 favorites]
As a scientist I would be appalled if anyone in my field tried to do this. In general, among biologists it's hard to recruit substantive comments on paper even when you have editors hunting people down and harassing them to turn in their reviews. On Reddit? Good lord no, that's a recipe for disaster.
posted by sciatrix at 2:33 PM on November 16, 2015 [9 favorites]
posted by sciatrix at 2:33 PM on November 16, 2015 [9 favorites]
Can I remember the name of that rule about headlines that ask questions like this?
No.
posted by uosuaq at 2:35 PM on November 16, 2015 [7 favorites]
No.
posted by uosuaq at 2:35 PM on November 16, 2015 [7 favorites]
Dear Editor/Redittor
In order to address the points made by One_Giant_Nostril and AllNiteLoŋŋŋŋŋ we reanalyzed our data using different post-hoc comparions and found no difference in our tests. Further we included the covariates suggested by both walrus_waffle and EmancipatedEmpanada and likewise found no changes.
The suggestions of IWantToTouchYourBum were ignored as it was clear they hadn't even bothered to read the methods section where the concerneds they raised had already been addressed.
SwordFaggots suggestions for what we should put in our conclusion were too speculative and possibly physically impossible given the space constraints.
We would like to thank MartyMcFYI for suggestions on how we could more deeply explore causality in both future and past research designs but must await sufficient funding for the purchase of the required capacitor.
posted by srboisvert at 2:42 PM on November 16, 2015 [33 favorites]
In order to address the points made by One_Giant_Nostril and AllNiteLoŋŋŋŋŋ we reanalyzed our data using different post-hoc comparions and found no difference in our tests. Further we included the covariates suggested by both walrus_waffle and EmancipatedEmpanada and likewise found no changes.
The suggestions of IWantToTouchYourBum were ignored as it was clear they hadn't even bothered to read the methods section where the concerneds they raised had already been addressed.
SwordFaggots suggestions for what we should put in our conclusion were too speculative and possibly physically impossible given the space constraints.
We would like to thank MartyMcFYI for suggestions on how we could more deeply explore causality in both future and past research designs but must await sufficient funding for the purchase of the required capacitor.
posted by srboisvert at 2:42 PM on November 16, 2015 [33 favorites]
Have you ever been on Ask Science? Its full of the dumbest dumbfucks imaginable. 90 percent of the responses are, "correlations does not equal causation!!!!!1"
posted by MisantropicPainforest at 2:43 PM on November 16, 2015 [4 favorites]
posted by MisantropicPainforest at 2:43 PM on November 16, 2015 [4 favorites]
I joined Researchgate, which is the start of a "social network" for scientists. It's quite interesting to post your publication and see if anyone reads it, if anyone cites it, if anyone messages you to ask a question. I like it a lot. It might be the start of a new way of doing things.
You can also just say "hi" to an old friend.
posted by acrasis at 2:44 PM on November 16, 2015
You can also just say "hi" to an old friend.
posted by acrasis at 2:44 PM on November 16, 2015
I'm sorry, but between reddit and "researchgate", all I can think is : IT'S ABOUT ETHICS IN SCIENTIFIC PUBLICATION
posted by idiopath at 3:00 PM on November 16, 2015 [2 favorites]
posted by idiopath at 3:00 PM on November 16, 2015 [2 favorites]
What could possibly go SHITCOCKS
posted by boo_radley at 3:02 PM on November 16, 2015 [14 favorites]
posted by boo_radley at 3:02 PM on November 16, 2015 [14 favorites]
Reddit in general is terrible, but it is possible to maintain a quality subreddit. AskHistorians is routinely hailed as a sterling example of what a subreddit can be. However, it achieves this by ignoring the general reddit ethos of "MUH FREEDOMS" and enacting a very strong moderation regime with clear rules about topics, discussion, and harassment, and has dedicated mods who constantly and consistently strictly enforce them.
If this ADAPT subreddit could do the same it could work, but it takes a lot dedication and vigilance.
posted by Sangermaine at 2:26 PM on November 16 [+] [!]
The question this raises is whether or not there's advantages to hosting a given community on reddit, and if those advantages are drowned out by the disadvantages. As I see it there is one chief advantage to using reddit: being on reddit might raise your community's profile, resulting in an influx of users that might be necessary to get the community to a sustainable size. However, there are two chief disadvantages: first, the new users will tend to be, well, redditors. Second, because reddit uses a rating system that surfaces high-rated comments and buries low-rated comments, preferential attachment effects come into play. Users upboat the comments and posts they like, making those comments and posts more visible, making them more likely to get upboated, making them more visible, rinse, lather, repeat. Meanwhile, comments that get downgoated are pushed to the bottom and greyed out, meaning that most users don't even look at downgoated comments at all. As a result of this, whichever viewpoints are held by a plurality of your userbase will tend to be visible and whichever viewpoints contradict the viewpoints of a plurality of your userbase will tend to disappear altogether. This is even if your plurality-holding faction is relatively small in absolute terms, because the preferential attachment machine will still magnify their views and suppress everyone else's.
So to recap: the advantage of putting your community on reddit is you might get a lot of users. the disadvantages are that the views of a plurality of your users will drown out the views of everyone else — and, no matter how good your moderation is, a plurality of those users are going to be redditors.
tl;dr: oh god don't put things like this on reddit it's the worst idea.
posted by You Can't Tip a Buick at 3:07 PM on November 16, 2015 [5 favorites]
If this ADAPT subreddit could do the same it could work, but it takes a lot dedication and vigilance.
posted by Sangermaine at 2:26 PM on November 16 [+] [!]
The question this raises is whether or not there's advantages to hosting a given community on reddit, and if those advantages are drowned out by the disadvantages. As I see it there is one chief advantage to using reddit: being on reddit might raise your community's profile, resulting in an influx of users that might be necessary to get the community to a sustainable size. However, there are two chief disadvantages: first, the new users will tend to be, well, redditors. Second, because reddit uses a rating system that surfaces high-rated comments and buries low-rated comments, preferential attachment effects come into play. Users upboat the comments and posts they like, making those comments and posts more visible, making them more likely to get upboated, making them more visible, rinse, lather, repeat. Meanwhile, comments that get downgoated are pushed to the bottom and greyed out, meaning that most users don't even look at downgoated comments at all. As a result of this, whichever viewpoints are held by a plurality of your userbase will tend to be visible and whichever viewpoints contradict the viewpoints of a plurality of your userbase will tend to disappear altogether. This is even if your plurality-holding faction is relatively small in absolute terms, because the preferential attachment machine will still magnify their views and suppress everyone else's.
So to recap: the advantage of putting your community on reddit is you might get a lot of users. the disadvantages are that the views of a plurality of your users will drown out the views of everyone else — and, no matter how good your moderation is, a plurality of those users are going to be redditors.
tl;dr: oh god don't put things like this on reddit it's the worst idea.
posted by You Can't Tip a Buick at 3:07 PM on November 16, 2015 [5 favorites]
Even researchgate is kind of crap. They're incredibly spammy.
I suppose if I put the work in to load all the pubs, and keyword them and devoted a portion of my week to it to keep current, it might work. But the question I keep coming back to is why? What benefit would it have to my work? Right now it just seems like a timesink.
posted by bonehead at 3:10 PM on November 16, 2015
I suppose if I put the work in to load all the pubs, and keyword them and devoted a portion of my week to it to keep current, it might work. But the question I keep coming back to is why? What benefit would it have to my work? Right now it just seems like a timesink.
posted by bonehead at 3:10 PM on November 16, 2015
"Well, sure...but how are you going to field 'I WANT TO FUCK YOUR FACE', within the context of criticism."
"We appreciate the reviewer's input, but this work is beyond the scope of the present report."
posted by mr_roboto at 3:12 PM on November 16, 2015 [7 favorites]
"We appreciate the reviewer's input, but this work is beyond the scope of the present report."
posted by mr_roboto at 3:12 PM on November 16, 2015 [7 favorites]
…those users are going to be redditors.
To be fair, they're going to be redditors who read computer science articles for fun in their spare time. So, probably people in your field with an interest in reading new papers. If anything, it democratizes academia. It also provides an entry for participation from people without access, that is people who are not in graduate school for all kinds of unfortunate reasons.
posted by esprit de l'escalier at 3:14 PM on November 16, 2015 [2 favorites]
To be fair, they're going to be redditors who read computer science articles for fun in their spare time. So, probably people in your field with an interest in reading new papers. If anything, it democratizes academia. It also provides an entry for participation from people without access, that is people who are not in graduate school for all kinds of unfortunate reasons.
posted by esprit de l'escalier at 3:14 PM on November 16, 2015 [2 favorites]
If your work is available in full text via a platform like ResearchGate, it is more likely to be read - or at least it's more likely to be cited. Multiple studies of open access have found this to be true. This boosts your h-index which helps get you tenure and then you're finally free to ask the controversial questions.
So basically it's:
1. Open access
2. ???
3. Profit
I'm bring intentionally snarky. It's my view that publishing is the most important thing we do as scientists as it advances knowledge. Doing experiments and learning things is meaningless unless those results are reviewed and shared. Making our work available is almost a moral imperative in my view. It takes time, but it matters very much.
posted by sockermom at 3:16 PM on November 16, 2015 [1 favorite]
So basically it's:
1. Open access
2. ???
3. Profit
I'm bring intentionally snarky. It's my view that publishing is the most important thing we do as scientists as it advances knowledge. Doing experiments and learning things is meaningless unless those results are reviewed and shared. Making our work available is almost a moral imperative in my view. It takes time, but it matters very much.
posted by sockermom at 3:16 PM on November 16, 2015 [1 favorite]
Reddit is not a social network. Reddit is a web site.
posted by If only I had a penguin... at 3:17 PM on November 16, 2015 [1 favorite]
posted by If only I had a penguin... at 3:17 PM on November 16, 2015 [1 favorite]
Reddit is a social network in exactly the same way MetaFilter is a social network.
It's just a whole lot of message boards with a large ratio of noise to signal. It's Usenet for the web-only generation.
posted by rokusan at 3:18 PM on November 16, 2015
It's just a whole lot of message boards with a large ratio of noise to signal. It's Usenet for the web-only generation.
posted by rokusan at 3:18 PM on November 16, 2015
If it matters, host it yourself, on machines you control, in a domain you own.
In the words of Dr. Bronner, "Exceptions eternally? Absolute none!"
posted by sourcequench at 3:19 PM on November 16, 2015 [2 favorites]
In the words of Dr. Bronner, "Exceptions eternally? Absolute none!"
posted by sourcequench at 3:19 PM on November 16, 2015 [2 favorites]
Okay, Reddit peer review. Can you replicate our results?
¯\_(ツ)_/¯
posted by rokusan at 3:20 PM on November 16, 2015 [3 favorites]
¯\_(ツ)_/¯
posted by rokusan at 3:20 PM on November 16, 2015 [3 favorites]
As a result of this, whichever viewpoints are held by a plurality of your userbase will tend to be visible and whichever viewpoints contradict the viewpoints of a plurality of your userbase will tend to disappear altogether.
And this would magnify the problem of a pervasive bias among the general population - which may be Reddit specifically here, but also would be a problem where reviewers do not have to be vetted.
A really good example of how this can be a problem is the way Reddit reacts to any research that can be interpreted as having a pro-feminist conclusion, e.g. research on language discrimination. In my experience, Redditors are primed to find any fault - even nonexistent faults - with any research that has to do with unequal treatment of women. They are, to put it bluntly, really fucking defensive. And this isn't something you can simply moderate away, because it is often expressed more as a trend of unfairly harsh treatment than of a single comment or user stepping over the line.
Crowd-sourcing peer review won't have as severe a problem with this when it's a subject that doesn't challenge societal biases. There may be some people who are really, really attached to certain theoretical models in computer science - but in general, it's not so challenging emotionally.
posted by Kutsuwamushi at 3:26 PM on November 16, 2015 [5 favorites]
And this would magnify the problem of a pervasive bias among the general population - which may be Reddit specifically here, but also would be a problem where reviewers do not have to be vetted.
A really good example of how this can be a problem is the way Reddit reacts to any research that can be interpreted as having a pro-feminist conclusion, e.g. research on language discrimination. In my experience, Redditors are primed to find any fault - even nonexistent faults - with any research that has to do with unequal treatment of women. They are, to put it bluntly, really fucking defensive. And this isn't something you can simply moderate away, because it is often expressed more as a trend of unfairly harsh treatment than of a single comment or user stepping over the line.
Crowd-sourcing peer review won't have as severe a problem with this when it's a subject that doesn't challenge societal biases. There may be some people who are really, really attached to certain theoretical models in computer science - but in general, it's not so challenging emotionally.
posted by Kutsuwamushi at 3:26 PM on November 16, 2015 [5 favorites]
I'm not sure exactly what the definition of "social network" is, because I'm one of those dinosaurs who pines for the 1990s web. but generally (just to short circuit the standard discussion that comes up) it's best to understand reddit as a web community with a distinct ethos and userbase, rather than as just a platform for web communities. this is because in practical terms, despite the existence of web communities on reddit that consciously work differently from reddit as a whole (askhistorians, shitredditsays, a few others), the site does in fact have an overarching ethos. Generally reddit admins and pr people promote this aspect of reddit (when talking about how much money reddit raises for charities, when talking about how you know it's reddit when the narwhal bacons at midnight or whatever), except when it's tactically necessary to distance themselves from aspects of the reddit community and its ethos (as when they claimed to just be a platform when people called them out on the creepshots subreddit and the other subreddits that had been moderated by violentacrez).
So long story short, reddit is a platform for a web community, managed by people who sometimes have to present it as a platform full stop in order to distance themselves from the actions of the community.
posted by You Can't Tip a Buick at 3:26 PM on November 16, 2015 [2 favorites]
So long story short, reddit is a platform for a web community, managed by people who sometimes have to present it as a platform full stop in order to distance themselves from the actions of the community.
posted by You Can't Tip a Buick at 3:26 PM on November 16, 2015 [2 favorites]
Yes, social networks can substitute for peer review because peer review itself is simply a highly specialized social network, or perhaps asocial network if you prefer . There is a different formulation actually being asked here though :
Now there is no more reason to expect a discussion form to out perform a more well thought out design than to expect reddit or ask.mefi to outperform stackoverflow.org at answering subtle programming question. It'll still be useful to see how the discussion form handles the job though. Anyone know if the stackexchange guys ever spoke about their experiences with sites like reddit or ask.mefi that predated starting stackoverflow?
posted by jeffburdges at 3:39 PM on November 16, 2015 [2 favorites]
What should an electronic social network designed to replace anonymous peer review and editorial work look like?There is a strong case that the existing system of peer review is overly rigid, wasteful, etc. We could presumably adopt some rigid formulation like mathoverflow.org does for questions, that's still much less rigid than the current system. We do not however know what the right formulation is, perhaps even a simple discussion form with social norms and moderators could work, or at least shed light on the question.
Now there is no more reason to expect a discussion form to out perform a more well thought out design than to expect reddit or ask.mefi to outperform stackoverflow.org at answering subtle programming question. It'll still be useful to see how the discussion form handles the job though. Anyone know if the stackexchange guys ever spoke about their experiences with sites like reddit or ask.mefi that predated starting stackoverflow?
posted by jeffburdges at 3:39 PM on November 16, 2015 [2 favorites]
It's worth noting that Reddit is subject to manipulation by monied interests, often at large scale. At this point, I suspect Reddit's funders see this as a feature rather than a bug.
Also, the pedantry can drive me insane.
How about a funded grant to do this on MeFi? It's a much better model for discussion.
posted by underflow at 3:41 PM on November 16, 2015 [4 favorites]
Also, the pedantry can drive me insane.
How about a funded grant to do this on MeFi? It's a much better model for discussion.
posted by underflow at 3:41 PM on November 16, 2015 [4 favorites]
metafilter doesn't seem remotely suited to anything like this — the userbase is small and the pace of the site is slow, and even more than reddit there is a very distinct site culture (well, okay, two really distinct site cultures, one primarily colored blue, the other primarily colored green). Metafilter is good at Metafiltering, reddit is good at redditing, and neither is particularly good for peer review.
(note: contemporary peer review processes in most fields aren't particularly good at peer review either, but that's neither here nor there.)
posted by You Can't Tip a Buick at 3:55 PM on November 16, 2015 [1 favorite]
(note: contemporary peer review processes in most fields aren't particularly good at peer review either, but that's neither here nor there.)
posted by You Can't Tip a Buick at 3:55 PM on November 16, 2015 [1 favorite]
Anyone interested in building a more advanced social network for peer review should check out arxiv.org, peruse the meta.SE list of open source SE clones, and spend some quality time playing around with issue trackers, github.com, etc.
I'd suspect you'll want basically an issue tracker in which issue visibility is optimized to keep people from reading crap, and where issues are closed by either the creator or a trusted-user voting process, not the article submitter.
In particular, the issue closing power should be reversed from how issues get resolved in normal issue trackers, github, etc. where the project maintainers hold the reins. And authors should use github for writing the article, not the review system itself, for that reason.
posted by jeffburdges at 4:03 PM on November 16, 2015 [3 favorites]
I'd suspect you'll want basically an issue tracker in which issue visibility is optimized to keep people from reading crap, and where issues are closed by either the creator or a trusted-user voting process, not the article submitter.
In particular, the issue closing power should be reversed from how issues get resolved in normal issue trackers, github, etc. where the project maintainers hold the reins. And authors should use github for writing the article, not the review system itself, for that reason.
posted by jeffburdges at 4:03 PM on November 16, 2015 [3 favorites]
I recall the nonlinear interest group at AGeologicalU a few years ago proposing a "bubbling up" publication system - everything on arXiv first, discussion and revision and publication public, with not -necessarily-isomorphic groups filtering and improving and publicizing papers in their own hierarchies of importance. Can't find mention of it tho.
posted by clew at 4:28 PM on November 16, 2015 [1 favorite]
posted by clew at 4:28 PM on November 16, 2015 [1 favorite]
To be fair, they're going to be redditors who read computer science articles for fun in their spare time. So, probably people in your field with an interest in reading new papers. If anything, it democratizes academia. It also provides an entry for participation from people without access, that is people who are not in graduate school for all kinds of unfortunate reasons.
So I am sort of interested in this as someone in economics, where research is both highly relevant to peoples' lives and is complicated. The comment section on any given economics blog is 90% likely to be terrible; you'll have some comments that are substantive, and the rest are some version of pet issue pushing, crankery, and political screed. It's unfortunate because economists should really try to make their stuff understood by the public and it's usually relevant to current debate over policy but the existing experiment of ``democratizing'' whereby someone - a blogger on their own space or a newspaper, say - publishes a link to the article with commenting enabled, doesn't work well at all, in my view. (Some blogs are better than others, obviously. But you get a lot of crypto-antisemitism in anything about banking, for example).
Reddit's platform seems like it would be particularly vulnerable to groups of users pushing some particular idea and letting a monetary policy article's failure to consider the gold standard or bitcoin or something appear to casual (nonacademic) readers as if it were a major flaw. So like, even if academics would know to filter it out, the general public may not. I don't quite know what to do about this because keeping the research between academics and, basically, technocrats (which is a lot of what seems to happen nowadays) doesn't help the public either.
posted by dismas at 4:34 PM on November 16, 2015 [4 favorites]
So I am sort of interested in this as someone in economics, where research is both highly relevant to peoples' lives and is complicated. The comment section on any given economics blog is 90% likely to be terrible; you'll have some comments that are substantive, and the rest are some version of pet issue pushing, crankery, and political screed. It's unfortunate because economists should really try to make their stuff understood by the public and it's usually relevant to current debate over policy but the existing experiment of ``democratizing'' whereby someone - a blogger on their own space or a newspaper, say - publishes a link to the article with commenting enabled, doesn't work well at all, in my view. (Some blogs are better than others, obviously. But you get a lot of crypto-antisemitism in anything about banking, for example).
Reddit's platform seems like it would be particularly vulnerable to groups of users pushing some particular idea and letting a monetary policy article's failure to consider the gold standard or bitcoin or something appear to casual (nonacademic) readers as if it were a major flaw. So like, even if academics would know to filter it out, the general public may not. I don't quite know what to do about this because keeping the research between academics and, basically, technocrats (which is a lot of what seems to happen nowadays) doesn't help the public either.
posted by dismas at 4:34 PM on November 16, 2015 [4 favorites]
In particular, the issue closing power should be reversed from how issues get resolved in normal issue trackers, github, etc. where the project maintainers hold the reins. And authors should use github for writing the article, not the review system itself, for that reason.
Peer review looks like a natural fit for GitHub or similar to me, though I'm in the computational world where the metaphors line up a little more easily. The workflow would be to write a paper with your co-authors in your own repo, then submit a pull request (perhaps with the edit history "squashed") containing your manuscript to a repo operated by a journal. The peer review would take place in the issue thread generated by that pull request. The "maintainers" (editors) of the journal repo would decide when/if to accept the manuscript / pull request. GitHub webhooks could be used to trigger the print publication workflow on the backend. Anonymous comments would have to be done through e-mails to the editor, though perhaps an extension for this could eventually be written.
The way to do print copies would probably be through branching ("Q4 2015" branch, etc.), so that the publishing backend knows which files to collect, as well as handling cases where corrigenda (corrections) get issued in the future.
As an avid user of Git, I can imagine how it'd be a pretty nice system, but the learning curve of Git largely precludes near-term adoption short of a Hercluean effort to educate everyone on version control. (Software Carpentry is trying; God help them.)
posted by anifinder at 4:35 PM on November 16, 2015 [3 favorites]
Peer review looks like a natural fit for GitHub or similar to me, though I'm in the computational world where the metaphors line up a little more easily. The workflow would be to write a paper with your co-authors in your own repo, then submit a pull request (perhaps with the edit history "squashed") containing your manuscript to a repo operated by a journal. The peer review would take place in the issue thread generated by that pull request. The "maintainers" (editors) of the journal repo would decide when/if to accept the manuscript / pull request. GitHub webhooks could be used to trigger the print publication workflow on the backend. Anonymous comments would have to be done through e-mails to the editor, though perhaps an extension for this could eventually be written.
The way to do print copies would probably be through branching ("Q4 2015" branch, etc.), so that the publishing backend knows which files to collect, as well as handling cases where corrigenda (corrections) get issued in the future.
As an avid user of Git, I can imagine how it'd be a pretty nice system, but the learning curve of Git largely precludes near-term adoption short of a Hercluean effort to educate everyone on version control. (Software Carpentry is trying; God help them.)
posted by anifinder at 4:35 PM on November 16, 2015 [3 favorites]
Reddit is a social network in exactly the same way MetaFilter is a social network.
It's just a whole lot of message boards with a large ratio of noise to signal. It's Usenet for the web-only generation.
Metafilter is not a social network. It's a website. Reddit, facebook, and twitter are web sites. Metafilter, Reddit, facebook, twitter, Starbucks , the park down the street, workplaces and my living room, are places where social networks can form and people in social networks can engage one another, but they are not social networks, they are web sites, parks, workplaces and living rooms.
And no, reddit should not substitute for peer review.
posted by If only I had a penguin... at 4:47 PM on November 16, 2015
It's just a whole lot of message boards with a large ratio of noise to signal. It's Usenet for the web-only generation.
Metafilter is not a social network. It's a website. Reddit, facebook, and twitter are web sites. Metafilter, Reddit, facebook, twitter, Starbucks , the park down the street, workplaces and my living room, are places where social networks can form and people in social networks can engage one another, but they are not social networks, they are web sites, parks, workplaces and living rooms.
And no, reddit should not substitute for peer review.
posted by If only I had a penguin... at 4:47 PM on November 16, 2015
I'm dubious that github pull requests would work better than a sub-reddit, anifinder. Wouldn't it get confusing with all those issues associated to the journal instead of associated to the articles? At least the sub-reddit separates them cleanly.
Ain't clear if you want a journal in any normal sense either, maybe just an approval process sans any editing together, or maybe the editing should constitute a "who should read this?" recommendation/ranking. You might want a smooth system for issues to subdivide too.
posted by jeffburdges at 5:29 PM on November 16, 2015
Ain't clear if you want a journal in any normal sense either, maybe just an approval process sans any editing together, or maybe the editing should constitute a "who should read this?" recommendation/ranking. You might want a smooth system for issues to subdivide too.
posted by jeffburdges at 5:29 PM on November 16, 2015
On GitHub you can use the Issue Dashboard to view all the issues you're associated with in any way: https://github.com/issues
It's possible to sort and filter the list by organization and other criteria. The key feature for me over a subreddit is that using Git, you're able to perfectly track, line by line, all of the changes made during the review process.
I don't have a problem with the concept of a curated periodic publication curated by subject-matter experts. I see a Git-based approach mostly about making the existing workflow more transparent and traceable, for accountability / credit purposes.
posted by anifinder at 6:15 PM on November 16, 2015
It's possible to sort and filter the list by organization and other criteria. The key feature for me over a subreddit is that using Git, you're able to perfectly track, line by line, all of the changes made during the review process.
I don't have a problem with the concept of a curated periodic publication curated by subject-matter experts. I see a Git-based approach mostly about making the existing workflow more transparent and traceable, for accountability / credit purposes.
posted by anifinder at 6:15 PM on November 16, 2015
I should add that at least one good argument against this idea would be that software has to operate as a coherent whole whereas academic publications are largely expected to stand alone with respect to others in the same issue of a journal, so perhaps the fork+pull request workflow doesn't really make sense.
posted by anifinder at 6:21 PM on November 16, 2015
posted by anifinder at 6:21 PM on November 16, 2015
And this isn't just anecdotal: social scientists studying Reddit have made a good case for why the design of the system actually fosters unpleasant and often mysognistic behavior.
Man I would really like to read the paper but it's paywalled.
This research is informed by the results of a long-term participant-observation and ethnographic study into Reddit’s culture and community and is grounded in actor-network theory.
i.e. "I have been using Reddit for years and decided to make a paper out of it." Not really making fun, this is just in itself an interesting picture of the difference between submitting a theory about Reddit on Reddit and submitting a theory about Reddit to an academic publication.
posted by atoxyl at 6:40 PM on November 16, 2015
Man I would really like to read the paper but it's paywalled.
This research is informed by the results of a long-term participant-observation and ethnographic study into Reddit’s culture and community and is grounded in actor-network theory.
i.e. "I have been using Reddit for years and decided to make a paper out of it." Not really making fun, this is just in itself an interesting picture of the difference between submitting a theory about Reddit on Reddit and submitting a theory about Reddit to an academic publication.
posted by atoxyl at 6:40 PM on November 16, 2015
As an avid user of Git, I can imagine how it'd be a pretty nice system, but the learning curve of Git largely precludes near-term adoption short of a Hercluean effort to educate everyone on version control. (Software Carpentry is trying; God help them.)
The problem here is not really that version control is hard to learn but that Git is - not just because of its (powerful but not perhaps entirely necessary for your purposes) complexity but its particularly bizarre and inconsistent interface. Github's GUI helps hide some of this but I'm sure you could do much better with a system built from scratch for non-programmers.
posted by atoxyl at 6:48 PM on November 16, 2015
The problem here is not really that version control is hard to learn but that Git is - not just because of its (powerful but not perhaps entirely necessary for your purposes) complexity but its particularly bizarre and inconsistent interface. Github's GUI helps hide some of this but I'm sure you could do much better with a system built from scratch for non-programmers.
posted by atoxyl at 6:48 PM on November 16, 2015
I don't care if it's good software or bad software, hosting a computer science peer-review on Reddit is just a giant flashing sign saying "WHITE BOYS ONLY" (or something way, way, way less polite than that.)
Reddit may be many things to many people, but it has a reputation, and I certainly wouldn't want to associate with it if I were a professional science organization, and I would be wary of any such organization that did so because I would assume they either (a) approved of Reddit's reputation or (b) were blissfully unaware of it.
posted by mmoncur at 7:13 PM on November 16, 2015 [5 favorites]
Reddit may be many things to many people, but it has a reputation, and I certainly wouldn't want to associate with it if I were a professional science organization, and I would be wary of any such organization that did so because I would assume they either (a) approved of Reddit's reputation or (b) were blissfully unaware of it.
posted by mmoncur at 7:13 PM on November 16, 2015 [5 favorites]
This research is informed by the results of a long-term participant-observation and ethnographic study into Reddit’s culture and community and is grounded in actor-network theory.
i.e. "I have been using Reddit for years and decided to make a paper out of it."
No, that's not what that means.
posted by If only I had a penguin... at 7:15 PM on November 16, 2015 [2 favorites]
i.e. "I have been using Reddit for years and decided to make a paper out of it."
No, that's not what that means.
posted by If only I had a penguin... at 7:15 PM on November 16, 2015 [2 favorites]
Yes, we'd definitely want to "track line by line, all of the changes made during the review process", so git and github's line comments makes a good basis.
There are however critical differences : Authors are sometimes maliciously trying to game the system, so they cannot be trusted with control over the issues database. At the same time, reviewers and editors interact only in a temporary way with the work, and have no independent interest in it, so they must detect deceit far more quickly than OSS collaborators.
I'd propose this github workflow :
Authors write article either using github to collaborate or posting a repo for submission. Authors submit the article by sending a fork request to a journal, which forks the article's repo into their queue account. Issue resolution is not under the author's control in this fork, but neither is the fork a part of a larger project with unrelated issues.
Articles are considered done when commenters reach a consensus that all issues were handled reasonably, deemed out of scope, etc. Article wikis are used to "rank" articles by reaching a consensus as to who should read the article. Articles are forked into the journal's accepted account once they're considered both done and ranked with a suitable recommended readership.
posted by jeffburdges at 8:02 PM on November 16, 2015 [1 favorite]
There are however critical differences : Authors are sometimes maliciously trying to game the system, so they cannot be trusted with control over the issues database. At the same time, reviewers and editors interact only in a temporary way with the work, and have no independent interest in it, so they must detect deceit far more quickly than OSS collaborators.
I'd propose this github workflow :
Authors write article either using github to collaborate or posting a repo for submission. Authors submit the article by sending a fork request to a journal, which forks the article's repo into their queue account. Issue resolution is not under the author's control in this fork, but neither is the fork a part of a larger project with unrelated issues.
Articles are considered done when commenters reach a consensus that all issues were handled reasonably, deemed out of scope, etc. Article wikis are used to "rank" articles by reaching a consensus as to who should read the article. Articles are forked into the journal's accepted account once they're considered both done and ranked with a suitable recommended readership.
posted by jeffburdges at 8:02 PM on November 16, 2015 [1 favorite]
I think this Reddit review system could really work. Then again, I was considering publishing in The Journal of Entitled Misogynist Manbabies.
posted by benzenedream at 9:14 PM on November 16, 2015 [2 favorites]
posted by benzenedream at 9:14 PM on November 16, 2015 [2 favorites]
Also, github is a commercial entity that one day will go the way of sourceforge. Adding features to arXive for collaboration would be better in the long run.
posted by benzenedream at 9:25 PM on November 16, 2015 [2 favorites]
posted by benzenedream at 9:25 PM on November 16, 2015 [2 favorites]
Reddit in general is terrible, but it is possible to maintain a quality subreddit.
I dunno, man. You can also install your toilet with the drain angled uphill. Both strategies take a lot more work than the established norm, require active intervention at all times, and tend to revert to a base state of "covered in feces."
posted by Mayor West at 4:46 AM on November 17, 2015 [1 favorite]
I dunno, man. You can also install your toilet with the drain angled uphill. Both strategies take a lot more work than the established norm, require active intervention at all times, and tend to revert to a base state of "covered in feces."
posted by Mayor West at 4:46 AM on November 17, 2015 [1 favorite]
« Older Football is a country | Basically, a chance to get wasted dancing... Newer »
This thread has been archived and is closed to new comments
I know that Reddit isn't just one thing, but as a woman who was mildly harassed there for literally no reason other than that I dared to say I was a woman on the Internet (some creep went through my entire posting history and then posted my old comments whenever I made a new comment to "prove" that I wasn't "trustworthy") I would not want to use Reddit to make comments about science. No thanks. And this isn't just anecdotal: social scientists studying Reddit have made a good case for why the design of the system actually fosters unpleasant and often mysognistic behavior.
These guys are smart enough to develop their own system for peer review, one that doesn't hinge on a broken piece of social software (Reddit is not really a social network at all, in my view) to get it done. Why not grow their own?
Our values are echoed by the systems we choose to use when we express those values. I think this is more about a lack of critical thinking about the system they chose, but I wish they would choose a different platform.
posted by sockermom at 2:14 PM on November 16, 2015 [24 favorites]