Taking peer review to the Internet

November 9, 2011 5:16 PM

Something that allows you to easily slice through misinformation online online could make misleading the public a lot tougher. The concept isn't new (previously & many others) but unlike those efforts, Hypothes.is is open, standards-based, and backed by some web heavy-weights.
posted by ATXile (59 comments total)

I reserved my username and am looking mighty forward to see what develops
posted by Renoroc at 5:22 PM on November 9, 2011 [2 favorites]

What about people who don't trust experts for fear they are running a conspiracy, like Climate Change deniers?

People will believe what they want to believe. But if this works, I think it could be a great resource for people willing to learn.
posted by mccarty.tim at 5:24 PM on November 9, 2011

So who wants to start the pool for the inevitable Conservapedia analogue to this?
posted by meinvt at 5:34 PM on November 9, 2011 [6 favorites]

"...the inevitable Conservapedia analogue..."

Oh man, I wish I had made that comment.
posted by bz at 5:48 PM on November 9, 2011

Looks promising. Though given comments in most internet threads, we can assume annotations will be equally puerile.
posted by stbalbach at 5:49 PM on November 9, 2011

Something that allows you to easily slice through misinformation online online could make misleading the public a lot tougher iff the public gave two hoots about being misled. Most people vastly prefer a feeling of certainty to actual knowledge, which is why Fox and friends continue to thrive.

That said, I've reserved my username. Let's see what happens.
posted by flabdablet at 5:53 PM on November 9, 2011 [1 favorite]

I'm intrigued by the ubiquity of this and the tech platform but I skeptical that this is a problem that wants/can be solved. Still, I reserved my username and look forward to try this on.
posted by Foci for Analysis at 5:57 PM on November 9, 2011

The project in this post, Hypothes.is, is raising a $100k round on Kickstarter right now, currently at $68k. Today we just announced that Sunil Paul will match everything we raise there. All gifts are 100% tax-deductible, as Hypothes.is is a non-profit. Dan Whaley, Founder. Thank you for helping us turn this into a reality. http://kck.st/nbvi0G
posted by dwhly at 5:59 PM on November 9, 2011 [12 favorites]

An earlier announcement about this technology. (1993, Marc Andreessen). "We've got ~50MB free for this; I certainly wouldn't mind filling it up." It is actually really cool to read the thread that this email started and see what was being imagined at the time.
posted by cgk at 6:06 PM on November 9, 2011 [2 favorites]

Looks like someone there might be reading this thread; they tweeted cgk's link 6 minutes after it was posted here :)
posted by finite at 6:14 PM on November 9, 2011

looks cool, i'm going to check it out. thanks!
posted by facetious at 6:15 PM on November 9, 2011

"Looks like someone there might be reading this thread; they tweeted cgk's link 6 minutes after it was posted here :)
posted by finite at 12:14 on November 10"

Yup, the founder is. And he's not just reading.
posted by moody cow at 6:19 PM on November 9, 2011 [1 favorite]

The "12 Principles" are all spot-on. I was also glad to see the video acknowledge the multitude of previous failed attempts at building something like this.

I'm excited!
posted by finite at 6:21 PM on November 9, 2011

Does this thing actually exist yet in any reasonably discussable form? The current website seems like a glorified press release or elevator pitch, without even a real description of the product, just a video full of earnest naivete about politics, the philosophy of language, and rhetoric.

What is this "information" and how do we "evaluate" it, exactly? Who gets to decide what information is mis- ? Not only do I not see anything approaching an answer to that question on the Hypothesis website, I don't see any signs of understanding how fundamental a question it is. It just seems like yet another incarnation of the eternal rationalist misunderstanding: if only we could sanitize all that ambiguity and disagreement out of language, and be left with the real facts, then no one would ever be able to lie to us again! But language doesn't work that way, and of course people don't agree about what the facts are.

If whatever Hypothesis ends up being works at all in practice, it will be because its users form a community with its own strong norms and viable consensus about what counts as a fact and what counts as good information, not because their norms are the universal ones.
posted by RogerB at 6:22 PM on November 9, 2011 [3 favorites]

I thought we weren't doing Kickstarter projects as FPP's anymore. Until they release it or at least describe it in much greater detail, this is just a fundraiser. There's nothing to look at, nothing to discuss, no reason for a post. Don't get me wrong, I've been waiting for somebody with resources to develop this for years now. I fully support the concept behind it; I would've built it myself if I had the resources. I'm sure I'll be an avid user. It just has no place here, at least not until it's actually released.
posted by scalefree at 6:22 PM on November 9, 2011 [4 favorites]

@finite. most definitely reading this thread. Just tweeted that Metafilter is driving more traffic than TC and RWW posts on us did. Never understood the power of this site. BTW... the Marc Andreessen piece is awesome. cgk has awesome recall!
posted by dwhly at 6:26 PM on November 9, 2011

Welcome, Dan. This looks like an interesting project. I had a bunch of comments, I will just throw out a couple, they are basically somewhat along the lines of RogerB's comments above, which is that the actual 'truth content' of many many sites is usually (deliberately) negotiable; so I'm not sure how annotations will lead to a convergence on a form of truth, if that is indeed what you are trying to achieve (and I may be wrong in this assumption).

Anyway: user name reserved ...
posted by carter at 6:29 PM on November 9, 2011

RogerB -- I understand that what we've got up right now is a glossy press release at best. Please forgive the early stage in the process, and that we're pulling together the best partners to make this happen. Development will begin in earnest shortly. As far as your question about what is "information", perhaps this piece on the Argumentative Theory of Reasoning will give you a bit more insight as to how we're approaching this. We'll be posting much more over the next few months.
posted by dwhly at 6:31 PM on November 9, 2011

If I was building it, it'd be a reputation-based voting scheme, where the value of your vote is weighted according to the number of other users supporting you, calculated recursively. Think PageRank for people. There's lots of ways to architect the information schema, from read-only snippets off other Internet sites to collaborative statements developed by affinity groups all the way up to complete manifestos or declarations of expertise across a broad domain of knowledge. But the underlying mechanism's still the same.

But that's just how I'd do it. We have no idea how these guys are doing it until they say something meaningful.
posted by scalefree at 6:31 PM on November 9, 2011 [1 favorite]

@Scalefree. That's exactly what we're doing. Reputation weighted voting. We're contemplating implementing that via meta-moderation (ala slashdot) but are not settled on that. Some complain about the meta-moderation overhead. We'll be very attuned to our early alpha community here.
posted by dwhly at 6:34 PM on November 9, 2011

Work everywhere
To the extent practical. Without consent.

This could be great. And a great source of lawsuits.
posted by DU at 6:36 PM on November 9, 2011

Previously.
posted by scalefree at 6:38 PM on November 9, 2011

@DU... the lawsuit phase is beyond us I think. Thirdvoice dealt with that, but more recent trial precedents around browser-based ad blockers seem to signal clear sailing here.
posted by dwhly at 6:49 PM on November 9, 2011 [1 favorite]

Oh, it's like Advogato plus Snopes!
posted by miyabo at 6:52 PM on November 9, 2011 [1 favorite]

Dan, have you thought about adding "suggested revision" to the list of annotation types? I want the the whole web to turn into a big distributed wiki!

I'd like anyone to be able to publish a diff which fixes a typo in, say, someone else's metafilter comment. People browsing through this software could have a configuration option which causes them to see the edited version by default (after it has received enough votes, perhaps specifically from people nearby the reader's web of trust).

I did notice "make a correction" in your list of annotation types in the video, but I really want this type of annotation to be an actual patch/diff so that the above scenario is possible and also so that a website owner can easily accept and apply the patch (if their CMS supports it).
posted by finite at 6:53 PM on November 9, 2011 [1 favorite]

Slideshare link with a bit more detail, for the video-allergic.

I think we should try things like this, and it's important. I also think it's a historically thorny problem, as RogerB points out, and dates back to Socrates. Something like Wikipedia is truth-by-consensus (or sometimes truth by most-recently-edited) which is one reality but not the only possible one. I think it's worthwhile to experiment with other belief systems and tools for helping people fit existing ideas into those.

What I have been unable to find is even a term that describes such systems (reason assistive technology?) or a place where people go to pontificate about such things. Do these things and these people exist?
posted by RobotVoodooPower at 6:57 PM on November 9, 2011 [2 favorites]

Hold up there, finite. I don't approve of any crowdsourced editing of my Metafilter comments.
posted by JaredSeth at 6:58 PM on November 9, 2011 [5 favorites]

s/nearby the reader's web of trust/nearby in the reader's web of trust/
posted by finite at 7:00 PM on November 9, 2011

Advogato trust metric.
posted by scalefree at 7:12 PM on November 9, 2011

They tweeted that this post generated more traffic than Reddit.

(Thus we're smarty-ier than Reddit. Nah nah boo boo.)
posted by k8t at 7:14 PM on November 9, 2011 [5 favorites]

People believe what they'd like to.
posted by a robot made out of meat at 7:23 PM on November 9, 2011

Wow, I'm amazed Advogato still exists. I swear I haven't visited that site since Kuro5hin was in vogue.
posted by miyabo at 7:26 PM on November 9, 2011 [2 favorites]

Ok - it wasn't their official twitter. Hmm.
posted by k8t at 7:28 PM on November 9, 2011

Hypothes.is could become a crucial tool for skeptics

Would Your Blog Stand up To Criticism? Here Comes Peer Review
posted by scalefree at 7:29 PM on November 9, 2011 [1 favorite]

Renoroc: "I reserved my username and am looking mighty forward to see what develops"

Did you get any notification of the reservation? The way it processed it with no feedback makes me think it didn't take.
posted by Samizdata at 8:24 PM on November 9, 2011

> detail, for the video-allergic

Thank you.
posted by hank at 8:32 PM on November 9, 2011

Well, ran the reservation in Chrome and it seemed to work.
posted by Samizdata at 8:34 PM on November 9, 2011

@finite We're totally open to new annotation "types". We call them 'stances' which comes from the rhetorical device. Suggested revision sounds like an interesting one. And I'll note that in our pending list. We'll be allowing new stances to be organically derived from the community. So we hope to rapidly converge on a list of the most obvious ones.

As far as integrating w/ CMS to auto-accept patch-diff? Hmm... interesting. Will noodle on that. Not sure most CMS systems are set up that way... though I'm not a student of what's available these days.
posted by dwhly at 9:01 PM on November 9, 2011 [1 favorite]

@samizdata. Confirmation should come by email. Let us know if that's not happening!
posted by dwhly at 9:02 PM on November 9, 2011

@RobotVoodooPower What is the term of art? There are a few... Collaborative Annotation. Collaborative Reasoning... Crowdsourced peer-review. Others? Yours?
posted by dwhly at 9:03 PM on November 9, 2011

Reputation Systems used to be a term for this in computer science...it's a somewhat discredited area these days though, a lot of the systems turned out to be easy to game.
posted by miyabo at 9:25 PM on November 9, 2011

Sounds like a badly needed service. I can think of a lot of questionable claims I'd like to mark up, such as nearly every sentence on Hypothes.is.
posted by michaelh at 9:26 PM on November 9, 2011 [2 favorites]

This seems like it would make a better post when it is a thing.
posted by Kwine at 10:01 PM on November 9, 2011 [1 favorite]

the internet is not about facts, it's about opinion

opinions are not facts; even if you 'prove' one of the rare factually-evaluatable opinions with a literal fact you haven't done very much except be 'right'
posted by This, of course, alludes to you at 10:49 PM on November 9, 2011

If someone had told me about this idea
in 1995 I would have though it not possible.
I did not believe that Wikipedia would make it.
Hell...I'm amazed that the public library works.
Bring it on, count me as a believer!
posted by quazichimp at 10:49 PM on November 9, 2011

Never understood the power of this site.

I think it would be worth your while to do so before hypothes.is goes live.

Crowd-sourced reputation systems are a fertile field, and if not carefully cultivated by skilled mods they will grow an awful lot of weeds and rubbish.
posted by flabdablet at 11:59 PM on November 9, 2011 [1 favorite]

Crowd-sourced reputation systems

shudder

if not carefully cultivated by skilled mods they will grow an awful lot of weeds and rubbish.

flinch, gag
posted by This, of course, alludes to you at 4:02 AM on November 10, 2011 [1 favorite]

Well I, for one, look forward to this. Reserved a username (not this one), received my confirmation email immediately, and hope to be able to participate soon.

Why not, right? I am constantly seeking a place to go that is not opinions. For opinions, I have a zillion places on the web to go to.
posted by sundrop at 4:16 AM on November 10, 2011

i am not going to argue that the concept of a "crowd-sourced reputation system"* is inherently vile, abusive and corrupt, but i can think of a few really decent reasons why that might be the case.

also,

I think it would be worth your while to do so before hypothes.is goes live.

sounds kind of threatening, as if i run the risk of being negatively evaluated should i refuse to operate on this thing's terms.

you talk about "cultivation", but i have to wonder exactly what you are growing here, and why something that is Strictly Factual** requires so much skill to deal with

also, reputations are "aesthetic" and therefore relative. i am not sure how you are going to "factually evaluate" a reputation at all, let alone without baking opinion into that evaluation

i would be a lot less antsy if i knew who got to be a weed or rubbish, you know?

*hey wasn't encyclopedia dramatica one of those
**not to say Rational
posted by This, of course, alludes to you at 4:22 AM on November 10, 2011 [1 favorite]

i am not sure how you are going to "factually evaluate" a reputation at all, let alone without baking opinion into that evaluation

The reputation of a contributor is supposed to factor into the credence one gives to the factual evaluations made by a contributor. No one else is talking about factually evaluating reputations themselves.
posted by Jpfed at 5:02 AM on November 10, 2011

The reputation of a contributor is supposed to factor into the credence one gives to the factual evaluations made by a contributor

incoming quote mark storm, but reputation with whom, though, and to what end? and doesn't everyone do that already? i am sorry if this is inane captious doubt or anything but this raises questions, to me. how do you "peer review" the internet? who says what the "best thinking" on something is? who are our "peers" and isn't statistically weighing them "objectively" what youtube does to select the best cat videos? what is new here?

also, "a powerful incentive for people to ensure that their works met a higher standard, and made it perceptibly harder to spread information that didn't meet that standard." doesn't actually state anything directly about factuality, but appears to be more of a generic call for some kind of judgement.

combine the fact that there is no real definition or elaboration on what that standard is beyond "the combined wisdom of the most informed people" with that bit about making it perceptibly harder to spread information has got my skin just crawling.

if the site is about evaluating academic claims, why not just say that? why be indirect and opaque? for some reason, this whole thing just has this pall around it, to me. shades of "curation".
posted by This, of course, alludes to you at 5:54 AM on November 10, 2011

It seems like a great project - I'm interested to see what will happen. Just one thought about the advisers; it would be interesting to have more ethnic and cultural diversity on the team so it can be globally relevant.
posted by a womble is an active kind of sloth at 5:56 AM on November 10, 2011

@womble as long as it's the right kind of diversity
posted by This, of course, alludes to you at 6:01 AM on November 10, 2011

sounds kind of threatening

Was absolutely not intended to be. All I'm saying is that there's a reason MeFi's influence is greater than obvious metrics would suggest it should be. In my opinion, it boils down to the fact that MeFi does operate as a genuine community, that its overarching value is generosity, and that the way moderation is done here plays a huge part in keeping that spirit healthy. It seems to me that a good understanding of the way MeFi operates could only be helpful to the designers of something like hypothes.is.
posted by flabdablet at 7:42 AM on November 10, 2011

I've only glanced at the 'prospectus', but the idea seems worth pursuing. It depends on what the annotations consist of. If confirmable *facts* float to the top, that'd be great. If "consulting" opinions float to the top - even informed opinions but without adequate *facts* - that would be better than, say, Youtube, but still not helpful.

The ever-lurking spoiler, of course, is invasions by paid, Ed Bernays schooled, spinmeisters. They've been all over the net for years... including (especially?) Wikipedia (where, for example, embarrassing, well-documented legal questions in bios of powerful figures may be quickly whitewashed or simply disappeared).

Will the "reputation" of economic "experts" with their cherry-picked "statistics" be weighted higher than balanced presentations by "disreputable" sources and "unwashed" outlyers whose POV are not fashionable? (At least pseudonyms are recognized - not the kind of "reputation" G+ endorses.)

The measure of reputation, I think, is decisive. Complex questions cannot be reduced so that everyone (unwilling or unable to learn) can understand them. Most of the problems we face today are complex. If reputation arises from glibness of expression, half-truths, or social connections... that's a fail. Facts are not democratic; the wisdom of the crowd that elected the current leadership of the US notwithstanding.
posted by Twang at 8:51 AM on November 10, 2011 [2 favorites]

Will the "reputation" of economic "experts" with their cherry-picked "statistics" be weighted higher than balanced presentations by "disreputable" sources and "unwashed" outlyers whose POV are not fashionable? (At least pseudonyms are recognized - not the kind of "reputation" G+ endorses.)

They should be weighted according to the number of people supporting their authority in the area under question, calculated recursively. You need to choose the right algorithm for resilience to various attacks (see here for an example).

In the end it'll still come down to "who do you trust?" but at least the lines of trust will be explicit & visible, putting more pressure on those selected as thought leaders.
posted by scalefree at 10:09 AM on November 10, 2011

Looks like a very fine idea. But it disturbs me a lot that the advisors and collaborators include so few women. If the founding leadership is willing to proceed without including a critical mass of at least 30% - 40% women, they're undermining their chances of success. I hesitate to invest in an organization which claims such lofty, aspirational goals, while their leadership roster demonstrates a willingness to perpetuate old errors. It's long past time to start doing better.
posted by Corvid at 6:25 PM on November 10, 2011

Bullsh.it
posted by quonsar II: smock fishpants and the temple of foon at 5:58 AM on November 11, 2011 [1 favorite]

Yes, its fun to be snarky, but ...

Bullsh.it

"Why exactly? Be more constructive with your feedback, please." - FOTC
posted by ATXile at 11:37 AM on November 12, 2011 [1 favorite]