Would a reputation/voting system oriented on peer-rewiew of scientific journals better fit science proposals than the current democratic system?

Question

I think I don't have to explain why and how peer review works in scientific journals.

While I think one user - one vote fits proposals where no long and higher education is needed pretty well, I see some downsides on scientific proposals. Esp. regarding questions in a BETA period.

I watched for some time the astronomy and philosophy BETAs here and a lot of questions are more on tricky black holes, dark matter physics, religion, god, death and speculative subjective reasoning than "real" astronomical and philosophical questions discussed in scientific journals.

My expectation is, that users with higher reputation due to good referenced answers and logic reasoning and higher vote count (for example, rep higher than 1000 means 2 points per vote (up/down) could much better regulate quality and development of such scientific BETAs than the mass of interested, but often without deeper background knowledge equipped, standard users.

Basically this is the function of peer-reviewers, and history has shown, that the scientific community needs them. Don't mix up peer-reviewers and moderators. They have similar functions, but not the same. A moderator has to filter and regulate, a peer-reviewer to filter and highlight important information/questions.

Also this might be a option to stop dividing of proposals in laymen and few experts - only experts (like physics.SE - theoreticalphysics.SE) I think such developments are a effect of this too democratic, linear voting, and reputations system for such proposals.

There are probably some downsides. But with so many different proposals in content, I strongly believe one voting and rep system can't fit all. Maybe it's technically not possible/feasible to set up different systems (I don't think so, it is basically just numbers) or Stack Exchange operators want that everybody doesn't have to be aware that his votes counts differently on different sites?

I don't think that more previliges of higher rep users compensate the curiosity for speculative and sensational questions. These privileges work mainly to filter out very bad questions, and this works. But a higher voting strength for higher rep users would highlight really on topic and interesting questions more.

It would be perfect if high rep users could decide by clicking from e.g. 1-3 time on a upvote arrow, how much votes they want to give a basic question, how much a very good on current research.

Plz give a comment or link to former similar discussion when downvoting, a downvote doesnt tell me whats wrong with this question, despite you dont like peer-review, which obviously works in scientific journals better than highly democratic systems like arxiv.

Answer 1

First of all, I just can't imagine SE team to change current voting scheme to something else for some sites... they would just stop being SE in that case.

Now back to merit. An upvote on SE is just a way to say "I like it" or (hopefully) "It was useful to me" and thus this is just flawed thinking to blindly assume that the highest upvoted answer is perfectly correct and the best solution -- as always, BTW not only in Internet. Also the correlation of vote score with actual truthness or usefulness is strongly dependent on the quality of the community. And it is obvious that at first it is dominated by "cool, X is on SE!" guys and the voting scores are very noisy.
However, good answers also appear, they attract Googlers really working with the subject and gradually the community becomes more serious and large enough to silence the noise that is inevitable on any open site. And the overall level should be increasing in such an evolutionary process.

Of course it is a theory (on AI good answers never appeared, Physics still drowns in discussions) but I can't really imagine a better system. Peer review is a very demanding thing coming along with great responsibility -- it would be very hard to recruit people to do this seriously. Your system of electing high-rep users has a problem that it is vulnerable to crooks that can earn rep on simple questions and use such gathered power to negate ideas they don't like.

Answer 2

Instead of proposing drastic, untested, and (many would say) unfair changes to SE. I would consider looking at successful research-level stack-exchanges, and figuring out what their secret sauce is. I know of only two such SEs:

cstheory and MO. I don't know nearly as much about the latter, and either way, it is an SE 1.0 that is hosted independently. However, I am active on cstheory and can comment a bit on why I think it is successful.

cstheory managed to attract and maintain the activity of many professors, and graduate students because it not only provides a Q&A form, but also a community. It was clearly focused enough in the beginning (instead of being unreasonably broad) and had a good number of dedicated academics that shaped it as an academic site. Now the community is almost exclusively academic and ruthlessly closes or migrates non-research level questions (or at least ones that would not be interesting to the scientists). This has stopped the community from being 'disolved' by non-experts as you say.

At no point did it use a different or alternative voting system.

If you want to make a research site, you don't need to make drastic and unfair voting systems. You just need a critical mass of academics that WANT a community and are willing to police it. If you want to create a site that caters to both academics and non-academics (such as the phyics.SE) then you have to tolerate low-level questions and you will lose academics. Some sites manage fine with this, for instance math.SE, and some not so well.

Answer 3

I would question just how much peer review actually contributes to the overall integrity of science in the long run. It's long been one of my areas of historical research. It's certainly a unfitting requirement for a general public site given that most peer reviewed journals are behind paywalls and/or requires a specialist knowledge to evaluate.

But the real reason is that "peer reviewed" doesn't mean "proven science." It means at best, "no big blunders" or at worse, "the voice of authorities of an entire field that has gone off the rails."

The real purpose of peer review is not to protect science but the reputation of the journals. It's basically proof reading to make sure the journals don't publish something blatantly wrong so their reader don't have to feel paranoid about everything they read.

Somewhere in the last 30 years, we've developed the idea that "peered reviewed" means "scientifically proven" and accepted by the scientific community. It doesn't. It just means, "Not overt quackery that would morbidly embarrass a publishers." Peer review checks form not facts.

Every bad idea in science in the last century has slipped right through peer review with glowing approval e.g. Eugenics. Go grab a copy of "Science" anytime before 1950 and just wade through the horror show.

The nigh worship of peer review has allowed some outright frauds to slip through and become accepted long before the real proof, reproduction of results, became known.

The prestigious Lancet got burned twice in the 90s, once by a completely false study claiming evidence that measles vaccinations caused autism. Turned out to be collusion between a corrupt researcher and a Ttrial lawyer. Yet in the years it took for followup studies to show the effect was not real, it had become widely accepted as true in the general public, courts, media and politics, largely because it was published in "peer reviewed" Lancet. I still see people refering to it long after the paper has been withdrawn.

Study did enormous damage, got some children killed and sailed through peer review.

It passed peer review because the faked data and methodology was faked with that intent. It's easy to slip faked data past peer review than than real world, warts and all data. Peer review does not reproduce the original research and rarely even looks at the raw data. It just looks for obvious mistakes in methodology, calculations and conclusions.

Circa 2000, Lancet published a study of infant mortality in Iraq under sanctions, showing staggering infant mortality. After the invasion/liberation, it was found the researchers had been shown bodies frozen and previewed over many months but told the infants died over just a few weeks. It passed through peer review because the data and methodology presented to the peer reviewers in valid form, just fake. The paper was withdrawn.

(The same team, not learning their lesson, tried again after the invasion/liberation. That paper remains on record despite it's findings never being reproduced by any other source. It's controversial.)

Peer review is just becoming a means of making arguments from authority. I've noted that their is a direct correlation between how often "peer review" shows up in public discussions of a science and that science's history of predictive power e.g. not often in physics, a lot in sociology.

I am an advocate of doing away with peer review entirely and replacing it with crowd reviewing. Peer review only existed in the first place because of the zero-sum nature of journal space and the expense of the old paper based workflow and distribution. Typos and mistakes of all kinds were expensive.

But in the internet age, why bother? Slap up the raw data, the raw methodology, your conclusion and let the crowd try to tear it to shreds. If it survives, then it can go on to reproduction.

The problem with garbage and noise on the internet isn't caused by a lack peer review, it's caused by 1) People wanting to believe the interesting or self-serving instead of disciplining themselves, and 2) By repeated arrogant authority overreach destroying a lack of trust in institutions.

As our problems get more complex, they require more and more narrow specialization on the part of experts but that leads to the problems of tunnel vision, insular subcultures, group think, hubris etc than can only be offset by a wider if naiver perspective. We need more eyes on the raw data and the raw science not fewer.

It won't be as pretty as the old Ivory tower blessing of authority of peer review but it won't make the same mistakes nor allow them to persist as long. Oliver Wendell Holmes Jr. might not have not been so quick to make "Three generations of morons is enough" the high law of the land had he had some other scientific input other than the most respective Ivy League scientist of the day (who turned out to have to missed a few things.)

This type of open forum is actually a good start. The real key to get people to think scientifically, is to get them to understand methodology, not to accept pronouncements of authority. Science is methodology, not fact. The accuracy and precision of scientific statement derive wholly from methodology. Teach people that, and you can forget the journals.