-2

I think I don't have to explain why and how peer review works in scientific journals.

While I think one user - one vote fits proposals where no long and higher education is needed pretty well, I see some downsides on scientific proposals. Esp. regarding questions in a BETA period.

I watched for some time the astronomy and philosophy BETAs here and a lot of questions are more on tricky black holes, dark matter physics, religion, god, death and speculative subjective reasoning than "real" astronomical and philosophical questions discussed in scientific journals.

My expectation is, that users with higher reputation due to good referenced answers and logic reasoning and higher vote count (for example, rep higher than 1000 means 2 points per vote (up/down) could much better regulate quality and development of such scientific BETAs than the mass of interested, but often without deeper background knowledge equipped, standard users.

Basically this is the function of peer-reviewers, and history has shown, that the scientific community needs them. Don't mix up peer-reviewers and moderators. They have similar functions, but not the same. A moderator has to filter and regulate, a peer-reviewer to filter and highlight important information/questions.

Also this might be a option to stop dividing of proposals in laymen and few experts - only experts (like physics.SE - theoreticalphysics.SE) I think such developments are a effect of this too democratic, linear voting, and reputations system for such proposals.

There are probably some downsides. But with so many different proposals in content, I strongly believe one voting and rep system can't fit all. Maybe it's technically not possible/feasible to set up different systems (I don't think so, it is basically just numbers) or Stack Exchange operators want that everybody doesn't have to be aware that his votes counts differently on different sites?

I don't think that more previliges of higher rep users compensate the curiosity for speculative and sensational questions. These privileges work mainly to filter out very bad questions, and this works. But a higher voting strength for higher rep users would highlight really on topic and interesting questions more.

It would be perfect if high rep users could decide by clicking from e.g. 1-3 time on a upvote arrow, how much votes they want to give a basic question, how much a very good on current research.

Plz give a comment or link to former similar discussion when downvoting, a downvote doesnt tell me whats wrong with this question, despite you dont like peer-review, which obviously works in scientific journals better than highly democratic systems like arxiv.

  • I think the problem with this is the initial assumption that the peer review system of academia actually does work... – Michael McGowan Aug 29 '11 at 2:53
  • @michael ? so why are top journals peer-reviewed and arxiv not? i dont say peer-review is the better system, but the better system for scientific topics, for topics based on learning by doing and alot experience like programming, i prefer the democratic 1 user - 1 vote system too. But in all metas of physics, astronomy, philosophy you read worries about level of questions and how to attract experts. As many proposals are commited by many programmers with deeper background knowledge in this topics, the only way to emphasize important questions more and attract more experts from outside – Werner Schmitt Aug 29 '11 at 13:02
  • the SE-world is a non-linear voting system imho. If there is/ws already a similar discussion, plz can someone give a link? The downvotes dont give me any hint whats actually wrong with this question, despite people dont like the word undemocratic voting... – Werner Schmitt Aug 29 '11 at 13:04
1

First of all, I just can't imagine SE team to change current voting scheme to something else for some sites... they would just stop being SE in that case.

Now back to merit. An upvote on SE is just a way to say "I like it" or (hopefully) "It was useful to me" and thus this is just flawed thinking to blindly assume that the highest upvoted answer is perfectly correct and the best solution -- as always, BTW not only in Internet. Also the correlation of vote score with actual truthness or usefulness is strongly dependent on the quality of the community. And it is obvious that at first it is dominated by "cool, X is on SE!" guys and the voting scores are very noisy.
However, good answers also appear, they attract Googlers really working with the subject and gradually the community becomes more serious and large enough to silence the noise that is inevitable on any open site. And the overall level should be increasing in such an evolutionary process.

Of course it is a theory (on AI good answers never appeared, Physics still drowns in discussions) but I can't really imagine a better system. Peer review is a very demanding thing coming along with great responsibility -- it would be very hard to recruit people to do this seriously. Your system of electing high-rep users has a problem that it is vulnerable to crooks that can earn rep on simple questions and use such gathered power to negate ideas they don't like.

  • 1
    imo you only have to change parameters for setting up a system like rep<1000 1upvote on question=1upvote,rep>1000 1upvote=2, rep>10000 1=3. This cannot be so tricky. Everything else keeps the same for now. The nonlinearity i want predominantly for highlighting of good modern questions to attract experts from outside. This way crooks dont have really more options than with current system. Everybody has to gain rep by answering simple questions, mostly answer of those questions get the most views and upvotes The main idea is how to get more experts to a proposal, if you have enough experts, – Werner Schmitt Aug 30 '11 at 15:39
  • the exact voting system actually plays a minor role! The quality of community is the crucial point. But if i understand you correctly this problems will solve itself with ongoing time automatically? meta.physics.stackexchange.com/questions/873/… meta.philosophy.stackexchange.com/questions/196/… Take a look at these links, imho you do a circular reasoning vs. the facts that proposals "for expert only" like cstheory or theo. physics have been made. – Werner Schmitt Aug 30 '11 at 15:41
  • Actually experts leave the too democratic sites ruled by majority of interested lawmen. With a nonlinear question upvote system maybe this would not happen. On ArXiv from Prof to student to lawman, everybody can upload his ideas. The signal/noise ratio is low. Peer review or following experts on citeulike is necessary for filtering. If SE is not able/or not will under any circumstances change upvoting of questions according to rep for whatever reasons, just say it and i immediatley stop asking – Werner Schmitt Aug 30 '11 at 15:42
  • But then this is imo the acknowledgement that SE democratic system works well for proposals based mainly on knowledge of experience like programming/cooking but will not yield ENDURING quality with growing user number on very tricky scientific topics that need a minimum of graduated academics like physics, philosophy, math... Then these guys will just create insulated communities like mathoverflow or cstheory now or search/create other places like scirate.com/citeulike/wikipedia. – Werner Schmitt Aug 30 '11 at 15:43
  • Plz dont underestimate how the exponential growth of the bigger lawmen user number will nonlinearly lower the quality of the community with time, as the smaller experts user number cannot grow the same way to keep the ratio of influence on quality constant, so the ratio will negatively shift You may say, if 10 non-physics experts come to SE for 1 expert, the ratio keeps the same. But the topic spectrum of SE is growing exponentially and attracting exponential more non-physics experts! – Werner Schmitt Aug 30 '11 at 15:44
  • If you dont compensate this effect of exp. growing lawmen commiters from non-science SE sites and give experts more vote count, the quality and number of frustrated experts will just fall on and on! You often see this effect here directly after some weeks in BETA when many of commited experts frequent less regularly the proposal and votes and quality drops. Another thing you imo overlook is that the spectrum of diff. user types reading Stackoverflow is much more narrow than physics/philosophy where nearly every lawmen has some interest. – Werner Schmitt Aug 30 '11 at 15:45
  • Programming is per se a insulated community of people practicing this profession/hobby. You will not find interested lawmen here not coding actively in contrast to physics/philosophy. – Werner Schmitt Aug 30 '11 at 15:45
1

Instead of proposing drastic, untested, and (many would say) unfair changes to SE. I would consider looking at successful research-level stack-exchanges, and figuring out what their secret sauce is. I know of only two such SEs:

cstheory and MO. I don't know nearly as much about the latter, and either way, it is an SE 1.0 that is hosted independently. However, I am active on cstheory and can comment a bit on why I think it is successful.

cstheory managed to attract and maintain the activity of many professors, and graduate students because it not only provides a Q&A form, but also a community. It was clearly focused enough in the beginning (instead of being unreasonably broad) and had a good number of dedicated academics that shaped it as an academic site. Now the community is almost exclusively academic and ruthlessly closes or migrates non-research level questions (or at least ones that would not be interesting to the scientists). This has stopped the community from being 'disolved' by non-experts as you say.

At no point did it use a different or alternative voting system.

If you want to make a research site, you don't need to make drastic and unfair voting systems. You just need a critical mass of academics that WANT a community and are willing to police it. If you want to create a site that caters to both academics and non-academics (such as the phyics.SE) then you have to tolerate low-level questions and you will lose academics. Some sites manage fine with this, for instance math.SE, and some not so well.

  • thx for your insights. I totally agree with you and am aware of this. We see diff. problems. Physics is a hobby very much laymen are interested in and ask low-level questions on physics.SE. This community faces a exp. growth of this laymen, as SE network is growing. Cstheory is per se a insulated topic and community, you will not find many not-experienced or not-students/academics here. This is the huge difference. The voting system works as mainly new user with expert or advanced background come to the community. Cstheory, MO and theophyiscs.SE are mainly about research, – Werner Schmitt Sep 5 '11 at 12:40
  • and mainly about research techniques, mathematical background. So its fine they have their own community, as it is a very specific topic. But something like experimentalphysics.SE IMHO would not work, as experimentalist will not share knowledge about prototypes, exact measurement setups, manufacturing recipes. This is a crucial diff. between proposals here related to math and highly experimental disciplines. So the question is to me, how to highlight question interesting to experts on physics.SE better. And then i can only think of "give them more question upvote strength" – Werner Schmitt Sep 5 '11 at 12:41
  • with a faster growing number of interested laymen. Making for every sub-branch a own proposal is kind of counter-productive to me here. I propose not a reinvention of the voting system, but a better adapting/weighting of the question upvote strength for high rep users according to the user distribution (laymen/student/expert) and topic. The voting system plays a minor role, but you have to give experts a filter/highlighting option to keep them on physics.SE or similar like biology/chemistry with many laymen. Its not the way to close every proposal that gathers with time too many laymen, – Werner Schmitt Sep 5 '11 at 12:42
  • that all experts leave as the signal/noise ratio explodes. meta.physics.stackexchange.com/questions/883/… – Werner Schmitt Sep 5 '11 at 12:42
0

I would question just how much peer review actually contributes to the overall integrity of science in the long run. It's long been one of my areas of historical research. It's certainly a unfitting requirement for a general public site given that most peer reviewed journals are behind paywalls and/or requires a specialist knowledge to evaluate.

But the real reason is that "peer reviewed" doesn't mean "proven science." It means at best, "no big blunders" or at worse, "the voice of authorities of an entire field that has gone off the rails."

The real purpose of peer review is not to protect science but the reputation of the journals. It's basically proof reading to make sure the journals don't publish something blatantly wrong so their reader don't have to feel paranoid about everything they read.

Somewhere in the last 30 years, we've developed the idea that "peered reviewed" means "scientifically proven" and accepted by the scientific community. It doesn't. It just means, "Not overt quackery that would morbidly embarrass a publishers." Peer review checks form not facts.

Every bad idea in science in the last century has slipped right through peer review with glowing approval e.g. Eugenics. Go grab a copy of "Science" anytime before 1950 and just wade through the horror show.

The nigh worship of peer review has allowed some outright frauds to slip through and become accepted long before the real proof, reproduction of results, became known.

The prestigious Lancet got burned twice in the 90s, once by a completely false study claiming evidence that measles vaccinations caused autism. Turned out to be collusion between a corrupt researcher and a Ttrial lawyer. Yet in the years it took for followup studies to show the effect was not real, it had become widely accepted as true in the general public, courts, media and politics, largely because it was published in "peer reviewed" Lancet. I still see people refering to it long after the paper has been withdrawn.

Study did enormous damage, got some children killed and sailed through peer review.

It passed peer review because the faked data and methodology was faked with that intent. It's easy to slip faked data past peer review than than real world, warts and all data. Peer review does not reproduce the original research and rarely even looks at the raw data. It just looks for obvious mistakes in methodology, calculations and conclusions.

Circa 2000, Lancet published a study of infant mortality in Iraq under sanctions, showing staggering infant mortality. After the invasion/liberation, it was found the researchers had been shown bodies frozen and previewed over many months but told the infants died over just a few weeks. It passed through peer review because the data and methodology presented to the peer reviewers in valid form, just fake. The paper was withdrawn.

(The same team, not learning their lesson, tried again after the invasion/liberation. That paper remains on record despite it's findings never being reproduced by any other source. It's controversial.)

Peer review is just becoming a means of making arguments from authority. I've noted that their is a direct correlation between how often "peer review" shows up in public discussions of a science and that science's history of predictive power e.g. not often in physics, a lot in sociology.

I am an advocate of doing away with peer review entirely and replacing it with crowd reviewing. Peer review only existed in the first place because of the zero-sum nature of journal space and the expense of the old paper based workflow and distribution. Typos and mistakes of all kinds were expensive.

But in the internet age, why bother? Slap up the raw data, the raw methodology, your conclusion and let the crowd try to tear it to shreds. If it survives, then it can go on to reproduction.

The problem with garbage and noise on the internet isn't caused by a lack peer review, it's caused by 1) People wanting to believe the interesting or self-serving instead of disciplining themselves, and 2) By repeated arrogant authority overreach destroying a lack of trust in institutions.

As our problems get more complex, they require more and more narrow specialization on the part of experts but that leads to the problems of tunnel vision, insular subcultures, group think, hubris etc than can only be offset by a wider if naiver perspective. We need more eyes on the raw data and the raw science not fewer.

It won't be as pretty as the old Ivory tower blessing of authority of peer review but it won't make the same mistakes nor allow them to persist as long. Oliver Wendell Holmes Jr. might not have not been so quick to make "Three generations of morons is enough" the high law of the land had he had some other scientific input other than the most respective Ivy League scientist of the day (who turned out to have to missed a few things.)

This type of open forum is actually a good start. The real key to get people to think scientifically, is to get them to understand methodology, not to accept pronouncements of authority. Science is methodology, not fact. The accuracy and precision of scientific statement derive wholly from methodology. Teach people that, and you can forget the journals.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .