Peer Review

[freeztar]

It seems appropriate to have a discussion on the scientific peer review process.

 

Reputable scientific journals require that all articles submitted are peer reviewed. Wikipedia has the following to say in the opening paragraph on the subject:

 

Peer review (also known as refereeing) is the process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform impartial review. Impartial review, especially of work in less narrowly defined or inter-disciplinary fields, may be difficult to accomplish; and the significance (good or bad) of an idea may never be widely appreciated among its contemporaries. Although generally considered essential to academic quality, peer review has been criticized as ineffective, slow, and misunderstood.

Peer review - Wikipedia, the free encyclopedia

 

Some questions come to mind:

 

• What constitutes a rigorous peer review and what doesn't?
  
• How does the process differ among disciplines?
  
• How is the process mediated, if at all?
  

 

Discuss!

[Jay-qu]

I think peer review is stupid in the way that the reviewers are anonymous but the person being reviewed is not. Science is supposed to be universal - it doesnt matter who it comes from, if its good science it should be published. In this way I think either both should be anonymous or both names are disclosed.

[Doctordick]

One of the major problems with peer review is the problem of, “subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field”. When Newton first published his theory of gravity, where were the “experts in the same field”? Back in 1982, I submitted my work to a number of scientific Journals and was rejected by everyone. These rejections were almost all given as “not of interest to our journal”. I don't believe any were ever actually submitted to a referee; I don't think they knew of any “experts in the field”. :rolleyes:

 

Have fun -- Dick :xx: :circle:

[freeztar]

You both raise valid concerns: anonymity is not conserved and peer experts may not exist

 

The anonymity issue is easy enough to fix in principle, but the "expert issue" is a bit of a showstopper eh?

[Flying Binghi]

Interesting views here -

 

Publish and be wrong

 

One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong

 

The fallibility of scientific journals | Publish and be wrong | The Economist

[freeztar]

Interesting views here -

 

Publish and be wrong

 

One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong

 

The fallibility of scientific journals | Publish and be wrong | The Economist

 

Interesting research in that article essay [1], but sadly, the article from The Economist made no mention of the dynamic (often rapidly changing) nature of scientific research.

 

I also fail to see what this has to do with peer review. :shrug:

[Jay-qu]

One of the major problems with peer review is the problem of, “subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field”. When Newton first published his theory of gravity, where were the “experts in the same field”? Back in 1982, I submitted my work to a number of scientific Journals and was rejected by everyone. These rejections were almost all given as “not of interest to our journal”. I don't believe any were ever actually submitted to a referee; I don't think they knew of any “experts in the field”. :shrug:

 

Have fun -- Dick :dogwalk: :circle:

I disagree - Newton made beautiful mathematical arguments that perfectly (to the days accuracy) agreed with observation. It then only requires another smart enough physicist to confirm its not rubbish.

 

DoctorDick your papers were probably rejected because they didt do real science - either you didnt present evidence to support your claims or didnt provide some experiments that could be done to falsify your argument.

[Turtle]

... Newton made beautiful mathematical arguments that perfectly (to the days accuracy) agreed with observation. It then only requires another smart enough physicist to confirm its not rubbish.

 

DoctorDick your papers were probably rejected because they didn't do real science - either you didn't present evidence to support your claims or didn't provide some experiments that could be done to falsify your argument.

 

I can't speak for Doc's experience or circumstance, but I once submitted a paper on my strange-numbers-sets work and it was rejected out of hand much as Doc's was. "Not our cup of tea" more or less, and no reason given. So, since we have all peer reviewed the heck out of my numbers right here at Hypog and found them beautimous and rigorous, are they any less so because some anonymous 'experts' paneling a journal don't fancy them?

 

Obviously not. :dogwalk: Peer review as we have known it is outdated & over rated. Facts is facts and that's that Jack. :shrug:

[InfiniteNow]

What a spurious and irrelevant argument Dick and Turtle have made.

"They rejected MY paper, so the entire process is flawed!" Bunch of childish whiners.

You most likely submitted it to the wrong journal(s). Nobody said it was wrong.

 

Of course if one writes a paper about something like neurotransmiiter receptivity in the left parietal cortex of rhesus monkeys and submits it to a journal which publishes data about finds in sediment layers of jurassic fossils across the arctic ridge... it's going to be rejected as "not their cup of tea."

 

Did you try handing in your english lit homework to your chemistry professor, too, and ***** out the school when they said it was not appropriate for the class?

 

Egotistical short-sighted morons, both of you. You failed to navigate the course properly, and the mirror would be the best target for your disgust and frustration, not the proven process of peer review.

 

 

 

 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~``

 

Well now... back to the topic at hand...

The below seemed relevant to this thread, so I'll post it again:

 

 

 

stand up for REAL science

 

REAL science...

 

 

R

ecognizes its limits -

Science only works with phenomena that can be independently verified by observations or empirical tests. This is a practical approach to the study of the natural world that has proven to be extremely conducive to the advancement of scientific knowledge. Since this approach does not rule out the existence of non-verifiable phenomena, any claims about the existence or non-existence of such phenomena are not scientific.

 

 

E

nriches our understanding of the universe -

Science enhances our appreciation of ourselves and of the world around us. It does not attempt to supplant or displace other ways of knowing, such as philosophy, religion, or ethics. Knowledge gained through scientific investigation is transferable to other contexts, but science does not invalidate other modes of human inquiry.

 

 

A

pproximates reality -

Science is not an attempt to prove hypotheses, but rather an attempt to falsify them. The purpose of a scientific model is to provide a conceptual framework that guides and directs future research. Although all foundational scientific models have been extensively tested, no scientific model should ever be viewed as absolute truth. All scientific models are tentative and subject to potential modification as new empirical evidence becomes available.

 

 

L

eads to new knowledge -

Science is a systematic approach to developing physical, mathematical, and conceptual models for understanding the natural world. The best scientific models are those that have both explanatory and predictive power. In other words, scientific models should not only explain what is currently known, they should routinely generate new hypotheses for further research.

 

Everyone in this debate wants a better world, and we can all benefit from the knowledge gained through the application of the scientific method. When it comes to science education, our goal should be to teach our children to understand and appreciate the scientific method so that it can be utilized for the benefit of all humanity. Consequently, we must not allow the quality of science education in our public schools to be compromised by the current culture war concerning who holds "the Truth."

 

We have an obligation to our children to pursue only the best available science. We should accept nothing less than REAL science in our public school science classrooms.

[modest]

I think peer review is stupid in the way that the reviewers are anonymous but the person being reviewed is not. Science is supposed to be universal - it doesnt matter who it comes from, if its good science it should be published. In this way I think either both should be anonymous or both names are disclosed.

 

It looks like the journal Nature did a trial-run for a more open peer review process in 2006. It's described here and apparently the trial ended just 6 months later which Nature gets into here:

 

Despite enthusiasm for the concept, open peer review was not widely popular, either among authors or by scientists invited to comment.

 

<... and some of the conclusions...>

• A general sense of indifference from key contacts in their fields to the trial, and that it was like 'pulling teeth' to obtain any comments.
  
• Direct attempts to solicit comments met with very limited success.
  
• Biologist editors in particular were not surprised that authors in very competitive areas did not wish to be involved.
  
• Anecdotally, some authors were reluctant to take part due to fear of scooping and patent applications.
  
• Anecdotally, potential commenters felt that open peer review is 'nice to do' but did not want to provide any feedback on the papers on the server.
  
• Editors felt that most of the comments provided were of limited use for decision-making. Most were general comments, such as "nice work", rather than adding to the review process.
  

 

Yet, another study found that an open process had no such problems:

 

http://www.bmj.com/cgi/content/abstract/318/7175/23?ijkey=8feca9dda2f29a07ec06f70a661120c97578c339&keytype2=tf_ipsecsha

 

:eek:

 

~modest

[Galapagos]

I think peer review is stupid in the way that the reviewers are anonymous but the person being reviewed is not. Science is supposed to be universal - it doesnt matter who it comes from, if its good science it should be published. In this way I think either both should be anonymous or both names are disclosed.

 

I'm with Jay-qu on this, and I think that instead of open peer review as posted by modest above, it should be anonymous on all sides. This seems like the best way to avoid bias.

 

Does anyone know how the anonymous reviewer/open author situation got started, and if it is that way for a reason?

[modest]

I'm with Jay-qu on this, and I think that instead of open peer review as posted by modest above, it should be anonymous on all sides. This seems like the best way to avoid bias.

 

Indeed, I was reading about that as well. There's some good commentary here:

 

Peer Review—The Newcomers' Perspective

 

DBPR, in which both the reviewers and the authors remain anonymous to each other, is thought to disentangle the peer-review process from non-scientific factors, thereby presenting an appealing alternative. The a priori case for masking and blinding is strong, and several studies have suggested that articles published in DBPR journals were cited significantly more often than articles published in non-DBPR journals [3,4]. However, other studies have been less convincing; critics of DBPR argue that it is difficult to hide the identity of the institution, laboratory, and/or authors of a paper from the reviewers, especially in smaller specializations. For instance, in a DBPR policy trial, despite explicit instructions to authors, 34% of prospectively evaluated manuscripts contained hints to unblind the authors, and editors correctly identified the authors or institutions of 25% of the manuscripts [5]. The disconnection between principle and practice is evident, and so far, few journals, and even fewer in biomedical sciences, have implemented DBPR policies. The reasons appear to be partly historical, as journals are used to SBPR, and partly intellectual, as the benefits of DBPR still remain controversial [6].

 

The sources it uses are listed in the footnotes, but I haven't looked at them yet.

 

It does seem like a double blind method would be conceptually better. I would assume referees would naturally be less critical if the source is someone they're familiar with and "trust" and more critical otherwise.

 

~modest