The only difference between the utterly laughable and the completely formalized is really that one thing is accepted and the other thing isn’t.

Just look around you at the history of discovery and ground breaking thinkers. So many of them were laughed right out of their fields.

I suppose I am saying that it bares at least a moment of consideration for the benefit of those who might take approaches which might not look like they make sense and in fact may be revolutionary and also to be aware that it may not be possible to clearly tell those from the sloppy at a certain stage of development.

John: Thanks for pointing out the funding aspect. I know of many professors who struggle to stay published in order to justify their existence at Universities, because teaching isn’t considered enough reason to keep them on the faculty.

Stacy and Chriggy: I’m glad to hear I’m not the only one guilty of this in the past. Some of my most embarrasing coding moments have been to look through someone’s code and think, “What moron wrote this???” Then look at the change history and think, “Oh. I was that moron.” : )

Chriggy: Sorry to hear about the security clearance requirement. That has always been a Catch-22 in IT. You can’t get a job without Security Clearance, and you can’t get security clearance without a job.

Re: Cargo Cult effect
I’ll admit, I myself have been guilty of this. Unfortunately(or maybe I should say fortunately) most of my work has dealt with real time systems, and I’ve spent many hours putting in timers and weeding out extraneous stuff from code I assimilated. Of course, this means that most of the jobs I qualify for now require an active security clearance, which is an issue, since I never needed one for my previous job.

Re- Stacy: incentive system

Funding is also important in this, and ones publication frequency has a direct correlation to ones ability to obtain funding, of course it helps if others reference your publications, but in smaller fields like my own, just getting material out there is important.

As far as peer review being for quantifiable valid data, eh, well, Newton kind of rocked the world with his ability to quantify things, but that doesn’t mean that Copernicus, Galileo and others shouldn’t have bothered to discuss or hypothesize their ideas into a generally critical community. Furthermore, Newton was not the end all be all (we are still doing science today!) and as we get more and more specialized, it is important to continue to publish and peer review material that is still far from quantifiable and mathematically perfect, acting as a sort of scientific living Monte Carlo simulation, until we find the actual factual truth, or technology advances enough to let us disregard old weak data.

It’s supposed to be, but scientists are people too. Worse, they work in an industry where the goal is to be right and the only reward generally is status. So there’s a powerful incentive system fighting against the ideal of ego-free application of the scientific method.

What I disagree with is calling theory peer reviewed, as this is indeed where bias and personal interests creep into things.

It’s either quantifiable or not, and until it’s quantifiable, it’s just an opinion, even if it may be useful opinion and get others to thinking along those lines.

The peer review process should be limited to the scope that someone hasn’t fudged their data, made a mathematical mistake, etc…

I can’t speak for a every field, but in my own, solar physics, there were plenty of published observations about the solar wind, some even including equations to describe the flow of solar plasma that after more information became available that showed the theories to be invalid, but stating them did not hinder the advancement of the scientific work because it helped to drive future work in the right direction when better tools became available to describe the phenomenon of the solar wind (mainly getting interplanetary probes out there with magnetometers and other such things)

All of these ideas about emerging fields are peer reviewed, and I’d say that is where a lot of the bias leaks in to the process.

As far as ideas to improve the peer review process, though i am sure it would introduce issues of its own, I think it would be cool to have a slashdot/digg style forum for scientific works, where registered members of a given field can ask questions and post comments.

The immediate problem to this is that a lot of groups want/need their publications protected until they have been fully accepted for publication for reasons like the ‘me first’ motivation, as well as intellectual property rights (think biomed etc..)

It would still be useful I think as a post published way to espouse ideas and get wider feedback.

It works most of the time. But if someone challenges the status quo — good luck. They’re going to have to go the extra mile to get their point across. Which is good when they are wrong, but bad when they are right. ;)

At least we all think the world is round nowadays! :)

The peer review process is great, but it has a lot of flaws too, especially if you are unfortunate enough to get a reviewer who is obviously blindly opposed to what you are publishing, though this is mostly a hassle as it seems the editors are often level headed enough to keep this limited.

It seems that far too much credit goes to those who do things ‘first’ rather than those who do things thoroughly.

I’ve seen stuff in my field that is published maybe a day after it is submitted and lacks depth, though the material is correct, but not thorough enough to really be considered a contribution, but it is published so that one group can get the ‘first’ over another which is working on the same material, and where the outcome is basically already known.

I am sure a lot of this is going on with the higgs boson folks as well.