Putting Away Magical Thinking
 |
The movement of troops through the islands of the South Pacific in World War II had a profound, unintended consequence for the native cultures living in them. These isolated aboriginal peoples were suddenly exposed to soldiers in the Japanese and Allied Forces, who brought incredible amounts of manufactured clothing, medicine, canned foods, tents, weapons, and other goods with them. These supplies, some of which were shared with the islanders, were even dropped miraculously from the sky.
Then the war was over, the airbases were quickly abandoned, and the wonderful cargo stopped falling from the heavens. The islanders, in an attempt to persuade their gods and ancestors to bring more cargo to them in planes, ships, and parachutes, began ritualistically imitating the behaviors of the soldiers. They waved landing signals while standing on the abandoned runways, now lit with torches. They carved wooden headphones and sat in replicas of control towers, all in an attempt to bring the wonderful riches from beyond.
From our technologically-advanced perspective, we may fall into the trap of looking down on these “Cargo Cults,” as they are known, but a bit of introspection finds that we may all be guilty of such magical thinking. In my own profession, we have the term cargo cult programming, where a novice programmer includes code that serves no purpose in their software simply because they don’t understand what it does. I know that I have been guilty of such a logical fallacy in my own coding as a novice, where I would incorporate a large block of another programmer’s code into my work because I lacked the skills to identify the smaller portion of it that I actually needed. I didn’t know how the code worked; I just knew that copying and pasting it into my software solved my problems.
Richard Feynman coined the term “Cargo Cult Science” in his 1974 commencement address at Caltech to describe scientific ideas that we accept because they are established dogma rather than because they provide evidentiary proof of their effectiveness:
There are big schools of reading methods and mathematics methods, and so forth, but if you notice, you’ll see the reading scores keep going down–or hardly going up in spite of the fact that we continually use these same people to improve the methods. There’s a witch doctor remedy that doesn’t work. It ought to be looked into; how do they know that their method should work? Another example is how to treat criminals. We obviously have made no progress–lots of theory, but no progress–in decreasing the amount of crime by the method that we use to handle criminals.
Yet the true essence of science is to perpetually challenge the dominant paradigm. Someone once told me they didn’t trust science because it was, “One guy saying he saw something and some other guy agreeing with him;” but this is the very antithesis of the scientific process.
Science doesn’t say “This is what I saw, worship me!” Science says, “This is what I saw, this is how I saw it, now go see for yourself.” The whole purpose of peer-review is to have others try and replicate your experiments, and if they don’t get the same results? Well, isn’t that interesting?
Feynman has a word of warning to scientists who will compromise scientific truth for personal gain:
We’ve learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.
The cargo cults of the south pacific mostly vanished when their newfound rituals failed to procure the favor of the gods, but one of these religions, the John Frum Cult, still exists today, and the inhabitants of Tanna island in Vanuatu still hold a parade every year, waiting for their god to return. In the meantime, the cult believes it has some empirical evidence validating their faith, as the influx of tourists to the island bring with them some of the legendary riches of the past.
Which societies are better off, the ones who abandoned their magical thinking, or the ones still living the illusion?
The cargo culture stuff is really interesting.
The peer review process is great, but it has a lot of flaws too, especially if you are unfortunate enough to get a reviewer who is obviously blindly opposed to what you are publishing, though this is mostly a hassle as it seems the editors are often level headed enough to keep this limited.
It seems that far too much credit goes to those who do things ‘first’ rather than those who do things thoroughly.
I’ve seen stuff in my field that is published maybe a day after it is submitted and lacks depth, though the material is correct, but not thorough enough to really be considered a contribution, but it is published so that one group can get the ‘first’ over another which is working on the same material, and where the outcome is basically already known.
I am sure a lot of this is going on with the higgs boson folks as well.
Comment by John — June 2, 2009 @ 5:56 pm
I gotta agree. Peer review is a great thing, but at the same time, it’s a tyranny of the majority. It’s basically peer pressure. Say something that’s not in with the in group, and now you’re not cool. At least the process depends on reason. But reason, like coolness, can end up being subjective in grey areas.
At least we all think the world is round nowadays! :)
Comment by ClintJCL — June 2, 2009 @ 6:57 pm
How would you suggest improving the peer review process, or what alternatives would you prefer to it?
Comment by ideonexus — June 2, 2009 @ 8:14 pm
No clue. But you don’t have to have the solution to a problem to state that there is one. (See also: Unsolved mathematical problems.) There’s simply no way to “truthify” a system. So we rely on peer pressure / herd mentality / conformity.
It works most of the time. But if someone challenges the status quo — good luck. They’re going to have to go the extra mile to get their point across. Which is good when they are wrong, but bad when they are right. ;)
Comment by ClintJCL — June 2, 2009 @ 8:47 pm
While some scientist are biased, for the most part scientific results are based on fact, not opinion. There are data sets than be quantified, and there are equations governing those data sets. If it cannot be proven to be correct or incorrect, then it’s just a theory(such as the arguments for and against global warning), which is not what the peer review process is about. The peer review process is simply there to make sure that your work is verifiable and repeatable, so it’s anything but peer pressure and conformity.
Comment by chriggy — June 2, 2009 @ 11:43 pm
Believe it or not, a lot of published material is theory, not equations defining fact. There are a lot of ways to interpret things, and as far as scientific literature goes, most of it is published so that everyone can have a look at it and through the tests of time, one of the interpretations can gain prominence or fizzle out.
I can’t speak for a every field, but in my own, solar physics, there were plenty of published observations about the solar wind, some even including equations to describe the flow of solar plasma that after more information became available that showed the theories to be invalid, but stating them did not hinder the advancement of the scientific work because it helped to drive future work in the right direction when better tools became available to describe the phenomenon of the solar wind (mainly getting interplanetary probes out there with magnetometers and other such things)
All of these ideas about emerging fields are peer reviewed, and I’d say that is where a lot of the bias leaks in to the process.
As far as ideas to improve the peer review process, though i am sure it would introduce issues of its own, I think it would be cool to have a slashdot/digg style forum for scientific works, where registered members of a given field can ask questions and post comments.
The immediate problem to this is that a lot of groups want/need their publications protected until they have been fully accepted for publication for reasons like the ‘me first’ motivation, as well as intellectual property rights (think biomed etc..)
It would still be useful I think as a post published way to espouse ideas and get wider feedback.
Comment by John — June 3, 2009 @ 12:28 am
John, I agree with you 100% that theory should indeed be published, as there is no other way to advance science. Every proof begins with a hypothesis. Eventually, every theory will be proven correct or incorrect.
What I disagree with is calling theory peer reviewed, as this is indeed where bias and personal interests creep into things.
It’s either quantifiable or not, and until it’s quantifiable, it’s just an opinion, even if it may be useful opinion and get others to thinking along those lines.
The peer review process should be limited to the scope that someone hasn’t fudged their data, made a mathematical mistake, etc…
Comment by chriggy — June 3, 2009 @ 1:14 am
“The peer review process should be limited to the scope that someone hasn’t fudged their data, made a mathematical mistake, etc…”
It’s supposed to be, but scientists are people too. Worse, they work in an industry where the goal is to be right and the only reward generally is status. So there’s a powerful incentive system fighting against the ideal of ego-free application of the scientific method.
Comment by Stacy — June 3, 2009 @ 9:28 pm
Oh, and Ryan — interesting you should mention the cargo cult effect in coding. I hadn’t heard the term before, but have definitely seen it. In my web developer days I did a little of it myself and ultimately ended up cutting over 80% of the code on some pages because the previous developer had literally included every function on every page. The excess weight was killing page-load times, and I learned a hell of a lot in the weeding-out process!
Comment by Stacy — June 3, 2009 @ 9:32 pm
When I learn new languages, based on reference material, I also tend to totally adopt stuff that I see just because it is there. I realize I am doing it and often I go back and make changes that are appropriate, but at times, if I have other work that is more important (which I hope I do!) I will just leave it as it was, it works, it doesn’t get in the way of what I am doing, I am not a professional coder, so it’s ok that I don’t care!
Re- Stacy: incentive system
Funding is also important in this, and ones publication frequency has a direct correlation to ones ability to obtain funding, of course it helps if others reference your publications, but in smaller fields like my own, just getting material out there is important.
As far as peer review being for quantifiable valid data, eh, well, Newton kind of rocked the world with his ability to quantify things, but that doesn’t mean that Copernicus, Galileo and others shouldn’t have bothered to discuss or hypothesize their ideas into a generally critical community. Furthermore, Newton was not the end all be all (we are still doing science today!) and as we get more and more specialized, it is important to continue to publish and peer review material that is still far from quantifiable and mathematically perfect, acting as a sort of scientific living Monte Carlo simulation, until we find the actual factual truth, or technology advances enough to let us disregard old weak data.
Comment by John — June 4, 2009 @ 1:20 am
John, I’m going to have to disagree with you there. Like I said, theories should be published, and need to be published. But accepting or discounting a theory that is not quantifiable based on peer review alone goes against the scientific method. At that point, we may as well be discussing the existence of god. There is no proof either way. If it’s not quantifiable, just because peers discount it does not mean it’s not valid, and likewise, just because peers accept it does not mean it is valid. Theories should not be peer reviewed. Proofs should.
Re: Cargo Cult effect
I’ll admit, I myself have been guilty of this. Unfortunately(or maybe I should say fortunately) most of my work has dealt with real time systems, and I’ve spent many hours putting in timers and weeding out extraneous stuff from code I assimilated. Of course, this means that most of the jobs I qualify for now require an active security clearance, which is an issue, since I never needed one for my previous job.
Comment by chriggy — June 4, 2009 @ 2:47 am
Wow. Glad to see this post stirred up so much conversation, and I can definately see the validity in everyone’s points on this thread. I see this all as matters of degree: tyranny of the majority, egos overcoming truth, and the like… I agree with Clint that they are all important aspects to be wary of in scientific research. I don’t think there are any solutions, except to keep pursuing the ideal.
John: Thanks for pointing out the funding aspect. I know of many professors who struggle to stay published in order to justify their existence at Universities, because teaching isn’t considered enough reason to keep them on the faculty.
Stacy and Chriggy: I’m glad to hear I’m not the only one guilty of this in the past. Some of my most embarrasing coding moments have been to look through someone’s code and think, “What moron wrote this???” Then look at the change history and think, “Oh. I was that moron.” : )
Chriggy: Sorry to hear about the security clearance requirement. That has always been a Catch-22 in IT. You can’t get a job without Security Clearance, and you can’t get security clearance without a job.
Comment by ideonexus — June 4, 2009 @ 8:51 pm
LOL. I’ve had those “Oh yeah, I was that moron” moments. You’re not alone :)
Comment by chriggy — June 4, 2009 @ 10:47 pm
I think that there is something missing here, and as a mere passerby you may take or leave my perspective, however at the onset of any process of discovery in a completely unknown area of knowledge so little is known about the laws of the subject yet to be discovered that the Cargo Culture approach may really be the only viable one at that point.
The only difference between the utterly laughable and the completely formalized is really that one thing is accepted and the other thing isn’t.
Just look around you at the history of discovery and ground breaking thinkers. So many of them were laughed right out of their fields.
I suppose I am saying that it bares at least a moment of consideration for the benefit of those who might take approaches which might not look like they make sense and in fact may be revolutionary and also to be aware that it may not be possible to clearly tell those from the sloppy at a certain stage of development.
Comment by I — June 25, 2009 @ 10:33 am