Response to the OSTP Request for Comment on Scientific Integrity

On March 9, 2009, the White House issued a Memorandum on Scientific Integrity, directing the Office of Science and Technology Policy (OSTP) to develop recommendations for improving scientific integrity based on six core principles. The OSTP opened a solicitation for public comment on their blog for ideas on how best to implement and enforce these principles.

There are a lot of thought-provoking and insightful ideas being posted, like exposing all software code used in simulations for public review, and there are some irrelevant comments, like the standard Obama-bashing or personal issues like “Legalize Marijuana!” Luckily, the OSTP is using a SlashDot-strategy for moderating comments, so the good ones float to the top and the trolls get banished to irrelevancy.

Here are my responses to the six questions posted:

Principle (a)

The selection and retention of candidates for science and technology positions in the executive branch should be based on the candidate’s knowledge, credentials, experience, and integrity;

What are the best metrics for the four listed criteria?

Knowledge is very specialized in scientific disciplines, and the only way to test if someone is an expert in their field is to have another expert interview them; however, there are flaws with this strategy, as many academics lack the social skills to perform well in interviews. I personally do very poorly in one-on-one situations; therefore, people like myself would much prefer to respond to a short-answer examination that would then be reviewed by an expert within the organization.

Credentials appear fairly straightforward, what institutions did they graduate from? How did they fair academically? What continuing education or professional accomplishments have they added to their cirriculum vitae since earning their degree?

Experience is more difficult to guage. Here again an expert is needed, not in the same field necessarily, but someone familiar with the background work environments. Two candidates could have spent the past five years developing software applications for the military, but differences between department development methodologies can make a world of difference between the quality of the two candidates’ backgrounds to ensure the past five years were spent doing things the correct way.

Integrity, I firmly believe, can only be measured on the job and should be measured by having the employee’s job performance rated by their coworkers and direct supervisor. As the employee’s work ethic and principles directly affect the individuals they interact with on a day to day basis, these individuals will have every incentive to review them truthfully.

Principle (b)

(b) Each agency should have appropriate rules and procedures to ensure the integrity of the scientific process within the agency

How can the integrity of scientific processes be assured? What are some good examples to learn from?

Ensuring the openess of the scientific process is the primary means of assuring its integrity. So long as everything is documented and published for the world to see online (including, as McNamara notes, software code), many eyes will review the process over and over again for years to come, ensuring the research will be reviewed in light of new knowlege and discoveries. The Public Library of Science is a good example of the advantages of publishing online, as research gets disseminated and reviewed within days.

It is important to remember that this is the scientific integrity of the United States Government that we are discussing. We citizens have a right to this research, and the Government has a responsibility to include us in it. Data.gov is going to be a fantastic step in this direction, and scientific integrity will certainly benefit from continuing along this course of online, easy inclusion of interested citizens.

Principle (c)

(c) When scientific or technological information is considered in policy decisions, the information should be subject to well-established scientific processes, including peer review where appropriate, and each agency should appropriately and accurately reflect that information in complying with and applying relevant statutory standards

What are the most effective processes and organizational structures for assuring that scientific and technological information is reliable? How can the processes and structures used in each case best be disclosed as part of the public record?

The basic peer review process is still the most effective strategy for ensuring reliable scientific and technological information. The peer review process should be conducted as any quality assurance process is administered, and formally documented every step of the way. This documentation should be published real-time as it happens, so that the public can review the results and identify errors before they continue into implementation, where they will become much more costly.

Publishing this process will open it to analysis and review by the public, better methodologies will result as academics are able to find metrics and contribute improvements to process. This publication will also bring citizenry into the process, where they can learn how it opperates and perhaps adopt portions of it into their organization’s own standard operating procedures.

Principle (d)

(d) Except for information that is properly restricted from disclosure under procedures established in accordance with statute, regulation, Executive Order, or Presidential Memorandum, each agency should make available to the public the scientific or technological findings or conclusions considered or relied on in policy decisions;

What are the best ways to maximize the legitimate public release of scientific and technological information relied upon by agencies?

Multiple strategies are required for releasing scientific and technological information agencies rely upon for public access and review. Firstly, this information must be released as quickly as possible, including finalized reports, their earlier drafts, and other documents used in their production. The public needs to see the entire history of how these reports came to exist in their final draft form. This will prevent agencies from editing out facts inconvenient to political ideologies and personal motivations. This first step is a data-dump to a regularly updated website, something very few citizens will have any interest in for themselves, but some citizens will scrutinize for impropriety.

Secondly, the information, the finalized reports and supporting data, should be given tags, described with XML, categorized, and published online in such a way that search engines will list it, citizens will read it, and students can cite it in their research papers. In the first stage of releasing the information, the Government is publishing it for the sake of transparency, in the second stage, the Government is publishing it as a resource of which citizens can make use.

Principle (e)

(e) Each agency should have in place procedures to identify and address instances in which the scientific process or the integrity of scientific and technological information may be compromised

How can agencies best ensure that they will know when scientific or technological integrity has been compromised?

Public transparency will allow private citizens to review the scientific process in government and blow the whistle should they see any misinformation, poorly-designed methodologies, or researchers accidentally using out-of-date information. It is important to keep in mind that the citizens are raising alerts, but not neccessarily to dishonesty or inethical behaviors. By making the scientific process within government transparent, we are opening the whole process to peer-review from academics from all over the world, who will take interest in it to further their own knowledge and understanding.

There is a concern that this transparency, allowing anyone, even the uneducated or misinformed, to review the process could bog government down in false alerts; however, in the open-source community, such false alarms are raised daily, and, in having to respond to them, researchers come to a better understanding of how to document their processes to avoid future misunderstandings.

Principle (f)

(f) Each agency should adopt such additional procedures, including any appropriate whistleblower protections, as are necessary to ensure the integrity of scientific and technological information and processes on which the agency relies in its decisionmaking or otherwise uses or prepares.

What are the best ways to make sure that the science and technology an agency relies on is reliable?

While transparency will open the scientific process within government to the public, it won’t ensure the public will actively get involved. Agencies should provide incentives, awards perhaps, to citizens that can find concrete faults in research and data. Offering an award of $100 to anyone who can find a mathematical error in a report’s research will give dozens, perhaps hundreds, of citizens the incentive to spend hundreds of collective hours reviewing the report for errors. Such an incentive not only provides quality assurance on the cheap, but also includes citizens in the process. Students would learn about the mathematics and research methodologies in reports as they strive to attain a monetary award.