Evidence Regarding Subjectivity in Approval Process

I’m going to deviate from my previous mentioned plan for today’s post.

I came across the article below in a weekly e-newsletter from AAAS / Science:
Editor’s Choice: Highlights of the Recent Literature, April 30, 2010

It asserts a finding that there is a 10% drop in publications and funding when a “superstar” researcher is subtracted abruptly from a team due to death.

“The authors’ analyses of these consequences favor a causal explanation in which the critical factor in these downward trends was being deprived of the intellectual input from these superstars, as opposed to a loss of collective experimental expertise or of privileged channels of communication to funding panels and journal editors.”

I urge you to consider, however, if this is not evidence that the selection of research for peer-reviewed publication is subjective, and driven by politics and personality over the substance of the ideas. Following is the article from the weekly e-newsletter.

By Gilbert Chin

It is no longer uncommon to see multi-authored original research papers, and in many instances, these studies represent the fruits of collaborations between multiple laboratories, especially in the biomedical sciences. How important are the lead researchers in these social and scientific networks? Answering this question empirically appears at first glance to be intractable, but Azoulay et al. have compiled a data set that enables them to take advantage of natural events—when still-active superstar researchers are subtracted from collaborations via death. Of the roughly 230,000 U.S. medical school faculty members, 10,000 were classified as elite according to seven objective professional criteria; during the last two decades of the 20th century, 112 of these scientists died suddenly. The effect on the productivity of the surviving faculty-level collaborators in these superstar-coauthor dyads was unambiguous and persistent: They suffered decrements of almost 10% in publications and funding. The authors’ analyses of these consequences favora causal explanation in which the critical factor in these downward trends was being deprived of the intellectual input from these superstars, as opposed to a loss of collective experimental expertise or of privileged channels of communication to funding panels and journal editors.

Q. J. Econ. 125, 549 (2010).  IMAGE: Leena Peltonen-Palotie, 1952–2010. Image Credit: PASI HYTTI

_________________________________________________

An alternate possibility is that the publishers and funders were willing to publish or fund boring or minor research because of the star, but when the star was gone, the dynamic changed. This has not been ruled out.

Why don’t I buy the conclusion of the analyses of Azuoulay et al? A team of researchers do most of the big decisions at the beginning of undertaking their project: the subject of the investigation, the method of investigation, key assumptions, etc. Following this, the research team and their investigation has a lot of inertia (resistance to change). The superstar on the team leaving abruptly might have changed aspects of presentation, or might recommend some adjustments to the actions of the team, but I find it hard to believe that an abrupt exit due to the death of the superstar researcher would change the main thrust or substance of the investigation. The substance of the investigation includes the inquiry into trying to answer some question about how or why the world works the way it does. This inquiry is what we expect to be the primary interest of the funding panel or journal editor. Do the other participants become any less professional? The dependence of research publication and funding on superstar researchers points to human factors independent from the research itself. Does the politics of research – as it is currently done – help us achieve the next level of discovery and knowledge?

In the end, the pursuit of knowledge is an activity involving people working together. And this will means that there is no escaping the human aspects of conflict, collaboration, prestige and intuitive judgements.

Advertisement

Competition of Theories, Where We Are Now

We have many reasons to feel good about the progress that has been made with respect to understanding our physical world. Gaining understanding about gravity, electro-magnetism, and statistical mechanics has enabled technologies for space travel, computers, and refrigerators. Continued progress, however, is being limited by the screening process for new theories.

Any explanation of how the world works involves a theory. A theory makes certain fundamental assumptions (axioms), has special key terms, and uses language in careful way in order to foster clarity and preserve consistency (i.e. it is not the case that a statement and it’s negation are both true). We like and want theories that let us know more about the world around us. We  especially like theories that lead to more power, more health, more free time, and other benefits.

What many people may be unaware of is this: the world of science has plenty of competing theories but unlike professional sports there are no formal rules that govern the competition.

In professional American football, does the enforcement of the rules (via referees) matter for which team wins? Yes. So how does it work in professional science? How does one theory win over another?

First, a person who is advancing a theory must be a grad student, post-doc, or professor at a college or university. For example, the movie Lorenzo’s Oil tells the story of parents with a son who is dying and they make a medical breakthrough. The breakthrough halts the progressive deterioration of their son. The process and politics involved in  academic medical knowledge rejects their breakthrough.  The benefit of the breakthrough is denied to other families who rely on and trust the medical establishment.

Being a grad student, post-doc or professor is not a guarantee that a person’s new theory will be considered, however. The person’s academic background, his or her thesis advisor, or the school where he/she works are all potential disqualifiers.  In order to be allowed onto the field of competition at all, the creator of a new theory must be from the best of schools and/or working with other top professors in the field. Theoretical work is not qualified based on its merits, rather the status of its creator.

We wouldn’t want to have to consider all possible theories — any theory proposed by anyone — but consider how this situation puts so much power in the hands of a very small group of people, say professors from the top 10-15 universities. (The size of the group depends on the field of study and the discipline that the theory is related to.) This small group of people are usually older and apt to hold on to the mental pathways that they have been successful with all their lives. This can potentially inhibit discovery. This also cultivates an environment where the politics of human relationships matters in a way that is contrary to the level of objectivity we expect in human intercourse related to science.

A theory is advanced in some written form, say a journal article or a book. People that work for the journal or the book publisher provide an initial filter to see if a work should be  accepted for peer review. They evaluate if the person is a reasonable candidate to be regarded well by others in the field and they consider the abstract to see if it seems reasonable and offers something new or interesting. This is another part of the process that fails to meet the level of objectivity desired, since it relies on the subjective opinion of a few people .

Next, the trusted professionals who perform the peer review look over the article or book and make sure that it meets basic standards: are claims supported, was the research done properly, does it fit with what is known, and according to their professional opinion is the paper acceptable. There are some basic standards in this process, but no rules to govern why one theory should win over another.

After a paper or book is published, then the competition begins in earnest.

For scientists who have familiarized themselves with paradigms and theories and the fact that they are subject to change, there is at least one principle governing the competition. If a proposed theory is more simple, explains what is known, and provides new results – new understanding – then it should win. (The power of incumbency is very strong as it probably should be.)

The comparable situation in American football would be putting 2 teams on a football field and saying that the winner is the one with the most points gained in one hour by touchdowns, field goals, safetys and extra points, with no rules regarding acceptable ways to advance the ball or change possession of the ball, no rules against roughing the kicker or pass interference. However, even this situation has more clarity than the competition between theories, because the definition of each method for gaining points: a touchdown, field goal, safety or extra point kick is much better defined than “what is known” or the property of being “more simple.”

You ask what kind of rules could be applied to theories?

I have some ideas, but that’s a post for another day.

Hopefully, you see that the current state of working together on scientific knowledge has a lot of room for improvement.