Scientific advancement includes two types: one is incremental and the other is transformative. The incremental one is typically consists of conservative small improvements based on widely agreed frameworks or principles. Obviously, effort of this type is easy to gain support from the community. Contrarily, the transformative one involves disruptive / unorthodox ideas which more often lie outside of the box of our collective thinking or even break some existing principles. Not surprisingly, these kind of so-called high-risk / high-reward ideas and studies are difficult to get funded, especially in the beginning. To balance both types of efforts, open science practices are in need.
Current review procedures by various funding agencies are not friendly to high-risk or unorthodox proposals (see my earlier discussions). Private foundations that are supposed to support such high-risk high-payoff projects are basically operated in a black box. A transparent pro-risk review system (similar to Venture Capital & Angel Investment in the business world) is urgently needed for future innovation in science.
Before we begin to discuss detailed review mechanisms, it is worth clarifying the confusing concept of high risk. There are possibly two kinds of risks implied depending on situations. One is related to the possibility of the idea being correct under the current understanding within the scientific community and this meaning for high risk will be used in this article. A novel idea, if it is eccentric enough, could be regarded as high-risk or low-possibility in this sense. However, such a perceived low possibility could become an obvious certainty with hindsight. It just shows how stubborn a human being’s mindset can be.
Another meaning of high risk is about the technical feasibility of carrying out the idea. It is more related to the capability of the team who have proposed the idea and/or the availability of required resources. We’ll refer to this kind of risk as feasibility or capability later.
The reward factor of a proposal for original research is reflected on its impact or significance on science if successfully done (I’ll use physics as an example below). Obviously, the most significant ones are truly revolutionary like what Newton and Einstein accomplished, roughly one of a kind for every 500-1000 years in a field. More likely, we would see disruptive ideas like Quantum Mechanics by Heisenberg, Standard Model by Weinberg, Glashow and Salam, something occurring every 50-100 years or a person’s lifetime. The full quantum theory itself is probably the most revolutionary idea in physics deserving the highest mark. But its individual milestones (QM, QED, SM, etc.) established by a large number of physicists maybe are better qualified in this category. The next level is breakthroughs we hear now and then every a few years that are worthy of a good Nobel prize. Proposals of these three significance levels should be considered as high risk/high reward by scientific VC and Angels.
The next two levels of significance (typically corresponding to mainstream studies), on the other hand, are what the traditional review system can focus on. That is, the substantial works that could result in good PRL-level publications, and incremental studies – routine ordinary works by typical scientists.
Then the last two levels are bottom-dwellers that should never be funded. One is useless works that does not provide any new results or improvement. The other is detrimental to science if non-scientific or inappropriate approaches are advocated. Now we see the danger of mixing this last detrimental level with the three top high-risk levels by reviewers with biased glasses. To prevent such problems, we need to introduce some type of scientific standards for all proposals.
Such scientific standards should be MINIMUM just like how a government should treat startups in business world. A business license is simple and easy enough for any worthy startup to have a chance to thrive. For a scientific proposal, we just need to make sure that it follows typical social standards (no abuse, no plagiarism, etc.) and is largely self-consistent. No judgement on a proposal’s hypotheses should be made as far as it is logically consistent or sound within itself in the eyes of the most open-minded scientists. Only the ones that don’t meet the minimum scientific standards will be set to the detrimental level.
The two factors of risk and significance are often correlated, i.e., high risk frequently implies high reward, and vice versa. But not always. Both has its subjective downsides due to human’s limitations. The risk (eccentricity) looks low to open-minded scientists while seems to be much higher to narrow-minded ones. The impact of an idea might not be fully recognized when it was first proposed. Taking both factors into account would give us the best chance to save the truly transformative ideas. The best strategy for scientific venture investment is to consider top three levels of each of these two factors equally, or at least not as the primary parameter for funding priority.
Then what criteria should we use for priority of support for the top three levels in risk and/or significance? The best option would be testability for scientific proposals. The testability criteria should consider three aspects: can it provide new predictions; how unique are the predictions; and how ready are the technologies for the test. It has no value for venture investment if it can not make any new predictions that are outside of our current understanding. It is more valuable if the new predictions are unique (i.e., compatible with none of other competing models). If the needed technologies exist or are ready in near future, then the proposal should have higher priority for support. (See the point system below)
In summary, the review panel should first single out proposals that meet the minimum scientific standards. Then they should grade the proposals based on the factors of risk and significance in order to find non-mainstream ideas. The high-risk high-reward ones are then selected if they score three points or more in either category (i.e., the top three levels). The funding priority is then seated based on their testability. With the same testability score, capability and feasibility points (possibly also significance/risk level) could be used for the funding order in the same tier.
Now the question becomes how we should form the review panel. If we use the experts in the same field of the proposal, we’ll probably miss most of truly transformative proposals if not all. A better way is to choose experts in a different sub-field or field or maybe in a totally different discipline, or even mix them all. And the grade of a proposal is assigned with the highest point it received from any reviewer. And it is probably better to replace the panel more often than not. Fresh eyes and different perspectives will help us uncover more gems out of all proposals.
We also need to keep the two avenues of venture investment for science. One is from government (e.g., NSF/DOE) similar to VC and the other is from private foundations and individuals like Angel Investors in business. For balanced support between mainstream and non-mainstream studies, we could learn from the business world. If venture capital and angel investment in business occupy 5% of total investment (just a guess, I am not sure about the ratio), then we could allocate similar portion in venture investment of science. My guess is that we probably have spent much less on truly high-risk high-reward scientific projects than we should. With the assistance of the above-proposed review system, hopefully we can provide the best environment to foster more and more truly transformative ideas as we should.
POINT SYSTEM: (points can also be assigned with any real number in between)
Scientific standards: minimum – self-consistent (even self-consistency is debatable; at least not have to be fully self-consistent to all details; probably good enough if largely logically consistent or no major flaws in the view of most open-minded scientists)
Significance: revolutionary (5 points) (Newton & Einstein, every ~500-1000 years), disruptive (4) (Heisenberg, QM, ~50-100 yrs), breakthrough (3) (Nobel prize-level, ~5-10 yrs), substantial (2) (PRL-level, ~1yr), incremental (1) (ordinary routine works), useless (0), detrimental (-1 point)
Risk/possibility: impossible (5), nearly impossible (4), unlikely (3), possible (2), likely (1), certain (0), inevitable (-1 point)
Capability: master (4) (leading), expert (3) (established), experienced (2) (Ph.D-level), student/new (1) (bachelor-level), untrained (0) (no post-secondary education in related discipline).
Feasibility: dream team and resources (4), well-balanced team and resources (3), capable team and resources (2), inexperienced and not enough resources (1), no experience and no resources (0).
Testability: unique anomalous predictions testable with existing technologies (5); unique anomalous predictions testable with near future technologies (4); anomalous predictions (compatible between some known models) testable with existing technologies (4); anomalous predictions (compatible between some known models) testable with near future technologies (3); unique anomalous predictions testable with far future technologies (3); anomalous predictions (compatible between some known models) testable with far future technologies (2); unique anomalous predictions that are impossible to test (2); anomalous predictions (compatible between some known models) that are impossible to test (1); no new predictions / not testable at all (0).
No comments:
Post a Comment