A guide to the functioning of the modern scientific ecosystem, excluding issues related to publication bias
This post provides a quick overview of claim articles in New Things Under the Sun related to the functioning of the modern scientific ecosystem, but not related specifically to publication bias (see this index for a guide to claim articles on that topic). Click one of the following links to jump to an article overview, or simply scroll down. Click on the title of any given overview to jump to the associated claim article on New Things Under the Sun.
Citation counts are often used to measure the impact of scientific knowledge, but critics argue they may not accurately reflect the influence of scientific ideas.
A recent survey found that highly cited references are more likely to reflect significant influence on scientists' own work than less cited references
Alternative methods, such as natural language processing, have been explored to measure the influence of scientific papers and are also correlated with citations
Highly cited papers are more likely to be cited outside of academia.
Positive peer review reports are also predictive of more citations
The correlation between citation counts and other measures of impact is positive but not strong; you need a lot of data.
Professional success in new scientific research topics requires a critical mass of scholars to be active in the field. This can be achieved with credible, scarce, and public coordinating mechanisms, such as prestigious prizes and honors.
Research on the Howard Hughes Medical Institute (HHMI) investigatorship has shown prizes can steer citations towards specific research topics and increase the attention received by the research of both the award recipient and their peers working on similar topics, at least if the topic is recent and not already well known.
Other work has found topics that receive scientific prizes go on to receive significantly more research attention than those that do not, based on the number of citations, articles published, number of scientists working on the topic, and number of elite scientists working on the topic.
Memorials for deceased scientists do not face the same issues related to the non-random nature of prizes, and also lead to an increase in citations for their work.
It is expensive to get researchers to change their research focus
Personal views on what is important can affect what scientists choose to research, as demonstrated by the strong response of the scientific community to the COVID-19 pandemic by shifting focus to researching COVID-19 related topics.
There may not be a strong response to changing research needs outside of global coordinating events though, for instance, there is a weak link between the impact of changing disease burden on research focus.
Scientists may be less likely to change their research focus due to the challenges of coordinating with others in the field, as well as the difficulty of making significant contributions as an outsider to the field.
These barriers can be overcome by creating a new consensus that a new topic is important and changing career incentives to allow more flexibility in research focus.
The success of these approaches in promoting change in research focus is mixed, with some evidence that more freedom from grant-seeking pressures and tenure can lead to more exploration of new topics, but other evidence suggesting that these factors have little effect.
Scientists and inventors may be influenced by their personal experiences in deciding what to work on, including their gender (though gender may also matter for other reasons).
Patents developed by a majority of male inventors are more likely to relate to male-focused topics and patents developed by a majority of female inventors are more likely to relate to female-focused topics. This pattern is also seen in the products developed by startups with female founders.
Research articles with more women as authors are more likely to focus on female-related topics in biomedical research, and papers with more women coauthors are also more likely to include gender and sex analysis. In the field of history, research articles with women as authors have also tended to focus on different topics than papers by men.
There is evidence that increasing representation of women in science leads to a shift in research priorities by male authors, potentially due to increased awareness and empathy among male scientists or a mainstreaming of ideas and perspectives of women.
An increase in representation of women at all-male universities led to an increase in research related to gender, potentially due to increased hiring of women faculty and changes in research preferences of pre-existing faculty.
It appears that novel ideas in science may face challenges in the publication and funding process.
Studies have found that highly novel work is more likely to be among the top cited papers in its field, but it may be less likely to be published in prestigious journals or to receive grant funding.
This may be because reviewers have a harder time evaluating the quality of highly novel proposals and may be more confident in their assessments of less novel ideas that are closer to their own areas of expertise.
Additionally, when a highly influential researcher (referred to as a "superstar") dies, their ideas may fade in importance and create space for alternative approaches to gain resources.
This does not mean that novel ideas cannot succeed, but rather that there may be biases against them that need to be acknowledged and addressed.
It can be difficult for new paradigms or methods to take hold in a field if people are biased against them and are more likely to favor those who adhere to the current paradigm.
However, change can occur if an outside group sympathetic to the innovation provides space for its proponents to grow, and if the new paradigm or method is able to establish its utility and is able to be framed in a way that is consistent with previous work.
In the case of the applied turn in economics, the policy world's preference for quasi-experimental methods and their ability to address key questions or anomalies in the field helped establish their credibility and allowed them to take root in the academic realm, aided by their compatibility with earlier work in economics.
Proximity was (and perhaps still is) important for forming collaborative working relationships in academia, but these relationships remain productive even when academics are far away from each other.
At the same time, the ever-rising set of knowledge needed to push the frontier and the increasing specialization of academics has pushed them to collaborate remotely more often.
Falling travel and communication costs have also favored building teams with remote colleagues with the right specialization.
While it is feasible to collaborate productively at a distance in academia, face-to-face interactions are still important for meeting new people and forming relationships.
Citation is a measure of the value of scientific work that can incentivize scientists to promptly disclose their results and accelerate discovery, but it can also be influenced by factors other than the true usefulness of the work, leading to potential bubbles in the citation market that can steer research away from truth-seeking and towards citation-seeking.
In economics research the pursuit of citations has been shown to be partially aligned with producing work that is useful outside the field, suggesting that the citation market in economics is not purely a bubble.
Patents that directly cite scientific research also tend to be more valuable and closer to science and patents that introduce new and unusual words, which may be inspired by concepts discovered through scientific research, also tend to be more valuable.
The value of patents that cite scientific work is correlated with the number of citations the cited articles receive from other articles, suggesting that the citation market in science may not be a bubble and that science may incentivize the creation of knowledge that is broadly useful.
When research doesn't produce desired results, scientists may change directions or continue down the same path.
It can be difficult to determine when research "doesn't pan out," but one way to assess the impact of this is through the process of retraction, where papers are removed due to errors or misconduct.
Studies have shown that retracted papers receive fewer citations than unretracted control papers.
However, if the author of the retracted paper was the one who reported the problem leading to the retraction, there is no significant effect on their other work.
Additionally, papers on similar topics as retracted papers may also suffer citation penalties, particularly when the retraction calls into question the validity of the retracted paper's findings.
Incentives play a role in the quality of science.
Some evidence has been found to support this:
simulations show how simple models of science can lead to the proliferation of sloppy research practices
priority races in structural biology seem to lead to lower quality research
industrial end-users perceive the quality of academic and non-academic scientific research differently
large scale observational studies on the factors associated with retraction and publication bias
However, the magnitude of these effects is not as severe as to make science irredeemably broken.
Other factors may also contribute to the problem of replication in science.