Creative Commons Attribution 4.0 International License (CC-BY 4.0)?
No comments here
Why not start the discussion?
Skip to main content
How Science Works (or Doesn't)
A guide to the functioning of the modern scientific ecosystem, excluding issues related to publication bias
Published onFeb 13, 2023
How Science Works (or Doesn't)
This post provides a quick overview of claim articles in New Things Under the Sun related to the functioning of the modern scientific ecosystem, but not related specifically to publication bias (see this index for a guide to claim articles on that topic).
Can’t find what you’re looking for? The easiest thing is to just ask me: I’m happy to point you to the best article, if there is a relevant one.
Do academic citations measure the impact of new ideas?
What does peer review know?
Can taste beat peer review?
Steering science with prizes
Building a new research field
Literature reviews and innovation
Gender and what gets researched
Biases against risky research
Conservatism in science
How a field fixes itself: the applied turn in economics
An example of successful innovation by distributed teams: academia
Science is good at making useful knowledge
Science responds quickly to retraction
Publish-or-perish and the quality of science
Do academic citations measure the impact of new ideas?
Citation counts are often used to measure the impact of scientific knowledge, but critics argue they may not accurately reflect the influence of scientific ideas.
A recent survey found that highly cited references are more likely to reflect significant influence on scientists' own work than less cited references
Alternative methods, such as natural language processing, have been explored to measure the influence of scientific papers and are also correlated with citations
Highly cited papers are more likely to be cited outside of academia.
Positive peer review reports are also predictive of more citations
The correlation between citation counts and other measures of impact is positive but not strong; you need a lot of data.
What does peer review know?
Looks at how well correlated are peer review scores with the eventual impact of scientific research (measured in terms of journal publications, patents, and citations)
At the NIH, research proposals with lower scores tend to perform worse by these metrics, as do proposals that were funded despite scores too low to normally get funding
At journals, higher peer review scores tend to be associated with more citations down the road, though not in all cases
In general though, peer review is only weakly correlated with later impact.
Can taste beat peer review?
There are various reasons individuals might outperform peer reviewers at selecting scientific grants.
At both the NSF and ARPA-E, programs where individual program officers can go against peer review seem to perform reasonably well (though they may be underused at NSF).
Another place where we see empowered individual decision-makers is among journal editors.
In economics, journal editor taste does seem predictive of future citations, even after taking into account peer review recommendations
At Cell (though not some other journals studied), editors favor more novel submissions, which helps correct for peer review’s apparent dislike of novel papers at this journal
The little evidence we have suggests a model of empowered decision-making can outperform peer review, but mostly we just know very little.
Steering science with prizes
Professional success in new scientific research topics requires a critical mass of scholars to be active in the field. This can be achieved with credible, scarce, and public coordinating mechanisms, such as prestigious prizes and honors.
Research on the Howard Hughes Medical Institute (HHMI) investigatorship has shown prizes can steer citations towards specific research topics and increase the attention received by the research of both the award recipient and their peers working on similar topics, at least if the topic is recent and not already well known.
Other work has found topics that receive scientific prizes go on to receive significantly more research attention than those that do not, based on the number of citations, articles published, number of scientists working on the topic, and number of elite scientists working on the topic.
Memorials for deceased scientists do not face the same issues related to the non-random nature of prizes, and also lead to an increase in citations for their work.
Building a new research field
It is expensive to get researchers to change their research focus
Personal views on what is important can affect what scientists choose to research, as demonstrated by the strong response of the scientific community to the COVID-19 pandemic by shifting focus to researching COVID-19 related topics.
There may not be a strong response to changing research needs outside of global coordinating events though, for instance, there is a weak link between the impact of changing disease burden on research focus.
Scientists may be less likely to change their research focus due to the challenges of coordinating with others in the field, as well as the difficulty of making significant contributions as an outsider to the field.
These barriers can be overcome by creating a new consensus that a new topic is important and changing career incentives to allow more flexibility in research focus.
The success of these approaches in promoting change in research focus is mixed, with some evidence that more freedom from grant-seeking pressures and tenure can lead to more exploration of new topics, but other evidence suggesting that these factors have little effect.
Literature reviews and innovation
Literature reviews are cited at about twice the rate of original research within both academia and the policy world (but not general media)
Measuring academic fields as networks of joint citations (two articles are related if cited together) publication of a literature review is associated with observable changes in field structure such as a reduction in fragmentation.
An experiment with wikipedia provides some evidence these changes are not spurious - citation patterns change when some topics (randomly) receive new wikipedia articles and others do not.
Outside of academia, an experiment on policymakers finds those who hear a briefing on several studies allocate more money in the direction recommended by the studies than policymakers who only hear about a single study (with the same finding).
Gender and what gets researched
Scientists and inventors may be influenced by their personal experiences in deciding what to work on, including their gender (though gender may also matter for other reasons).
Patents developed by a majority of male inventors are more likely to relate to male-focused topics and patents developed by a majority of female inventors are more likely to relate to female-focused topics. This pattern is also seen in the products developed by startups with female founders.
Research articles with more women as authors are more likely to focus on female-related topics in biomedical research, and papers with more women coauthors are also more likely to include gender and sex analysis. In the field of history, research articles with women as authors have also tended to focus on different topics than papers by men.
There is evidence that increasing representation of women in science leads to a shift in research priorities by male authors, potentially due to increased awareness and empathy among male scientists or a mainstreaming of ideas and perspectives of women.
An increase in representation of women at all-male universities led to an increase in research related to gender, potentially due to increased hiring of women faculty and changes in research preferences of pre-existing faculty.
Biases against risky research
Suppose high-risk, high-reward research tends to be polarizing: some people love them, others hate them.
This can lead to biases against high-risk, high reward research in a few ways:
One study suggests scientists themselves may dislike uneven research, where some aspects of a project are strong and others weak.
When averaging peer review scores, proposals that receive more reviews or more polarized reviews may be funded at lower rates
When reviewers can share their assessments about research with each other, there is some evidence this is more likely to pull consensus reviews down to the lower scores than to pull them up towards the higher ones.
Argues that perhaps science funders do want to reduce risk aversion, but don’t know how, because risk aversion stems from many individually small and hard to notice biases, like the ones discussed above.
Conservatism in science
It appears that novel ideas in science may face challenges in the publication and funding process.
Studies have found that highly novel work is more likely to be among the top cited papers in its field, but it may be less likely to be published in prestigious journals or to receive grant funding.
This may be because reviewers have a harder time evaluating the quality of highly novel proposals and may be more confident in their assessments of less novel ideas that are closer to their own areas of expertise.
Additionally, when a highly influential researcher (referred to as a "superstar") dies, their ideas may fade in importance and create space for alternative approaches to gain resources.
This does not mean that novel ideas cannot succeed, but rather that there may be biases against them that need to be acknowledged and addressed.
How a field fixes itself: the applied turn in economics
It can be difficult for new paradigms or methods to take hold in a field if people are biased against them and are more likely to favor those who adhere to the current paradigm.
However, change can occur if an outside group sympathetic to the innovation provides space for its proponents to grow, and if the new paradigm or method is able to establish its utility and is able to be framed in a way that is consistent with previous work.
In the case of the applied turn in economics, the policy world's preference for quasi-experimental methods and their ability to address key questions or anomalies in the field helped establish their credibility and allowed them to take root in the academic realm, aided by their compatibility with earlier work in economics.
An example of successful innovation by distributed teams: academia
Proximity was (and perhaps still is) important for forming collaborative working relationships in academia, but these relationships remain productive even when academics are far away from each other.
At the same time, the ever-rising set of knowledge needed to push the frontier and the increasing specialization of academics has pushed them to collaborate remotely more often.
Falling travel and communication costs have also favored building teams with remote colleagues with the right specialization.
While it is feasible to collaborate productively at a distance in academia, face-to-face interactions are still important for meeting new people and forming relationships.
Science is good at making useful knowledge
Citation is a measure of the value of scientific work that can incentivize scientists to promptly disclose their results and accelerate discovery, but it can also be influenced by factors other than the true usefulness of the work, leading to potential bubbles in the citation market that can steer research away from truth-seeking and towards citation-seeking.
In economics research the pursuit of citations has been shown to be partially aligned with producing work that is useful outside the field, suggesting that the citation market in economics is not purely a bubble.
Patents that directly cite scientific research also tend to be more valuable and closer to science and patents that introduce new and unusual words, which may be inspired by concepts discovered through scientific research, also tend to be more valuable.
The value of patents that cite scientific work is correlated with the number of citations the cited articles receive from other articles, suggesting that the citation market in science may not be a bubble and that science may incentivize the creation of knowledge that is broadly useful.
Science responds quickly to retraction
When research doesn't produce desired results, scientists may change directions or continue down the same path.
It can be difficult to determine when research "doesn't pan out," but one way to assess the impact of this is through the process of retraction, where papers are removed due to errors or misconduct.
Studies have shown that retracted papers receive fewer citations than unretracted control papers.
However, if the author of the retracted paper was the one who reported the problem leading to the retraction, there is no significant effect on their other work.
Additionally, papers on similar topics as retracted papers may also suffer citation penalties, particularly when the retraction calls into question the validity of the retracted paper's findings.
Publish-or-perish and the quality of science
Incentives play a role in the quality of science.
Some evidence has been found to support this:
simulations show how simple models of science can lead to the proliferation of sloppy research practices
priority races in structural biology seem to lead to lower quality research
industrial end-users perceive the quality of academic and non-academic scientific research differently
large scale observational studies on the factors associated with retraction and publication bias
However, the magnitude of these effects is not as severe as to make science irredeemably broken.
Other factors may also contribute to the problem of replication in science.
