Study of Innovation

This post provides a quick overview of claim articles in New Things Under the Sun related to methodological questions in the study of innovation.

Can’t find what you’re looking for? The easiest thing is to just ask me: I’m happy to point you to the best article, if there is a relevant one.

Do Academic Citations Measure the Impact of New Ideas?
- (summary) (article)
Measuring Knowledge Spillovers: The Trouble With Patent Citations
- (summary) (article)
How many inventions are patented?
- (summary) (article)
Patents (Weakly) Predict Innovation
- (summary) (article)
Do studies based on patents get different results?
- (summary) (article)
Can we learn about innovation from patent data?
- (summary) (article)

Do academic citations measure the impact of new ideas?

Citation counts are often used to measure the impact of scientific knowledge, but critics argue they may not accurately reflect the influence of scientific ideas.
A recent survey found that highly cited references are more likely to reflect significant influence on scientists' own work than less cited references
Alternative methods, such as natural language processing, have been explored to measure the influence of scientific papers and are also correlated with citations
Highly cited papers are more likely to be cited outside of academia.
Positive peer review reports are also predictive of more citations
The correlation between citation counts and other measures of impact is positive but not strong; you need a lot of data.
(go to article)

It is tempting to treat the citations patents make to each other as indicators of knowledge flows, but this is potentially misleading. Patent citations may not reflect genuine knowledge flows for several reasons:
- Many citations are not added by the inventor, or are added after the invention is completed, as part of the patent application process
- Some important citations may be purposefully omitted to try and game the patent examination process
- Some irrelevant citations may be purposefully added to try and game the patent examination process
There is some evidence the signal-to-noise ratio of patent citations has begun to deteriorate markedly
Nonetheless, there is signal with the noise, and used carefully citations can be informative
(go to article)

Most US R&D is performed by firms that have patents.
If you ask manufacturing firms how many of their inventions are protected by patents, they say:
- About half of product inventions
- About one third of process inventions
If you have a list of inventions are try to identify how many are patented, either by asking the inventors or manually searching for related patents:
- Most FDA-approved drugs have some form of patent protection.
- Typically 10-30% of entrants or winners of innovation contests can be matched to patents.
- Around 20% of new consumer products in drug and grocery stores are introduced by firms that have patents in related product categories.
(go to article)

Counts of patents are correlated with several measures of innovation, though this correlation is pretty weak. These include:
- Introduction of new consumer products
- Release of semiconductors with more information storage capacity
- The rate of improvement of performance metrics for various technologies
- R&D spending
Patent counts are not robustly correlated with total factor productivity growth though.
However, various approaches for identifying high quality patents do find counts of these are associated with total factor productivity growth.
(go to article)

Surveys posts on New Things Under the Sun (up through March 2024) that cover studies based on both patent and non-patent data, and looks for substantive disagreement between studies.
In about 85% of posts, there isn’t any substantive disagreement between patent and non-patent data analysis, in my judgment.
- Note the high degree of agreement applies to cases where both myself and the paper authors thought a dataset that was appropriate to the problem was used.
To test for selection effects driven by my choice on what to write about, I repeated this exercise for updates to my posts (since updates are less likely to be selected for agreement), obtaining similar results.
(go to article)

An argument post that synthesizes and extends the other posts about using patents as data.
Discusses:
- evidence on the share of innovations that receive patent protection.
- evidence on how many patents describe valuable inventions, as well as ways to identify these patents.
- how well patents predict other measures of innovation
- sources of bias in the patent data that differentiate it from a random sample of invention
Argues that many of these biases are correctable, and that appropriately handled, patents are a useful source of data on innovation.
That said, as with most of social science, multiple lines of evidence are desirable.
(go to article)