How to Accelerate Technological Progress

Matt Clancy

This essay is a map of the content on New Things Under the Sun that relate to accelerating technological progress. For this essay I’ll basically take it as a given that technological progress is desirable. While I think that’s true, I don’t necessarily think it is obvious (see the article “What are the returns to R&D?” for one theoretical argument for why we should be spending more on R&D and “When technology goes bad” for some reasons progress could be bad). The main message is that there are lots of factors that collectively set the rate of technological progress - it’s not down to any one thing.

Unless otherwise noted, all of the below is founded on a base of evidence derived from academic studies. That doesn’t mean it’s correct: academic knowledge isn’t received from on high; it is always contested and always evolving. But it does represent a real effort to think through these issues with data and careful thought by a community of experts. So I think it’s worth hearing.

The raw inputs of innovation: human beings

Innovation comes from human beings (at least for now). So one of the most important inputs to innovation is, simply, more people. And indeed, we do have some evidence that larger populations innovate at a faster rate, though most of this evidence relates to innovation that predates the modern world. The article “More people leads to more ideas” reviews some evidence consistent with this view from isolated islands, lab experiments, and the history of the human race.

So anything that increases our population could increase the rate of technological progress.

Of course, not all people become innovators. One reason may be that they lack the resources to innovate (see below, on incentives). But another might be that they lack the very idea of trying new things. People take their cues about what range of options they should consider from the people around them. And if the people around you simply accept problems as unsolvable, you might do the same and not try to innovate yourself. That’s one way that innovation can become self-perpetuating: if lots of people are trying to innovate, that can inspire others to view problems as things to be solved rather than mitigated.

The best evidence we have that this notion is true comes from a variety of studies about entrepreneurship. As discussed in “Entrepreneurship is contagious”, we tend to find entrepreneurs in social clusters, and when we expose people who are not themselves predisposed to be entrepreneurs to entrepreneurship, it seems to increase the probability they go on to found businesses. “The ‘idea’ of being an entrepreneur” provides further related evidence: the impact of entrepreneurial peers is highest when they occupy a similar social position and lowest when someone probably already has the “idea” of being an entrepreneur. Moreover, entrepreneurs self-report that family/friend role models are important reasons for them starting a business. Finally, we have a bit of evidence you can transmit these ideas even through media. That all suggests people might obtain the “idea” of trying new things from seeing people like themselves try new things.

That implies one way to accelerate technological progress is to expose people to examples of people like themselves innovating.

Now let’s return to an assumption we made at the outset of this section, that humans are necessary inputs to innovation. We’re living in an era of rapidly advancing artificial intelligence; what if we drop that assumption? There’s not much in the way of empirical data on what happens when machines take over the task of innovating because it hasn’t happened yet, but the article “What if we could automate invention?” looks at what one particular class of economic models say on this question. These models imply that if you really could replace all the human inventors with equally capable robots, then growth accelerates ever faster into something wild and unparalleled in human history.

But that singularity-style result requires fully replacing humans with robots. A more nuanced model thinks of innovation as consisting of a large number of different tasks, some of which are automated, some of which can only be done by humans, but all of which are necessary for innovation to occur.1 When humans are still the only ones capable of doing some of the necessary innovation work, then the accelerating dynamic described doesn’t work.

Instead, these models predict the rate of progress depends not on how much of the innovation pipeline is automated, but the rate at which we automate more and more of it.

Innovation with and without a map

So now you’ve got some people who are open to new and better ways of doing things. How successful are they likely to be?

People can actually get pretty far innovating without deep causal knowledge of how things “really” work. This occurs via a process analogous to biological evolution; tinker and retain any improvements. The article “Progress in programming as evolution” provides the most direct evidence of innovation as a sort of evolutionary process, at least in the context of coding. The article “Learning curves are tough to use” looks into another example of this phenomenon, wherein experience in a particular production process drives progress (though note that a lot of the literature on this kind of innovation greatly overstates how predictable and useful it is - see “Standard evidence for learning curves is not good enough”).

So, to encourage innovation (at least to some extent), just encourage people to get started using new production processes.

But to move really quickly and to make the kinds of big jumps that you simply can’t arrive at via a process of blind tinkering, it helps to have a map of the terrain. Knowledge can be that map. The article “Science as a map of unfamiliar terrain” reviews evidence from a few papers that patents which use science are more valuable, especially when the patented invention occupies relatively unfamiliar knowledge terrain.

More generally, while modern science has its share of problems, on the whole it generates knowledge that helps us better understand the world. A series of natural experiments where scientific progress was rapidly and unexpectedly sped up or slowed down, discussed in “More science leads to more innovation,” goes on to show that basic scientific discoveries are, indeed, spun out into new and valuable technologies. The article “Science is good at making useful knowledge” provides some additional, more correlational, evidence: patents that cite science seem to outperform those that don’t. And even technologies that do not directly rely on new scientific discoveries frequently rely indirectly on them. Evidence for this, suggested by the predictive power of patent citations, is discussed in “Ripples in the river of knowledge.”

Lastly, the articles “How common is independent discovery?” and “Contingency and science” imply there is plenty of unexplored scientific terrain out there, waiting to be discovered. “How common is independent discovery?” argues it is, in fact, not particularly common for multiple groups to independently discover the same idea. That suggests we have not reached a saturation point wherein additional researchers would mostly end up repeating the work done by others. The article “Contingency and science” complements this line of evidence by reviewing evidence that the direction taken by science has a lot of contingency; that suggests the existence of paths not taken, which might be fruitfully explored by additional scientists.

That means another obvious way to accelerate technological progress is to increase our knowledge of how the world works by expanding basic scientific research.

Steering Science

More science is better, but assuming there are limits, how should we choose which scientific topics get more support and which get less?

In general, targeting the science most likely to be useful for technological progress is actually quite challenging. The article “Knowledge spillovers are a big deal” looks at several examples where the intended beneficiary of a specific research project accounted for only a small part of the total benefits. In other words, the R&D had many unintended benefits, implying that precise targeting is challenging. Relatedly, there tends to be quite a long time lag between when scientific discoveries are made and when technological applications become available. The article “How long does it take to go from science to technology” looks at citations from patented inventions to academic work, as well as statistical correlations between scientific research and industry productivity, to estimate that the time between a scientific discovery and a technological implementation is often on the order of 20 years.

All told, given a long enough time horizon, scientific funding should not strive to be too targeted to specific problems since spillovers may provide unexpected solutions.

One of the most important ways we actually allocate funds to different scientific priorities at present is via academia. The academic ecosystem has created a system of incentives that pushes researchers to do novel work within a framework of methods and background knowledge sufficiently well understood by a community of experts that its merit can be evaluated. This system works ok - as noted above, science does seem to lead to technological advance - but it’s far from perfect.

To begin, who are the scientists choosing what to work on? The articles “Gender and what gets researched” and “Geography and what gets researched” provide some evidence that who is able to do research has a measurable impact on what gets researched. Specifically, having more women inventors and researchers in a field helps a field avoid gender-related blindspots, both because women tend to be more likely to work on different things then men and because men are more likely to be responsive to the priorities of women when there are more female researchers in a field. A similar dynamic plays out on a geographical axis: researchers seem to be disproportionately drawn to local problems (though this affinity can be overcome).

In general, if we want research to cater to the priorities of society at large, then it helps to have scientist at large represented in the research community.

Beyond the identity of the scientific community, the processes are also not perfect. Consider anonymous peer review, which forms an important input into how most science funding is allocated. The article “What does peer review know?” presents some evidence that peer review does generate useful (albeit pretty noisy) information about the likely success of research projects. However, “Biases against risky research” presents some reasons to believe this system might also be biased against certain kinds of risky (but ultimately more valuable) research streams. “Can taste beat peer review?” argues that some problems with peer review might be overcome by relying on well incentivized and carefully selected individual decision-makers, but the evidence base here is thin.

The article “Conservatism in science” looks at other evidence that science has a bias against very novel research questions, preferring work that is more closely tied to existing paradigms. That article suggests this might, in part, be due to budget constraints; if it’s hard for very unusual proposals to score top marks, and the budget is so tight that only proposals with top marks get funded, then novel research might be underfunded.

Peer review does provide (limited) information on how well research proposals will fare, but may be subject to biases against some kinds of risky or novel research.

Setting aside the questions of what research projects get supported, there are a different set of issues related to the conduct of research and dissemination of results. The articles “One question many answers” and “Publication bias is real” for example, document serious issues with the research and publication process. The articles “Publication bias without editors? The case of preprint servers” and “Publish-or-perish and the quality of science” look at some evidence that these problems are at least partially due to factors related to the publication system (and the associated prestige of publishing). Both articles argue these effects are real, but probably not that large in their magnitude. The article “Why is publication bias worse in some disciplines than in others” points instead to issues related to the quality of empirical methods as an important cause of publication bias. Essentially, to the extent that empirical work is “more art than science” in a field, the easier it is to justify not publishing work that goes against the prevailing wisdom.

All in all, it seems almost certain that we can improve our processes for selecting which research projects to fund, and how we disseminate and communicate findings. But the evidence base is thin enough that I’m nervous recommending any particular course of action.

To close, let’s assume we have identified a domain where the traditional scientific incentive system is not working well; specifically, we’ll assume it is underinvesting in research on an important topic. Now what?

The article “Building a new research field” looks at two specific challenges that come with trying to encourage development of an understudied topic. First, because it is hard to do great science in isolation, there is a coordination challenge with getting a critical mass of scientists to begin work in a new field. One tool for solving this challenge is scientific prizes, discussed at length in the article “Steering science with prizes.”

It seems scientific prizes can sometimes galvanize research in overlooked areas by creating credible, public, signals of areas of promising research.

A second challenge to building a new research field is that it is challenging to do great work in a new field, at least on average. That creates a strong incentive for scientists to “stay in their lane” rather than branch out and try new things. If we want to encourage scientists to try new things regardless, one approach may be to offer some insulation from the usual demands of academia to continuously produce high profile publications. Alas, as discussed in “Building a new research field” the evidence on the efficacy of this strategy is mixed.

Humans + knowledge = faster innovation

Now, we’ve got people looking to solve problems, and knowledge to help them find solutions. That knowledge isn’t much good if it’s not in the head of the people who need it. For that reason, it’s also important to circulate knowledge. You can do this in a variety of ways.

The article “Free knowledge and innovation” looks at several experimental and quasi-experimental studies on different forms of free knowledge depositories (think libraries and wikipedia) to show making knowledge freely accessible has a measurable positive impact on the rate of innovation.

One way to accelerate innovation is to create free knowledge resources.

Not all knowledge is codified in text though; some of it lives primarily in people’s heads. One way to get at this knowledge is just to pick knowledgeable people up and drop them down in a community that lacks the requisite knowledge. The article “Importing knowledge” looks at this in the context of immigration, examining a variety of cases where an influx of immigrants with a specific knowledge advantage led to that knowledge advantage being picked up in the host country.

Policies that encourages high skilled immigration, especially from people with different knowledge advantages, can also accelerate innovation.

Lastly, the article “Urban social infrastructure and innovation” looks at some evidence that creating ways for people to mix locally - in bars, cafes, and in dense neighborhoods - helps circulate local knowledge. The article “Innovation at the Office” surveys similar evidence for colleagues working within the same building and the article “Why proximity matters: who you know” complements both articles, highlighting the ways that proximity seems to be important especially for encouraging meetings between people who might not normally interact. The article “Local Learning” provides some additional evidence that local interactions facilitate the exchange of novel ideas among academics working together at the same university. Lastly, “Academic conferences and collaboration” indicates that social mixing doesn’t only mean where you live and work - it looks at how conferences encourage the formation of new collaborative relationships.

Another way to accelerate innovation is to encourage social mixing of people with different sets of knowledge.

Incentivizing Innovation

Once you’ve got people who are open to trying to solve problems, and who have the knowledge to help them innovate, innovation is now in their choice set. But for them to choose to innovate (instead of doing some other thing in their choice set), you’ve got to make it an attractive option, either by lowering the costs of innovating or raising the benefits of it.

One simple way to lower the costs of innovation is simply to provide the funding for innovation activities, e.g., from the government.

That can be risky. The government might not know who to fund, and recipients might not use the money as intended. But we do have pretty good evidence that this is not an insurmountable problem for governments. “An example of high returns to publicly funded R&D” looks at some quasi-experimental evaluations of programs like the US government’s small business innovation and research grants. These studies show the funds make a difference, and probably have quite good returns on investment.

With some caveats, we also have decent evidence that private sector R&D is pretty responsive to market signals. “When extreme necessity is the mother of invention” presents some straight-forward evidence that innovation is highly responsive to demand, in this case the kind of very extreme demand that arises in the wake of global crises (think global pandemics, world wars, and energy crises). For some evidence from more “normal” times, “Medicine and the limits of market-driven innovation” looks at a number of studies in medicine that show the R&D of pharma and other private sector health companies is quite responsive to market signals. When there’s more money in curing a disease, there’s more R&D. But at the same time, that R&D tends to be limited to stuff that’s already close to the market. More fundamental research doesn’t seem very responsive to market signals. Similarly, “Pulling more fuel efficient cars into existence” reviews studies that use patents and direct measures of technological progress in fuel efficiency to argue carmakers are responsive to market signals as well. Specifically, higher fuel prices and explicit emissions standards (enforced by fines for failing to meet them) seem to work quite reliably at inducing incremental innovation, though it is less clear they work for more radical kinds of innovation.

Still, to rapidly pull technologies close to viability onto the market, making it more profitable to do so seems to work.

Competitive markets are good at efficiently allocating resources in an economy, by aggregating information and equating the value people derive from a product with the marginal cost of producing it. But not all products are well served by the market system, especially public goods (of which knowledge is just one example). It can be hard to efficiently allocate resources to public goods without market signals. One alternative that has been proposed is Buterin, Hitzig, and Weyl’s proposal for a mechanism to fund public goods in a decentralized fashion, discussed in the article “Optimal Kickstarter.”

As a general principle, the way we incentivize innovation is itself a (social) technology, and we should try to make progress on this technology as well, by trying new things.

Organizing Innovators

Now you’ve got knowledgeable people open to solving problems and incentivized to do so. Can we say anything about the way you should approach solving a problem?

One common finding in studies of innovation is that “The best new ideas combine disparate old ideas.” In other words, creating an environment where inventors are exposed to disparate ideas can sometimes lead to unexpected breakthroughs. As noted earlier, this can be one rationale for innovators to physically locate in a space with many other knowledge workers whether that be a city or building. But there may be other ways to encourage workers to become exposed to unexpected ideas, such as by shuffling workers between project types, allocating some portion of their time to undirected exploration/personal projects, or creating a culture of open communication between different divisions (though note these suggestions are speculations on my part, not derived from studies reviewed on this site).

Important, but perhaps unexpected, innovations may be more likely when inventors are exposed to many different kinds of idea.

Another basic factor to consider is the size of the team that works on a problem. As described in the article “Are ideas getting harder to find because of the burden of knowledge?” there has been a trend towards greater use of teams in the innovation process, possibly because the amount of knowledge that must be directed at a problem tends to grow over time. The article “Highly cited innovation takes a team” surveys some work on the efficacy of innovative teams in different domains, where impact is mostly measured by creating highly cited patents or projects. It tends to be the case that bigger teams are associated with more impact, with some evidence that avoiding “weak links” in a team is more important than including a super-productive “star.” But a big caveat here is that all this evidence is correlational - we don’t have much in the way of experiments (natural or artificial) to draw on here.

Many problems seem to be best solved with a team.

We also have some work (just one study, so be cautious putting too much weight on it) that while teams are good at producing highly cited work. Coincidentally, you see the same dynamic at the level of the organization. The article “The size of firms and the nature of innovation” documents that larger firms seem to develop more incremental innovations than smaller ones, on average (even as they seem to spend on R&D at the same proportional rate as small firms).

A variety of reasons have been proposed for this correlation between the size of an innovating group and its incremental nature. The article “Big firms have different incentives” argues, at least for large organizations, part of the problem may be simply be the differing incentives faced by large firms. Since they sell more units, incremental cost reductions are more valuable for large firms than for small ones. On the sales side, large incumbent firms partially cannibalize their own sales when they invent new and superior products, which further disincentivizes this kind of innovation.

Large teams and large organizations appear to produce more incremental innovations, at least on average.

A final question to consider is whether a team of inventors (or scientists) should work together in the same physical space, or remotely. In some cases, there is no question, because work requires physical proximity to research instruments. But in other cases, it is entirely possible for a team to collaborate at a distance most of the time, though likely gathering in one place on at least a few occasions. As discussed in the article “Remote breakthroughs” the conventional wisdom that colocation is ideal for generating breakthrough ideas seems to have been true for most of the twentieth century, but is probably less applicable today in the era of digital collaboration. Indeed, some evidence suggests remote teams today are more capable of generating remote breakthroughs, possibly because they can draw on a more varied set of local knowledge environments or possibly because remote communication disrupts some of the social dynamics that normally hinder productive collaboration.

Where it is feasible, remote collaboration among team members should no longer be assumed to be inferior to working together in the same physical space.

Summary

To summarize. Innovation happens when people apply knowledge to problems. There are lots of points where we can nudge that process along faster. More people (or more robots to take on some of their tasks). More people considering innovation. More knowledge. Better organization of science. More communication of knowledge. More resources to innovation. More benefits from innovating. Better management of innovators. Each of those might only have a small impact individually; but they stack up. Moreover, they compound, year on year.

There is so much more to say about all this; the academic literature is vast. So this essay will grow, bit by bit, as New Things Under the Sun gets more and more new articles added. But hopefully this has given us some places to start.

Note: this essay is continuously updated as relevant articles are added to New Things Under the Sun. To keep abreast of updates, subscribe to the site newsletter.

Outline

How to Accelerate Technological Progress

The raw inputs of innovation: human beings

Innovation with and without a map

Steering Science

Humans + knowledge = faster innovation

Incentivizing Innovation

Organizing Innovators

Summary

Outline

Introduction

The raw inputs of innovation: human beings

Innovation with and without a map

Steering Science

Human + Knowledge = faster innovation

Incentivizing Innovation

Organizing Innovators