Evidence that local interaction leads to deeper engagement with new ideas
In my experience, the internet can’t be beat for encountering a diversity of ideas. But often, that encounter is at a pretty surface level. You read a tweet; a headline; a blog post synthesizing some studies, etc. Nothing wrong with surface level engagement - you can’t engage in everything deeply. But pushing the innovation frontier increasingly requires deep engagement with at least some domain of knowledge. And there are reasons to think that offline/in-person interaction might be better for forging that kind of deep engagement with new ideas.
To start, let’s look at van der Wouden and Youn (2023), which wants to see if in-person collaboration on academic projects more reliably leads to the transfer of knowledge between coauthors than remote collaboration. To answer that question, the authors gather data on 1.7mn academics who, at some point over the period 1975-2015, produce a sequence of three papers that exhibit a very specific pattern. In reverse order, they need:
The last paper in the sequence to be solo-authored
The second-to-last paper to be coauthored with at least one other author
At least one more prior paper.
They’re going to pull all that information from the Microsoft Academic Graph.
Next, they want an estimate of what knowledge domains the academic is fluent enough in to publish an original research paper in. To get those, they leverage the 292 subdisciplines that the Microsoft Academic Graph tags papers with. By looking at the subdisciplines tagged to your work, they can get an idea about what you are an expert in, and also how your areas of expertise grow over time. Moreover, by focusing specifically on solo-authored work, they can be most sure that it’s really you who is the expert, and not one of your coauthors.
The main idea of the paper is to figure out an academic’s areas of expertise based on all papers they’ve published, up to and including the first one in the sequence of three alluded to above. Next, they look to see if the second paper in the above sequence was conducted with local or remote collaborators. Finally, they look at the final paper in the sequence, which was solo-authored, and see if it is tagged with any new subdisciplines, relative to all your papers up-to-and-including the first one in the sequence. If so, they take that as evidence that the author gained expertise in a new subject in between the first and third paper, possibly via their interaction with their collaborators on the second paper. Lastly, they can see if this “learning” effect is more common when you work with local or remote coauthors.
In the following figure, we can see how the probability of writing a solo-authored paper tagged with a new subdiscipline changes when you work with increasingly distant colleagues on your previous paper. van der Wouden and Youn call this the “learning rate.” If your collaborators were local (under 700m away, a 10 minute walk), then about 7.5% of the time, your next paper is on something you haven’t written about before. If your collaborators are out of town, say more than 25km, the probability drops to more like 4.5%.
This pattern is consistent across fields, though stronger in some fields than others. For example, the relative probability of pivoting to a new topic after a local collaboration compared to a distant one is generally higher in STEM fields than in non-STEM fields. Moreover, while the figure above is raw data, you get a similar effects when you toss in a bunch of additional control variables: the number of coauthors, the career stage of the academic, the ranking of the institution they are affiliated with, and so on.
The evidence here is a bit circumstantial: we don’t actually know what happened in between the first and third paper. But Duede et al. (2022) provides some complementary evidence that colocation is important for facilitating deep engagement with new ideas from our peers.
Duede and coauthors have a big survey where they ask thousands of academics across many fields about citations they made in some of their recent work. Among other things, they asked respondents how well they knew the cited paper, as well as how influential was the citation to the respondent’s work. In the latter case, respondents rated their citations on a scale from “very minor influence”, which meant the respondent’s paper would have been basically unchanged without knowledge of the cited reference, to “very major influence”, which meant the cited reference motivated the entire project.
If we have a way to measure the geographic distance between the authors and the “intellectual distance” between the citation and the author’s normal expertise, we can see how the two are related: does being close in space facilitate learning about ideas you wouldn’t normally know about? Computing distance in space is straightforward: Duede and coauthors just code whether authors are in the same department, same institution, same city, or same country. To measure intellectual distance, they rely on the similarity of the title and abstract of the citing and cited paper, as judged by natural language processing algorithms. This algorithm judges papers to be more similar if they contain words that are themselves more closely related to each other.
Duede and coauthors find if you and the author of a paper you cite are at the same university, then you are indeed more likely to say you know the cited work well and that it was influential on you. That’s consistent with the notion that being around other people facilitates deep engagement in their work.
But what’s interesting is that the strength of this relationship is stronger if the cited and citing paper are less similar to each other. In other words, if you cite a paper that’s surprising, given the topic you are working on, you are more likely to say you know that paper well and that it influenced you if the author is at the same university. That’s quite consistent with colocation being a useful way to learn about ideas you wouldn’t otherwise encounter in the course of your normal knowledge work. And it means it is more plausible that coauthors working together in person might be more likely to learn about intellectually distant work, and later start writing on new topics as a result, as van der Wouden and Youn find.
I started this post with a riff on the internet. But the internet has steadily gotten better at enabling communication; we can communicate with video instead of email, we can “meet” lots of people on twitter, we can collaborate over GitHub. That would seem to make it easier to engage deeply with new ideas over a distance; perhaps the advantages of local communication are waning?
In other places, I have argued this is indeed the case. Knowledge spillovers appear to be less local, and the internet is probably one reason why. But Duede and coauthors survey is about papers published in 2015; so at least as of 2015, it was still the case that people were more likely to have been influenced by intellectually distant work if the author was local.
van der Wouden and Youn also look into how the returns to local collaboration have evolved over time. This is a bit tricky to interpret, because the rate at which people jump to any new topic has steadily declined over time1, but we can at least compare the rate at which people jump into a new topic after collaborating locally, compared to remotely. If the gap between the two is shrinking, that implies it’s getting easier to get what you get from local collaboration at a distance.
Below, is their estimate of the “learning premium” of local collaboration. Basically it is how much more likely you are to start writing on a new (to you) topic if your previous collaborators were local, compared to your probability of writing on a new topic if they were remote. In fact, this learning premium has been on the rise since around the birth of the internet!
How can this be, given that it’s easier to exchange ideas today at a distance, than it was in the 20th century?
I think what’s going on here is that local collaboration affects research through two channels: first, it affects which kinds of projects get started in the first place; second, it affects how much knowledge is shared during the performance of a research project.
I take it as a working assumption that better remote collaboration technology has reduced the impact of the second channel: once you get going on a project, it probably matters less today if you collaborate locally or at a distance (though it could still matter). But since we actually observe the impact of local collaboration rising over this period, if my assumptions are right, that means this must be due to some kind of change happening in the nature of projects that are initiated locally and remotely.
Here’s one story. The share of articles that involve non-local collaboration has risen from around 25% in 1975 to around 50% in 2015, according to van der Wouden and Youn (and others find similar increases). Let’s imagine there are two kinds of research articles you can write: routine articles, which are in your usual research groove, and stretch articles, that are a bit outside your comfort zone. My guess is that the increase in remote articles has been disproportionately of the routine type, and that routine articles are less likely to lead to writing on new topics in the future.
Why might that be the case? Suppose you only ever have ideas for stretch articles when you deeply engage in a new area; for example, when you get into a long chat with someone who works on stuff a bit outside your normal line of research.2 Suppose these kinds of long chats with people working on different topics happen with greater frequency in person (Duede et al. 2022 provides some evidence this is true).3 For these reasons, assume stretch papers mostly arise from local collaboration over the entire time period 1975-2015.
In contrast, suppose you can generate your own routine research ideas (for example, by reading people’s work in your field, encountering unforeseen problems in your existing research, etc.). Once you have a routine idea, do you develop it with a local colleague or a remote one? Well, back in 1975, there’s not much choice in the matter: it’s hard to collaborate at a distance and so you’re most likely to recruit a local colleague to help you. Fast forward to 2015 and the story is quite different; if it’s easy to work at a distance, it may well be that the best coauthor is someone you know who isn’t local.
In this simple story, local collaborations in 1975 are a mix of stretch and routine papers. By 2015, you’re still doing most of your stretch work locally, but you no longer do nearly so much routine work with local colleagues. And maybe that shows up in the data as an increasing probability that local collaboration leads to writing on new research topics in the future.
However, even if this story is true, it still suggests there is something special about local knowledge clusters; they are unusually good at forcing you to encounter new ideas.
New articles and updates to existing articles are typically added to this site every two weeks. To learn what’s new on New Things Under the Sun, subscribe to the newsletter.
Are ideas getting harder to find because of the burden of knowledge?
An example of successful innovation by distributed teams: academia
Increasingly distant knowledge spillovers
The internet, the postal service, and access to distant ideas
Remote work and the future of innovation
Why proximity matters: who you know
van der Wouden, Frank and Hyejin Youn. 2023. The impact of geographical distance on learning through collaboration. Research Policy 52(2): 104698. https://doi.org/10.1016/j.respol.2022.104698
Duede, Eamon, Misha Teplitskiy, Karim Lakhani, and James Evans. 2021. Being Together in Place as a Catalyst for Scientific Advance. arXiv:2107.04165. https://doi.org/10.48550/arXiv.2107.04165