The Rise of Preprint Servers in Life Sciences

When and how did preprint publishing begin?

Richard Sever: I'm one of those people who left the lab pretty soon after doing a PhD. So most of my working life has been as an editor of journals. But most recently I've been spending my time on preprint servers, and specifically the preprint server bioRxiv, which now has a sibling, medRxiv. It's an opportune time to chat because we just celebrated our 10th anniversary, and it's interesting looking back. I think there's quite a few people who didn't think we would ever get to 10 years.

Nicole Quinn: I remember attending an AAAS panel discussion in 2016 or '17 about open access and preprints, and it was the first time I'd ever heard of a preprint. At the time, I remember thinking, I hope it works, but I don't know if it's going to. So congratulations on 10 years. It's made a huge impact in how science is communicated. What were some of the hurdles and milestones along the way?

RS: We have more than a quarter of a million preprints across bioRxiv and medRxiv, so that's a nice landmark to pass after 10 years. It's been an interesting journey and one that you couldn't necessarily have forecasted. It's really important to remember that bioRxiv was predated by arXiv in physics and computational science and math, and they've been doing it for a long time. That started in 1991. One with Paul Ginsparg, who launched arXiv. There were a number of conversations over the years about, could this be done in biology? Could this be done in medicine? A lot of people said no. Not only did people say no, people tried and failed. Then we launched bioRxiv in 2013, and it's interesting to think about why it succeeded [when past attempts had failed]. People had said in the past that biology is different; physics produces easily verifiable conclusions whereas biology is messy and completely different. They thought that everyone would get totally confused and that the field would disintegrate. Then it was interesting talking to people like Paul Ginsparg, who said, “Well, these people don't know anything about physics. Physics is not easily verifiable. It's messy, too.” We were lucky to be in Cold Spring Harbor because we had people who had been physicists, who then became biologists. They thought we ought to be doing this in biology as well. That was a real turning point for me.

There was also a lot of fear about how journals might react, for the individual users. People were worried that if they put their preprint online, journals were not going to look at it because it was previously published. So that was a big fear, and that fear for some people continues to this day even though, in the interim, it became very clear that all the journals were not only fine with preprints, but they had to be fine with it because the movement is growing.

RS: My colleague, John Inglis, and I talked more about the idea and decided to give it a go, while really listening to the community. We got a lot of good reception, particularly in the population genetics community; it was clear that they had been thinking about this, too. I think there's a general level of dissatisfaction with science publishing that everybody hears about, and in particular, that it takes so long. You go to a conference and you hear somebody talking about work, and you think, wow, that's amazing. But then the paper would appear a year later. I think people were just generally thinking, with the World Wide Web, why does it have to be like this? Of course, the answer is, it doesn't. You can decouple making the information available from the process of peer review, which had all been conflated in the journal world.

What has the impact been on publishing in the life sciences?

RS: A lot of scientists, particularly in genomics, evolutionary biology, and computational neuroscience, got immediately involved. There was a wave of those kinds of people who got involved and started posting papers on bioRxiv. Then there was almost an osmotic process throughout the community. A stem cell biologist would say, “Oh, I just read this genomics paper about some aspect of development,” and then people in developmental biology and cell biology began to do it. Next, funders started realizing that it was a good thing.

It became most evident [that preprints were needed] during the COVID-19 pandemic. We had launched medRxiv, for clinical information, six months before, expecting that it would have a slow growth like bioRxiv did and ramp up over a number of years. However, six months later, there were 10 million people looking at it every month, and we had 4,000 papers because everybody understood that for COVID work and understanding the virus, getting the information out really, really quickly was important. That became a real demonstration of the effectiveness of preprints. That demonstrated that it could work in medicine.

Addressing Concerns About Preprints in Life Sciences

Why do we need preprint servers?

NQ: I think a year is generous when you say it takes a year to publish, as sometimes it’s a lot more than that. You also mentioned conferences, and I think we're lucky if we see something at a conference because they are a pretty exclusive place to be as well, which highlights how research can be hidden for some time.

Daylon James: You underscored it there, both of you, it's the speed. I think one of the major responses to the question, “Why do we need a preprint server?” is that it takes too long to get the stuff out. I heard you in a talk you gave online about how Steve Quake at the Biohub Network had done some back-of-the-envelope calculations and came around to the idea that preprints can accelerate science by roughly 5 times over the course of a decade. It's obvious why it seeds collaboration and benefits young scientists, as it’s great for visibility, and there's no paywall on these preprints. It seems like a no-brainer. But there's the counterargument that I think a lot of people make about speed, also considered as haste, or that maybe the work is a little bit less vetted or there is less rigor. It's more about just getting it out there, making the claim, than actually supporting it. I'm sure there have been a lot of haters out there who have been telling you why it's a bad idea or why it's going to fail or how it's going to lower the standard.

What are the arguments against preprints?

DJ: What's your response to those people? When people talk about the muddying of the waters or the reduction of rigor, is anyone pointing towards any specific evidence? Or are they all just saying, “Oh, this could happen?” What are the arguments that you hear against the preprint?

RS: I think the possibility that there's information out there that is wrong is a concern. But my feeling is that that is already the case without preprints. When we were planning medRxiv [a few years after launching bioRxiv], I was talking to my dad, who's an older generation physician, about this concern, and he said to me, “Well, I get what you're doing with bioRxiv, but I'm not sure about medRxiv. I worry about the potential for misinformation coming out.” We talked about it a little bit, and then he came back to me about three weeks later. When I spoke with him on the phone, he said, “Oh, I've been thinking about your medRxiv thing. I've decided it isn't a problem, actually, because all the crap gets published somewhere anyway.” It's flippant, and I don't want to seem like I don't care about this, but I do think we have to set it against the backdrop of the fact that you can now put any information on the web. By not doing preprints, you aren’t stopping things from getting out.

But not only that, you can pretty much publish anything somewhere these days. There's really been a race to the bottom with journal publications. There was a discussion on X [formerly Twitter] recently about this dreadful paper that's appeared in a peer-reviewed journal, which everybody thinks is complete nonsense. There are lots of examples of that. The good thing about a preprint is it comes with a massive sign on the front, basically saying, this has not been peer-reviewed. We're not making any claims about it at all. It could all be wrong. So I think it focuses people's attention on this issue, but it's very transparent in that aspect. I do find it hard to find examples of preprints that provide evidence that this is a problem. I actually find it quite easy to find examples of journal articles which purport to be peer-reviewed and are incredibly misleading.

It’s also worth emphasizing that one of our major concerns with bioRxiv and medRxiv was always that some papers might be dangerous, and that if they were wrong, there could be bad consequences, and so we read all the papers. People sometimes think we just put every paper we get online, but actually everything is looked at by a scientist. It's not peer review, but we look to see whether there's something that's potentially dangerous if it were wrong. In those cases, we turn them away.

How do you dispel myths about preprint servers?

RS: Really, the worry is that it might take you down the wrong direction experimentally. But people point out that preprints are being read for the most part by people who are perfectly equipped to peer review them anyway. It’s like a conference, but with six billion potential people in the audience. Most of the people reading preprints are reading arcane bits of science and biology or health science. So most people who will read them, will be experts, and will be equipped to evaluate them. If they aren't, then they should ask somebody else. I find it hard to point to specific instances where there's been a problem.

DJ: I want to elevate that point because I think that's something that's lost on many scientists that I've spoken to about preprints; there’s this idea that they are like a beefed-up Twitter [now X] where everyone can post whatever they think or whatever they've found. Whereas, in fact, there is a screening process. I think it's important to emphasize, with medRxiv in particular, that misinformation that might be exploited by bad actors is excluded, where, as you said, sometimes you have peer-reviewed journals that are happy to publish it.

The Evolution of Preprints

Are the types of articles that are going to bioRxiv changing?

RS: One change that’s clear is that different disciplines adopt preprints at different rates. Looking back at the original arXiv, it started with high energy physicists, and then condensed matter physicists came later and various other people later. There's still some areas of physics that haven’t adopted preprints. We saw something very similar with bioRxiv. Initially, it was mainly genomics and genetics, and then bioinformatics. But as I said, the developmental biologists came along and then the cell biologists. So the population of papers is changing, but mainly because more people are getting on board in different subjects. I think around 70 to 80% of these papers go on to appear in journals.

What’s happening with the 20 to 30% of papers in preprint that aren’t published later in journals?

RS: That 70 to 80% figure is calculated after two years. So some fraction of the 20 to 30% will appear in a journal [because, as mentioned earlier, we know it often takes longer than two years to publish]. But there will be a class of paper that doesn't appear in a journal. What are those? Some of them may be so bad that they never get into a journal, although I don’t think it’s many. Some of them may be papers where they morphed and they became part of something else. But some will be in a category of paper where people say, “I want to get this information out there. I'm done with it now. I don't need to send it to journals. It doesn't need to be peer-reviewed.” There are certain types of study where people know that it’s enough that it's out there for people to read. I think it's going to be really interesting to watch that fraction. Some people predict that that could become 50% of papers [in preprint]. If we lose this obsession with the branding and impact factor, you may have a lab that puts half of their papers on bioRxiv and sends the others to a journal because there's some reason that they need to go through the formal peer-review process.

Clearly, during COVID, there was a certain type of paper where people wanted to get the results out and read fast. By the time it could be peer reviewed it would have been totally out of date. They were of pretty variable quality. We saw some modeling of the epidemic. If a lab had started modeling the epidemic in March of 2020, how useful are those predictions if the paper is published in December of 2020? So there are various reasons why some papers never move on to formal peer review. It's always interesting when I talk to scientists about the future of peer review and I ask, “How many papers should be peer reviewed?” I get answers from zero to 100%. Some people think peer review is so broken and so wrong that we shouldn't do formal peer review of any papers. Other people think that absolutely every single thing should be peer reviewed and that the entire field will disintegrate if the papers aren't. I think that'll be very interesting to watch over the next few years.

Have preprints helped to debunk any published results?

DJ: I was interested in this semi-superconductor, LK-99. There was a series of preprint papers that nucleated this massive effort across many scientists who ultimately showed that it wasn't a superconductor, and they explained why it wasn't, and why the false result emerged. They figured it all out completely outside of the sphere of peer review. Although it was true peer review.

RS: Yes, that was on arXiv, as it’s basically a physics paper. But we have had examples on bioRxiv where people have repeated other people's experiments. There was a great example by a former Cold Spring Harbor scientist, Yaniv Erlich, who debunked a paper that Craig Venter had published in PNAS claiming that their group could predict appearance from a genome sequence. By the end of the same day that Venter’s paper appeared in PNAS, Erlich had posted a paper on bioRxiv debunking that finding. He showed that what they were really doing was predicting what the average ethnicity was of someone. That was a real-time peer review.

How did the practice of peer review originate?

RS: I wrote an article about this for PLOS Biology on the history of science publishing, recently. A lot of people think that 300 or 400 years ago, when journals were invented, peer review appeared on day one and was practiced like it is today. But actually, it's a relatively recent intervention. Most journals only started doing it after the Second World War. It's a formal process that's been enacted. But actually, peer review is the repetition and analysis over a period of time of work. That's really what peer review means. There is a possibility for a preprint server to do that because people can replicate the work or fail to replicate the work and explain what the problems were. There have been a handful of really great examples on bioRxiv where people have done that. One was about this tiny little eukaryote; there was a paper in PNAS that said it had huge amounts of horizontal gene transfer, the likes of which had never been seen in a eukaryote. Within a couple of months, there was a paper on bioRxiv that showed that this was completely wrong, and that it was some form of contamination in the sequencing. That was the best form of peer review.

Peer Review and Its Future in Preprint Publishing

Should we be moving towards preprints entirely?

NQ: Earlier, you talked about the race to the bottom, and that there are more and more journals, including predatory journals, that are just trying to publish anything. I’ve always wished that there was a journal of negative results showing what didn't work.

RS: I think there actually is a journal like that. What's interesting is that people don't put papers in it because they don't think it will benefit their careers.

NQ: Maybe it won't benefit their careers, but I believe the point is to benefit science. There's a huge amount of papers that could say, “This didn't work. Don't go down this road.” I think you're also onto something there with the small pieces of information that don't make it into that big paper that you end up submitting that are still useful pieces of information that should be learned and used and built upon by other scientists. Peer reviewing is also a huge endeavor. I know I don't peer review anymore, but back when I was doing research, I would dread that invitation to peer review something. Not only because it was a huge time commitment, but because I felt an enormous amount of pressure to get it right, knowing that there's only a couple of people that are going to be looking at the paper. Whereas on a preprint server, you're now doing this crowdsource review. I would say it’s true peer review instead of a handful of selected people.

RS: The great thing about bioRxiv is it doesn't constrain what you do next because it decouples the peer review from the dissemination. I think that's where the opportunities are. I think there's a real chance now for people to experiment and ask the question that you're asking, which is really, “What should peer review look like?”.

What should peer review look like?

RS: There are a couple of things that immediately spring to mind. One is the fact that peer review looks pretty standard whatever field you're in; you send a paper to two or three people and nag them for 14 days to get a decision or written analysis of the paper. I think back to what we were saying earlier about peer review; maybe if the information is out on the preprint server, you have much more freedom, both in time and mechanism for peer review. I always think of the examples of people doing synthetic biology and building tools. Maybe the best peer review of those is somebody actually testing them in a lab. If the information is already out there, you have the freedom in time to do that because you don't have to do it really quickly. Which touches on the burden of peer review.

We have these simultaneous problems with lots of scientists saying, “I'm so burdened with peer review. I'm asked all the time. Peer review is broken. The system is overloaded.” But, as a journal editor, I know that many journal editors get emails from people saying that they would love to be asked to peer review papers or that they never get asked. So there are a lot of early career scientists who are never asked, who would love to be asked. There are lots of people outside of Europe and North America who would like to be asked and are never asked. So that's another example of not necessarily crowdsourcing, but being able to have a much broader net cast. It could solve some of those problems and increase equity by involving more people. It could get us to a point where we can do a better job and decide what exactly it is that peer review is doing. I mentioned that article I wrote recently in PLOS Biology, but one of the key points of that article is that peer review conflates a huge number of things.

RS: The peer reviewer is examining the science and asking, “Is this correct? Or, do I think it's correct?” But there's also, “Is this any good? Is this important?” It becomes a filter of some sort. You can say, “I only want to read the good papers, so I read the ones in certain journals because I believe the peer review system has put them there.” However, what's interesting is that there have been a number of studies that have shown that most papers don't change very much from being a preprint to being in a peer-reviewed journal. So, you can argue that peer review is not doing anything, but it is doing something; it's deciding what journal papers go into. So even if the peer review doesn't change the paper, one thing it does is decide what journal that paper goes into, which people use as a filter to read. Lots of people will dispute whether that's a good filter, but it absolutely is a filter that people use.

That's something that we could look at and ask: Now we have decoupled the dissemination in the peer review, can we deconflate aspects of peer review around things like the quality of the article, as opposed to the level of broad interest, or impact as a word that people throw around? Another point I always want to make is that there are other things that journals and preprint servers do that are important and will be increasingly important around the verification of information. We need to do a better job. It's hard to assure that the person who said they wrote the article did write the article or that the data in the article are bona fide and are not created by ChatGPT or the latest large language model or something like that. So there are all sorts of things that I think are not talked about enough—the kinds of evaluation of content that are not quite peer review, but maybe increasingly important in the whole post-truth world.

Decoupling Impact Factor: Rethinking the Value of Scientific Publications

Where is publishing in academia going next?

NQ: The PLOS Biology paper that you published is a really comprehensive history of publishing and has some food for thought about where things are going next. Thank you for publishing that and putting the effort into that.

DJ: I loved reading that. I want to quote from the abstract: While it would seem to maybe be a disincentive to adopt the preprint model, it’s not mutually exclusive as you say, “Journal brand and Impact Factor have meanwhile become quality proxies that are widely used to filter articles and evaluate scientists in a hypercompetitive prestige economy.” That last point is what I want to focus on; the hypercompetitive prestige economy seems to be like a zero sum. There’s a loss to scientists who want to show they are doing esteemed research with their big result by publishing in a high-impact journal. But it's not mutually exclusive. I think what we're talking about here is that you can get results out in a preprint and get your line in the sand. Then you're saying that it is still important to review the results, for those proxy legacy reasons of deciding what results have high impact or importance. I don't love this as a scientist, but it's how we all live and breathe. It's how we advance in our careers as scientists. I would love to live in a world where we don't, but it seems like we need these proxies to judge ourselves in this hypercompetitive prestige economy.

I'm aware of this, but I didn't understand the timing of how the big prestige journals, Nature, Science, Cell, gobbled up all the more niche subject matter with their sibling journals, like Cell Stem Cell, Nature Medicine, Nature Biotech, for example, to the point where now essentially the big three or big four, they own all prestige science, which I think is a bit of a money grab on the publisher's side. I don't know why else they would do that. But it does raise the question that a similar thing could happen with preprints, as you get these specialty preprint servers, is there ultimately going to be a point where you have prestige preprint servers? Are there going to be papers that get into bioRxiv that have a different level of impact than other, more niche, preprint servers?

Will preprint servers ever have an impact factor?

RS: One of the things that people said to me early on is that there are great papers on bioRxiv. I said that I'm glad that there are great papers on bioRxiv, but let's just be clear, the goal is not to have only great papers on bioRxiv. The goal is to have all the papers on bioRxiv about biology and medicine. So that means that, on average, they'll be mediocre—because most ones are. But what you do is you decouple that filtering. And so maybe you can make it better. There are a number of problems that people have with this idea that if you get a paper in Nature, that you get a job immediately. Should those kinds of brands be used as the quality proxy? There's a whole debate about that. But even if you don't have too much problem with the concept itself, you ought to have a problem with the fact that that decision is made really soon by the opinions of two or three people.

Even if you don't mind that, there is the problem that you have this immutable quality indicator that is assigned before anybody gets to read the paper. Even 20 years later, a <