Pre-publication peer reviews became the dominant paradigm within the academia in the 20th century. In recent years, the system has been criticized and some attempts to correct the weaknesses of the system have been made, such as ventures into open post-publication peer review. However, the scientific community at large remains relatively satisfied with the prevailing paradigm.

In this post, I will examine the peer review paradigm from a point of view that I have not seen used in this context before, and which points at a potential for paradigm shift. This point of view relies on a combination of three key elements: Total quality management (TQM), Lean thinking, and social collaboration.

Weaknesses of peer review

Peer reviews have been criticized in many ways. Mark Ware’s 2008 report Peer review: benefits, perceptions and alternatives provides a rather good summary of the issues at hand.

Long lead times, the bane of Lean

In the Ware report, submission-to-acceptance times for journal articles were on average 130 days, with a single peer review taking around 30 days to complete (with a median of 5 hours of work and average of 9 hours of work per review). With 38% of respondents dissatisfied with the time it takes, this was the main gripe the surveyed academic community had against peer reviews.

In manufacturing, it has long been understood that wait times are very costly: in fact, waiting is one of the seven wastes that Lean attempts to eliminate. Show any production manager a process with a lead time of 130 days and the main task in it with a lead time of 30 days in order to complete 9 hours of work, and they would not only be dissatisfied, they would downright explode!

With companies constantly improving their performance, the scientific community cannot afford such huge inefficiency either. It may seem as not that big of a deal for the academia now, but that was exactly how the US car manufacturers felt about Japanese car manufacturers until one day they noticed that they had fumbled completely and had to go through great pains to reinvent themselves.

It is important, however, to note that nothing in the peer review paradigm itself causes these huge wait times. By organizing the work more effectively, these queues could be eliminated within the existing paradigm.

To improve quality, drive out the fear?

It has been extraordinarily difficult to establish that peer reviews actually improve the quality of the articles. Nonetheless, the academics surveyed for the Ware report almost universally considered it important for the quality of the published papers.

Furthermore, the majority of the respondents considered neither open peer review (one in which the reviewer knows the identity of the author and vice versa) nor post-publication peer review as effective. There was also similar opposition to ranking articles based on other post-publication criteria, such as citations, or to even provide open reviews at all.

These findings are deeply problematic. To understand why, let’s turn to the Total Quality Management movement as pioneered by W. Edwards Deming. One of Deming’s most famous quotes is “Drive out the fear.” It was Deming’s conviction that an enterprise can succeed only by collaboration, and collaboration is only possible in an environment where there is no fear.

I can only interpret the huge opposition to openness to arise from fear. After all, a peer reviewed article is not just a contribution to science, it is also a contribution to the author’s career. Disagreements over merits, brought to open, can undermine the careers of people if the community is one in which disagreement is viewed in a negative light. Ironically, the academic community is precisely like that (this is a rather informational take on the subject). If the wrath of the circles that decide on tenures, grants, and the like can only be avoided by anonymity, it is prudent to advocate the adoption of double-blind peer review in order to be able to criticize papers honestly.

However, does it have to be this way?

The existing solution: Public Library of Science (PLOS)

Groundbreaking work on refining the paradigm is already underway at the Public Library of Science, a non-profit publisher of academic articles.

The PLOS offerings include several improvements:

• Everything is open access, available to anyone from anywhere
• More frequent issues: peer-reviewed articles in main journals are published online weekly, and in print monthly
• Rapid publishing on selected subjects: peer reviewed articles on selected subjects (PLOS currents) are published within 24 hours of submission
• Post-publication reviews: all articles can be commented by anyone
• Article metrics: the impact of articles online is measured

So, is this a problem solved? Lead times, check. Drive out the fear, check. I think this is awesome. However, perhaps there could be even more to improve?

The new solution: death of the article, rise of social collaboration

I have written before about how social reading may become a significant tool for research groups. However, this time I want to take a step further and consider changing the whole landscape.

In the discussion on peer reviews, neither the proponents nor the critics have questioned one thing: the scholarly article itself. However, I do not think there is anything holy about the article as a format. It was not too long ago that the main format was an entire book, perhaps supplemented by letters. And just as we have progressed from books to articles, it would not surprise me if the next step we take is to an even smaller increment, just like Lean manufacturing strives to move from large batches towards a single piece flow.

Just like companies are moving towards more open, decentralized, and collaborative innovation, the academic community could also move that way. Instead of crafting a detailed, complete article on a subject, perhaps the next paradigm is to work on ideas, hypotheses, or preliminary test results and analyze them collaboratively online with a large number of other people interested in the same subject.

Crucially, this group of people would be formed on an open, ad hoc basis with no entry requirements. The weight of a person is based on the weight of his arguments and nothing else. This would enable even more rapid access to the most current research and even faster improvement of this research through open discussion.

While improvements brought about by PLOS improve the lead time of publishing science, a collaborative way of doing science would strive to improve the lead time of doing science. As a side effect, it would also eliminate the need for peer review: when science is done collaboratively, there is nothing left to review.

Potential problems with social collaboration

Of course, such a shift would not be without problems.

First, the power of social collaboration depends on the people who are collaborating. If the scientific community, or a major portion of it, opposes such an open methodology, the critical mass will not be reached.

Second, not all research is open, not because of people conducting the research, but because of requirements from funders, for example private companies. To solve this issue, the new model would have to prove superior. Companies can work even with their competitors on select cases when it is in their best interest. Solve a problem collaboratively even though it means your competition also solves it, or try on your own and risk being left behind? Open research is not impossible, even for companies.

Third, peer reviewed publications serve as a merit ranking in the academia. If the information is diluted into even smaller increments, many of them incomplete until refined through collaboration, how can merit be awarded? Ironically, this is something Wikipedia struggles with today, and the reason why PLOS has introduced topic pages that are static pages at PLOS that are transferred into Wikipedia – the static PLOS topic page can then be cited and used as merit for its authors whereas the Wikipedia page will continue its life. It would take new types of metrics to measure the impact of a researcher if this static model is retired.

Fourth, if there are no complete publications, is there any knowledge either? How can you tell what is well-established and what is conjecture? Is this new type of article like a blog post, or like a wiki? This issue is, actually, not new at all. How can we tell what is well-established nowadays? There are articles aiming to prove contradictory positions. It is through arguments and ultimately use of information that a view becomes established. This new paradigm would actually bring doing science and publishing science much closer together.

Can doing science and publishing science be one and the same?

Nowadays, it is one thing to do science and quite another to publish what you have done. The question is, is this distinction something that we should hold on to in the future as well? I don’t think it is.

Science is well-suited for collaboration instead of competition, because the ultimate goal of science is to understand the world. Ultimately, all scientists share the same goal. This is important, because sharing a common goal provides a good basis for collaboration.

The difference between doing science and publishing science could be all but eliminated by a move to global, collaborative work. There would still be a need to summarize and popularize the results, but peer reviewed articles could be replaced by open, collaborative workspaces.

I admit that it is highly unlikely that such a change could take place rapidly, and I welcome the improvements PLOS has brought to the prevalent paradigm. Nonetheless, such a change could significantly increase the efficiency of scientific research.

Picture: Peer review monster by Gideon Burton @ Flickr (CC)