Churchill, Peer Review and All Which Will Not Pass Away

1 Leave a comment on paragraph 1 0 Peer review and its malcontents

2 Leave a comment on paragraph 2 0 “It makes much more sense in fact to publish everything and filter after the fact” – Cameron Neylon, quoted in Mandavilli (2011) “Trial by Twitter” Nature 469, 286-287

3 Leave a comment on paragraph 3 0 In 2011 I was quoted somewhat out of context in a news piece in Nature called “Trial by Twitter” – that discussed one of the early examples of a published research article being subjected to strong criticism online.  Ironically the quote sparked something of an online storm itself. Even though today this form of online criticism is common for high profile papers, traditional peer review remains a peculiar sacred cow for the scholarly community. Certain criticisms of peer review are commonplace and expected. But if you scratch deeper or suggest wholesale change it is easy to become regarded as a lunatic bent on the destruction of the entire scientific enterprise. My views on peer review and the value of filtering some forms of work out prior to publication have changed over the years. Nonetheless, while I would probably say something a little more nuanced today, I remain of the view that in general, filtering mechanisms that operate after a work becomes public are both more efficient and more effective than those applied prior to publication.

4 Leave a comment on paragraph 4 0 We have more evidence on the value that peer review creates than we did in 2011 but much of it remains weak or focuses on very specific cases. We have much more data on what researchers think about peer review than on what it does or does not actually achieve. Objectively showing how peer review has value is a challenging process but one which still deserves much greater attention and support than it has had to date. The interplay between community beliefs in peer review, researcher behaviour and trust by wider publics is also complex. Radical and unconsidered change can be dangerous, or at the very least counterproductive. But the real challenge is to apply our own standards of quality assurance to testing our own assumptions about the processes we use. Now more than ever we have an obligation to use all the tools at our disposal to ensure that the processes we use to anoint particular pieces of published work as “peer reviewed” are fit for purpose.

5 Leave a comment on paragraph 5 0 The science of peer review

6 Leave a comment on paragraph 6 0 It remains puzzling to me how un-scientific scientists are about the actual practice of science. In our own research specialities we tear arguments to pieces, critically analyse each piece for flaws, and argue incessantly over the data, the methodology, the analysis, and the conclusions that are being put forward. Ideally this happens with an open mind and a positive attitude. But shift attention onto our processes of review and all that goes out the window. Personal anecdote, gut feelings, half-baked calculations and sweeping statements suddenly become dominant.

7 Leave a comment on paragraph 7 0 Let me pick a toy example. Whenever peer review is discussed it seems inevitably to begin or end with someone quoting Churchill. I could have picked from a million examples but in that same article from 2011we have this quote:

8 Leave a comment on paragraph 8 0 “It’s exactly like what’s said about democracy,” he adds. “The peer-review process isn’t very good — but there really isn’t anything that’s better.”

9 Leave a comment on paragraph 9 0 There are two problems I have with this. Firstly it’s an appeal to authority, not something we’re supposed to respect in science. In any case its a kind of transplanted authority. Churchill never said anything about peer review. Even if he did, why should we care? But perhaps more importantly it is a misquotation. If we actually look at the original quote we see:

10 Leave a comment on paragraph 10 0 “Many forms of Government have been tried and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time.” – sourced from Wikiquotes, which cites: The Official Report, House of Commons (5th Series), 11 November 1947, vol. 444, cc. 206–07

11 Leave a comment on paragraph 11 0 The key here is “…apart from all those other[s…] tried from time to time…”. Churchill was arguing from historical evidence. To make this analogy we would need to show first that other systems have in fact been tried. What other systems have we tried? Experiments have been done on – looking at small changes to traditional processes in specific contexts. Many of these show quite positive results. Unfortunately few of these “experiments” are designed in a way that lets us draw firm conclusions – they have tended to be more of the “suck it and see” variety. But Churchill’s statement is stronger than “sometimes better”. Even in its negative form it still claims the superiority of democracy over all other systems.

12 Leave a comment on paragraph 12 0 But when we try to show that traditional peer review performs better than chance the evidence is, at best, patchy. It doesn’t effectively guarantee accuracy, it fails dismally at predicting importance, and is arguably not supporting effective filtering. If I appeal to authority I’ll go for one with some domain credibility, lets say the Cochrane Reviews which conclude the summary of a study of peer review with “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.” Or perhaps Richard Smith, a previous editor of the British Medical Journal, who describes the quite terrifying ineffectiveness of referees in finding errors deliberately inserted into a paper. Smith’s article is a good entry into to the relevant literature as is a Research Information Network study.

13 Leave a comment on paragraph 13 0 Does this matter? Many feel that peer review as currently practices works well enough to be left alone. “If it isn’t broken, don’t try to fix it”.  Perhaps more importantly, many feel that to question peer review is to question what makes the practice of science unique as a means of creating knowledge, to question the basis on which we receive public money. To my mind this is putting the cart before the horse. The argument that science has succeeded due to the criticism of peers does not mean that our current forms of criticism are as good they could be. Our systems of criticism and testing work may work in aggregate and over time, but there is little question that they could work better. Faster, with greater accuracy and precision, and most importantly more efficiently.

14 Leave a comment on paragraph 14 0 If we don’t apply our best critical analysis to our own systems of quality assurance then the justification for any claims about the value of our work is weak. Lets look at the evidence, see where the problems are, and see where the good is. Lets abandon the gut feelings and anecdotes and actually start applying some scientific thinking to the processes we use to do and communicate science. After all if science works as a means of testing and generating knowledge, then we can’t possibly lose can we?

15 Leave a comment on paragraph 15 0 Practical steps to reforming peer review processes

16 Leave a comment on paragraph 16 0 What then are the practical steps to testing, improving and ultimately reforming our review processes? Simply abandoning the current system is untenable and impractical. There are a range of important and valid concerns that can be raised about change and in particular about rapid change. These are worth looking at closely. We need to consider carefully what kinds of systems and what kinds of transition might work.

17 Leave a comment on paragraph 17 0 Let us start with a statement of belief. Peer review at its core is what makes science work. Questions of what qualify as knowledge, let alone truth, are challenging, but for our purposes we can adopt a simple pragmatic position based on two reasonably widely held views, at least amongst scientists. Humans are fallible, and the universe exhibits usably consistent behaviour.

18 Leave a comment on paragraph 18 0 There are two simple approaches we use in empirical research that help to explain why the laptop I’m using works, why I didn’t die as a child of infection, and how we are capable of global communications. One of these is the testing of our working models of the universe against the universe itself. If your theory of light requires the existence of a medium called “the ether” that should have measurable effects and you cannot measure them, then there is a good chance there is something wrong with your theory. If the prevailing theory of how stomach ulcers are formed means that you should neither be able to cause ulcers through infection with Heliobacter pylori nor cure them with antibiotics, and both of these things can be demonstrated, then the theory of ulcers will need to be changed. We use experiment and testing to give the universe the opportunity to prove our models wrong.

19 Leave a comment on paragraph 19 0 The second is that by exposing our models and ideas and our tests of those models and ideas to the harshest possible criticism of our peers that we can stress them to see what holds up to the best logical analysis available. The motto of the Royal Society “Nullius in verba” is generally loosely translated as “take no-one’s word for it”. The central idea of the Invisible College, the group that became the Royal Society was that they would present their experiments and their explanations to each other, relying on the criticism of peers to avoid the risk of fooling themselves. This second aspect is crucial. It embraces the fact that we have a tendency to fool ourselves – to see what we want to see. But it also embraces the human aspect of competition – the desire to prove someone else wrong. This system of peer review recognised and used these human weaknesses as a way of improving the quality of information generated.

20 Leave a comment on paragraph 20 0 It is the combination of both of these approaches, testing the experiment against the world’s behaviour and exposing that test directly to the critique of peers that leads to the success of the scientific process. This community of peers was small but this was in a real sense post-publication peer review; testing and critique was done in the presence of the whole community. We might even argue that our current systems are a pale imitation of this much more rigorous approach. Perhaps our current system is not even the least worst of those those that have been tried from time to time. Perhaps it is not even as good as where we started.

21 Leave a comment on paragraph 21 0 But the review approaches employed by a few tens of wealthy men do not scale to todays global scientific enterprise and our community has developed different systems to manage this. Those systems involve approaches that have broad similarities across disciplines, but also important differences. In the following discussion I focus on changes to traditional review processes that occur prior to publication, on the advice of a small number of expert peers who are active researchers. The process of review is generally managed by editors who may be active researchers or professionals depending on the journal. It is unnecessary to go over the well trodden issues with these systems but I will note what I hope should be be three fairly uncontroversial issues.

22 Leave a comment on paragraph 22 0 Firstly that pre-publication peer review as the only formal process of review runs a severe risk of not finding the correct diversity and expertise of reviewers to identify specific technical issues. The degree of that risk is more contentious but many recent examples illustrate that it is real. Second, although review and critique after publication is important, indeed critical to maintaining a consistent literature there are currently few incentives to engage in public review after publication and little expectation by the research community that members should engage in this kind of activity. We have few mechanisms to encourage high quality review after publication, nor to track the current status or context of published work. Finally, peer review has a significant financial cost, both in direct terms of its management but also the non-cash cost in researchers’ time, and we should address whether this money is being used as efficiently as it could be.

23 Leave a comment on paragraph 23 0 It is entirely possible to imagine utopian schemes in which these problems are solved. I have been guilty of proposing a few myself in my time. This doesn’t work and I don’t propose to explore any such schemes in detail. The intent here is to explore how the system and the community within it could change in ways that would allow improvements.

24 Leave a comment on paragraph 24 0 Arguments and fears

25 Leave a comment on paragraph 25 0 The prospect of radical change to our current process of peer review provokes very strong and largely negative responses. Most of these are based on fears of what would happen if the protection that our current pre-publication peer review system offers us is ripped away. My personal view is that these protections are largely illusory but I could well be wrong and that doesn’t mean we shouldn’t treat these fears seriously. Fear is a barrier to change and if we it can be neutralized with evidence then we are also making a case for change. In most cases that evidence will also offer us guidance on the best specific routes for that change.

26 Leave a comment on paragraph 26 0 These fears broadly fall into two classes. The first is the classic information overload problem. Researchers already have too much to track and read. How can they be expected to deal with the apparent flood of additional information? One answer to this is to ask how much more information would be released. This is difficult to answer. Probably somewhere between 50 and 95% of all papers that are submitted somewhere do eventually get published [12 (pdf), 34] suggesting that the total volume would not increase radically. However it is certainly arguable that reducing barriers would increase this. Different barriers, such as cost could be introduced but since my position is that we need to reduce these barriers to minimise the opportunity cost inherent in not making research outputs public I wouldn’t argue for that. However we could imagine a world in which small pieces of research output get published for near zero cost but turning those pieces into an argument, something that would look a lot like the current formally published paper, would cost more either in terms of commitment or financial costs.

27 Leave a comment on paragraph 27 0 An alternative argument, and one I have made in the past is that our discovery tools are already broken and part of the reason for that is there is not enough of an information substrate to build better ones. This argument holds that by publishing more we can make discovery tools better and actually solve the overload problem by bringing the right information to each user as and when they need it. But while I make this argument and believe it, it is conceptually very difficult for most researchers to grasp. I hesitate to suggest that this has something to do with the best data scientists, the people who could solve this problem, eschewing science for the more interesting and financially rewarding worlds of Amazon, Google, and Facebook.

28 Leave a comment on paragraph 28 0 The second broad class of argument against change is that the currently validated and recognized literature will be flooded with rubbish. In particular a common, and strongly held, view is that the wider community will no longer be able to rely on the quality mark that the peer reviewed literature provides in making important health, environmental, and policy decisions. Putting aside the question of whether in fact peer review does achieve an increase in accuracy or reliability there is a serious issue here to be dealt with respect to how the ongoing results of scientific research are presented to the public.

29 Leave a comment on paragraph 29 0 There are real and serious risks in making public the results of research into medicine, public health, and the environment. Equally patronising the wider community is also inappropriate. The responsible media and other interested members of the community, who can’t be always be expected to delve into, or be equipped to critique, all of the detail of any specific claim, need some clear mark or statement of the level of confidence the research community has in a finding or claim. Regardless of what we do the irresponsible media will make stuff up anyway so its not clear to me that there is much that can be done there. But responsible reporters on science benefit from being able to reference and rely on the quality mark that peer review brings. It gives them an (at least from their perspective) an objective criterion on which to base the value of a story.

30 Leave a comment on paragraph 30 0 It isn’t of course just the great unwashed that appreciate a quality control process. For any researcher moving out of their central area of expertise to look at a new area there is a bewildering quantity of contradictory statements to parse. How much worse would this be without the validation of peer review? How would the researcher know who to trust?

31 Leave a comment on paragraph 31 0 It is my belief that the emotional response to criticism of traditional pre-publication peer review is tightly connected to this question of quality, and its relation to wider publics. Peer review is what makes us different. It is why we have a special relationship with the media, and the wider community that funds us, who can trust us because of their reliance on the rigour of our quality marks. Criticism of peer review is perceived as an attack at the centre of what makes the research community special.

32 Leave a comment on paragraph 32 0 The problem of course is that the trust has all but evaporated. Scandals, brought on in part by an over-reliance on the meaning and value of peer review, and in part on the pressures placed on individual researchers to perform, have reduced the credibility that was there. Nonetheless, there remains a clear need for systems that provide some measure of the reliability of scientific findings.

33 Leave a comment on paragraph 33 0 Applying findings in the real world will often mean moving before things are settled. Delays in applying the results of medical research can kill people just as much as rushing in ahead of the evidence can. There is always a choice to be made as to when the evidence is strong enough and the downside risks low enough for research results to be applied. These are not easy decisions and my own view is that we do the wider community and ourselves a disservice by pretending that a single binary criterion with a single, largely hidden, process is good enough to universally make that decision.

34 Leave a comment on paragraph 34 0 Confidence is always a moving target and will continue to be. That is the nature of science. However an effective science communication system will provide some guide to the current level of confidence in specific claims.  In the longer term there is a need to re-negotiate the understanding around confidence between the responsible media and the research community. In the shorter term we need to be clearer in communicating levels of confidence and risk, something which is in any case a broader issue for the whole community.

35 Leave a comment on paragraph 35 0 Charting a way forward

36 Leave a comment on paragraph 36 0 So in practical terms what are the routes forward? There is a rhetorical technique of persuasion that uses a three-part structure in arguing for change. Essentially this is to lay out the argument in three parts, firstly that nothing (important) will change, second that there are opportunities for improvement that we can take, and third that everything will change. This approach is supposed to appeal to three types of person, those who are worried about the risks of change, those in the middle who can see some value in change but are not excited by it, and finally those who are excited by the possibilities of radical change. However, beyond being a device this structure suits the issues here, there are significant risks in change, there are widely accepted problems with the current system, and there is the possibility for small scale structural changes to allow an evolution to a situation where radical change can occur if momentum builds behind it.

37 Leave a comment on paragraph 37 0 Nothing need change

38 Leave a comment on paragraph 38 0 At the core of concerns around changing peer review is the issue of validation. “Peer reviewed” is a strong brand that has good currency. It stands for a process that is widely respected and, at least broadly speaking, held to be understood by government and the media. In an environment where mis-reporting of medical or environmental research can easily lead to lost lives this element of validation and certification is critical. There is no need in system we might propose for this function to go away. Indeed we aim to strengthen it. Nor is there a need to abandon specific publication venues that are marked as having been peer reviewed and contain material that is explicitly marked as having been through a defined quality assurance process.

39 Leave a comment on paragraph 39 0 The key to managing the changes imposed on science communication by the rise of the web, while maintaining the trust and value of traditional review systems is to strengthen and clarify the certification and validation provided by peer review. This includes having a set of specific publication venues that guarantee those standards and procedures of review. These venues, speaking as they will to both domain specific and more general scientific audience, as well as to the wider community will focus on stories and ideas. They will, in fact look very like our current journals and have contents that look the same as our current papers.

40 Leave a comment on paragraph 40 0 These journals will have a defined and transparent review process with objective standards and reasonable timeframes. This will necessarily involve obtaining opinions from a relatively small number of people and a final decision made by a central editor who might be a practising researcher or a professional editor. In short all the value that is created by the current system can, and should be retained.

41 Leave a comment on paragraph 41 0 Room for improvement

42 Leave a comment on paragraph 42 0 If we are to strengthen the validation process of peer review we need to address a number of issues. The first of these is transparency. A core problem with peer review is that it is in many cases not clear what process was followed. How many external referees were used? Did they have substantive criticisms, and did disagreements remain? Did the editors over-rule the referees or follow their recommendation? Is this section of the journal peer reviewed at all?

43 Leave a comment on paragraph 43 0 Transparency is key. Along with providing confidence to readers such transparency could support quantitative quality control and would provide the data that would help us to identify where peer review succeeds and where it is failing. This is data that we desperately to move beyond assertions and anecdote that characterise the current debate.

44 Leave a comment on paragraph 44 0 A number of publishers have experimented with a range of open peer review processes. While these remain largely experiments a number of journals, particularly those in medical fields, will publish all the revisions of a paper along with the review reports at each stage. For those who wish to know whether their concerns were covered in the peer review process this is a great help.

45 Leave a comment on paragraph 45 0 Transparency can also support an effective post publication review process. Post-publication review has occurred at ArXiv for many years where a pre-print will often be the subject of informal discussion and comment before it is submitted for formal review at a journal. However it could be argued that the lack of transparency that results from this review happening informally makes it harder to identify the best papers in the ArXiv.

46 Leave a comment on paragraph 46 0 A more formal process of publication, then validation and certification has been adopted by Atmospheric Chemistry and Physics and other Copernicus publications. Here the submitted manuscript is published in ACP Discussions (after a “sanity check” review), and then subject to peer review, both traditional by selected referees and in an open forum. If the paper is accepted it is published, along with links to the original submission and commentary in the main journal. The validation provided by review is retained while providing enhanced transparency.

47 Leave a comment on paragraph 47 0 In addition this approach addresses the concerns of delays in publication, whether due to malicious referees or simply the mechanics of the process, and the opportunity costs for further research that they incur. By publishing first, in a clearly non-certificated form, the material is available for those who might find them of value but in a form that is clearly marked as non-validated, use at own risk. This is made clear by retaining the traditional journal, but adding to it at the front end. This kind of approach can even support the traditional system of tiered journals with the papers and reviews trickling down from the top forming a complete record of which journal rejected which papers in which form.

48 Leave a comment on paragraph 48 0 The objection to this style of approach is that this approach doesn’t support the validation needs of biomedical and chemical scientists to be “first to publish in peer reviewed journal”. There is a significant cultural distinction between the physical sciences that use ArXiv and the biosciences in particular best illustrated by a story that I think I first heard from Michael Nielsen.

49 Leave a comment on paragraph 49 0 A biologist is talking to a physicist and says, “I don’t understand how you can put your work in the ArXiv as a preprint. What if someone comes along and takes your results and then publishes them before you get your work to a peer reviewed journal?”

50 Leave a comment on paragraph 50 0 The physicist thinks a little about this before responding, “I don’t understand how you can not put your work in the ArXiv as a preprint. What if someone comes along and takes your result and then publishes them before you get your work to a peer reviewed journal?”

51 Leave a comment on paragraph 51 0 There is a cultural gulf here that can not be easily jumped. However this is happening by stealth anyway with a variety of journals that have subtle differences in the peer review process that are not always clearly and explicitly surfaced. Even within highly prestigious journals it can be unclear whether particular classes of article are peer reviewed (see for example these comments [1, 23] on this blog post from Neil Saunders). The two orthogonal concepts of “peer reviewed” and “formally published” that were once tightly coupled appear to be drifting apart. Priority will continue to be established by publication. The question of what kind of publication will “count” is likely to continue to shift but how fast and in what disciplines remains a big question.

52 Leave a comment on paragraph 52 0 This shift has already lead to a situation where the reference list of many peer reviewed paper include a significant proportion of non-peer reviewed work. Again the issue of transparency arises, how should this be marked? But equally there will be some elements that are not worthy of peer review, or perhaps only merit automated validation such as some types of dataset. Is every PDB or Genbank entry “peer reviewed”? Not in the commonly meant sense, but is there a quality assurance process? Yes. What processes are required for what objects? And how should these be marked?

53 Leave a comment on paragraph 53 0 What is required, and what is not a substantial change in practice is to provide clear and transparent information on what quality assurance procedures have been applied to any given object. As we create new systems for publishing new objects, or for publishing old objects in new forms or more rapidly, we also need clear marking of whether and how those objects have been reviewed.  This will provides real opportunities to make valuable information available more rapidly. It will enhance transparent and fair review, and to provide a framework for communicating effectively the level of confidence the wider community has in a particular claim.

54 Leave a comment on paragraph 54 0 These new publication mechanisms and the increasing diversity of published research outputs are occurring anyway. All I am really arguing for is a recognition and acceptance that this is happening at different rates and in different fields. The evidence from ArXiv, ACP, and to a lesser extent conferences and online notebooks is that the sky will not fall in as long as there is clarity as to how and whether review has been carried out. The key therefore is much more transparent systems for marking what is reviewed, and what is not, and how review has been carried out.

55 Leave a comment on paragraph 55 0 Radical Changes

56 Leave a comment on paragraph 56 0 A system that accepts that there is more than one version of a particularly communication opens the world up to radical change. Re-publication following (further) review becomes possible as do updates and much more sophisticated retractions. Papers where particular parts are questioned become possible as review becomes more flexible and disagreement, and the process of reaching agreement no longer need to be binary issues.

57 Leave a comment on paragraph 57 0 Reviewing different aspects of a communication leads in turn to the feasibility of publishing different parts for review at different times. Re-aggregating different sets of evidence and analysis to provide a dissenting view becomes feasible. The possibilities of publishing and validating portions of a whole story offer great opportunities for increased efficiency and for much more public engagement and information with thecurrent version of the story. Much is made of poor media reporting of “X cures/causes cancer” style stories but how credible would these be if the communication in question was updated to make it clear that the media coverage was overblown or just plain wrong? Maybe this wouldn’t make a huge difference but at some level what more can we be asked to do?

58 Leave a comment on paragraph 58 0 Above all the blurring of the lines between what is published and what is just available and an increasing need to be transparent about what has been reviewed and how will create a market for these services. Markets will ultimately help to both drive down the costs of scholarly communication and to identify where and how review actually does add value. Whole classes of publication will cease to be reviewed at all as the (lack of) value of this becomes clear. Equally high quality review can be re-focussed where it is needed, including the retrospective or even continuous review of important published material. Smaller ecosystems will naturally grow up where networks of researchers have an understanding of how much they trust each others results.

59 Leave a comment on paragraph 59 0 The cultural chasm between the pre-review publication culture by users of the ArXiv and the chemical and biomedical sciences will not be closed rapidly but as the pressures of government demands for rapid exploitation and the possibilities of losing opportunities by failing to communicate rise there will be a gradual move towards more rapid publication mechanisms. In parallel as the pressures to quantitatively demonstrate efficient and effective use of government funding rise opportunities will arise for services to create low barrier publication mechanisms. If the case can be made for measurement of re-use then this pressure has the potential to lead to effective communication as well as just dumping of the research record.

60 Leave a comment on paragraph 60 0 Conclusion

61 Leave a comment on paragraph 61 0 Above all other things the major trend I see is the breakage of the direct link between publication and peer review. Formal publication in the print based world required a filtering mechanism to be financially viable. The web removes that requirement, but it does not remove the requirement for quality assurance. The ArXiv, PLOS ONE and other experiments with simplifying peer review processes, Institutional Repositories, and other data repositories and the explosion of freely available research content and commentary on the web are all signs of a move towards lower barriers in publishing a much more diverse range of research outputs.

62 Leave a comment on paragraph 62 0 None of this removes the need for quality assurance. Indeed it is precisely this lowering of barriers that has brought such a strong focus on the weaknesses of our current review processes. We need to take the best of both the branding and the practice of these processes and adapt them or we will lose both the confidence of our own community and the wider public. Close examination of the strengths and weaknesses and serious evidence gathering is required to adapt and evolve the current systems for the future. Transparency, even radical transparency of review processes may well be something that is no longer a choice for us to make. But if we move in this direction now, seriously and with real intent, then we may as a research community be able to retain control.

63 Leave a comment on paragraph 63 0 As we do this we must instrument and measure the performance of our systems. We must understand the tradeoffs between rapid release and extensive quality assurance processes. We need to understand what certifications systems work, and what it is that different users want to know about the process that was followed. We need to understand the costs of the system, both direct and indirect, as well as the downstream costs in wasted time and bad decisions of releasing poor information. We can’t design new systems from scratch, but as we adapt and evolve our existing systems, we should be testing our assumptions and engaging all of our critical faculties on making our quality assurance procedures as good as they can be.

64 Leave a comment on paragraph 64 0 The status quo is not an option unless we choose to abandon the web entirely as a place for research communication and leave it for fringe elements. This to me is a deeply retrograde step. Rather, we should take our standards and our discourse, and the best quality control we can bring to bear out into the wider world. Science benefits from a diversity of views and backgrounds. That is the whole point of peer review. The members of the Invisible College knew that they might mislead themselves and took the radical approach of seeking out dissenting and critical views. We need to acknowledge our weaknesses, celebrate our strengths and above all state clearly where we are unsure. It’s just good science.

65 Leave a comment on paragraph 65 0 Notes

66 Leave a comment on paragraph 66 0 Orginally published as two posts, “What is it with researchers and peer review? or; Why misquoting Churchill does not an argument make” and “Reforming Peer Review. What are the practical steps?” on Science in the Open. Other relevant material are posts from Richard Grant, Sylvia McClain, Cath Ennis (including many of the comments). 

Page 44

Source: http://book-shaped-object.cameronneylon.net/wp/discarded/assessment-and-metrics/peer-review-and-its-malcontents/?replytopara=57