Peer review and its malcontents: Reform or revolution?

1 Leave a comment on paragraph 1 0 Ah peer review. Never have so many gallons of ink (or fingers and pixels) been spent so futilely as the ongoing discussion of peer review. It is fascinating to trace my own thinking. Most of these essays are early ones, brimming with fury or excited by the possibilities a new demonstration seemed to show.

2 Leave a comment on paragraph 2 0 To be honest, I then got bored. Traditional pre-publication peer review is so entrenched and so much of a sacred cow that tackling it head on seemed futile. Even those that seek change are ham strung by a first mover disadvantage so great, and so frightening that any move is either tentative or so far out on the fringe as to make no difference.

3 Leave a comment on paragraph 3 0 At the same time some of the things that traditional pre-publication peer review, or at least pre-publication filtering, does achieve have started to come into focus. Scandals around non-existent review, around reviewer cartels and even fake reviewers illustrate the challenges of maintaining any quality standards at the scale of literature. Experts have a chance of catching some of these issues in a way that automated systems do not. The question is really whether the time experts would need to spend is justified.

4 Leave a comment on paragraph 4 0 What is peer review really? It’s a social construct, the thing that draws a line around formal scholarly content and declares it “real”. As a result it is also the thing that draws a line around the community. Traditional peer review is a mechanism for excluding writing, and people, deemed to be “non-scholarly”. As I discuss later in “Who do you get to say I am?” this is about power and community definition. For our purposes here we can focus on the way in which it reinforces a sense of community.

5 Leave a comment on paragraph 5 0 That peer review does reinforce our sense of community can be easily seen in the way that peer review is a thing, perhaps the only thing, that unites scholarly communities. Peer Review is held as our common practice, even though the actual practices are not common between disciplines at all. Peer review in philosophy has almost nothing in common with the practice in mathematics, which is entirely different to that expected in bioscience. Only once you recognise that we are a community of practice defined by something that is not even a common practice, do you really understand the totemic significance of the concept.

6 Leave a comment on paragraph 6 0 This is what makes change so hard. Not that there is a lack of evidence about what might work better, not that practice needs to be different for different fields, not due to any lack of technology or systems.  But that the whole self-conception of the scholarly community is bound up in this practice.

7 Leave a comment on paragraph 7 0 In turn of course, such loyal adherence leads those of us who want to change to adopt a radical stance and radical language. Peer review is “broken”, a practice of “cargo-cult worship” of a “golden calf”. Such language, does not make us friends of course.

8 Leave a comment on paragraph 8 0 The quote which starts this section illustrates this. My throw away comment “It makes more sense to publish first and filter later” could be read as totally innocuous. “It makes more sense…”, not “things are broken” and not “the world must change tomorrow” and yet the response to that short sentence was a storm, or at least a squall, of social media attention that might lead you to believe I was holding the whole establishment hostage.

9 Leave a comment on paragraph 9 0 I still believe this. It would be more efficient and probably more effective if those works that deserved more attention got it, and those that did not, didn’t but were available for examination. The essay I wrote examining the response to Deolalikar’s claim of a proof that P != NP shows how effective public review can be, as did later, but perhaps rather less polite examples from #ArsenicLife to the STAP stem cell papers. The high profile public take down of a controversial paper has become common. And we clearly do not have the social conventions in place to manage that process appropriately.

10 Leave a comment on paragraph 10 0 As I explore in that first essay the practical approach is incremental. Find those places, and those communities looking for change, looking for new possibilities. Work to make the process more transparent, although this has its own issues, not least revealing more of the underlying problems in our traditional process. Ultimately we should be using the review process to improve the review process. We know that the value of peer review is hard to measure, so creating a positive feedback loop through examination of the process itself is surely the way forward?

11 Leave a comment on paragraph 11 0 In the end we will hopefully iterate towards systems of review that work with the web, not against it. That use the ease of publishing effectively but also the potential ease of granular and efficient certification. In the end it is the earliest post in this section, the one that I finish with, that still holds for me the promise of effective reform. Clay Shirky saw how this worked a long time ago, and his aphorism that “it’s not information overload, it’s filter failure” still should give us pause in today’s world of listicles and BuzzFeed.

12 Leave a comment on paragraph 12 0 But it’s easy to engage with the surface meaning and assume that peer review is, has been, and always should be “our” filter for managing our community. Shirky is actually saying something different. Something that the scholarly community still has failed to tackle head on. The question of effective discovery.

13 Leave a comment on paragraph 13 0 While I stopped speaking and writing so much on peer review, that discovery question remains a thread through much of the later writing. The questions of power, discovery and filtering reappear in many places, perhaps most strongly in that same essay I mentioned above. Today I understand much more about the role of review in our culture and the way it reinforces and protects definitions of what counts as scholarly. That doesn’t mean we don’t still need radical change if we are to do to actually protect that space effectively.

Page 41

Source: http://book-shaped-object.cameronneylon.net/wp/discarded/assessment-and-metrics/