The main points I made in this blog post from 2021 seem to hold up pretty well. In the time since it was written, I continue to observe the Peer Review process through the lens of QA, which I continue to think provides a helpful metaphor.
Peer Review, a Critical Step in Scholarly & Scientific Publishing
However, we work assiduously to keep up on our reading around here:
One blog I read recently suggests (with a pinch of hyperbole) that while peer review is a time-hallowed and important process, a more fully sustainable method for it has never actually been tried, while another writer suggests that the whole business of peer review continues to be in crisis and that nobody has solved it yet.
I feel they each have a point there. Perhaps peer view is well thought of as (in the old expression) “reformata, sed semper reformanda” (‘reformed, but ever reforming’)
Dave Davis revisits his 2021 blog post observing the Peer Review process through the lens of QA (quality assurance).(Disclaimer: I’m not a scientist, nor have I have ever edited or peer-reviewed a scientific paper, although in my background research for this post I communicated with several colleagues who do fulfill such descriptions. My degrees are in History, which may be conducted as social science, or as a form of literature, or both, depending on how you go about it.)
For several years now, I’ve been honored to work with The Concord Review, a journal featuring top-quality academic essays by high school students. My role is similar to that of the history teacher to whom a term paper is submitted: I read the work closely and offer a brief, constructive critique along several dimensions, focusing on evidence, internal logic, writing, and the writer’s application of valid historical methods.
In the paragraph above, I used the word “quality,” a multifaceted and rich term, and one whose meaning can slip around like a fish pulled from a stream. Checking with Merriam-Webster, I find the sense I want at 2(a): “degree of excellence.” A graded, or ranked, paper is seen as of high quality if it excels on those points upon which it is assessed.
A similar (but distinct) usage comes in from software engineering: “Quality Assurance” (QA) is a critical part of the development process where the programs under study are rigorously tested to determine that they function as designed in the specs and do not make critical errors, such as failing randomly or messing up the data.
Scientific peer review may be thought of as something a lot like the QA component of the software development process – that is, it is essentially the QA step for the production of high-quality articles which are published in high quality journals.
In early 2020, CCC (in cooperation with Outsell) published a map of the scientific publishing ecosystem, available for download here. The following graphic is a simplified version of the research-authoring-review-publishing cycle:
On this idealized representation, Peer Review appears in the middle right of the cycle, the step at which formal peer reviewers —who, as a rule, are independent, i.e. not employees of the publisher —are sent the submitted paper. Depending on the publisher’s expectations (administered in the person of the editor), at this stage the peer reviewers typically would —minimally, although they may do more— check for (a) appropriate application of method, (b) apparent valid application of the data presented, and (c) internal consistency/logical validity. Considered at a more granular level of detail, though, there are QA checks in each of the five main phases: at Research and Discovery, when the team conducts a literature review and then designs the study itself, in accord with known and published methods; at Authoring and Output, when the results are checked (and possibly triple-checked) such that the assertions and conclusions are fully constrained by what the data says – and doesn’t say; at Editorial Submission and Acceptance; at Publishing and Distribution; and then in the article’s Post-Publication life when it is weighed as a contribution to the domain’s literature. In short, for top-tier articles and journals at least, the pursuit of quality is “baked into” each phase of the cycle.
There are many examples of when these efforts have failed, either due to intentional misleading on the part of the researchers, or when the all-too-human peer reviewers have dropped the ball. An early example from modern times, the “Desktop Fusion” controversy of the late 1980’s, led to a frenzy in the popular media, and then to recriminations and retractions, when the Fleischmann-Pons results could not be replicated by any other researchers. A few years later, in 1988, a paper by British researcher Andrew Wakefield, which claimed to find a correlative link between vaccines and autism, was published in a major biological journal. Later, Wakefield’s data was shown to have been falsified and his results utterly invalid. The paper was not retracted for 12 years, however, during which time a massive anti-vaccination movement took hold and gained popularity, often citing Wakefield’s paper in support of their efforts. Both the incident of Fleishmann-Pons, and that of the Wakefield debacle, can be considered failures of the peer review process. These massive publication failures are rightly seen as outliers, to be sure; but they also should stand as warnings, and spurs to continually striving to improve the peer-review process. The stakes can sometimes be high indeed.
As I was writing this post, I learned that the STM Organization (in cooperation with NISO) has a pilot project underway, assessing a potentially standard Peer Review Taxonomy. As a fan of taxonomies —my other degree being in Library & Information Science— I experienced a little thrill when I heard about that. There are many aspects of peer review – e.g., internal versus external — that can be clarified by implementing a controlled vocabulary and concept hierarchy. Here’s hoping!
There is a pithy old chestnut, attributed (apparently erroneously) to 19th century art critic and aesthete John Ruskin, which runs “Quality is never an accident. It is always the result of intelligent effort. There must be the will to produce a superior thing.” Even if Ruskin never said it, someone did, and they were on to something. Quality, in the sense we’ve been discussing here, is the effort to get it right, as much as possible, as soon as possible – whatever “it” is.