Peer assist review, American Idol, and the future of academic writing

In past posts, both here and during my guest stint at ELS, I discussed comparisons between law review and peer review publishing. A set of similar movements have posited somewhat different forms of determining what gets published in academic journals.

On TaxProf Blog, Paul Caron relays a recent press release on an intriguing alternative method:

In a radically new interactive approach to legal scholarship, more than 100 leading scholars are debating the fundamental questions of modern criminal law through a law professor’s version of the TV show American Idol.

Professor Stephen P. Garvey of Cornell Law School, along with Paul Robinson of Pennsylvania Law School and Kimberly Ferzan, professor and associate dean at Rutgers School of Law-Camden, are the guiding professors in a 10-month online effort to create a new method of processing scholarship. In this new project, called Criminal Law Conversations, authors of the top-rated essays can defend their ideas against criticism from the judges, who are other law professors. The essays that receive too few votes get kicked off the stage, which in reality is the University of Pennsylvania Law School Web site, which hosts the Criminal Law Conversations project.

On First Movers, Anthony Ciolli discusses a possible alternative method associated with a proposed journal that apparently never came to be, The Wharton Law Review. Here’s the description of the method:

Our proposed solution to these problems is the Peer-Assist system. This system seeks to tap into the collective wisdom of academics worldwide prior to selecting articles for publication. A journal using the Peer-Assist method would have the same structure as a traditional student-edited journal; only the process would differ. When an article is submitted to the Wharton Law Review, it will be stripped of its identifying information and posted to a secure, password-protected website. Access to this website will be limited to professors of any discipline at any recognized university who request access, all members of the journal (including associate editors), and alumni of the journal. Readers will have the option of browsing articles by subject matter or by keyword, and also will have the option to receive direct email alerts for new articles in certain fields. All readers may submit comments about the articles, which can be as detailed or as frequent as the reader desires. No anonymous comments will be allowed, as the reviewer’s name will be automatically added to every comment he or she submits. Additionally, the content of each comment will be only accessible by the editorial board of the Wharton Law Review, so incoming readers will not be able to see what else has been written about a pending article. Any author may opt out of the Peer-Assist system at his or her discretion, in which case that author’s submission will be reviewed by the editorial board alone. Ideally the Peer-Assist system will include academics from outside the field of law, so that an expert on, say, economics or philosophy can weigh in on cross-disciplinary articles.

It is important to remember that the comments solicited through the Peer-Assist system are simply a tool to aid the editorial board of the Wharton Law Review in its publication decisions. The students of the editorial board will retain final decision-making power over all articles, and will have the option to accept or reject an article at any time, regardless of the feedback received from Peer-Assist.

I’m not sure if I am ready to endorse or flunk this new idea, but a number of concerns come to mind. Perhaps foremost are problems with this system becoming a popularity contest (not that this doesn’t occur to some degree in traditional peer review) and that there appears to be little assurance of the review capabilities of comment providers. Ciolli addresses these two concerns as follows:

A second common critique concerns the faculty reviewers themselves. Some professors worried that the reviewers would give good reviews of their colleagues’ articles in a quid pro quo arrangement, and that these reviews would unduly influence the article selection process. This problem assumes that the editorial board of the Wharton Law Review would be unable to distinguish legitimate criticism from puffery. It is our belief that the editorial board is capable of making these judgments, as distinguishing a good argument from a bad one is what Articles Editors already do. It is an important part of the learning process, and Peer-Assist will simply create another avenue for that skill to be developed. Additionally, we presume that the editorial board of the Wharton Law Review will naturally give more weight to objective comments (e.g. “This argument is pre-empted by X paper,” or “The mathematics in this piece are flawed”) or well-reasoned subjective comments as opposed to comments offering unsupported praise or criticism.

A third and final common criticism of Peer-Assist is that the readers may not be sufficiently qualified to give informed commentary. Peer-Assist will open up the peer-review process to all academics, not just the pre-selected ones that populate traditional peer-reviewed journals. This does present a greater risk of relatively less-qualified commentators, but we believe that the market effects of the Peer-Assist system will outweigh the risk. It is also quite likely that readers will only comment on articles within their field of interest, whereas the current student-edited system relies on only a handful of third-year law students to critique articles in a wide array of fields. Informed commentary from a professor would be a good addition.

While these are reasonable refutations, it still seems that the process is somewhat cliquish, not that this doesn’t occur in other academic contexts. I imagine that the process might become more refined as it progresses, assuming that such a journal review process is implemented somewhere.