One Pill Makes You Larger: Flaws in Sisk’s Westlaw Methodology Illustrated With Leiter’s Citations
2018 Wisconsin Law Review Forward, 27-33 (2018).
9 Pages Posted: 28 Aug 2018 Last revised: 29 Apr 2019
Date Written: August 17, 2018
The Sisk-Leiter rankings of scholarly impact use a Westlaw search to determine a scholar's citation count. However, the search does not review the citations to determine if they actually are citations to a scholar's work rather than other hits such as blog posts (whether authored by the scholar or by another on the scholar's eponymous blog), citations to works by others in books that the scholar edited, citations to the work of other scholars who only mention the scholar under study (such as a citation in a work to the work of a scholar that had reviewed the book of the scholar under study), media mentions, or author acknowledgements for comments not eliminated by the search term, such as those that appear in footnotes or the body of the article. I use citations to Brian Leiter's work to show that the Sisk-Leiter Westlaw citation count is overstated by about 40% in Leiter's case, with only 398 of 557 attributed cites being to citations to Leiter's academic work. This is a very low signal-to-noise ratio that is 100% curable by more careful study. While Leiter's case may be more upwardly biased because of his popular industry blog, the fact is that media mentions, citations to the works of others in edited volumes, and citations to works that discuss the scholar under study but are not cites directly to that work have an unknown bias that cannot be assumed away with the wave of the hand and an appeal to randomly-distributed (is it?) noise. Moreover, by including such low-quality cites, but ignoring cites in judicial opinions to scholarly work, the method as applied by Sisk significantly undervalues the impact of scholars in some fields and, therefore, likely the scholarly impact and ranking of faculties with strong scholars in antitrust, bankruptcy, corporate and securities law. If this exercise is worth doing - which is not clear - then a combination of Westlaw (with judicial citations) and Google Scholar probably would provide more reliable results.
Keywords: Citation Studies, Faculty Rankings
Suggested Citation: Suggested Citation