Since the boycott of the U.S. News and World Report Rankings (“U.S. News Rankings”) by a group of law schools, the U.S. News Rankings have gone a bit haywire. The top law schools, all highly ranked, refused to submit data to U.S. News. Other schools followed their lead. As a result, U.S. News changed its rankings, perhaps focusing a bit more on publicly available data.
But that has led to speculation of potentially wild results. In the past, such dramatic ranking shifts were somewhat limited, but with the changing rankings, and the apparent new norm of frequent changes to methodologies, such radical changes might be in our future. It is striking, though, how the fluctuations rarely seem to affect the higher ranked schools.
Meanwhile, prospective law students, perhaps largely unaware of these fluctuations because the top-ranked law schools barely change, are likely still invested in the U.S. News Rankings for determination of what makes a good law school. A quick check assures that the schools they know are prestigious like Harvard and Yale are in the top ten, with little variation year to year.
And, despite the understanding in the legal academy about the inconsistency and other problems, the U.S. News Rankings are still used by some schools as an aspirational goal and a publication goal (with some schools offering anti-intellectual publication bonuses for high-rank placement). Others use the rankings for marketing materials to prospective faculty candidates and students.
In this essay, I list five problems with the U.S. News Rankings, and I offer a few concrete solutions.
Problem 1: Information Asymmetries between Prospective Students and U.S. News
Prospective students know little about the ranking methodology or its impact. For example, students may know what percentage of the rankings is based upon peer assessment, but not know that the peer assessment has problems, such as the monopoly power a few schools wield in the law professor labor market. Students likely do not know of relative changes in rankings over time, so are unable to determine whether a rapid change is a mere blip or a genuine trend downward or upward. In short, the prospective student may be deceived by the nature of the rankings and the changes in how rankings are calculated.
Problem 2: Information Asymmetries between the Law Schools and U.S. News
As the dominant player in the market for rankings, U.S. News has little incentive to expend resources to monitor the data that law schools provide, to correct inaccurate data, or to make algorithmic adjustments unless the results produced by its formula are egregiously false or schools flagrantly manipulate the data that they submit. In fact, the value of the rankings endures by virtue of having little change at the top of the list. Should a school experience an unexpected drop in ranking, however, dramatic effects may occur, including dean resignations. Schools seeking to climb the rankings to attract high-quality students, or faced with habitually low rankings, may succumb to pressure to manipulate data to improve their rank. For example, it has been past practice for schools to employ their former students to inflate post-graduation employment statistics.
Problem 3: Favoring Well-Endowed Schools
To the extent that U.S. News alters variables as to what makes a “good school,” it favors wealthier schools that can deploy more resources and adapt quickly to the moving goal posts. But those at the top end of the rankings do not need to worry, as they are virtually guaranteed their spot. Other schools are subject to the potential of rapid fluctuations.
Schools also make significant time investments by creating committees tasked with benchmarking competitive schools, collecting employment data from recent graduates, and grappling with the impact of how varying analyses of the information might affect the U.S. News Rankings.
Part of the methodological shift as such is the post-boycott need to access publicly available data. That need, however, may not be a great basis for the methodology put forward. It may merely be a streetlight effect on methodology road.
Problem 4: The Problem of Potential Retaliation
To the extent that the variables that inform U.S. News Rankings are subject to change, they are potentially subject to abuse. It would be easy to add variables punishing schools that complain about the ranking’s methodology.
Just two examples should suffice. When Reed College refused to provide data to U.S. News for the general college rankings, instead of omitting the institution from its ranking, U.S. News “arbitrarily assigned the lowest possible value to each of Reed’s missing variables, with the result that Reed dropped in one year from the second quartile to the bottom quartile.” In subsequent rankings, U.S. News allegedly ranked Reed College on information available from other sources, noting that the institution did not complete the requested survey. In the law school context, when Pepperdine discovered an error in its submission, it sought to correct it. As a result of its innocent mistake, its ranking plummeted for a year.
Problem 5: Lack of Competition in Rankings
There are no substitutes for U.S. News Rankings, at least for U.S. law schools. The vast majority of prospective students use them. The vast majority of schools look to them. To the extent it is a monopoly, it has a unique role in shaping legal education. Indeed, not only is it a unique ranking, much effort is made by Spivey Consulting and others to predict those rankings, rather than create new ones.
As Sahaj Sharda has pointed out in The Sling, rankings matter. And it is rife with the possibility of hijinks, both from the university side and from the side of U.S. News.
Solution: An FTC Unfair or Deceptive Act or Practice (UDAP) Rule
Section 5 of the FTC Act states that “unfair or deceptive acts or practices in or affecting commerce . . . are . . . declared unlawful.” Deceptive acts or practices require a “material representation, omission or practice that is likely to mislead a consumer acting reasonably in the circumstances.” An act or practice is “unfair” if it “causes or is likely to cause substantial injury to consumers which is not reasonably avoidable by consumers themselves and not outweighed by countervailing benefits to consumers or to competition.”
The FTC UDAP rules cover common consumer experiences, such as wash-and-dry labels, octane ratings for gasoline, funeral prices, eyeglasses, auto fuel ratings, used cars, energy ratings, door-to-door sales, identity theft, free credit report, contact lenses and eyeglasses, and child protection. In each of these instances, informational and other barriers barred consumers from realizing important information or availing themselves of alternatives.
Why not apply the same principles to rankings? An FTC UDAP rule related to law school rankings could involve three components.
First, the FTC should require U.S. News to disclose the algorithm it uses or to explain in greater detail the amount of weight it puts on each factor that goes into its overall rankings. An algorithm is a mathematical formula that allows data factors to be compiled into an order, based on the relative importance of each input. If an end user has different values or priorities than U.S. News, the algorithm makes a major difference in the list’s outcome. Disclosing the algorithm protects the consumer from false, deceptive, and misleading information.
The FTC should mandate that any alteration of methodology should provide law schools with at least two year’s notice. At present, much of the methodology could be ex-post, surprising schools attempting to climb the rankings.
Second, the rule should eliminate conflict of interest voting and should mandate disclosure of other data. Schools are not in a position to rank themselves; nor are faculty of a school that is dominant in the production of law professors. The bulk of law professors hail from a few schools. Absent a horrific experience, it stands that the bulk of reputation scores favor those schools. This self-perpetuating cycle is not known to prospective students, and ought to be halted.
To the extent post-graduation employment is used, schools ought to be forced to disclose that number. Thus far, U.S. News has been reluctant and unable to determine the extent of such gaming. Subjecting such data submission to a UDAP rule raises the potential risk to a school that such manipulation might become subject to an investigation and an amplified public notice.
Third, the FTC’s rule should impose penalties on schools and U.S. News for violations of the rule. Another feature of a UDAP rule would be consistent deterrence, as opposed to arbitrary punishments that U.S. News might impose upon schools. If such penalties are not linked to the ranking itself, a UDAP rule would still benefit consumers of the ranking, whereas displacing the ranking of a school that misbehaves might penalize beyond the group involved in the decision to manipulate the ranking.
If the FTC were to adopt such a rule, it would bring some much-needed relief to law school applicants and the schools themselves.
The views expressed in this piece do not reflect the views of my employer.
Correction 2/28/24: Since publication, Spivey Consulting reached out to correct an entry error related to Buffalo Law School’s ranking in their post, when Spivey converted that information from their software to the blog. We have corrected the entry here as well. Other schools will still face such drops.