Dismissive reviews and citation cartels in education policy research -- A list

Dismissive reviews in education policy research -- A list

Nonpartisan Education Review / Resources

Dismissive Reviews and Citation Cartels in Education Policy Research – A List

Richard P. Phelps


The celebrity professor is a new phenomenon and not a good one. In celebrity-driven academia, "getting ahead" means beating other people, which means establishing a personal reputation and denying it, to the extent possible, to rivals. (Harry Lewis*, in Russell (2007))

No man who bothers about originality will ever be original: whereas if you simply try to tell the truth...you will, nine times out of ten, become original without ever having noticed it. (C.S. Lewis)

The main principle of groupthink ... is this: The more amiability and esprit de corps there is among the members of a policy-making ingroup, the greater the danger that independent critical thinking will be replaced by groupthink, which is likely to result in irrational and dehumanizing actions directed against outgroups. (Irving Janis)

In scholarly terms, a review of the literature or literature review is a summation of the previous research conducted on a particular topic. With a dismissive literature review, a researcher assures the public that no one has yet studied a topic or that very little has been done on it. Dismissive reviews can be accurate, for example with genuinely new scientific discoveries or technical inventions. But, often, and perhaps usually, they are not.

Included in these web pages are the statements—the dismissive reviews—of some prominent education policy researchers. Most of their statements are inaccurate; it is possible that all of them are. Certainly, they are misleading. Each linked file includes the dismissive statements, the names of the lead authors (in bold when known) and co-authors, title, source, date, and, page numbers for the statement and hyperlink to the source, when available, all listed in reverse chronological order.

"Dismissive review" is the general term. In the "type" column of the files, a finer distinction is made among simply "dismissive"—meaning a claim that there is no or little previous research, "denigrating"—meaning a claim that previous research exists but is so inferior it is not worth even citing, and "firstness"—a claim to be the first in the history of the world to ever conduct such a study.

Dismissive review claims are typically made ex cathedra. That is, they are simple declarations made without explanation as to how they were derived. No mention is made of where the authors looked for sources, or how (or, even if).** Celebrity researchers, whose claim to fame is their allegedly superior research skill, provide no evidence or analysis whatsoever for their literature review claims. By contrast, meta-analysts, researchers who specialize in research literature summaries, provide thorough descriptions of where and how they look for source material, identifying in detail the bibliographic data bases and search engines used, the search algorithms, keywords, time periods, and geographic coverage. Meta-analyst claims are easily verified because another researcher can trace the steps taken. Celebrity researchers' dismissive reviews are meant to be accepted on faith.

When a group of dismissive reviewers joins to support each other, they form a "citation cartel". Essentially, they cite and reference each other and ignore, dismiss, or denigrate other research. Citation cartels can advance researchers' careers substantially. Each member of the cartel receives more exposure in general while, at the same time, their professional rivals receive less. Universities consider the numbers of citations as evidence of research productivity and influence, and use those numbers in appointment and advancement decisions.

For the most part, I have included statements made by "serial dismissers", researchers who dismiss repeatedly on a variety of topics. This is done to help counter the argument that they might be innocent, actually did make an effort to look for previous research, and simply could not find it. In some cases they dismiss a research literature that is hundreds or thousands of studies deep. And, when they do that repeatedly across a variety of topics, the odds that their dismissive behavior could be innocent fade to miniscule.

Moreover, I have restricted the list to the statements of those whose dismissive reviews can inflict real harm. If a master's student at a middling, not particularly well known college writes a dismissive review in his or her thesis, the effect on the world may not even be noticeable. But, the people included here, as some of them are fond of telling us themselves, are "influential" (e.g., Edu-Scholar Rankings ). They rank among the most widely quoted and cited researchers in education policy. They advise presidents, international education organizations, federal education secretaries, and legislators. Journalists have their telephone numbers on speed dial. They have been profusely awarded with professional sinecures, prizes, titles, and other manner of prestige. What they say and write matters even when it is accurate, but probably even more so when it is not.

Finally, I only have time to read so much, and only so much money to retrieve documents from behind paywalls, so many possible sources of dismissive reviews are not included here. Had I read everything these people have written, I would imagine that these lists would be much longer.

The dismissive review files linked below are:

Daniel M. Koretz A longtime associate of the federally-funded Center for Research on Educational Standards and Student Testing (CRESST) while working for the Rand Corporation and, subsequently, as a professor of education, first at Boston College and then at Harvard University.

CRESST_BC Statements from researchers affiliated with the federally-funded Center for Research on Educational Standards and Student Testing (CRESST), their occasional junior partners, the faculty of the education school's testing and measurement program at Boston College (not to be confused with the very different and separate TIMSS/PIRLS international test research center, also housed at Boston College), and the captured Board on Testing and Assessment at the National Research Council. The Rand Corporation has been a member of CRESST for decades.

Rand Corporation Statements from those working directly for long-time CRESST member, the Rand Corporation.

National Research Council Statements from reports of the Board on Testing and Assessment at the National Research Council.

Laura S. Hamilton A longtime associate of the federally-funded Center for Research on Educational Standards and Student Testing (CRESST) and a protege of Daniel Koretz, who worked at the Rand Corporaton for decades before accepting her current position at the Educational Testing Service.

Jennifer L. Jennings Another protege of Daniel Koretz. Currently on the faculty at Princeton University, after a stint on the faculty at New York University.

Martin R. West PhD graduate of Harvard's Government program with Paul Peterson as an advisor. He has held faculty positions at Brown University and, more recently, at Harvard's Graduate School of Education. He is also a nonresident senior fellow at the Brookings Institution.

Matthew M. Chingos Another PhD graduate of Harvard's Government program with Paul Peterson as an advisor. He has worked at and published with the Brookings Institution and the Urban Institute.

Brian A. Jacob PhD graduate of the University of Chicago. He taught at Harvard's Kennedy School of Government for several years before accepting a tenured faculty appointment at the Ford Public Policy School of the University of Michigan. He continues his affiliation with the Program on Education Policy and Governance at Harvard, which is led by Paul Peterson.

Frederick M. Hess A PhD graduate of Harvard's Government (i.e., political science) program with Paul Peterson as an advisor. After working as a college professor for a few years, he has settled in as the chief education policy researcher at the think tank, the American Enterprise Institute (AEI).

Jay P. Greene/U. Arkansas graduates Greene is a PhD graduate of Harvard's Government program with Paul Peterson as an advisor. After working as a college professor for a while, he published for several years from the think tank, the Manhattan Institute. In more recent years, he has chaired the Walton-funded Department of Education Reform at the University of Arkansas where, one might say, a new generation of Peterson grand-students is trained.

David N. Figlio Received a PhD in economics from the University of Wisconsin. He has held faculty appointments at the Universities of Florida and Oregon and, more recently, Northwestern University, where he directs the Institute for Policy Research. He has also served as president of the Association for Education Finance and Policy.

Journalists Some journalists who report on education research accept the dismissive reviews and firstness claims of their chosen sources at face value without much, if any, effort to verify. These journalists work for Chalkbeat, Hechinger Report, The 74, and Education Week..

Journalists as Publicists Some journalists step even further into the role of publicist by characterizing their sources in complimentary terms, again without much, if any, effort to verify if the self-promotional claims are valid. .

Other Harvard Professors Includes the statements of other Ivy League professors.

National Bureau of Economic Research (NBER) The organization's name is deceptive; it is more Harvard than it is "national."

Eric A Hanushek Considered by some the preeminent Education Economist of all time, ...and a prolific dismissive reviewer.

Other Stanford Professors Includes the statements of other Stanford University professors.

Other Think Tank Elite Includes the statements of other prominent think tankers and economists who often work with the individuals listed above and share similar backgrounds, funding sources, and affiliations.

Education Next A favorite publication venue for the dismissive reviewers of the Ed Reform Citation Cartel.

*At the time, Harry Lewis was dean of Harvard College. Source: Russell, J. H. (2007). A million little writers: Welcome to the world of celebrity academics, and the behind-the-scenes scribes who help make their fame and fortune possible. 02138 Magazine (November/December). Retrieved from https://velvelonnationalaffairs.blogspot.co.uk/2007/11/read-em-and-weep-for-harvard.html

**Among the thousands of dismissive reviews collected in these pages lies a single exception. Morgan Polikoff claims in a May 2020 Commentary published in Educational Measurement: Issues and Practice (p. 1) to have conducted "a quick scan" of the post-2010 editions of three US education measurement journals and found nine articles on alignment, which he asserts represents a "dated," "few," and "limited and weak" collection. This "quick scan" in just three journals in just one decade in just one country apparently justified to the editors publishing his many global declarations of an alleged lack of research. No matter that there exist hundreds of other countries, a century's worth of research prior to 2010, literally thousands of other journals that might publish such a article, and a large "grey literature" of alignment studies conducted as routine parts of test development. Virtually any standards-based, large-scale test development includes an alignment study, not to be found in a scholarly journal. A short time before this journal published Polikoff's commentary consisting of nothing but his thinly substantiated claims of a lack of research, I proposed a commentary criticising dismissive reviews; it was rejected.

For more on dismissive reviews and citation cartels, see:
Dismissive reviews: A cancer on the body of knowledge. Retraction Watch.

Down the memory hole, evidence on educational testing Academic Questions

Dismissive reviews: Academe's memory hole, Academic Questions

Gaming the system, scientific 'cartels' band together to cite each others' work, STAT

How citation cartels give 'strategic scholars' an advantage: A simple model. Retraction Watch.

Subtractive Scholarship. Academe Blog.

The hitchhiker's guide to scholarly research integrityProceedings of the Association of Information Science and Technology

Worse than plagiarism? Firstness claims and dismissive reviews [slide show]

The Failed Spacing Effect 30 Years Later (Part 1), The Learning Scientists

The dissolution of education knowledge, Educational Horizons

Citation amnesia: The results, TheScientist

Citation Cartels: The Mafia of Scientific Publishing, Enago

Italy's rise in research impact pinned on 'citation doping': Citation of Italian-authored papers by Italian researchers rose after the introduction of metrics-based thresholds for promotions, Nature

T Hacking Peter McGraw blog