Ranking Shenanigans – The Drivers behind the Manipulation Temptation

News broke last week that the U.S. News & World Report ninth ranked liberal arts college, Claremont McKenna College, has been fudging its reported incoming student SAT scores by ten to 20 points for the past six years. While you might not think that such a nominal overstatement in test performance would move the ranking needle materially (test scores are weighted 7.5% in the ranking formula), the inflated scores helped Claremont McKenna rise through the pack and break into the coveted top ten in 2011. Why such pressure on colleges to improve their place in the rankings pecking order, even to the extent that a highly recognized and respected admission dean would risk his job and career by perpetuating a fraud all of these years? A higher ranking translates into an increase in applications and higher yields (the number of accepted students who actually matriculate). Once colleges figured out that rankings can directly affect admission outcomes, the temptation to game the system became too much for some to resist.

After reading the Claremont McKenna news and speculation that many other colleges are finding ways to manipulate their numbers, I was struck by a thought; the group that most influences the rankings are not the colleges themselves, but rather the segment of the population that has come to rely on these rankings: parents and students. As much as I try to dissuade them, I do understand why parents focus and obsess over the low and still falling acceptance rates of the most selective colleges. Nevertheless, I encourage them to broaden their scope to include the many quality schools that are lower ranked and more robust in their admission numbers. Every time we rely on the U.S. News & World Report rankings as the quality standard bearer, we are perpetuating the misconceptions about excellence and motivating the actions that affect rankings. While reader perception of worth does not have its own separate weighting in the U.S. News & World Report formula the way that college peer opinion does, we are the ultimate drivers.

Think about this. If I use the rankings as my guide about where to apply, I, like everyone else who is employing the same skewed logic, will actually contribute to the ranking frenzy. The “top” colleges will draw more applications (good for rankings), have to reject a greater number of students (a huge boost for the rankings), which drives down their acceptance rates (a rankings game changer) and ultimately jacks up their yields (rankings home-run!). Never did I think that I would feel amazingly empowered by the realization that if I read and react to the U.S. News & World Report I could exercise such influence! Unfortunately, I am just one person. It is our collective action that will change this game.

Following Claremont McKenna’s disclosure last week, the finance magazine, Kiplinger, removed the school from its list of best buy colleges. One man erred in judgment and now this highly respected academic institution is no longer a smart educational investment? How can that be? Did anything else change at the school since last week? Sure, the average SATs are slightly lower than we thought and that, no doubt, will cause the college to fall in the 2012 rankings. Have we all been duped into thinking that Claremont McKenna was a better school than it really is? Perhaps…if you buy into the rankings.

It's Mid-August - Must Be College Ranking Season

Today the U.S. News & World Report published its annual Best Colleges 2011, the granddaddy of school rankings that students and their families too often use as the definitive source and arbiter for academic quality. Sure, USNWR provides a detailed explanation of its methodology as if to suggest transparency. Yet it still fails to convince this reader that the formula inputs offer anything but a highly subjective assessment that does little to assist students in finding schools that are the right fit and that will provide a meaningful and fulfilling college experience.

This year USNWR tweaked its criteria, with the most significant adjustment being the weighting of input from other education professionals. Though this now accounts for 22.5% rather than 25% of the total ranking, the methodology still places far too much emphasis on some subjective measure of a school's reputation. The good news is that only 15% is determined by administrators from competing institutions who I’m certain rarely if ever step foot on the college campuses they are judging. Even if they do, is this not a conflict of interest? For the first time high school guidance counselors have a say in the rankings and make up 7.5% of the 22.5% reputational factor. I am skeptical whether this actually helps students make informed college decisions. Most high school counselors know that students would be better served by considering fit over someone else’s opinion of a college’s place on a list that has questionable criteria.

Here is my question for every student and parent who feels compelled to look at the college rankings: what is there about this list of colleges and the way it is presented that gives you any insight about the academic strengths, the quality of life, the social scene, or the support services, academic and otherwise available at each school? I am not suggesting that all of the factors are meaningless. For example, I also believe that graduation and retention rates are important. However, a couple of statistics do not tell the whole story. The top schools on each of these ranking lists, for the most part, are all well endowed and can provide substantial need based aid. What about colleges of quality that cannot afford to be as generous with financial aid? In this economy many students are forced to take a leave from their education due to lack of adequate funds to pay tuition and support themselves through school. In other words, the economy is hurting retention numbers for some fine colleges that cannot spend as much as they would like on aid. My point is, too little information about what drives the numbers often leads one to draw the wrong conclusions.

But is it a “good” college? My own 17 year old has finally learned to stop asking me that question, though when she occasionally slips, she quickly recovers with, “I know, I know…don’t confuse selectivity with quality.” She’s finally catching on! Isn’t it curious that the colleges with the lowest rankings also have the lowest acceptance rates? The correlation between acceptance rate and a ranking which is supposed to order schools by some measure of “best” is especially troubling to me.

U.S. News & World Report is not the only publication to issue its assessment of colleges this month. Forbes.com recently published its list of America’s Best Colleges which has many surprises that will have you scratching your head, especially when compared with the USNWR rankings. And lets not short change Sierra Magazine’s current ranking of the Greenest Colleges. If you are not happy where your own alma mater ranks on any of these lists, be patient. Another ranking is sure to be released shortly.

Best Values in Public Colleges

In my November 23, 2009 blog posting I wrote about finding value in public colleges and universities. As a follow up, I wanted to share the link to Kiplinger’s Personal Finance magazine which recently published its 100 Best Values in Public Colleges 2009-2010. You can find the full list at http://www.kiplinger.com/magazine/archives/best-values-in-public-colleges-200910.html.

The Kiplinger report also includes a one page guide to the methodology used in the rankings, explaining that both academic quality and affordability come into play. The first cut to make the list was academic quality, which for purposes of the listing includes SAT or ACT scores, admission and retention rates, student-faculty ratios, and four and six-year graduation rates. Academics, in fact, are weighted almost two-thirds more than affordability. The schools were then ranked based on cost and financial aid. The cost assessment takes into consideration total expenses for in-state students, and then looks at the average cost for both need and merit aid recipients after subtracting grants. A similar exercise was done to determine out-of-state rankings.

Why is the primary emphasis on academic quality rather than cost? Following the old adage, “you get what you pay for,” Kiplinger is rightfully concerned that state schools are following prudent strategies to maintain or even improve quality while keeping costs down. Are colleges successfully eliminating the fat in their budgets or rather, cutting into their core mission initiatives (reducing or eliminating academic programs that benefit students, for example)? That is an important factor in the quality assessment.

In my prior posting about state universities, I mentioned that many of these institutions, in an effort to increase revenues, are trying to grow their out-of-state enrollment for obvious reasons: non-resident students pay more. As a result, several public “Ivies” and other flagship state universities, unlike some of their private, highly selective counterparts, offer merit aid to entice top non-resident students (A topic, perhaps, for another posting is the criticism public universities have recently received as a result of these policies: see the January 14, 2010 Washington Post http://www.washingtonpost.com/wp-dyn/content/article/2010/01/13/AR2010011302643.html). Schools such as the University of Maryland and UNC-Chapel Hill seek to increase non-resident enrollment and are using merit aid to bring in out-of-state tuition and to shape a class (though North Carolina has an 18% cap on the percentage of students from outside the state).

If you are exploring public colleges for value, then certainly take a look at the Kiplinger list. However, I will offer my usual caveat about rankings. Determining “value” based on a selection of criteria that include such measures as standardized test scores will probably yield results that should be viewed with a critical, if not skeptical eye. The conclusions drawn are not scientific. As I have stated in the past, “value” will largely be influenced by how good a fit the school is for the students attending. Student engagement is often what contributes most to value.

College Rankings....Drum Role, Please!

Last Thursday the U.S. News & World Report published its annual rankings of U.S. colleges and universities. The release of the rankings is always accompanied by speculation and anticipation…who will win the coveted top spot? The aftermath is equally predictable: schools that have moved up in the rankings tout their good news, other educators criticize the emphasis on misguided measures to rate schools, and parents spin into a frenzy and fret that their child may not get into one of the “top 25” schools. What amuses and frustrates me about these rankings and the clout that U.S. New & World Report commands is that no matter how questionable the criteria or poor the participation in the surveys which are used to rate schools, people from all sides of the aisle still look to the rankings as some kind of authoritative assessment of quality across the spectrum of higher education institutions.

Today I was reading the newly released college ranking issue while my freshman daughter, who heads off to college in a week, looked on. “Can I see this?” she asked as she grabbed the magazine from my hand. I knew exactly where she was headed…straight to the page with the rankings of liberal arts colleges. She quickly scanned the list, starting at the top, of course, and followed her finger down the page, glancing nervously for her school. “It isn’t here,” she said to me in a panic. “It’s there,” I assured her, and then pointed to it on the page, much to her relief, but not to mine. Despite everything I preach about “good” being what’s good for the student, my own daughter still falls victim to the ranking hysteria (I guess I do too since I already knew where her college ranked).

I, like most college consultants and counselors, make a point of talking to families and students about right fit and the fallacy of thinking one can actually meaningfully rank colleges. Do these rankings measure where your child is most likely to thrive, find the optimal social environment, get the best education that meets his or her needs and interests and at the best value, or whether the college he or she attends will predict future success in life (however one chooses to measure that)? Of course not! Then why are we overly fixated on them? We get caught up in prestige, name recognition and factors that have nothing to do with whether or not our children will receive a quality education that may open their eyes to the many possibilities available to them.

Here are some truths about the rankings. The U.S. News & World Report bases its rankings on 7 key measures, with the single highest weighted factor being that of peer assessment (in other words, the impression held by presidents, provosts or admission deans at other, unaffiliated institutions), which accounts for 25% of the ranking. Many schools choose to ignore these peer assessment surveys and only 48% actually filled them out this year. And among those that did, some are alleged to have manipulated their answers with the sole purpose of boosting their own rankings! The August 19 issue of Inside Higher Ed (http://www.insidehighered.com/news/2009/08/19/rankings) has a disturbing article that points out how schools may be gaming the system just to climb up the rankings.

Here’s my point: the flaws in these rankings are so obvious to so many, yet we still get caught up with them and ascribe undeserved value to the ranking order. Even with no intended manipulation, why would an admission dean at another college know or have any say in the quality of the education or experience a student will have at a college he or she may have never even visited and why is this given so much weight in the rankings?

So please, ask yourself as you assist your sons and daughters through the college admission process about the utility of college rankings and whether this is a good way to choose a college where your child will excel and be happy. What makes one institution better than the one ranked directly below it and who decided that college A should be listed higher than college B? Some of you will read this and continue to give undue attention and importance to these rankings. At a minimum, I hope you will think twice before you make assumptions about the value of a four year college or university based upon where it stacks up according to U.S. News & World Report. I very much welcome the thoughts of my readers on this topic!