Exposé of the Manipulation of Data used to Justify the Reinvention

Below we present a meticulously researched document that exposes the way in which the quality of education at the Chicago City Colleges (CCC) has been misrepresented by manipulating and selectively choosing the Colleges performance data. These are data that have been professionally gathered over many years and that present an opposite picture to that loudly peddled by Chancellor Hyman and her crew. The author brilliantly demonstrates how the solution came first, the Reinvention, and the problem was identified afterwards via the use of spurious facts and misinformation.

Furthermore, the author provides important insights as to how those “selfless” consultants of the Reinvention, initially provide free services so they can later frolic in the pie of juicy goodies being showered by Hyman on newly hired upper level administrators. In a previous post, we have warned readers about this topic, as it pertained to the limbic relationship that the CCC and Accenure have developed.

We thank the author for sharing with us this clarifying analysis.

PEARL

The Case for Change: Reinventing the Wheel

Shortly after she was appointed appointed Chancellor of the City Colleges of Chicago (CCC) in April 2010 by Mayor Richard M. Daley, Cheryl Hyman acquired the services of two consulting firms, McKinsey & Co. and the Civic Consulting Alliance.  The work contributed by these companies has been described as pro bono—that is, loosely, free of charge.  In November, however, Chancellor Hyman hired Alvin Bisarya, a consultant from McKinsey, and Donald Laackman, a member of the Civic Consulting Alliance.  Both formerly “free” advisors, they have assumed positions that pay $130,000 and $140,000, respectively.  In short, pro bono was transformed into quid pro quo.

Nothing Is What It Seems

It is important to understand this fact because it illustrates a major theme of the Hyman administration: Nothing is what it seems.  Besides the  Chancellor’s hiring of twenty-five executives at a cost of more than $100,000 each and then claiming that her goal is to  save big money, the best example of this theme is the rationale for the Chancellor’s program for redesigning the entire CCC system, which she and Mayor Daley call Reinvention.  Keep in mind that, according to a Civic Consulting Alliance document, Hyman launched her investigation into the state-of-the-colleges in the summer of 2010, but it wasn’t as much an investigation of a problem as it was a justification for a solution.  That is, McKinsey and the Alliance “helped City Colleges of Chicago build their ‘Case for Change.’” 

Thus, while the Alliance document claims that the consultants took “a deep dive into the metrics that tell the [colleges] how they are doing today and where they want to head in order to deliver on student success,” they didn’t dive very deeply, after all.  They cherry-picked the data and then used it to provide an excuse for transforming a system that, as all the data show, didn’t need to be transformed.

After briefly stepping into the shallow waters of pre-chosen statistics, the Chancellor embarked on a city-wide tour of businesses, industries, civic associations, and professional societies to make her Case for Change. The tale she has consistently told is that the colleges in CCC are failing in every performance area.  The aforementioned audiences heard that

·                      only 7% of CCC students graduate;

·                     only 16% of them transfer to four-year institutions;

·                     only 4-5% earn a bachelor’s degree;

·                     and more than half drop out before completing 15 credit hours.

Presumably, it was these “dismal” statistics that led to the “dismissal” of six college presidents in February 2011.  After all, the Board of Trustees redefined the job of college president to include the ability to meet “specific performance measures and goals.”  Evidently, the sitting presidents failed to demonstrate that ability. 

In this regard, it’s important to note that, according to CCC’s July 14, 2010, news release, the people who were added to the senior administrative staff under Chancellor Hyman brought “expertise and experience which will increase the City Colleges’ ability to strengthen accountability and oversight and to provide high quality services to our students.”   It looks as if the competent are coming in, and the incompetent are going out.  If that’s the case, then Chancellor Hyman’s claim that “nobody has been fired” (quoted in Inside Higher Ed) sounds suspect.  However, it may only turn out to be just another example of the kind of sleight of hand that turns “free” into costly.  Nothing is what it seems.

Ignoring the Data

The July 14 CCC press release stated that “Management oversight will improve by better utilizing data and organizational intelligence to guide district strategy.”   Ironically, but not surprisingly, CCC has a pile of data, collected over many years, demonstrating that the Chancellor’s doom-and-gloom diagnosis of the colleges’ performance is completely wrong.  Worse yet, the advisory teams seem to have missed this data, much of which is long-term, comprehensive, and generally favorable. 

For example, in 2009, one year before Cheryl Hyman was appointed, the District Office of CCC issued a summary of student success in the seven colleges across a six-year period, from 2002 to 2008.  Based on a “nationally recognized” benchmark (from the National Student Clearinghouse) for judging how well a college is doing in serving the needs of its students, this longitudinal study measured, in addition to the percentage of students transferring to and graduating from four-year institutions, the percentage of students returning, completing degrees or certificates, and attaining a 2.0 grade point average.

The advantage of this analysis is that it’s not based on data that have been selected to prove a point.  The CCC Office of Research and Evaluation (ORE) evidently proceeded with the hope that the seven colleges in the CCC system would demonstrate that they were performing well, but there were no guarantees.  At the end of the six years, the ORE found that almost 67% of the nearly 7,500 credit students in the cohort were successful in at least one area of achievement.  The combined transfer/degree completion figure was 32%.  And the percentage of students who had successfully completed their courses was almost 30%, not counting graduates and transfers.

Also demonstrating that the colleges are far better than the Chancellor claims is the information released by the ORE in March 2011 showing that term-to-term retention improved dramatically over a five-year period, from 2006 to 2010.  District-wide, the jump went from 63.0% to 67.3%.  Olive-Harvey, Wright, and Kennedy-King each increased the numbers of returnees by five percentage points.  The colleges had been encouraged to focus on retention, and they did so.  Wright, for example, joined an organization called Foundations of Excellence, implemented a program called the First-Year Experience, and reaped the benefits.  Other colleges made similar efforts.  In 2008, the Integrated Postsecondary Education Data System (IPEDS) reported that Wright College beat the schools in its 29-member cohort in retention, 59% to 56.4%.

These positive results, which reflect a long-standing interest on the part of college administrators, faculty, and staff to improve the academic performance of CCC students, are the product of much soul-searching, hard work, and intense focus dating back to 1998.  According to the Illinois Board of Higher Education (IBHE) Results Report for FY 2003, Wright College was at that time “in its fifth year of implementing educational programs and practices”  that met the goals of a state-wide program called The Illinois Commitment and Illinois Community College Board’s “requirement to incorporate information regarding performance indicators and the assessment of student learning.”  That is, every college in the state was mandated to improve student achievement by assessing its methods and evaluating its results.  Clearly, the community colleges of Chicago met that challenge.   

In defense of the criteria used in the ORE study, it is important to remember two points.  First, students at community colleges typically take far longer than four-year-college students to complete a degree or earn a certificate because most of them attend part-time (60%, according to the 2009 Community College Survey of Student Engagement [CCSSE] report), most of them work either full-time (one out of five, according to this report) or part-time, and many of them must take one or more remedial courses.  As a result, degree-completion and transfer are typically slowed down quite significantly.

Second, community colleges like CCC, in the words of the ORE’s interim 2007 report, have “the mission of delivering learning opportunities and educational services for diverse student populations.”   Thus, as the report says, although “graduation rates are often used as the single measure of student success … this measure fails to document outcomes for students with multiple objectives and career paths.”  The CCSSE report lists as goals of community college students—in addition to graduation, certificate completion, and transfer—obtaining or updating job-related skills, self-improvement or personal enjoyment, and changing careers. 

This means that many students take one or more courses and stop when their goal is satisfied—well short of the time required for graduation or transfer.  And such students are not indications of system failure, but of system success.  That’s why the ORE warned in its 2009 study, “Reporting one single outcome in isolation is a biased and incomplete method for reporting student outcome for community college students.”  So much for the Chancellor’s frequent use of the 7% graduation rate as a justification for change.  Anyone who actually read either CCSSE’s list of student goals or the District’s own ORE report would know better.

Misusing the Data

When reporters and bloggers get their information exclusively from CCC under Chancellor Hyman, they are too often likely to ignore this caveat.  Worse yet, they are likely to get the numbers wrong.  Blogger Joanne Jacobs, for example, said on August 12, 2010, that “20 percent of City Colleges students complete a certificate or degree or transfer to a four-year institution.”   The Joyce Foundation reported in February 2011: “City Colleges has struggled with boosting its student outcomes and local businesses have been hesitant to hire its graduates.  The institution has also suffered from a lack of quality data to guide reforms or set policy standards.”   Progress Illinois stated in November 2010, “It’s no secret that the system is in serious need of repair; new data compiled by the city found that only 16 percent of Chicago’s 120,000 student transfer to four-year colleges.”  

Ms. Jacobs needs to know that the latest available graduation rate (2008) is 32%, not 20%.  The Joyce Foundation should refer to the extensive study mentioned above, which was intended to “allow CCC to document student success, identify at-risk student populations, and enhance academic and student services,” precisely the goals of  the current CCC administration, who act and speak as if nobody in the District ever thought of using research for such purposes.  Missing in the Progress Illinois claim is the fact that well over 50% of the 120,000 students, of which “only” 16% graduate, are not in credit programs.  Enrolled in Adult Education or Continuing Education, they can neither graduate nor transfer to four-year colleges.  The courses they take are either pre-credit or non-credit.   

 It’s quite possible that the Chancellor has made the same mistake, since all of her data turn out to be unrelated to the 2009 longitudinal study.  She has 4-5% of CCC students graduating from four-year institutions, whereas the report has 7.5%.  She has 16% transferring, although the report has this number at nearly 19%.  Her most famous statistic, that only 7% graduate, is contradicted in the report, which puts the graduation rate at 13%. 

It may be that Chancellor Hyman and her highly paid advisors either failed to add in certificate earners or calculated percentages based on the total student figure of 120,000 instead of the actual number of credit students, 42,000.  However, no matter how we interpret the Chancellor’s use of statistics, it’s clear that it is, at the very least, misleading, if not flagrantly and purposely deceptive.  A press release from CCC says that Cheryl Hyman “began to measure City Colleges’ performance based on student outcomes quickly after being appointed to her position by Mayor Daley.”  But perhaps she proceeded too quickly and, therefore, missed the 2009 report by her own Office of Research and Evaluation.

Comparison and Context

The worst thing about the data being used by CCC to fix a system that’s not actually broken is that it’s non-comparative and non-contextual.  By non-comparative, I mean that the statistics for graduation and transfer don’t mean very much in isolation.  They have to be compared to the statistics for other two-year colleges in order to determine whether they reflect success or failure.  Tossing around such data (especially when it’s inaccurate) doesn’t reveal anything, except, of course, a desire to prove the unprovable.

By non-contextual, I mean that the data used by the Chancellor and her advisors have been interpreted outside the context of factors that influence school success, but which are beyond the control of the schools themselves.  The two main factors in this regard are student preparedness and program resources.   We have to consider, first, the quality of CCC students’ preparation for college work, mostly in the Chicago Public Schools.  Beyond any doubt, CCC is at least partly a product of this system.  That is, it is very much the context in which CCC operates.  In addition, we have to assess the amount of money actually invested in the education of students in the CCC system, which depends on state funding and District Office allocations.

While data on comparisons between CCC, as a whole, and other community colleges is lacking, the individual colleges maintain such information, sometimes at the behest of their accrediting agency.  (Each college is separately and independently accredited.)  Wright College, which has been placed in a national cohort of 29 demographically similar community colleges, uses data from the IPEDS, whose 2008 report shows that Wright granted more than 1,200 degrees and certificates in 2008, versus an average of slightly more than 1,100 at the 29 comparable institutions.  The combined graduation and transfer rate at Wright is also above the national average—substantially.  The 2008 IPEDS study shows that Wright’s rate was 42%, versus an average of 36% for schools in its cohort.

Wright also measures the academic success of its students by using the Collegiate Assessment of Academic Proficiency (CAAP) to test students in five areas of study.  In 2008, Wright students scored well above the national average on CAAP tests in math, slightly above the average in science, slightly below average in critical thinking and reading, and average in writing.  The amazing thing is, the individual City Colleges are comparatively successful despite the fact that many of their students are graduates of one of the worst public school systems in the country, evidenced not only by the notoriously large dropout rate in Chicago’s high schools, but also by the sizable number (90%) of incoming CCC students who need at least one remedial course before they can take credit classes in that field. 

The City Colleges of Chicago have open enrollment, which means that, unlike most four-year institutions, they can’t pick and choose their students.  More often than not, they’re forced to work with the students who graduated in the bottom half of their class rather than the top.  Furthermore, remediation can only go so far.  If students are reading at an elementary school level, it’s difficult to bring them up to the college level in a short amount of time.  And it’s easy for them to exhaust their student aid if the process takes too long.  The result, all too frequently, is that students who have been successfully remediated and are therefore prepared to take college courses can’t afford to continue.

The other part of the context in which CCC student achievement must be understood is resources.  Not only are CCC students underprepared, they are also underfunded, at least if Wright College can be used as a standard for the District.  According to Wright’s interpretation of IPEDS data, the college “employs fewer total FTE [full-time equivalent] staff than do institutions within the peer group.  Wright has also tended to employ fewer persons in instructional and administrative/managerial positions than institutions within the peer group.”  As a result, Wright spent only $48 M on “core expenses” in 2008, compared with an average of $56 M in the cohort.

These claims are also verified in the 2010 IPEDS report, which says that Wright had 539 employees in 2009, while the cohort averaged 874.  Most significantly, the instructional staff at Wright numbered 210, against the cohort’s average of 288.  There were 61 employees in support services at Wright, but 99, on average, at comparable institutions.  The cost per student “full-time equivalent” was $6,385 at Wright, but $9,026 at other community colleges.     

According to the 2003 IBHE report, even in that year Wright, operating “at a high degree of productivity and accountability,” had “few, if any, efficiencies left to be achieved.”   The college had a higher average class size than the State of Illinois average.  And the cost of instruction was significantly lower.  In short—again, if Wright can be taken as typical of the colleges in the CCC system—these schools have, for a long time, operated at a disadvantage compared to other colleges in Illinois and nationally.  Ironically, as a cure for underfunding, the Chancellor recently called for a 10% reduction in instructional expenditures across the board.  Each school lost an average of thirty employees and “saved” the District millions of dollars.  To what end?   

What’s Going On?

The Chancellor’s invalid claim that the community colleges of Chicago are failing has created two very serious problems.   Based as it is on non-comparative and non-contextualized evidence, the claim is insulting to faculty, staff, and students.  One would expect an analysis of the strengths and weaknesses of any operating system to be based on a careful, thorough, and long-term study, such as was conducted by the CCC’s own Office of Research and Evaluation from 2002 to 2008 and such as is now being conducted by the sixty-member task forces (created by District Office in January) that are meeting for 30 hours a week over a period of one semester—no doubt at a cost of approximately one million dollars—to come up with a solution to a problem that should have been studied in the same way. 

The result is that faculty members, who have shown themselves to be dedicated to ongoing and comprehensive self-assessment as a means of self-improvement, have been told that their efforts have been inadequate.  It is a safe bet that neither the Chancellor nor the members of the CCC Board of Trustees have any idea how much time and energy City College faculty have put into assessment efforts over the past five to ten years or even the slightest notion of the ability and dedication of both faculty and staff.  The reward for the college presidents, whose hard work has resulted in measurable achievement over the past few years, is termination.  Who’s next?

Furthermore, the mere fact that the Chancellor’s claims are baseless raises questions about her motivation.  Was she the recipient of misinformation?  Did a staff member give her bogus numbers and also neglect to provide the comparative and contextual data that would allow even the real numbers to be understood in their proper perspective?  Or were the numbers deliberately chosen and then interpreted solely to make a case for change—change, by the way, that has until now only resulted in  bitterness, shock, anger, frustration, disappointment, and confusion on the part of college administrators, faculty, staff and students.

Strange to say, the net effect of the Chancellor’s Reinvention has so far been a massive expenditure for high-priced marketing consultants and new six-figure hires for CCC’s non-instructional District Office.  In the meantime, thanks to the Chancellor’s presentation of questionable data, CCC has not been improved; it has been discredited.      

For the (so far) uninquiring minds of those who have supported the Chancellor’s sweeping decisions affecting the entire CCC system, the figure of 7% has been taken to justify massive changes.  What will happen when they discover that the rationale for the Chancellor’s program of radical change, Reinvention, is nothing more than a fabrication?  Will the program be terminated?  Will the Chancellor be fired?  That is, after millions of taxpayer dollars have been spent solving a problem that doesn’t exist, will heads roll?  One assumes that these issues will be raised at the Chancellor’s” annual review of performance,” required by CCC Board Rule 3.19.  And one assumes that the Case for Change will be subjected to the kind of scrutiny it deserves.