Multi-State Collaborative on Learning Outcomes Assessment Completes Pilot Study Gathering Data from 59 Institutions of Higher Education in 9 Public State Systems

Washington, DC— September 24, 2015—With support from the Bill & Melinda Gates Foundation, the Association of American Colleges and Universities (AAC&U) and the State Higher Education Executive Officers (SHEEO) Association launched in 2011 the Multi-State Collaborative to Advance Learning Outcomes Assessment (MSC).  In its pilot year, the project initially engaged faculty in 59 institutions in nine participating states.  The nine states participating in the MSC during the pilot year were: Connecticut, Indiana, Kentucky, Massachusetts, Minnesota, Missouri, Oregon, Rhode Island, and Utah.  See:www.sheeo.org/msc for the full list of participating institutions involved in both the pilot and continuing phases of the MSC work.

The pilot study successfully demonstrated that rubric-based assessment can be taken to scale and can produce valid findings with credible and actionable information about student learning that can be used to improve curricular and assignment designs and to increase effectiveness of programs and classes in advancing the most important learning outcomes of college.

“What this pilot study showed is that faculty from a variety of disciplines, from dozens of colleges and universities, from nine different states across the nation could assess the work students had done and evaluate it in a consistent and reliable way,” said SHEEO President George Pernsteiner.  “There was no special test.  There was no time away from the classroom.  There was, however, a common understanding by faculty from diverse places and backgrounds of what constituted learning and whether students had demonstrated it.”

As part of the pilot study, more than 7,000 samples of student work produced for course assignments in students’ regular courses were uploaded to a Web platform developed by Taskstream.  126 faculty members were trained, and then independently scored students’ work to produce a preliminary landscape analysis of student achievement at the participating schools.  Samples of student work were collected and evaluated for achievement in three important learning outcome areas: written communication, critical thinking, and quantitative literacy.  The faculty members used common scoring rubrics—called VALUE rubrics—that were developed and validated by faculty as part of AAC&U’s Liberal Education and America’s Promise (LEAP) initiative.

The MSC is part of AAC&U’s ongoing VALUE (Valid Assessment of Learning in Undergraduate Education) initiative originally launched in 2007. In its pilot year, SHEEO and AAC&U tested the feasibility of cross-state and cross-institutional efforts to document student achievement without using standardized tests and without requiring students to do any additional work or testing outside their regular curricular requirements.  All the student work samples were assessed using common rubrics developed and tested against student work by teams of faculty at hundreds of individual institutions across the country.

In the MSC pilot study, 126 faculty from across the participating states and campuses used the common VALUE rubrics to evaluate student work and scored only work products produced by students from institutions that were not their own. 

The pilot successfully demonstrated that:

• A wide array of institutions can develop sampling plans to provide reliable samples of student work from across a variety of departments and that demonstrate achievement of key cross-cutting learning outcomes.

• Faculty can effectively use common rubrics to evaluate student work products—even those produced for courses outside their areas of expertise.

• Following training, faculty members can produce reliable results using a rubric-based assessment approach.  More than one-third of the student work products were double scored to establish inter-rater reliability evidence.

• Faculty report that the VALUE rubrics used in the study do encompass key elements of each learning outcome studied, and were very useful for assessing student work and for improving assignments.

• A Web-based platform can create an easily usable framework for uploading student work products and facilitating their assessment.

• Actionable data about student achievement of key learning outcomes on specific key dimensions of these important learning outcomes can be generated via a common rubric-based assessment approach.

While the findings from the pilot study are not generalizable across the entire population of students in the participating states or nationally, the study found within the cohort of participating institutions some clear patterns in students’ achievement levels.  Using a 0–4 rating scale, much higher percentages of student work products were rated at either a “3” or “4” in four-year institutions than were rated at those levels in two-year institutions in the project.  Significant numbers of students nearing degree completion at two-year institutions demonstrated high or very high levels of achievement on key outcomes.

Selected Findings on Critical Thinking

• Consistent with other national studies, many students who had earned at least 75 percent of credits toward their degrees still are not achieving high levels of important critical thinking skills.   Of the three outcomes evaluated, far fewer students were achieving at high levels on a variety of dimensions of critical thinking than did so for written communication or quantitative literacy.

• Using a 0 to 4 rubric scale, less than one-third of student work products collected from four-year institutions were scored a “3” or “4” on “using evidence to investigate a point of view or reach a conclusion.”

• Nearly 40 percent of work samples assessed at four-year institutions were rated only “0” or “1” on how well students “analyzed the influence of context and assumptions” to draw conclusions. 

• On the other hand, nearly 45 percent of work products from four-year institutions were rated “3” or “4” in terms of how well students “explained issues” in their analysis.

Selected Findings on Written Communication

• Nearly 50 percent of work products collected from two-year institutions were scored either “3” or “4” on “content development” in writing.

• About one-third of work products collected from two-year institutions were scored “3” or “4” on demonstrating the use of “sources and evidence” in writing.

• More than 50 percent of student work at four-year institutions was scored at either a “3” or “4” overall in terms of written communication, but when looking at the individual dimensions of the writing rubric, one still sees much room for improvement in students’ capacity to use “sources and evidence” to make arguments in their writing.

Selected Findings on Quantitative Literacy

• Far more student work products collected in both two-year and four-year institutions were rated “3” or “4” on students’ “calculation” skills than were rated similarly on students’ ability to “make judgments and draw appropriate conclusions based on quantitative analysis of data.”  Less than half of work products at four-year institutions and about one-third of work products at two-year institutions were rated “3” or “4” on this dimension of quantitative reasoning.

“The calls are mounting daily for higher education to be able to show what students can successfully do with their learning,” said AAC&U President Carol Geary Schneider.  “The VALUE Multi-State Collaborative is a very important step toward focusing assessment on the best evidence of all:  the work students produce in the course of their college studies.  It is exciting and inspiring to see the results of this project.  Higher education owes a debt of gratitude to the educational leaders and faculty members in the participating states who helped develop this pilot study and contributed to these illuminating results.”

See the MSC pilot summary slide deck for full information on demographics of students in the pilot study institutions, rubrics used in the assessments, and preliminary results of scoring of student work products.

For more information, see VALUE and Multi-State Collaborative on Learning Outcomes Assessment.