The Long-Term Effectiveness of Multiple Measures Assessment: Evidence From a Randomized Controlled Trial

Elizabeth Kopko, Hollie Daniels & Dan Cullinan | October 2023

This study—a follow-up to prior CAPR research using the same sample of students—employs a randomized controlled trial to investigate whether an algorithmic multiple measures assessment (MMA) system used at seven State University of New York (SUNY) community colleges led to better student outcomes—for up to four and a half years after randomization—than a system based on test scores alone. Nearly 13,000 incoming students who arrived at the seven colleges in fall 2016, spring 2017, and fall 2017 took placement tests and were randomly assigned to be placed using either the status quo method (business-as-usual group) or the alternative, algorithmic MMA method (program group). Using this sample, the authors estimate the overall treatment effects on placement into, enrollment in, and completion of college-level math and English as well as effects on other outcomes. They conduct similar analyses on race/ethnicity, Pell recipient status, and gender subgroups. They also descriptively examine the proportion of program group students who were bumped up (i.e., their placement changed from a developmental course placement to a college-level course placement) and bumped down (i.e., their placement changed from a college-level course placement to a developmental course placement) by the MMA algorithm. They perform a cost-effectiveness analysis as well.

The study finds that the MMA method used at the colleges improved access to and success in college-level courses and that lower cut scores in English rather than math are associated with larger and longer lasting impacts on completion of college-level coursework. In English, program group students had higher rates of placement into, enrollment in, and completion of (with a C or higher) college-level English courses. Impacts on enrollment and completion in English were statistically significant across all nine terms. In math, program group students had higher rates of placement into and enrollment in college-level math courses, and they had higher initial rates of completion. While impacts on enrollment persisted through term nine, there is no evidence that statistically significant gains in completion were sustained past the first term.

While MMA improved outcomes among student subgroups, it had little to no impact on gaps in outcomes between subgroups. The study also finds that bumped-up students had substantially better outcomes in both math and English, while bumped-down students had substantially worse outcomes. The results suggest that increased access to college-level courses is the driving factor in the positive outcomes experienced by program group students and that placement into standalone developmental courses can have detrimental effects on student outcomes. In the discussion of the study’s results, the authors make recommendations for adopting MMA at colleges.

A CAPR brief focusing on the “bump-zone” findings is also available: By the ninth term, program group students in the bump-up zone were about 9 percentage points more likely than their business-as-usual peers to complete college-level math or English.