Podcast: Explaining the Research on Multiple Measures Placement

MDRC and the Community College Research Center (CCRC) are working together on two major studies of the use of multiple measures to place students into developmental or college-level courses—one through CAPR in New York State and a separate project in Minnesota and Wisconsin. Evidence is growing that replacing test-only placement with multiple measures systems allows more students to enter directly into college-level courses while maintaining or increasing the number of students who pass gateway courses.

The two MDRC and CCRC studies are using rigorous research methods to get a truer picture of the effect of multiple measures and give colleges more definitive answers about its short and long term impacts on students. MDRC’s Katie Beal spoke with CCRC’s Elisabeth Barnett and MDRC’s Dan Cullinan about what they are learning for MDRC’s Evidence First podcast.

Transcript

Katie Beal, External Affairs Associate, MDRC: I’m Katie Beal, and welcome to Evidence First. In this episode, we’ll be talking about developmental education—in particular, how community colleges determine whether students need developmental courses to improve their skills before taking college-level courses. Most students are placed into developmental math and English courses based on their scores on standardized placement tests. However, research suggests that the tests are not a good predictor of whether students can actually succeed in college-level courses. In fact, the tests actually misplace students, meaning there are students who are placed into developmental education even though they could have succeeded in college-level courses. This is troubling, since we know that students who place into developmental education are less likely to graduate. One alternative strategy is to use multiple measures of college readiness to place students. These measures include high school GPA, time since high school graduation, and placement test scores, among others.

MDRC and the Community College Research Center, or CCRC, are currently conducting two studies to understand the impact of multiple measures placement systems on student success. One is through the Center for the Analysis of Postsecondary Readiness, or CAPR, which is a developmental education research center led by CCRC and MDRC. And the other is in partnership with several community colleges in the Midwest and with Ascendium Education Group. To learn more these projects, I spoke with Elisabeth Barnett, a senior research scientist at CCRC, and Dan Cullinan, a research associate at MDRC.

So why is this issue of placement so important? How can placing in a college-level course versus a developmental course affect students’ path?

Elisabeth Barnett, Senior Research Scientist, CCRC: It goes back to the reasons why we want students to be placing into college level if at all possible. So, if they’re being told when they come in that they’re ready for college-level material, first of all, it’s very encouraging. It gives them the sense that they are really on the path to college, and secondly, multiple studies have shown that students who start off in college-level courses are going to have a better shot at succeeding.

So students end up in developmental courses that are not allowing them to accrue credit and also delaying their ability to enter into college-level courses. Now, some of them may need that head start, and others of them could have been successful in college-level courses, and so we want to give every student that has a good probability of being successful in college courses the chance to start there.

Beal: So to get at this issue of placement, you’re conducting a study through the Center for the Analysis of Postsecondary Readiness. Tell me more about the goals of that study.

Barnett: The goal of the study is to make sure that we do a good job with placement. So the research is looking at using multiple measures assessment—so using more than one measure and, most importantly, the high school GPA as one of the criteria to try to do a better job of placing students. And what we’re interested in finding out is whether students who are placed more accurately actually end up having better college outcomes.

Beal: As part of this study, you worked very closely with seven community colleges in the State University of New York system to develop an algorithm that would predict students’ success in college courses based on historical data like high school GPA, placement test scores, time out of high school, and so on. And then you helped the colleges implement new multiple measures placement systems based on the algorithms. So what did you find?

Barnett: We’ve just had a chance to look at the impacts of the students who were placed using multiple measures and compare them with students who were placed using a placement test alone. What we’re seeing is that in both math and English, students who were placed using multiple measures were considerably more likely to be placed into college-level courses, and then to actually take those college-level courses, and finally to pass those college-level courses. And while that’s true in both math and English, we’re seeing much bigger impacts or differences between the two groups in English than we are in math.

Dan Cullinan, Research Associate, MDRC: And I think what will be really interesting to see is the longer term follow up—how persistence plays a role in this in subsequent semesters. This is really an indication that things are working as planned and expected so far. In the final report, we will have a better sense of the ultimate success of this.

Beal: And you also looked at the impact on different types of students. Tell me about that.

Barnett: We were able to also look at whether there were differences between students by race/ethnicity, by whether they received a Pell Grant or not, and gender. And what we found is that across all of the different outcomes we were concerned with, students in all groups were benefiting, which was very encouraging. When we looked at math, we actually saw women catching up to men in the proportion that were succeeding in their math courses, so that was encouraging as well. We aren’t able to say yet whether gaps are actually being reduced, but we hope to be able to look at that in the next phase of this research.

Beal: So Dan, MDRC and CCRC are working together on another multiple measures project with colleges in the Midwest. Tell me about that project.

Cullinan: We are working to roll out multiple measures assessment systems at a number of colleges in Minnesota and Wisconsin. We began by piloting at five colleges in each of those states, at a small scale, a system that would take into account high school GPA, but in addition to high school GPA, also a noncognitive assessment.
We really pulled from previous research, from the experience of states that have already begun this process, like North Carolina and California, and from the internal research that was being done at these colleges on their own students to come up with a sort of common-sense cutoff on high school GPA, and then to start thinking, you know, really brainstorming, how could we use a noncognitive assessment, particularly in cases where maybe a student is a nontraditional student and doesn’t have a high school GPA readily available, or at least one that’s recent.

We were able to take our experience from all this and generate a sort of playbook for colleges that are already interested in doing multiple measures assessment, to kind of walk them through, what are the kinds of things you need to consider before getting started? Who needs to be involved in the conversation? What are some potential obstacles to be prepared for? And how do you assess whether or not the new placement system is being implemented as intended?

Beal: Great, so as a result of these two projects, you have worked closely with several colleges now to develop and implement placement systems using multiple measures. What are some key early lessons or advice you have for colleges?

Cullinan: I think the early lessons from the work in Minnesota and Wisconsin really boil down to allowing adequate time to set something like this up, making sure that really everyone who’s touched by this at the college is involved early on in planning and making key decisions. That includes faculty, admissions, advisors, testing staff, registrars, IT, administrators. . . depending on the college, there may be others.

Barnett: One specific area of communication that’s important is related to the high school GPA itself. We’re asking people to consider using the high school GPA as the major predictor in many cases, and there are a lot of concerns about, first of all, the quality of local high schools and whether a GPA from all of them means the same thing. There are concerns about whether it still has validity a certain number of years after high school, there are concerns about just the logistics of getting ahold of it. We think that it’s worth taking the time to really help people to understand what the experience has been in other places, both in terms of kind of working through the barriers, but also in terms of its predictive value.

So there has been a lot of prior experience in other states where they’ve examined high school GPA in relation to placement test scores and almost universally have found that no matter which high school students came from, high school GPA is doing a good job of predicting their future success.

Beal: And how do you see these multiple measures placement reforms fitting into other developmental education reforms that colleges are taking on?

Barnett: Community colleges are working very hard but still graduating small proportions of students, and that’s of great concern both to colleges and students themselves, and to society. So, many colleges are also in the process right now of thinking deeply about the changes they can make. Different reform movements that are emerging are focused on guided pathways, so trying to structure students’ experiences so that they can more easily make good decisions about how to fulfill the requirements for graduation. There are new math pathways being used, there are new approaches to developmental education that involve companion courses, corequisite courses, and you know, there are a number of people that are trying these placement reforms.

So, many colleges are kind of working at them all at the same time, but what we’re seeing is that, from these early results, taking on the placement reforms is really a good idea. That you can have a relatively large impact, at least on the short-term outcomes, which are the ones we’ve looked at so far, by making reasonably small changes to college procedures.

Cullinan: And I think it’s also an access question, ultimately, because while a lot has happened to get more students access to particularly community colleges, if they are taking a lot of developmental or remedial courses and not persisting, they really aren’t getting access to that college education. The hope is that this will improve that.

Beal: Thanks to Dan and Elisabeth for joining me. In 2019, MDRC and CCRC will release longer term findings from the CAPR study and early findings from the work with the Wisconsin and Minnesota colleges. If you’d like to learn more about CAPR, visit postsecondaryreadiness.org.