National Partnership Begins Work on Large-Scale Efficacy Trials of Promising Afterschool Programs
In recent years, there has been a movement to strengthen the quality of afterschool programs by tying these programs to the academic subjects of the regular school day to improve student achievement. At the same time, according to Dr. Robert Stonehill, director of the U. S. Department of Education’s 21st Century Learning Centers program, there have been thousands of studies conducted about various aspects of afterschool programs, many of which suggest that afterschool programs have benefits (SEDL Letter, May 2006). The problem is that few of these studies were rigorous studies that used randomized controlled trials. Those that were rigorous attempts to study the impact of afterschool programs on student achievement outcomes have yielded mixed results. So, an important question remains: Can fully developed, well implemented, promising afterschool programs have an impact on student achievement?
To address this question, SEDL, as part of the National Partnership for Quality Afterschool Learning, funded by the U.S. Department of Education, recently ran a competition focused on randomized controlled trials (RCTs) that could provide rigorous evidence in the debate about the benefits of afterschool interventions on student achievement. Below, we describe the projects that were funded and discuss the development of a research consortium to support their efforts over the 2-year funding period.
Three RCTs Focus on Reading Programs
A Randomized Evaluation of the Adventure Island Afterschool Reading Program with English Language Learners, conducted by the Success for All Foundation, evaluates the Adventure Island afterschool reading program with English language learners (ELLs) and other language minority children. Adventure Island, an adaptation of the Success for All program, is currently being evaluated in a large-scale randomized experiment, but that experiment includes few ELLs. This experiment provides a definitive evaluation of a promising approach to afterschool education with a population of great interest. Adventure Island will be studied in 14 majority-Hispanic schools in Alabama, Texas, and Utah, with approximately 1260 children in grades 2–4.
What Works in Afterschool Programs: The Impact of a Reading Intervention on Student Achievement in the Brockton Public Schools, undertaken by MPR Associates, Inc., in partnership with Scholastic Publications and Brockton (MA) Public Schools, is designed to compare Scholastic’s READ 180 program with Brockton’s standard services for afterschool programs. READ 180 will be studied with approximately 1,100 students in grades 4, 5, and 6 to capture gains in reading skills and other outcomes. The study will provide methodologically rigorous information about READ 180, a program that has several characteristics that have been associated with positive academic outcomes.
Afterschool Randomized Controlled Trials: The Voyager Passport Program in Kentucky 21st Century Community Learning Centers, is designed by The Center for Evaluation and Education Policy at Indiana University, in collaboration with the Kentucky Department of Education, to compare the impact of previously established 21st Century Community Learning Center (CCLC) programs with the impact of CCLC programs that include the Voyager Passport Reading program. The study tests the hypothesis that economically disadvantaged youths in grades 2–5 who participate in a high-quality afterschool program with Passport achieve significantly greater learning gains over a 2-year period than do disadvantaged youth who participate in the same afterschool program without Passport.
Afterschool Research Consortium to Support Implementation of Research Designs
SEDL will provide analytic and technical support to projects through the Afterschool Research Consortium (ARC). The ARC brings together SEDL staff, key project staff, Technical Working Group (TWG) members, and representatives from the Department of Education on a regular basis to facilitate a cross-fertilization of ideas and provide strategies that can help ensure full implementation of RCT designs in afterschool settings.
The ARC focuses on substantive content issues in the afterschool field, technical and analytic issues in conducting rigorous RCTs, programmatic and research-to-practice issues, and policy issues. Project teams are brought together to discuss ongoing implementation, data collection, and analytic challenges presented by the work as well as to explore opportunities to collaborate and disseminate the findings to the field.
Next Article: Better Research, Better Schools: Making Today's Research Relevant to the Classroom
