What Rigorous Research and Reviews Tell Us: Impacts of Afterschool Programs and Parent Involvement on Student Outcomes

Systematic reviews are becoming more important in the field of the education. These rigorous reviews are especially important in areas such as afterschool or parent involvement where there are a limited number of stringent experimental or quasi-experimental studies and where there are mixed findings.

Long used in the field of medicine, systematic reviews are scientific investigations in themselves. The reviews identify, assess, and synthesize all relevant studies in order to answer a question or set of questions. Unlike traditional reviews, systematic reviews aim to minimize bias in locating, selecting, coding, and aggregating data from individual studies. They allow for more accurate assessments of the effectiveness of an intervention and evidence of knowledge gaps in the research literature. Because the reviews are looking at particular interventions, the studies included in the reviews are generally experimental or quasi-experimental research designs.

Here we summarize the findings of two systematic reviews—one focused on afterschool programs; the other on parent involvement. The reviews may be found online on the Campbell Collaboration Web site. The Campbell Collaboration is an international network of social scientists committed to producing, maintaining, and disseminating systematic reviews of research evidence on the effectiveness of social interventions. Currently there are three Campbell Collaboration coordinating groups: crime and justice, education, and social welfare.

Impacts of Afterschool Programs

In 2006, a systematic review was completed on afterschool programs. “Impacts of After-School Programs on Student Outcomes: A Systematic Review for the Campbell Collaboration” by Susan Goerlich Zief, Sherri Lauver, and Rebecca A. Maynard may be viewed in its entirety at http://www.sfi.dk/graphics/Campbell/reviews/afterschool_review.pdf. What follows is a brief review of their work.

Zief, Lauver, and Maynard’s systematic review was guided by three questions:

1. To what extent and in what ways does access to afterschool programs impact student context (i.e., student location, supervision, and safety), participation in enriching activities, behaviors, social and emotional development, and academic outcomes for youth?
2. Do the effects of after-school programs vary among subgroups of youth defined by their baseline characteristics?
3. Among the program models and settings evaluated, do some seem more beneficial to youth than others? What are the distinguishing characteristics of those more and less successful programs?

The authors included only well-implemented experimental design studies due to strong evidence documenting the unreliability and in some cases bias of recent quasi-experimental studies. They also developed a formal approach to assess the quality of the relevant experimental design studies identified.

Four specific standards needed to be met for a study’s inclusion:

1. No specific evidence of control group contamination
2. Neither overall study attrition or differential attrition would bias the impact estimates
3. Appropriate statistical measures were used for the analyses
4. Primary impact analyses were conducted on all available sample members at followup

After reviewing 88 studies, five were included in the review (see sidebar). The impact estimates presented by these studies were analyzed in two ways, including meta-analysis, although incongruent measures between the studies limited the possibilities for meta-analysis. Ninety-seven different outcomes were measured by the five studies and 79 percent of those were measured by only one study out of the five. All five evaluated programs that operated in urban, school-based environments and served primarily low-income minority students in low-performing schools. Three of the five studies were of programs intended to reduce negative behaviors such as delinquency and drug use. All five included activities to promote positive outcomes such as academic growth and exposure to enrichment activities. The programs also used a similar mix of activities that included academic support and recreational activities.

Systematic Review Findings to the Three Research Questions

The programs each had a different emphasis on promoting positive behaviors and reducing negative ones. However, the authors did not find that any one approach was more or less effective at contributing to “improved behavioral outcomes or other estimated effects.” The authors also noted that the studies reviewed “provide no evidence that any one program model is more effective at changing students’ context or improving academic outcomes.”

The authors found that standardized reading test scores showed that the programs reviewed did not contribute to higher reading achievement for participants, but that afterschool programs may have more of an impact on raising grades, although impacts were small and not statistically significant.

Looking at the 97 outcomes measured by the five studies included in the review revealed that 84 percent showed no significant differences between the program and control youth. The authors explained these null impacts could be a function of limited duration of the intervention or the relatively low participation rates across studies. They also said that the null impacts could suggest that similar afterschool programs “may not be an effective means to achieve the outcomes that afterschool programs in theory hold such promise to impact.”

The authors conclude that this systematic review can be a guide for the conduct of a systematic review and analysis of like programs and evaluations, and a benchmark from which future afterschool program efforts can begin. It also helped identify the knowledge that is lacking and what should be done to gain that knowledge. Their suggestions included rigorously testing alternative models, promoting replication of experimental design studies, and extending the data collection so that longer-term impacts of afterschool programs are studied.

Effects of Parent Involvement on Academic Performance

Chad Nye, Herb Turner, and Jamie Schwartz are authors of the systematic review, “Approaches to Parent Involvement for Improving the Academic Performance of Elementary School Age Children,” available online at http://www.campbellcollaboration.org/lib/download/63/.

Nye, Turner, and Schwartz note that quantitative evidence on the effect of parent involvement on student achievement has been mixed since 1966, when an evaluation of the Head Start Program focused on outcomes related to parent involvement. That evaluation suggested a “substantial relationship” between the parent’s involvement and their child’s academic success. The authors note that discrepancies across parent involvement studies can be explained by the nature of data collection and research design. In some cases, the discrepancies are related to the outcomes measured—such as the effects of parent involvement on reading achievement or math achievement. Another problem centers around how parent involvement is defined; in some studies it is so broadly defined that it is difficult to understand how to consistently measure it.

For the review, the authors defined parent involvement as “the active engagement of a parent with their child outside of the school day in an activity which centers on enhancing academic performance.” They limited their review to randomized controlled trial (RCT) studies with the following intervention characteristics: 1) parent involvement with their child in academic support activities outside of school (for example, reading or completing supplemental math problems with the child), and 2) parent involvement for a minimum of 20 days. The authors included 18 RCTs in their review, all published between 1964 and 2000. About two thirds of the studies were dissertations. The length of the parent intervention programs ranged from 4 to 104 weeks; the most frequently reported method of parent involvement was collaborative reading. All but one study was conducted in public schools—that particular study did not indicate what type of school was involved. Of the studies that reported socioeconomic status (SES) of the parents, 73% were from mixed SES, 9% from middle SES, and 18% were from lower SES. Of those that reported ethnicity, 25% were African American, 25% were Caucasian, and 50% were of mixed ethnicities.

For the meta-analysis, the first question addressed was general: “Does parent involvement result in an improvement in children’s academic performance?” The analysis suggests that parent involvement programs of the kind reviewed have a positive and significant effect on student achievement and that the effect has remained unchanged for three decades.

Then the researchers looked at specific effects of parent involvement. Parent involvement had a positive and significant effect on children’s reading performance. The results for math were also positive and significant but the authors believed the result was biased by one study, which produced an effect size that was three times as large as the effect size for any other study. When they removed that study from the analysis, the impact of parent involvement was still positive, but not statistically significant.

The authors also wanted to determine the parent involvement intervention program that produced the largest effect. Three program approaches were identified:

• Collaborative reading – Parents and children read together as a structured activity.
• Education and training – A specific program designed to provide parents with appropriate teaching or skills-based activities, materials, or information to be used with their child outside the school day.
• Parent rewards and incentives – Parents provide rewards or incentives to their child outside the school day for their child’s performance in school.

Nye, Turner, and Schwartz found that reward and incentives produced the largest impact on children’s academic performance, followed by education and training.

The authors concluded that there is a need to improve the quality of RCTs on parent involvement and a need to replicate high-quality RCTs they reviewed that produced unusually large and significant effects. They also cautioned that although their review indicated positive and significant effects measured immediately following an intervention program, there is little evidence with regard to the sustainability of the effects.

Next Article: Afterschool Programs Make a Difference: Findings From the Harvard Family Research Project