Preliminary Findings From Promising Practices Site Identification for the 21st Century Community Learning Centers

Afterschool programs have long served as venues for providing enrichment, social development, and arts-related activities as well as childcare at the end of the school day and special help to under-served and at-risk populations. More recently, the No Child Left Behind Act has focused on the potential of afterschool programs to supplement and enhance academic learning. The 21st Century Community Learning Centers Program (CCLC) plays a key role in this paradigm shift, working to improve the quality of afterschool programs and enhance student success. CCLC programs provide expanded academic enrichment opportunities for children attending low-performing schools. They also provide tutoring and activities to help students meet local and state academic standards in core subjects like reading and math. Additionally, CCLC programs often include youth development activities; drug and violence prevention programs; technology education; art, music, and recreation courses; counseling; and character education.

Students from Kansas's South Hutchinson Elementary School enjoy music after school as well as academics.

However, CCLC programs vary in their structure and curriculum and the extent to which they focus on academics. Many are centered on enrichment and social development alone. Many go no further than providing tutoring and homework help, often with unqualified staff. Still others have academic content that is neither rigorous nor research-based. In an effort to identify and incorporate promising practices into existing and future afterschool programs, the U.S. Department of Education (USDE) commissioned a large-scale evaluation of the program. After the evaluation, the USDE identified the need for effective approaches to strengthen core content within afterschool programming. Through a competitive solicitation process, the National Partnership for Quality Afterschool Learning—comprising the Southwest Educational Laboratory (SEDL); the National Center for Research on Evaluation, Standards, and Student Testing (CRESST); the Mid-continent Research for Education and Learning (McREL); the Northwest Regional Educational Laboratory (NWREL); the WGBH Educational Foundation; and the SERVE Center at the University of North Carolina at Greensboro—contracted with the USDE to support program quality for the growing number of CCLC programs operating throughout the United States. The 5-year project offers strategies, tools, and technical assistance to address two continuing challenges for afterschool programs: 1) ensuring that programs offer high-quality, research-based academic content using appropriate methods of teaching and learning, and 2) ensuring that programs are able to attract and retain students who participate regularly and thus can benefit fully from the programs and services offered.

To accomplish these two goals, a series of tasks was identified as key to improving the delivery and quality of academic content, teaching, and professional development (see the article on page 3 to find out more about the National Partnership's work). The initial tasks included identifying afterschool sites across the country with promising practices and then validating their afterschool success in reading and math using site visits and data analysis.

This article summarizes the preliminary findings based on these initial tasks. Data included interviews, surveys, and direct observation from 18 programs identified as promising. SEDL staff and partners collected the data in the spring and fall of 2004 using instruments provided by CRESST. CRESST then completed all data analysis and a preliminary report.

Site Selection

A comprehensive process was established to select exceptional CCLC programs that showed evidence of success. Using the 2002 Annual Performance Report, teacher survey results, participant academic performance data, and recommendations from regional partners, an initial pool of more than 1,600 grantees was culled down to 20 grantees (10 math and 10 reading). The process began with a number of prerequisites for consideration: the potential promising practices site must serve 100 or more students, have operated for at least 3 years, and provide math or reading instruction at least three times per week. The next step involved an in-depth analysis of program objectives, grade levels served, number of students served, student demographics, days per week that reading and math content curricula were offered, number of staff in the program, and percentage of credentialed staff. It also included analysis of a composite ranking based on academic performance data, which included grade gains, percentile ranking, and percentage meeting project goals. Programs were selected that exceeded one or more of their goals and that scored highly in the previously mentioned areas. After the initial selection process, non-CCLC afterschool programs that were popularly identified as exemplary were added to the list to ensure more diversity.

The initial selection process from the formal data analysis yielded 15 math and 15 reading sites, in addition to seven non-CCLC programs. Further validation of the selected grantees occurred through a telephone screening process that included a formalized protocol and a request for additional supporting materials. An additional 10 programs (5 math and 5 reading) that did not meet the selection criteria were added to the pool before phone screening took place. None of the randomly selected programs passed the phone-screening process. The validation process presented further credible evidence that the selection process was successful in identifying strong programs, particularly as selected programs scored higher than all the randomly selected programs in both the formal data analysis and the phone screening.

Although it has been recognized for promising practices in the arts, Santo Domingo School's CCLC program devotes time each day to homework help and tutoring.

A third form of validation involved aggregating the 10-question teacher survey data from the annual performance report (APR) and comparing the results for the selected grantees to the general population of grantees. This analysis favored the selected grantees, providing a third source of validation. Finally, the National Partnership leadership team and steering committee reviewed and approved the list with the USDE and contacted the programs to ask for their participation. In the end, 11 reading and 7 math programs agreed to participate. In composite, these validation procedures make us comfortable that the 18 grantees studied in this report are outperforming the average CCLC grantee and are, in fact, among the best in the country working in an afterschool setting today.

Data Analysis Methodology

CRESST used a multi-method approach to data collection and analysis, combining quantitative and qualitative data—including staff and parent surveys; in-depth interviews with program directors, site coordinators, principals, and instructors (lasting approximately 1 hour on average); and direct observation of classroom instruction. The staff survey included questions on content-specific curriculum, general instructional practices and activities, and organizational characteristics. The parent survey focused on perceived program satisfaction, opportunities for involvement, and impact on students. The interview was more comprehensive and covered numerous areas, including grantee background, content-area instructional strategies, other student-based activities, organizational structure and operation, internal and external communication, and interaction, evaluation, professional development, parental involvement, and impact on students. Finally, SEDL staff and partners observed afterschool instructors. Their observations were based on a structured observation protocol that included scales, checklists, and open-ended questions focused primarily on content and quality of instructional practices.

The following three broad themes emerged as central to effective afterschool programming:

• Goals and evaluation–Involves using theory- and research-based practices, implementing effective program structure, setting desired outcomes, and conducting continual evaluation to make sure goals are met
• Curricular quality–Comprises strong academic collaboration between day school and afterschool staff, availing students with opportunities to practice skills in groups and individually, and fostering motivation and engagement
• Program environment–Includes establishing mechanisms for social collaboration and communication, having adequate physical and human resources, using qualified staff, implementing effective professional development, and offering positive attitudes and high expectations for students.

Within these three broad categories, CRESST then designated the following 13 indicators of an effective afterschool program:

• Establishing clear goals for content-area practice
• Assigning research-based activities to achieve goals
• Aligning content materials with state standards
• Developing links between content and school-day activities
• Using research-based curriculum and teaching strategies
• Providing a positive program environment
• Employing motivational strategies to engage students in learning
• Promoting student engagement (e.g., encouraging meaningful experiences)
• Providing effective program management, support, and resources (e.g., staff–student ratio, staff educational experience, ongoing evaluation)
• Providing opportunities for student practice
• Assessing program effectiveness periodically
• Reviewing student progress periodically
• Resetting goals according to assessment results

CRESST analyzed the qualitative data using these indicators. This involved a preliminary factor analysis combined with the earlier literature review to establish five constructs as key. These included links to the school day, research-based practices, periodic assessment, parental involvement, and parental satisfaction. These constructs, together with the indicators above, then informed the rest of the data analysis, from the coding of interviews to the quantitative analysis of the survey results and APR.

Afterschool classes may provide students access to technology they might not otherwise use.

The quantitative analysis involved using grantee-level performance data to approximate individual student achievement, which was unavailable in this case. The proxies for student achievement came from staff and parent surveys collected during the site visits and teacher surveys from the APR. Rather than using individual student data, we used site grade-level data and normalized to the mean among the programs studied, establishing a baseline from which comparison could take place.

Program Overview and Characteristics

The 18 programs studied cover a cross-section of the country and include a variety of program styles and foci. Most programs were in operation for fewer than 10 years when the data were collected, with nearly all sites reporting 3–7 years of experience. Most grantees served ethnically diverse populations comprising primarily lower-income students.

The majority of programs offered three or more activities a day, generally combining academic content with homework help, tutoring, and some form of enrichment or social development. Math and reading instruction generally took place 2–4 times a week, lasting from 30 minutes to 2 hours. Most programs also allotted 30 minutes–1 hour for homework help several times a week. And all had some form of recreation that allowed students to release energy and enjoy themselves.

Common Practices in Program Structure and Process

The 18 grantees had similar patterns of program operation. They all had clear goals and aligned the program structure and content to meet those goals. All established some relationship with their corresponding day schools. Most of the grantees used research-based strategies, and all had staff members who related well to the students and were able to build rapport, maintain high expectations, and keep students motivated and engaged.

Children in the afterschool program at the Mirabel Sisters Campus in New York City perform Little Shop of Horrors.

Most programs made an effort to link the program to state standards. Half of the programs reported building on the regular school curriculum. Many others noted that the curriculum, whether purchased or self-designed, was specifically developed to incorporate standards. Some sites went even further, with staff specifically assigned to develop curriculum and, in some cases, coordinate with day school teachers in creating and modifying it. Homework was also used to connect to the regular day curriculum, and many afterschool staff reported using it as a guide to their own instruction.

In general, the staff members at the 18 grantees appeared to be more highly qualified than the general CCLC staff population. Math program site coordinators had more than 5 years' experience on average, and reading program site coordinators slightly more than 4. Instructors in both reading and math had 3.5 years of experience on average. The grantees provided leadership styles that empowered staff members and helped maintain low turnover rates, with most balancing centralized and decentralized control.

Afterschool programs can nurture a child's natural talents such as in this choral class at the Mirabel Sisters Campus in New York City.

The programs were also systematic in providing professional development and tailoring it to fit both the program and staff needs. Professional development generally centered on classroom management and on reading and mathematics instruction. Some focused on giving staff ideas for enrichment and hands-on activities; knowledge in assessing student progress; and strategies for academic, enrichment, and recreational activities (including ways to incorporate learning into all three). In some cases, afterschool staff had the chance to attend district-wide professional development with day school staff. The majority of instructors reported all of these opportunities as extremely helpful to their professional growth.

Another striking feature of the programs was the strident efforts to maintain open channels of communication, both between staff within the program and informally with day school teachers. Regular meetings and communication took place between afterschool instructors and site coordinators for sharing successes and problems and collaboratively working to improve overall program quality. Most also had informal mechanisms for communication between afterschool and day school teachers, including phone calls, notes, and meetings as needed, and sharing progress reports and report cards.

All of the grantees had some structures in place for evaluation. Teachers at many programs reported monitoring grades, test scores, and interest among students frequently throughout the year, often informally discussing progress with the day school teachers. Broader internal and external evaluation also took place at many of the programs. About one third of the grantees completed a formal external evaluation at least once a year. These evaluations typically involved pre-post testing and observation, sometimes with comparison groups or longitudinally over a number of years. Internal evaluation was also common, with student assessment data often used to inform instructional focus and content, monitor student progress, and measure program impact. Some programs also used student, staff, and teacher surveys to gain other valuable information.

Lastly, the 18 grantees appeared to be successful in keeping students focused and engaged. Parents and staff noted the reluctance of students to leave the program at the end of the day, and observation and anecdotal evidence supports strong student engagement. One aspect that may facilitate this excitement and engagement in the children was the relatively high expectations staff reported for their students. High expectations are particularly important for under-served and at-risk students, who are often the victims of low expectations that hinder their motivation to succeed. Another factor may be the effort by many programs to give the children a voice in decision making and allow them to actively and democratically participate in their education. Additionally, the programs made strong efforts to balance learning with fun activities, relating the activities to the children's personal lives, and using a variety of learning strategies that kept students engaged.

Common Practices in Teaching and Learning

There were a number of common instructional practices among the 18 grantees. All supplemented the core academic curriculum with homework help, tutoring, and some form of enrichment activities. All attempted to make learning more engaging by using nontraditional learning strategies, diversifying activities, and connecting the content to students' personal lives. Most allowed children to get out into the surrounding community through service learning projects, field trips, and other educational experiences away from campus.

Mathematics

Following the general theme of engaging learners, the seven programs that showed promising practices in mathematics attempted to make learning more interesting and fun by offering opportunities to explore mathematics in new and innovative ways and capitalize on intentional learning opportunities. These included games, sports, and cross-content activities related to science, art, and reading. Programs also used visual and tactile tools and computer software on a regular basis to supplement learning. All sites attempted to connect content to real-world experiences and provide some inquiry-based activities. Lastly, all employed scaffolding and multiple grouping strategies to take advantage of the variable knowledge and skills of students. Traditional mathematics learning was part of the curriculum at most sites, but instructors attempted to supplement and enrich the regular school day curriculum to foster interest and excitement about math while offering a more relaxed learning environment after a long day at school. This focus may have also resulted from a recognition that afterschool populations are generally the most under served and at risk, and often lag behind other students under traditional learning regimes.

Reading

Similar themes emerged with reading instruction, where programs offered real-world examples and connected content to children's personal lives. Journal writing was a popular approach, as was relating learning to popular culture. A third approach was addressing multicultural themes. Cross-content learning was also common, with 10 of the 11 grantees connecting reading to math, science, art, or social studies. The vast majority used multi-modal learning, including read-alouds, paired reading, and read-alones, and often combined fiction with nonfiction texts such as newspapers and magazines. All included differentiated instruction and multiple grouping strategies to facilitate learning across reading and language proficiency levels. Many grantees focused on fluency and comprehension, with some attention to vocabulary building. There was more variability in reading than math instruction, with some programs using computer software and others emphasizing cooperative and dialogical learning. Five grantees focused on phonics, and several used games to supplement learning.

Enrichment

All 18 grantees reported offering some form of enrichment or social development in their program. Most incorporated it as a core component with academics and homework help/tutoring; one program made it the primary focus of instruction. Enrichment included arts activities, sports, drug and violence prevention programs, cooperative learning, and social and life skills development. Arts activities helped foster creativity in children by giving them avenues for self-reflection and coming to voice, helped build self-esteem, and allowed them to work in teams. Social development of students was also key with all sites offering some life skills activities including teamwork, self-esteem building, character development, and working together and getting along. Communication and cooperative learning were two other key aspects of almost all the programs, as were attempts to bring the children together and offer them the opportunity to develop interpersonal and public speaking skills. Finally, over half of the grantees specifically mentioned incorporating discussion of tolerance and other cultures into their curriculum and activities.

Afterschool activities such as soccer and basketball keep students fit.

Motivational Support

In addition to positive relationships, motivational support is important in supporting students' academic achievement. Such traits can be fostered in afterschool programs through personal interest in a topic; the desire to please a teacher, peers, parents, and other adults; the knowledge that success has long-term rewards; and a desire to increase a sense of their own capabilities. Researchers noted that a variety of motivational strategies were implemented at the 18 grantees in the study sample.

Enhancing Program Retention Through Student Engagement
Most programs reported recruitment and retention as not being of major concerns, exemplified concretely by waiting lists for participation in many programs. No specific strategies were thus offered for recruitment or retention, although some grantees did mention having a day for student or parent sign-up, sending notices or fliers to parents, getting referrals from school day teachers, or, in one case, introducing a fee to try to increase parent buy-in to the program.

Regarding student engagement, observers found students captivated in learning and activities at the sites they visited. Interview data revealed the common features that most staff seemed to care about the students, had high expectations of them, and worked to establish good relationships with them. Instructors attempted to develop curriculum and activities that were captivating and related to students' daily lives, and parent survey responses indicated high satisfaction with the general functioning of the program. Staff also reported that many students did not want to leave the program at the end of the day.

Several key strategies were used to engage students, including the following:

• Empowering the students: The programs embraced student-centered learning. One third of the grantees specifically mentioned providing opportunities for student input into the program, and most offered opportunities for students to make choices throughout the day.
• Meaningful experiences: Several programs set aside time for open dialogue and for connecting learning to personal experiences and problems. The enrichment aspect of programs generally cultivated the development of the whole student, including social skill growth, preventative intervention, self-esteem building, employment skill training, and fostering creativity and imagination.
• Active learning: Inquiry-based approaches and cooperative learning were also common strategies, and instructors at most of the grantees employed problem-solving and scaffolding. We believe these approaches offer great potential as mechanisms to supplement and enrich regular school day activities and improve the educational and future opportunities of participating students.

Program Impact

Although evidence of impact is a key aspect of any study of program effectiveness, a number of barriers existed in accessing individual student achievement. One limitation was the availability of data, which was not accessible in all cases and tended to vary in content and form across programs and, in some cases, even across sites within the same program. Another problem was the difficulty in controlling for all other variables that might affect school day performance. Finally, there was the issue of selection bias, where students who chose to participate in afterschool education might be more motivated to succeed in school in general.

Given these limitations, a number of findings did demonstrate the effectiveness of these programs. As indicated above, the comprehensive selection process and validation of programs make us comfortable that all 18 programs out-performed the average afterschool program. Overall student performance was measured and improvement demonstrated in all programs. All also exceeded at least one of their primary goals. This was further solidified by the annual performance report data, staff and parent surveys, in-depth interviews, and observation data.

The program demonstrated effectiveness in a number of areas. The first was the ability to attract students and maintain consistent program attendance, supported by daily attendance records and the fact that almost all the programs had waiting lists for entry. The second area of effectiveness was student engagement. Almost all the programs reported that students were happy and enthusiastic, and observation tended to support this claim. Additionally, parental satisfaction was high, as was staff satisfaction with the program. A third area is student achievement. Most of the programs reported improvement in students' academic achievement supported by growth in overall test scores and grades. There was also considerable secondary anecdotal evidence from day school teachers who claimed higher grades, more homework completion, and better overall behavior. Internal and external evaluation at most programs tended to validate these claims.

Promising practices sites often connected reading content to children's personal lives.

Denise Huang is a researcher at the National Center for Research on Evaluations, Standards, and Student Testing (CRESST). Her work involves conducting evaluation studies for LA's BEST, an afterschool program established in Los Angeles to combat obstacles to educational achievement by offering an alternative support system to at-risk students. Dr. Huang would like to acknowledge that a number of CRESST staff members have worked on the National Partnership for Quality Afterschool Learning research and made valuable contributions to this summary and to the research project.

Next Article: The Future of Federal Afterschool Initiatives - Q&A: Robert Stonehill