Learn more about the basics of evaluation, as well as how to find an evaluator for your program.
View Afterschool Alliance resources, including a glossary of terms used in the database, evaluation-related blogs, webinars and more.
Our list of evaluation resources from other organizations, including how to collect and work with data.
Want to find what we know about afterschool programs more broadly, not just individual programs? Head to our Afterschool Research page!
Year Published: 2020
A longitudinal study of voluntary summer learning programs, led by five school districts located in urban communities across the country, followed students from 3rd to 7th grade and found positive academic gains among the randomly assigned students to the program compared to the control group of their nonparticipating peers. After the first and second summer of program participation, program attendees outperformed control-group students. Statistically significant differences were found in math achievement after the first summer, and math, language arts, and social and emotional skills after the second summer. Researchers followed up on program participants three years after the second summer of program participation and found that while academic gains compared to average gains made in a year were no longer statistically significant, they were still large enough to be meaningful.
Program Description:
The National Summer Learning Project (NSLP) was a study conducted to examine the ways that voluntary summer learning programs can benefit youth. It was launched in five school districts—Boston, Massachusetts; Dallas, Texas; Duval County, Florida; Pittsburgh, Pennsylvania; and Rochester, New York. All programs, which were free of charge, implemented full day programming that included both academic instruction and enrichment activities, at least three hours of language arts and math per day, small class sizes, and free transportation and meals.
Scope of the Evaluation: National
Program Type: Summer
Location: Boston, Massachusetts; Dallas, Texas; Duval County, Florida; Pittsburgh, Pennsylvania; and Rochester, New York
Grade level: Elementary School, Middle School
Program Demographics:
Across all school districts involved, 89 percent of participants qualified for the free and reduced price lunch (FRPL) program, 31 percent were English language learners (ELL), and 10 percent were on an Individualized Education Plan (IEP). Regarding race, 47 percent identified as African American, 40 percent as Hispanic, 7 percent as White, and 3 percent as Asian.
Evaluator: McCombs, J. S., Augustine, C. H., Pane, J. F., & Schweig, J. RAND Corporation.
Evaluation Methods:
This study is the third phase of a research project commissioned by the Wallace Foundation to examine implementation and outcomes associated with the National Summer Learning Project (NSLP). Summer program applicants were randomly assigned into two groups: a treatment group that participated in two consecutive summers of voluntary summer programming (2013 and 2014) and a control group that did not participate. At the start of the study during the spring 2013, 5,639 third grade students were randomly selected to participate, with more than 3,000 of whom assigned to the treatment group. Student outcomes were measured through spring 2015 for phase II, and the current follow up (phase III) looks at student outcomes in the spring 2017, three school years after the second summer of programming. Data—such as language arts and math state assessment results, course grades, social and emotional assessments, and suspension and attendance—were collected through classroom observations, teacher surveys, teacher and administrator interviews, and administrative attendance records.
Evaluation Type: Experimental
Summary of Outcomes:
A longitudinal study of summer learning programs that followed students from 3rd through 7th grade found positive academic gains among the students randomly assigned to the summer programs compared to the control group comprised of nonparticipants. After the first summer of programming, program participants outperformed control group students on their fall math assessments. Among the students who were “high attenders” in the summer program (attending for 20 days or more), the gains students experienced accounted for approximately 25 percent of an average annual gain for fall assessments, and 13 percent of the annual gain in the spring. However, program participation was not found to have a significant effect on language arts assessments, or social-emotional skills and school-year attendance after the first year.
After the second year of summer programming, high attenders outperformed the control group in math, language arts, and social-emotional skills. These estimated gains were between 14 percent to 21 percent of typical annual gains in math, and 17 percent to 25 percent of the typical annual gains in language arts. High attenders also scored higher on the teacher administered Devereux Student Strengths Assessment-RAND Research Edition (DESSA-RRE), demonstrating stronger social-emotional skills than the control group when they returned to school in fall 2014.
Consecutive attendance was also found to have a positive impact on students’ academic performance, with those who participated at least 20 days both summers having greater gains in math and language arts assessments compared to the control group of students. The study also found that greater amount of time on task (defined as students who received a minimum of 25 hours of summer math instruction and 34 hours of language arts instruction) during both summers were positively correlated with better academic outcomes in math state assessments in the fall of 2014 and spring of 2015, and on language arts assessments in the spring of 2015.
In the follow up study three years after the students participated in the two consecutive summers of programming, it was found that the academic benefits were no longer statistically significant, however, still large enough to be important when benchmarked against typical grade level achievement gains. High attenders who participated in both summers of programming saw 19 percent of typical annual growth in language arts and 23 percent in math.
Associated Evaluation:
https://www.wallacefoundation.org/knowledge-center/pages/every-summer-counts-a-longitudinal-analysis-of-outcomes-from-the-national-summer-learning-project.aspx
Date Added: August 23, 2021