A project of the Afterschool Alliance.

The Dynamic Effects of a Summer Learning Program on Behavioral Engagement in School

Year Published: 2020

This quasi-experimental study examines students who attended Aim High, a voluntary summer learning program that provided academic and social and emotional learning (SEL) supports, during 2013-2014 and/or 2014-2015 to middle school students in the San Francisco Unified School District. The study analyzes how students’ participation in the program impacts both behavioral engagement and academic achievement. Evaluators found that program participants were significantly less likely to be chronically absent and suspended, as well as have slight improvements in English/language arts state assessments and school-day attendance than their peers who did not participate in Aim High. Additionally, this study found that these effects are greatest for Aim High participants who are boys and Latinx students.

Program Name: Aim High

Program Description:

Aim High is a voluntary summer learning program offered to middle school students (from grades 6-8) in the San Francisco Unified School District (SFUSD) and surrounding communities for seven hours a day for five weeks. The program was created specifically for students from low-income neighborhoods. The curriculum consists of core subjects like math, science, and English as well as a social and emotional learning (SEL) component called “Issues and Choices”. The SEL class includes a variety of topics like mindfulness, community building, creating a growth mindset, stopping bullying, and more.

Scope of the Evaluation: Local

Program Type: Summer

Location: San Francisco Unified School District

Community Type: Urban

Grade level: Middle School

Program Demographics:

Findings in the evaluation are based on Aim High program participants who were 92 percent children of color, 22 percent English language learners, 9 percent designated as special education students, and 49 percent girls. Among Aim High participants overall, 71 percent qualify for free or reduced-price lunch. 

Program Website: https://aimhigh.org/

Evaluator: Pyne, J., Messner, E, & Dee, T. S. Stanford Center for Education Policy Analysis.

Evaluation Methods:

Evaluators used administrative data from SFUSD records and Aim High program sites to identify “intent-to-treat” (ITT) samples: two cohorts of fifth graders who have just finished the school year (the 2013-14 and 2014-15 school years) and were in their first eligible summer to participate in Aim High. Then, evaluators were able to collect data on these two cohorts, tracking them back from the first grade, as well as through eighth grade. Additionally, this study identified another ITT sample of students who took the California Assessment of Student Performance and Progress (CAASPP) English language arts (ELA) and math assessments and were able to collect student performance on these tests from fourth through seventh grade. Evaluators were then able to determine how many students in each of these ITT samples attended Aim High programming. Lastly, evaluators used additional SFUSD data to calculate and compare outcome measures for Aim High participants—including school-day attendance, chronic absenteeism, suspension rates, and academic achievement in ELA and math—to that of their non-participating peers. 

Evaluation Type: Quasi-experimental

Summary of Outcomes:

Comparing SFUSD students who did and did not participate in the Aim High program through their cohort samples, evaluators found that participation in the Aim High program contributed to improvements in students’ behavior. Evaluators also found slight academic gains for Aim High participants in English language arts state standardized test scores.

The study examined Aim High’s impact on students’ absence rates by looking at overall absence rate, excused absence rate, and unexcused absence rates. Regarding effect size, evaluators equated the lower unexcused absence rate after three years to almost an additional two days of attendance during a school year for middle schoolers. Aim High participants were also less likely to be chronically absent than their non-participating peers (defined in this study as missing at least 10 percent of school days during the school year), especially when they took part in the program for more than one summer. For example, among students who participated in the Aim High program for three years, chronic absenteeism went down by 58 percent. Students who were not a part of Aim High were labeled “chronically absent” at about 2.3 times the rate of participants.

Aim High participants were also significantly less likely to be suspended than their peers who did not participate in the program, with results suggesting the longer a student was in the program, the less likely they were to be suspended. The study found that students who participated in Aim High for at least two summers, or who were in their second or third year after participating, saw a 37 percent reduction in the likelihood of being suspended.

In terms of academic outcomes, evaluators obtained one cohort’s California Assessment of Student Performance and Progress (CAASPP) English language arts (ELA) and math assessments from fourth through seventh grade. Participants had a slight increase in ELA test scores in the year or years after they took part in programming, but saw no significant increases in math test scores.

Evaluators found that outcomes are stronger the more a student participates in Aim High, and were most prevalent among boys  and Latinx students. For instance, the probability of chronic absenteeism decreased by 4.6 percentage points among Latinx students and by 2.3 percentage points for boys, compared to 1.4 percentage points for Aim High students overall. Evaluators conclude that the Aim High program’s combination of academic and SEL curriculum, “…can generate meaningful improvements in important measures of behavioral engagement longer-run success.”