The $190B question: Did federal pandemic funds improve test scores?

This story was originally published by Chalkbeat. Sign up for their newsletters at ckbe.at/newsletters.

Did federal pandemic relief dollars improve student achievement?

It’s the $190 billion question hanging over American schools as the deadline looms for them to spend the last of the money intended to help them weather pandemic disruptions.

Two new research papers released Wednesday attempt to isolate the effects of federal relief spending on student test scores. Both analyses, which were conducted independently, find that spending under the relief programs known as ESSER improved test scores in reading and math, and that the improvements were in line with other research showing that more spending boosts student achievement.

But the studies could not answer which specific spending decisions — which approach to tutoring or summer school, how many social workers and counselors, what investments in better attendance and student engagement — actually made a difference. That’s because the federal government gave districts significant flexibility and did not require detailed reporting on how the money was spent.

The studies come as education activists and politicians debate the impact of federal spending, and as states and school districts consider which programs to continue. Many students’ test scores remain below pre-pandemic levels, and educational inequality has grown.

“We learned that no-strings-attached dollars do have some positive effect on student achievement,” said Thomas Kane, an economics professor at Harvard University’s Center for Education Policy Research and co-author of one of the studies. “But to have bigger impacts on student achievement, we need to know more about which interventions are working. And we blew an opportunity to learn more.”

Independent Studies Found COVID Relief Improved Achievement

Kane wrote his paper with Sean Reardon, a professor of poverty and inequality in education at Stanford University. The other paper came from Dan Goldhaber, director of the Center for Analysis of Longitudinal Data in Education Research, or Calder, and University of Washington researcher Grace Falken. Both teams of researchers hope to inform short-term debates about continuing pandemic-era programs as well as the larger debate over whether money matters in education.

Congress sent schools $190 billion through three separate aid packages. Money from the largest and final package — the $120 billion American Rescue Plan package — must be spent or committed by September. Schools were required to spend at least 20% of that package on academic recovery, but local leaders had broad discretion about how to define academic recovery and how to spend the rest of the money. District officials generally know how they spent their allotments, but that information hasn’t been collected centrally or in a consistent fashion.

The money was distributed through the same formula as the federal Title I program, which gives extra money to schools serving more students living in poverty, but the total aid package was about 10 times as large as the annual Title I allocation. That magnified quirks of the Title I formula, meaning that districts serving similar populations with similar academic profiles received very different amounts of money, with some receiving thousands of dollars more per student than others.

Both sets of researchers took advantage of these differences to isolate the effects of ESSER spending, though there were some other differences in their methodology.

Kane and Reardon found that each $1,000 in ESSER spending was worth about three-hundredths of a grade level of student progress in math and two-hundredths of a grade level of student progress in literacy. The effects were larger in high-poverty districts that received more money and where students on average suffered more academic setbacks, as well as more economic and family disruptions and more death and serious illness from COVID, the analysis found.

The analysis found that between a third and a half of academic recovery in the highest-poverty districts could be attributed to ESSER. That means federal money helped reduce inequality, Reardon said.

The analysis also found that school districts that had spent a larger share of their ESSER money in the 2022-23 school year showed more academic recovery on tests given that spring than those that received a similar allocation but hadn’t yet spent it. Test results from spring 2024 are not yet available in many states, nor are 2024 national standardized test results.

Previous research from Kane and Reardon found that students in high-poverty districts lost more ground during the pandemic and remain further behind their peers in more affluent districts, despite making greater than average progress in many cases. Students in more affluent districts are ahead academically despite their districts receiving less money, but Kane and Reardon said that doesn’t mean they spent their money more efficiently. Those students didn’t fall as far behind in the first place.

Goldhaber’s analysis found similar effects in math and more progress in literacy than Kane and Reardon’s, though the literacy results were not statistically significant. He found that ESSER dollars had a greater impact on districts that previously had lower spending per student and on districts that served fewer Black and Hispanic students. Goldhaber also found the dollars had a greater impact in towns and rural districts, which often serve more white students than larger cities, which may explain some of the difference. Kane and Reardon did not break down results by race or ethnicity.

Goldhaber found that the impact of ESSER dollars appeared smaller once differences among states were accounted for, something he hasn’t been able to explain yet by looking at differences in state policies or other factors. States were not allowed to tell districts how to spend their money but did have to approve plans and could still have influenced local decisions.

“It might be something as simple as the fact that states have different orientations around these test-based measures of academic achievement, and so the districts in those states invested their money in different ways,” he said.

Goldhaber said it’s hard to say whether the impact of ESSER dollars was large or small.

“I think it is disappointing to have spent so much money and not see greater recovery,” he said. “If there was more focus on academic recovery and some specific measures of what that means, then we probably would have seen larger academic effects.”

At the same time, the impact was greater than some other research on the effects of increased funding, most of which involves smaller funding increases that are spread out over a longer time frame, he said. And Goldhaber noted his study doesn’t measure other benefits like improved mental health or student engagement.

Based on the impact of previous spending, Goldhaber estimated that it would take $9,000 more per student to see a full recovery in reading scores and $13,000 more to see a full recovery in math. Kane and Reardon estimated that it could take as much $904 billion to pay for a full recovery — five times the original relief packages.

In contrast, President Joe Biden’s budget proposal calls for $8 billion to extend tutoring, summer school, and attendance outreach. Republicans in Congress have repeatedly rebuffed efforts to increase Title I spending.

Reardon said future support for schools most likely will have to come from the states and should be more targeted to students and districts with the most needs. It should also be better tracked, not just at the district level, but at the state and federal level so that policymakers can better understand what’s working.

“States’ challenge now is to generate even more growth per dollar spent,” Kane said. “And the way to do that is with targeted funding, rather than across-the-board funding.”

Where the Dollars Made a Difference in Birmingham, Alabama

For Mark Sullivan, superintendent of Birmingham City Schools in Alabama, federal relief dollars meant an opportunity to finally meet his students’ needs and to transform the orientation of his school district, where the large majority of students are Black and live in poverty.

Kane and Reardon’s Education Recovery Scorecard previously had found that Birmingham students made significantly more progress on math than the national average, while state test scores showed elementary students making significant progress on reading.

That outcome hardly seemed inevitable when Sullivan was looking at “horrific” state test scores in spring 2021. The district had toggled between virtual and hybrid settings, eight staff members had died of COVID — including two at the same school within a week of each other — and students learning at home were caring for siblings while their parents worked.

Sullivan originally wanted to move to year-round school using federal relief dollars, but it was clear that no one else did. Instead he started the school year weeks early and built into the calendar four week-long “intersessions” that offered voluntary academic remediation and enrichment. He offered teachers $60 an hour to work the sessions. The district paid for transportation, food, and afterschool care.

For the first session, just 1,800 of the district’s 20,000 students showed up. But by summer, fully half of students were participating. Research commissioned by the district found that students who attended intersessions surpassed classmates who had started the school year ahead but did not attend.

The district also partnered with local universities to have college students do high-dosage tutoring; hired social workers, counselors, and instructional coaches; and invested in iReady assessment programs and trained teachers in how to use the data to adjust instruction.

“Without this funding, we would not have been able to make the progress that we’ve made,” Sullivan said. “People often say you just can’t throw money at the problem. We were able to show that with additional funding and it being used wisely and for it to be used based on the needs of our students, you can see this type of progress.”

With COVID aid expiring, Sullivan said the district is going down to three intersessions, planning to charge again for afterschool programs, and cutting coaches, while looking for grants and community partners to maintain some programs.

“We will not be able to do things at the scale in which we’ve done it over the last three years and provide the kinds of supports that we provided for students,” he said. “And the thing that is most difficult and disheartening is that the needs are still there.”

Chalkbeat is a nonprofit news site covering educational change in public schools.