The $190B question: Did federal pandemic funds improve test scores?

Students head to school on the first day of class at Maple Elementary School in Fullerton, California, on August 14, 2023.

Students head to school on the first day of class at Maple Elementary School in Fullerton, California, on August 14, 2023. Paul Bersebach/MediaNews Group/Orange County Register via Getty Images

 

Connecting state and local government leaders

Two new studies attempt to answer that question as education activists and politicians debate the impact of federal spending, and as states and school districts consider which pandemic-era programs to continue.

This story was originally published by Chalkbeat. Sign up for their newsletters at ckbe.at/newsletters.

Did federal pandemic relief dollars improve student achievement?

It’s the $190 billion question hanging over American schools as the deadline looms for them to spend the last of the money intended to help them weather pandemic disruptions.

Two new research papers released Wednesday attempt to isolate the effects of federal relief spending on student test scores. Both analyses, which were conducted independently, find that spending under the relief programs known as ESSER improved test scores in reading and math, and that the improvements were in line with other research showing that more spending boosts student achievement.

But the studies could not answer which specific spending decisions — which approach to tutoring or summer school, how many social workers and counselors, what investments in better attendance and student engagement — actually made a difference. That’s because the federal government gave districts significant flexibility and did not require detailed reporting on how the money was spent.

The studies come as education activists and politicians debate the impact of federal spending, and as states and school districts consider which programs to continue. Many students’ test scores remain below pre-pandemic levels, and educational inequality has grown.

“We learned that no-strings-attached dollars do have some positive effect on student achievement,” said Thomas Kane, an economics professor at Harvard University’s Center for Education Policy Research and co-author of one of the studies. “But to have bigger impacts on student achievement, we need to know more about which interventions are working. And we blew an opportunity to learn more.”

Independent Studies Found COVID Relief Improved Achievement

Kane wrote his paper with Sean Reardon, a professor of poverty and inequality in education at Stanford University. The other paper came from Dan Goldhaber, director of the Center for Analysis of Longitudinal Data in Education Research, or Calder, and University of Washington researcher Grace Falken. Both teams of researchers hope to inform short-term debates about continuing pandemic-era programs as well as the larger debate over whether money matters in education.

Congress sent schools $190 billion through three separate aid packages. Money from the largest and final package — the $120 billion American Rescue Plan package — must be spent or committed by September. Schools were required to spend at least 20% of that package on academic recovery, but local leaders had broad discretion about how to define academic recovery and how to spend the rest of the money. District officials generally know how they spent their allotments, but that information hasn’t been collected centrally or in a consistent fashion.

The money was distributed through the same formula as the federal Title I program, which gives extra money to schools serving more students living in poverty, but the total aid package was about 10 times as large as the annual Title I allocation. That magnified quirks of the Title I formula, meaning that districts serving similar populations with similar academic profiles received very different amounts of money, with some receiving thousands of dollars more per student than others.

Both sets of researchers took advantage of these differences to isolate the effects of ESSER spending, though there were some other differences in their methodology.

Kane and Reardon found that each $1,000 in ESSER spending was worth about three-hundredths of a grade level of student progress in math and two-hundredths of a grade level of student progress in literacy. The effects were larger in high-poverty districts that received more money and where students on average suffered more academic setbacks, as well as more economic and family disruptions and more death and serious illness from COVID, the analysis found.

The analysis found that between a third and a half of academic recovery in the highest-poverty districts could be attributed to ESSER. That means federal money helped reduce inequality, Reardon said.

The analysis also found that school districts that had spent a larger share of their ESSER money in the 2022-23 school year showed more academic recovery on tests given that spring than those that received a similar allocation but hadn’t yet spent it. Test results from spring 2024 are not yet available in many states, nor are 2024 national standardized test results.

Previous research from Kane and Reardon found that students in high-poverty districts lost more ground during the pandemic and remain further behind their peers in more affluent districts, despite making greater than average progress in many cases. Students in more affluent districts are ahead academically despite their districts receiving less money, but Kane and Reardon said that doesn’t mean they spent their money more efficiently. Those students didn’t fall as far behind in the first place.

Goldhaber’s analysis found similar effects in math and more progress in literacy than Kane and Reardon’s, though the literacy results were not statistically significant. He found that ESSER dollars had a greater impact on districts that previously had lower spending per student and on districts that served fewer Black and Hispanic students. Goldhaber also found the dollars had a greater impact in towns and rural districts, which often serve more white students than larger cities, which may explain some of the difference. Kane and Reardon did not break down results by race or ethnicity.

Goldhaber found that the impact of ESSER dollars appeared smaller once differences among states were accounted for, something he hasn’t been able to explain yet by looking at differences in state policies or other factors. States were not allowed to tell districts how to spend their money but did have to approve plans and could still have influenced local decisions.

“It might be something as simple as the fact that states have different orientations around these test-based measures of academic achievement, and so the districts in those states invested their money in different ways,” he said.

Goldhaber said it’s hard to say whether the impact of ESSER dollars was large or small.

“I think it is disappointing to have spent so much money and not see greater recovery,” he said. “If there was more focus on academic recovery and some specific measures of what that means, then we probably would have seen larger academic effects.”

At the same time, the impact was greater than some other research on the effects of increased funding, most of which involves smaller funding increases that are spread out over a longer time frame, he said. And Goldhaber noted his study doesn’t measure other benefits like improved mental health or student engagement.

Based on the impact of previous spending, Goldhaber estimated that it would take $9,000 more per student to see a full recovery in reading scores and $13,000 more to see a full recovery in math. Kane and Reardon estimated that it could take as much $904 billion to pay for a full recovery — five times the original relief packages.

In contrast, President Joe Biden’s budget proposal calls for $8 billion to extend tutoring, summer school, and attendance outreach. Republicans in Congress have repeatedly rebuffed efforts to increase Title I spending.

Reardon said future support for schools most likely will have to come from the states and should be more targeted to students and districts with the most needs. It should also be better tracked, not just at the district level, but at the state and federal level so that policymakers can better understand what’s working.

“States’ challenge now is to generate even more growth per dollar spent,” Kane said. “And the way to do that is with targeted funding, rather than across-the-board funding.”

Where the Dollars Made a Difference in Birmingham, Alabama

For Mark Sullivan, superintendent of Birmingham City Schools in Alabama, federal relief dollars meant an opportunity to finally meet his students’ needs and to transform the orientation of his school district, where the large majority of students are Black and live in poverty.

Kane and Reardon’s Education Recovery Scorecard previously had found that Birmingham students made significantly more progress on math than the national average, while state test scores showed elementary students making significant progress on reading.

That outcome hardly seemed inevitable when Sullivan was looking at “horrific” state test scores in spring 2021. The district had toggled between virtual and hybrid settings, eight staff members had died of COVID — including two at the same school within a week of each other — and students learning at home were caring for siblings while their parents worked.

Sullivan originally wanted to move to year-round school using federal relief dollars, but it was clear that no one else did. Instead he started the school year weeks early and built into the calendar four week-long “intersessions” that offered voluntary academic remediation and enrichment. He offered teachers $60 an hour to work the sessions. The district paid for transportation, food, and afterschool care.

For the first session, just 1,800 of the district’s 20,000 students showed up. But by summer, fully half of students were participating. Research commissioned by the district found that students who attended intersessions surpassed classmates who had started the school year ahead but did not attend.

The district also partnered with local universities to have college students do high-dosage tutoring; hired social workers, counselors, and instructional coaches; and invested in iReady assessment programs and trained teachers in how to use the data to adjust instruction.

“Without this funding, we would not have been able to make the progress that we’ve made,” Sullivan said. “People often say you just can’t throw money at the problem. We were able to show that with additional funding and it being used wisely and for it to be used based on the needs of our students, you can see this type of progress.”

With COVID aid expiring, Sullivan said the district is going down to three intersessions, planning to charge again for afterschool programs, and cutting coaches, while looking for grants and community partners to maintain some programs.

“We will not be able to do things at the scale in which we’ve done it over the last three years and provide the kinds of supports that we provided for students,” he said. “And the thing that is most difficult and disheartening is that the needs are still there.”

Chalkbeat is a nonprofit news site covering educational change in public schools.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.