This is a report on an empirical investigation into how many education projects funded by the National Science Foundation (NSF) include a randomized controlled trial (RCT) when possible.
One fruitful avenue might be asking NSF people what regulatory barriers there are to carrying out RCTs in schools? Are there any IRB issues / procedures NSF has to follow that make this burdensome? ideas for reforming them, etc.
I agree completely. RCTs are criminally underused within government social programs. Education is a perfect environment for implementing RCTs at scale because classrooms are a relatively controlled environment and the sample sizes are potentially massive.
I would go so far as to say that it should be the most important thing that the federal and state DOE can do. We should shift most DOE spending from subsidies to programs that might work to RCTs that give us definitive proof which policies actually work.
I understand why you find some of the initiatives impractical (occasionally preposterous).
Regarding at least some of the others, is it possible that the benefits are expected to become evident after a longer time period than grant policies allow for measurement? For instance, might a middle-school initiative to spark STEM interest show success later, when students engage in high school activities like science/STEM fairs?
I wonder how much long-term analysis is being conducted on the success of the grant initiatives, specifically tracking the progress and activities of large numbers of individual students over time.
For sure, that makes sense--I'm all for long-term follow-up on educational interventions (if it is reasonably efficient). But the vast majority of the time, these studies don't even have a rigorous way of determining causality in the first place.
What about RCT demand side? All these studies examine school and district level interventions. NSF would be wise to launch a ton at the individual teacher level. It's the ~500k math and science teachers more likely to be intrigued by RCT evidence (compared to school administrators or elementary school teachers).
Reminds me of this gwern post: https://gwern.net/doc/sociology/1987-rossi
One fruitful avenue might be asking NSF people what regulatory barriers there are to carrying out RCTs in schools? Are there any IRB issues / procedures NSF has to follow that make this burdensome? ideas for reforming them, etc.
I agree completely. RCTs are criminally underused within government social programs. Education is a perfect environment for implementing RCTs at scale because classrooms are a relatively controlled environment and the sample sizes are potentially massive.
I would go so far as to say that it should be the most important thing that the federal and state DOE can do. We should shift most DOE spending from subsidies to programs that might work to RCTs that give us definitive proof which policies actually work.
I have more thoughts on the topic here:
https://frompovertytoprogress.substack.com/p/the-case-for-randomized-trials-in
I understand why you find some of the initiatives impractical (occasionally preposterous).
Regarding at least some of the others, is it possible that the benefits are expected to become evident after a longer time period than grant policies allow for measurement? For instance, might a middle-school initiative to spark STEM interest show success later, when students engage in high school activities like science/STEM fairs?
I wonder how much long-term analysis is being conducted on the success of the grant initiatives, specifically tracking the progress and activities of large numbers of individual students over time.
For sure, that makes sense--I'm all for long-term follow-up on educational interventions (if it is reasonably efficient). But the vast majority of the time, these studies don't even have a rigorous way of determining causality in the first place.
Tremendous essay.
What about RCT demand side? All these studies examine school and district level interventions. NSF would be wise to launch a ton at the individual teacher level. It's the ~500k math and science teachers more likely to be intrigued by RCT evidence (compared to school administrators or elementary school teachers).
https://www.educationnext.org/studying-teacher-moves/