It’s also important to ask whether the signal can be manipulated. This seems moderately robust, but any signal that gets used for allocating funding will then be a target for manipulation, so you don’t want to just encourage people to generate the signal that you’re following.
Always necessary to think about gaming. We have thought very carefully about how this approach might be gamed, and have, to the extent possible, made design choices to mitigate against it. Most importantly, because of the underlying math, it's quite difficult to cast enough "votes" for yourself to trigger the signal. No matter how well designed the measure, though, it's important to keep humans in the loop for orthogonal validation.
This is important work, and I’ll say up front I’m on the side of the researchers here — I write as a research admin and as a mother who wants a better world for my kids and, so, for all kids. The Karikó pattern is real and infuriating, and getting good science backed before it’s five years too late is worth chasing. So I’m not arguing against the model. I’m curious what it optimizes for.
My concern is the signal's thinness. A model tuned to citation bursts rewards fields that already have people with the resources to flock to them — WEIRD, well-funded, mostly elite institutions generate that flurry; the ecosystems I volunteer in can’t, even when they’re solving everyday problems with huge returns for who actually gets to use science. So “merit” risks becoming a faster version of patronage. (That’s a throughline found in recent work by Musa al-Gharbi, Raj Chetty, and lately Ada Palmer’s Inventing the Renaissance — and the legitimate/illegitimate line is an old, engineered one, going back to the 12th-century Church deciding who could inherit, but found in examples throughout today and history touching on language, ethnicity, linguistics, religion, culture, gender, etc.)
A non-WEIRD scientist I worked with called this the “Marvel Effect”: elites hoard the resources, inequality compounds the problems, and then someone swoops in with an Iron Man fix and expects the hoarding that produced it to be forgiven because they’ve arrived to save the day — from a problem the system that funded them arguably helped create to their (albeit not intentional) benefit.
There’s also a distribution question your piece raised for me. You’d run the experiment through active program managers with authority to recruit — the incumbent apparatus. Al-Gharbi estimates that of every dollar aimed at redistribution, roughly 75 cents is absorbed by the institutions doing the redistributing before it reaches anyone. I’d genuinely want to know how a breakthrough fund avoids the same skim.
And this is where your call for “a framework that doesn’t repackage prestige” might go further. Economic complexity is the closest thing we have — the renowned César Hidalgo’s ECI outpredicts schooling and capital metrics because it measures latent, path-dependent capability, which is really your seed-corn point made rigorous. But even ECI has no column for care: it rewards what’s scarce and recombinable, so the abundant, unrecombinable work that keeps an ecosystem alive scores near zero. A citation model is thinner still. Researcher Angus Fletcher’s primal intelligence names what these metrics miss: the intuition, imagination, emotion, and common sense that actually move a field, which live in the very people a signal strips away.
So I’d love your read: is there a way to fund, as Fletcher puts it, the probable while protecting the possible — and to build a signal that can see the people the metric tends to erase? That, to me, is the difference between making our own luck and deciding what luck means for everyone else.
It’s also important to ask whether the signal can be manipulated. This seems moderately robust, but any signal that gets used for allocating funding will then be a target for manipulation, so you don’t want to just encourage people to generate the signal that you’re following.
Always necessary to think about gaming. We have thought very carefully about how this approach might be gamed, and have, to the extent possible, made design choices to mitigate against it. Most importantly, because of the underlying math, it's quite difficult to cast enough "votes" for yourself to trigger the signal. No matter how well designed the measure, though, it's important to keep humans in the loop for orthogonal validation.
This is important work, and I’ll say up front I’m on the side of the researchers here — I write as a research admin and as a mother who wants a better world for my kids and, so, for all kids. The Karikó pattern is real and infuriating, and getting good science backed before it’s five years too late is worth chasing. So I’m not arguing against the model. I’m curious what it optimizes for.
My concern is the signal's thinness. A model tuned to citation bursts rewards fields that already have people with the resources to flock to them — WEIRD, well-funded, mostly elite institutions generate that flurry; the ecosystems I volunteer in can’t, even when they’re solving everyday problems with huge returns for who actually gets to use science. So “merit” risks becoming a faster version of patronage. (That’s a throughline found in recent work by Musa al-Gharbi, Raj Chetty, and lately Ada Palmer’s Inventing the Renaissance — and the legitimate/illegitimate line is an old, engineered one, going back to the 12th-century Church deciding who could inherit, but found in examples throughout today and history touching on language, ethnicity, linguistics, religion, culture, gender, etc.)
A non-WEIRD scientist I worked with called this the “Marvel Effect”: elites hoard the resources, inequality compounds the problems, and then someone swoops in with an Iron Man fix and expects the hoarding that produced it to be forgiven because they’ve arrived to save the day — from a problem the system that funded them arguably helped create to their (albeit not intentional) benefit.
There’s also a distribution question your piece raised for me. You’d run the experiment through active program managers with authority to recruit — the incumbent apparatus. Al-Gharbi estimates that of every dollar aimed at redistribution, roughly 75 cents is absorbed by the institutions doing the redistributing before it reaches anyone. I’d genuinely want to know how a breakthrough fund avoids the same skim.
And this is where your call for “a framework that doesn’t repackage prestige” might go further. Economic complexity is the closest thing we have — the renowned César Hidalgo’s ECI outpredicts schooling and capital metrics because it measures latent, path-dependent capability, which is really your seed-corn point made rigorous. But even ECI has no column for care: it rewards what’s scarce and recombinable, so the abundant, unrecombinable work that keeps an ecosystem alive scores near zero. A citation model is thinner still. Researcher Angus Fletcher’s primal intelligence names what these metrics miss: the intuition, imagination, emotion, and common sense that actually move a field, which live in the very people a signal strips away.
So I’d love your read: is there a way to fund, as Fletcher puts it, the probable while protecting the possible — and to build a signal that can see the people the metric tends to erase? That, to me, is the difference between making our own luck and deciding what luck means for everyone else.