This was going to be a linkfest item, but I decided to turn it into a mini-post.
Here’s Jessica Hullman on whether “rigor-enhancing practices” (e.g., preregistration, large sample sizes, open data) are a distraction–or at least, an ineffective gateway–to thinking hard about tougher problems like “what are you even measuring, really?” Includes a link to Devezer et al. 2021, which claims that most arguments for improving methodological rigor in the sciences aren’t themselves all that rigorous.
I have mixed feelings on this.
On the one hand, I’m sympathetic to the argument that lack of statistical or methodological rigor isn’t all that big a deal in the grand scheme of things. I agree that the most widespread and important problems in scientific research are hard to fix, precisely because they’re not amenable to narrowly-focused technical or procedural fixes. There’s no checklist you can follow to do good science.
On the other hand, I’m reflexively suspicious of arguments against incremental, doable reforms, on the grounds that incremental, doable reforms are a distraction from attacking more important and challenging problems. I’m reflexively suspicious for two reasons. One, these sorts of arguments make the perfect the enemy of the good. Two, I think it’s generally just not true that effort being put into narrow, doable reform X can be redirected towards solving big, intractable problem Y. Those two things aren’t interconvertible substitutes, I don’t think. At least, not to most people. I’m reflexively suspicious when someone assumes that time, effort, or money being put towards X could be put towards Y instead. You need to establish that X and Y are in fact substitutes in most people’s eyes. Or else show that somebody has the power to incentivize or force people to substitute Y for X.