> those studies were some toy experiment design that had nothing to do with actual real-world conditions
Isn't that the nature of understanding and applying science? Science is not engineering: Science discovers new knowledge. Applying that knowledge to the real world is engineering.
Perhaps overcoming that barrier, to some degree, is worthwhile. In a sense, it's a well-known gap.
The question is whether spherical cow research tells you anything that holds up once you introduce the complications of reality into it. In physics, it clearly does. In economics, I think it does in a lot of cases (though with limits). In software engineering... well, like I say, there are areas where I'm sure it does, but research about e.g. strong typing or unit tests or PR review or whatever just doesn't have the juice, IME.
In real-world software development, managing complexity is often (usually) the core of the challenge. A simplified example, is leaving out the very thing that is the obstacle to most good software development. In fact, it is sometimes the case that doing something that helps with managing complexity, will impair performance as measured in some other way. For example, it may slow execution speed by some amount, but allow the software to be broken into smaller pieces each of which is more compehensible. Managing this tradeoff is the key to much software development. If you test with "toy experiment design", you may be throwing out the very thing that is most important to study.
Great point. I think that also emphasizes the necessity of the D in R&D: The research has to be adapted to the real world to be useful, for example to organizational frameworks and processes that manage complexity as you say.
Most software organizations I know don't have anything like the time to do D (to distinguish it from software development), except in a few clear high-ROI cases. Big software companies like Microsoft and Google have research divisions; I wonder how much they devote to D as opposed to R, and how much of that is released publicly.
> those studies were some toy experiment design that had nothing to do with actual real-world conditions
Isn't that the nature of understanding and applying science? Science is not engineering: Science discovers new knowledge. Applying that knowledge to the real world is engineering.
Perhaps overcoming that barrier, to some degree, is worthwhile. In a sense, it's a well-known gap.