Debunking Software Engineering Myths: Does the organization matter more than the programming?

I’ve been wanting to post a link to this article for a while, but ever since I discovered it, research.microsoft.com has been unreachable for me, so I’ll post a small summary:

Microsoft has done research on some popular conceptions about software engineering and come up with hard numbers on some factors affecting code quality.  Here are the main findings reported in the artice, with links to the research papers, in case the original is lost forever:

• More test coverage does not equal better code quality, as measured by number of post-release fixes.  Usage patterns and code complexity are the main reason test coverage is a poor predictor of quality.
• Test Driven Development increases quality, creating code that was up to 40% to 90% better in the Microsoft research. The cost is increased development time, 15% to 35% in the same studies. (Realizing quality improvement through test driven development: results and experiences of four industrial teams)
• Use of Assertions decrease program faults. However, this is also correlated with programmer experience, meaning more experienced programmers generally write more assertions and have less program faults. (Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation). And here is more about Assertions.
• Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. Not only that, but organizational structure was by far the best predictor of code quality, and was at least 8% percent better than the best predictor the researchers could get from code-based measurements. (The influence of organizational structure on software quality: an empirical case study)
• Organizational closeness is more important than geographical closeness, which has no effect on quality (Does Distributed Development Affect Software Quality? An Empirical Case Study of Windows Vista)

Here’s Microsoft’s article: Exploding Software-Engineering Myths (or if that doesn’t work, a Google Cache link).

One drawback with this research is that this is primarily based on case studies, which is a generally poor research method for drawing general conclusions. How valid are these observations for other organizations outside Microsoft? Is the organizational structure of your project or company actually more decisive than your programming methodology?

Also, how transferable is this to other programming frameworks. In dynamic typed languages like Perl, is test coverage more important? I often find that a sub-set of my tests do what a compiler could have done in a statically typed language, or even for Perl if I just had a more automatic testing tool. So maybe coverage would be more predictive of bugs if the compiler catches fewer mistakes? That would be a good candidate for further research.

However, this is excellent as a step towards more evidence-based software engineering.

Found through John Tells All.