Applying Retrospective Cohort Study Methodology in Mining Software Repositories Studies

Randomized controlled experiments are the most widely accepted design for studying causal relationships. However, they may not always be feasible due to ethical concerns or their limited capability of utilizing historical data. The latter is especially problematic for Mining Software Repositories (MSR) studies as they are often retrospective and observational in nature. Despite this, MSR currently lacks alternative study methodologies capable of providing a high level of evidence when controlled experiments are unattainable.

The goal of this thesis is to identify and adapt a study methodology capable of assessing causality utilizing existing observational data and is applicable for MSR studies. We identified suitable methodologies from epidemiology called analytical observational studies. Therefore, this thesis aims starting the process of adapting the analytical observational study types, specifically retrospective cohort studies, to MSR. For achieving the goal, we identified relevant characteristics and issues of MSR research, studied the analytical observational methodologies, and finally adapted and applied the retrospective cohort study method in practice.

MSR studies are characterized by having large data sets of historical observational time series data. However, the data is not fully utilized and the data processing raised concerns especially when pooling data from different projects. These findings support the need of analytical observational studies which provide solutions for the identified issues. We successfully created a preliminary process for conducting retrospective cohort studies in MSR and applied it in practise. Even we did identify aspects that still need clarification, the methodology showed potential for MSR research.

In conclusion, the analytical observational study types seem promising for both MSR and empirical software engineering. Especially the retrospective cohort study type is applicable to MSR studies. However, there is a need of process for conducting them properly in the MSR context. Further work is needed on several aspects of running analytical observational studies such as external variables.