How to Think About Validity in Effect Studies: A Case Study of the Effect-evidence Ranking Scheme of the National Board of Social Services

Joachim Skanderby Johansen

Student thesis: Master thesis


This thesis evaluates the effect-evidence ranking scheme used by The National Board of Social Services (NBSS) seen in light of discussions in the philosophy of science on internal and external validity. I start by introducing the effect-evidence scheme and its context. I then argue that we can best understand the effect-evidence ranking as an expression of how certain we are, based on conducted studies, that a social method will be effective when we employ it. Next, I analyse two central types of studies that the effect-evidence scheme ranks – variance (e.g. cohort studies, RCTs etc.) and process type studies - in regards to the possible threats to their internal and external validity. In terms of internal validity, I argue that confounders pose a threat for both variance and process studies, and discuss when which type will do better. In terms of external validity, I argue that both types of studies face a threat from lack of support factors in the target we want to extrapolate the result of the study to, although we often have better knowledge of support factors in process studies. I also analyse the external validity threat posed by different mechanisms being present in the population that we have studied and in the target we want to extrapolate to. Here, I will discuss strategies to mitigate this threat and why we should differ between processes and mechanisms. Lastly, I consider the threat to both internal and external validity of multiple causal paths, arguing that this is a relevant concern for both variance and process studies. In the conclusion, I argue that the NBSS effect-evidence ranking scheme does not provide a good guide for effect-evidence ranking and I outline an alternative scheme which ranks studies through probability assessments of three parameters; confounders, support factors and causal paths. Keywords: Internal validity, External Validity, Evidence, Mechanisms, Processes.

EducationsMsc in Business Administration and Philosophy, (Graduate Programme) Final Thesis
Publication date2016
Number of pages82
SupervisorsMorten Sørensen Thaning