Measuring the world of social phenomena
Economists working with Professor Marko Sarstedt from University of Magdeburg are demanding that the same scientific standards be applied to economics and the behavioral sciences in general as are used in the natural sciences. They argue that the inherent uncertainties in measurement must be identified and quantified in order to improve the reproducibility of research. Only in this way can the sources of errors be identified and eliminated.
In a recently published study, the economists shed light on the question why many scientific results produced in the behavioral sciences is are not reproducible - or only to a limited extent. They argue that established checklists that are intended to make important aspects of the studies transparent are not enough. According to the economist and co-author, Professor Marko Sarstedt, in physics no study would pass muster without stating the accuracy of the measurement. “Although providing corresponding calibration data for the measurement of social phenomena is difficult to conceive of, it is not impossible.” In the study the economists advocate transferring concepts from the natural sciences to the behavioral sciences and show, using a sample calculation, the major consequences that arise from measurement uncertainty for a seemingly statistically significant result.
Researchers in the behavioral sciences need to incorporate the impact of measurement uncertainty directly and fully into their results. “Physicists, engineers, physicians, and biologists do this by stating the measurement accuracy of their instruments, for example of a thermometer or scale,” explains Sarstedt. “Behavioral scientists such as economists and psychologists must do the same when measuring unobservable concepts such as satisfaction and happiness - even if it is difficult.”
Modeled on physics
Whilst physics generally deals with precisely defined and measurable quantities such as lengths, masses, and temperatures, concepts such as satisfaction and happiness are not clearly defined and are measured and interpreted differently depending on the team of researchers. But, according to Sarstedt, despite the disparity between the subjects being studied, reproducibility is an essential element of science. Only through the repeated confirmation of research results can they be accepted as fact.
“Protocols in physics or biology specify clear guidelines, which need to be adjusted to match the fuzzy nature of the concepts being studied in our discipline,” explains the economist. “As in physics, we need institutions that establish standards for measurements in the behavioral sciences and to quantify uncertainties in standard measurements.”
The background to the study is that, according to the authors, many areas of social science research find themselves in a replication crisis. In recent years, different research teams were not able to replicate fundamental effects in psychology, management, and marketing research in repeated studies under almost identical conditions. Since then, various groups of researchers around the Center for Open Science have made efforts to define standards for empirical studies in order to enhance their reproducibility. From this a variety of checklists emerged, that are intended to assist researchers in documenting important aspects of their studies and making them transparent.
The study by the authors, Edward E. Rigdon (Georgia State University), Marko Sarstedt (University of Magdeburg) and Jan-Michael Becker (University of Cologne) recently appeared in the renowned journal, Nature Human Behaviour under the title “Quantify uncertainty in behavioral research”.