mverse: How the R package is designed to help students explore the multiverse
CANSSI Ontario Statistical Software Conference
Michael Jongho Moon
PhD Student, DoSS, University of Toronto
November 10, 2022
Multiverse analysis
Example
Are darker-skinned soccer players more likely to receive a red card?
Data1
Card counts received by 2,053 soccer players who played in the first male divisions of England, France, and Spain in the 2012-2013 season throughout their careers
Contains skin tone information along with other player information
Which attributes are important to take into account as covariates?
1Silberzahn, R., Uhlmann, E.L., …, & Nosek, B.A. (2018). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. https://doi.org/10.1177/2515245917747646
Position
As-is
Fielder vs. goalkeeper
Fielder groups
Position
Which groupings should we use?
Size
Should we use height, weight, or both to represent size?
League country
Should we take the league country into account?
Reporting results from a single analysis doesn’t reveal the full scholarship involved in the analysis
Multiverse analysis2
involves performing and reporting all analyses across the whole set of reasonable alternatives
There are at least 29 unique analytic approaches with the same data set1
A multiverse analysis would consider all reasonable combinations of alternatives for each analytic decision
2Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016).
Increasing Transparency Through a Multiverse Analysis.
Perspectives on Psychological Science, 11(5), 702–712.
https://doi.org/10.1177/1745691616658637
1Silberzahn, R., Uhlmann, E.L., …, & Nosek, B.A. (2018).
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.
Advances in Methods and Practices in Psychological Science, 1(3), 337-356.
https://doi.org/10.1177/2515245917747646
3Figure by Richie King, Aug 19, 2015, Science Sin’t Broken.
FiveThirtyEight. https://fivethirtyeight.com/features/science-isnt-broken/
Exploring a multiverse is complex and difficult, especially for students with limited technical competency
Design adopted from Simonsohn, U., Simmons, J.P. & Nelson, L.D. Specification curve analysis. (2020). Nature Human Behaviour, 4, 1208–1214. https://doi.org/10.1038/s41562-020-0912-z
Thank you
The package is available on CRAN (install.packages("mverse"))
Experimental features are available on GitHub (remotes::install_github("mverseanalysis/mverse", ref = "test"))
For those more adventurous, see multiverse package by Abhraneel Sarma, Matthew Kay, et al. at https://mucollective.github.io/multiverse/ on which mverse is built
Demo codes for the slides available at https://github.com/mjmoon/presentations/tree/main/CANSSI-Ontario-Stat-SW-Conf-22