The lack of computational reproducibility threatens data science in several domains. In particular, it has been shown that different operating systems can lead to different analysis results. This study identifies and quantifies the effect of the operating system on neuroimaging analysis pipelines. We developed a framework to evaluate the reproducibility of these neuroimaging pipelines across operating systems. The framework themselves leverages software containerization and system-call interception to record
results provenance without having to instrument the pipelines. A tool (Repro-tools) compares results obtained under different conditions. We used our framework to evaluate the effect of the operating system on results produced by pipelines from the Human Connectome Project (HCP), a large open-data initiative to study the human
brain. In particular, we focused on pre-processing pipelines for anatomical and functional data, namely PreFreeSurfer, FreeSurfer, PostFreeSurfer, and fMRIVolume. We used data from five subjects released by the HCP. Results highlight substantial differences
in the output of the HCP pipelines obtained in two versions of Linux (CentOS6 and CentOS7). Inter-OS differences corresponding to normalized root mean square errors of up to 0.27 were observed, which corresponds to visually important differences.
We provide visualizations of the most important differences for various pipeline steps. No meaningful inter-run differences were observed, which shows that the inter-OS differences do not originate from the use of pseudo-random numbers or silent crashes of the pipelines. We hypothesize that the observed inter-OS differences come from numerical instabilities in the pipelines, triggered by rounding and truncation differences that originate in the update of mathematical libraries in different systems.
An apparent solution to this issue is to freeze the execution environment using, for example, software containers. However, this would only mask instabilities while they should ultimately be corrected in the pipelines.