Login | Register

A Framework To Evaluate Pipeline Reproducibility Across Operating Systems


A Framework To Evaluate Pipeline Reproducibility Across Operating Systems

Scaria, Lalet ORCID: https://orcid.org/0000-0002-5088-0830 (2018) A Framework To Evaluate Pipeline Reproducibility Across Operating Systems. Masters thesis, Concordia University.

[thumbnail of Scaria_MASc_F2018.pdf]
Text (application/pdf)
Scaria_MASc_F2018.pdf - Accepted Version
Available under License Spectrum Terms of Access.


The lack of computational reproducibility threatens data science in several domains. In particular, it has been shown that different operating systems can lead to different analysis results. This study identifies and quantifies the effect of the operating system on neuroimaging analysis pipelines. We developed a framework to evaluate the reproducibility of these neuroimaging pipelines across operating systems. The framework themselves leverages software containerization and system-call interception to record
results provenance without having to instrument the pipelines. A tool (Repro-tools) compares results obtained under different conditions. We used our framework to evaluate the effect of the operating system on results produced by pipelines from the Human Connectome Project (HCP), a large open-data initiative to study the human
brain. In particular, we focused on pre-processing pipelines for anatomical and functional data, namely PreFreeSurfer, FreeSurfer, PostFreeSurfer, and fMRIVolume. We used data from five subjects released by the HCP. Results highlight substantial differences
in the output of the HCP pipelines obtained in two versions of Linux (CentOS6 and CentOS7). Inter-OS differences corresponding to normalized root mean square errors of up to 0.27 were observed, which corresponds to visually important differences.
We provide visualizations of the most important differences for various pipeline steps. No meaningful inter-run differences were observed, which shows that the inter-OS differences do not originate from the use of pseudo-random numbers or silent crashes of the pipelines. We hypothesize that the observed inter-OS differences come from numerical instabilities in the pipelines, triggered by rounding and truncation differences that originate in the update of mathematical libraries in different systems.
An apparent solution to this issue is to freeze the execution environment using, for example, software containers. However, this would only mask instabilities while they should ultimately be corrected in the pipelines.

Divisions:Concordia University > Gina Cody School of Engineering and Computer Science > Computer Science and Software Engineering
Item Type:Thesis (Masters)
Authors:Scaria, Lalet
Institution:Concordia University
Degree Name:M.A. Sc.
Program:Software Engineering
Date:18 July 2018
Thesis Supervisor(s):Glatard, Tristan
Keywords:Reproducibility, Human Connectome Project, Neuroimaging, Big Data, Neuroinformatics, Neuroimaging Pipelines
ID Code:984061
Deposited By: Lalet Scaria
Deposited On:16 Nov 2018 16:48
Last Modified:16 Nov 2018 16:48
Related URLs:
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Research related to the current document (at the CORE website)
- Research related to the current document (at the CORE website)
Back to top Back to top