Abstract

Ultra-high resolution 3D brain imaging is of great importance to the field of neuroscience as it provides a deep insight into brain anatomy and function. Such images may range between a few 100 gigabytes to terabytes in size and do not typically fit into computer memory. Lack of accessibility to the processing of these images is a threat to open science. This thesis aims to design a web system that will handle the storage and processing of ultra-high resolution neuroimaging data.

The system architecture uses technologies such as Hadoop Distributed File System and Apache Spark. For the seamless integration neuroimaging pipelines into our system, we adopted NIfTI as our distributed data format and require that all neuroimaging pipelines be described in common formats such as Boutiques or BIDS.

The large images are split into chunks, and also, recreated from the chunks. The effects of 2D slices and 3D blocks are investigated. Different algorithms to minimize number of seeks were designed and implemented. Results indicate that clustered reading of blocks achieves a significant reduction in processing time, and partitioning data into slices is most effective.

The scalability of processing large images with Spark using a simple non-containerized and containerized pipeline was investigated. It was found that processing time of both algorithms scale well. As data may need to be written to and read from disk for containerized pipeline processing, the speedup provided by Spark's in-memory computing was also investigated. In-memory computing was found to provide significant speedup, however, this speedup may be less significant in more compute-intensive pipelines.