Abstract

Deep learning techniques have been shown to produce state-of-the-art performance in segmenting biomedical images. These techniques, however, are highly dependent on the quantity and quality of available training data, as well as their capacity to represent complex relationships, which is dependent on the size of the network and limited by the computational resources of the machine(s) on which they are trained. In this work, we performed two experiments. First, we explored multi-stream model configurations that can leverage available data from multiple unpaired biomedical imaging modalities to learn a shared representation. Specifically, segmentation of cardiac CT and MRI was done to see if this learns the shared anatomical features and thus performs better than individual models trained on each modality. Second, we compared our full deep learning segmentation pipeline as applied to paired multi-modal brain images against other existing publicly available pipelines. From these experiments, we found that (1) multi-stream architectures can achieve better results in unpaired multi-modal segmentation compared to single-stream models, however, the specific configuration with the best performance is in disagreement with previously published results; and (2) careful adjustment of deep learning pipeline configurations to our specific data set and hardware constraints yields improved segmentation accuracy over publicly available state-of-the-art solutions in paired multi-modal image segmentation.