DAX – The Next Generation: Towards One Million Processes on Commodity Hardware

Posted by Prasanna Parvathaneni on Tuesday, November 1, 2016 in Cloud Computing.

Stephen M. Damon, Brian D. Boyd, Andrew J. Plassard, Warren Taylor, Bennett A. Landman. “DAX – The Next Generation: Towards One Million Processes on Commodity Hardware” In Proceedings of the SPIE Medical Imaging Conference. Orlando, Florida, February 2017. Oral presentation.

Full Text: 

Abstract

Large-scale image processing demands a standardized way of storage and distribution. The eXtensible Neuroimaging Archive Toolkit (XNAT) is one such platform. Distribution Automation for XNAT (DAX) is a job distribution manager. Recent projects have revealed bottlenecks for projects with 37,000 pipelines. We have developed an API using a direct connection to the database. We have optimized DAX to keep track of processing status on disk, rather than on XNAT. Finally, we have integrated DAX into a Docker. Using our new API, we reduced the time to create assessors 270 fold. DISKQ, using pyXnat, decreased time to launch jobs 200 fold.