DAX – The Next Generation: Towards One Million Processes on Commodity Hardware
Stephen M. Damon, Brian D. Boyd, Andrew J. Plassard, Warren Taylor, Bennett A. Landman. “DAX – The Next Generation: Towards One Million Processes on Commodity Hardware” In Proceedings of the SPIE Medical Imaging Conference. Orlando, Florida, February 2017. Oral presentation.
Large-scale image processing demands a standardized way of storage and distribution. The eXtensible Neuroimaging Archive Toolkit (XNAT) is one such platform. Distribution Automation for XNAT (DAX) is a job distribution manager. Recent projects have revealed bottlenecks for projects with 37,000 pipelines. We have developed an API using a direct connection to the database. We have optimized DAX to keep track of processing status on disk, rather than on XNAT. Finally, we have integrated DAX into a Docker. Using our new API, we reduced the time to create assessors 270 fold. DISKQ, using pyXnat, decreased time to launch jobs 200 fold.