\n<\/strong><\/p>\n
Abstract<\/h2>\n
Traditional in-house, laboratory-based medical imaging studies use\u00a0hierarchical data structures (e.g., NFS file stores) or databases\u00a0(e.g., COINS, XNAT) for storage and retrieval. The resulting\u00a0performance from these approaches is, however, impeded by standard\u00a0network switches since they can saturate network bandwidth during\u00a0transfer from storage to processing nodes for even moderate-sized\u00a0studies. To that end, a cloud-based “medical image\u00a0processing-as-a-service” offers promise in utilizing the ecosystem of\u00a0Apache Hadoop, which is a flexible framework providing distributed,\u00a0scalable, fault tolerant storage and parallel computational modules,\u00a0and HBase, which is a NoSQL database built atop Hadoop’s distributed\u00a0file system. Despite this promise, HBase’s load distribution strategy\u00a0of region split and merge is detrimental to the hierarchical\u00a0organization of imaging data (e.g., project, subject, session, scan,\u00a0slice).<\/p>\n
This paper makes two contributions to address these concerns by\u00a0describing key cloud engineering principles and technology\u00a0enhancements we made to the Apache Hadoop ecosystem for medical\u00a0imaging applications. First, we propose a row-key design for\u00a0HBase, which is a necessary step that is driven by the hierarchical\u00a0organization of imaging data. Second, we propose a novel data\u00a0allocation policy within HBase to strongly enforce collocation of\u00a0hierarchically related imaging data. The proposed enhancements\u00a0accelerate data processing by minimizing network usage and localizing\u00a0processing to machines where the data already exist. Moreover, our\u00a0approach is amenable to the traditional scan, subject, and\u00a0project-level analysis procedures, and is compatible with standard\u00a0command line\/scriptable image processing software. Experimental\u00a0results for an illustrative sample of imaging data reveals that our\u00a0new HBase policy results in a three-fold time improvement in\u00a0conversion of classic DICOM to NiFTI file formats when compared with\u00a0the default HBase region split policy, and nearly a nine-fold\u00a0improvement over a commonly available network file system (NFS)\u00a0approach even for relatively small file sets. Moreover, file access\u00a0latency is lower than network attached storage.<\/p>\n Shunxing Bao, Andrew Plassard, Bennett Landman and Aniruddha Gokhale. “Cloud Engineering Principles and Technology Enablers for Medical Image Processing-as-a-Service.”\u00a0 IEEE International Conference on Cloud Engineering (IC2E), Vancouver, Canada, April 2017. Full text: NIHMSID Abstract Traditional in-house, laboratory-based medical imaging studies use\u00a0hierarchical data structures (e.g., NFS file stores) or databases\u00a0(e.g., COINS, XNAT) for storage and retrieval….<\/p>\n","protected":false},"author":1920,"featured_media":960,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,69,8],"tags":[126,127],"class_list":["post-957","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data","category-cloud-computing","category-informatics-big-data","tag-hadoop","tag-hdfs"],"_links":{"self":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/957","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/users\/1920"}],"replies":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/comments?post=957"}],"version-history":[{"count":6,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/957\/revisions"}],"predecessor-version":[{"id":967,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/957\/revisions\/967"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/media\/960"}],"wp:attachment":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/media?parent=957"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/categories?post=957"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/tags?post=957"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Throughput](\"https:\/\/my.vanderbilt.edu\/masi\/wp-content\/uploads\/sites\/2304\n2661\/2016\/11\/Screen-Shot-2016-11-15-at-8.50.24-PM.png\")
number of datasets processed per minute by each of the scenarios as a function
of the number of datasets selected for processing. (B) shows the fraction of
time spent on overhead relative to the number of datasets.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"