{"id":3059,"date":"2022-07-25T23:02:03","date_gmt":"2022-07-26T04:02:03","guid":{"rendered":"https:\/\/my.vanderbilt.edu\/masi\/?p=3059"},"modified":"2022-08-08T12:11:04","modified_gmt":"2022-08-08T17:11:04","slug":"self-supervised-pre-training-of-swin-transformers-for-3d-medical-image-analysis","status":"publish","type":"post","link":"https:\/\/my.vanderbilt.edu\/masi\/2022\/07\/self-supervised-pre-training-of-swin-transformers-for-3d-medical-image-analysis\/","title":{"rendered":"Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis"},"content":{"rendered":"
Tang, Yucheng, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. “Self-supervised pre-training of swin transformers for 3d medical image analysis.” In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition<\/i>, pp. 20730-20740. 2022.<\/div>\n
<\/div>\n
Full text:\u00a0<\/strong><\/div>\n
<\/div>\n
\n

Abstract<\/h2>\n

Vision Transformers (ViT)s have shown great performance <\/span>in self-supervised learning of global and local representations <\/span>that can be transferred to downstream applications. Inspired <\/span>by these results, we introduce a novel self-supervised learning <\/span>framework with tailored proxy tasks for medical image analy<\/span>sis. Specifically, we propose: (i) a new 3D transformer-based <\/span>model, dubbed Swin UNEt TRansformers (Swin UNETR), <\/span>with a hierarchical encoder for self-supervised pre-training; <\/span>(ii) tailored proxy tasks for learning the underlying pattern <\/span>of human anatomy. We demonstrate successful pre-training <\/span>of the proposed model on 5,050 publicly available computed <\/span>tomography (CT) images from various body organs. The ef<\/span>fectiveness of our approach is validated by fine-tuning the <\/span>pre-trained models on the Beyond the Cranial Vault (BTCV) <\/span>Segmentation Challenge with<\/span> 13<\/span> abdominal organs and seg<\/span>mentation tasks from the Medical Segmentation Decathlon <\/span>(MSD) dataset. Our model is currently the state-of-the-art <\/span>on the public test leaderboards of both MSD<\/span> and BTCV<\/span> datasets. Code: https:\/\/monai.io\/research\/swin unetr.<\/span><\/p>\n

\"Overview
Overview of our proposed pre-training framework. Input CT images are randomly cropped into sub-volumes and augmented with random inner cutout and rotation, then fed to the Swin UNETR encoder as input. We use masked volume inpainting, contrastive learning and rotation prediction as proxy tasks for learning contextual representations of input images<\/figcaption><\/figure>\n<\/div>\n

 <\/p>\n

\"Overview
Overview of the Swin UNETR architecture.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Tang, Yucheng, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. “Self-supervised pre-training of swin transformers for 3d medical image analysis.” In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 20730-20740. 2022. Full text:\u00a0 Abstract Vision Transformers (ViT)s have shown great performance in self-supervised…<\/p>\n","protected":false},"author":7582,"featured_media":3061,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,27,64,60,130],"tags":[137,21,190],"class_list":["post-3059","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-abdomen-imaging","category-big-data","category-body-wise","category-computed-tomography","category-deep-learning","tag-deep-learning","tag-segmentation","tag-transformer"],"_links":{"self":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/3059","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/users\/7582"}],"replies":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/comments?post=3059"}],"version-history":[{"count":2,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/3059\/revisions"}],"predecessor-version":[{"id":3063,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/posts\/3059\/revisions\/3063"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/media\/3061"}],"wp:attachment":[{"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/media?parent=3059"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/categories?post=3059"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/my.vanderbilt.edu\/masi\/wp-json\/wp\/v2\/tags?post=3059"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}