Multi-Label Medical Image Retrieval Via Learning Multi-Class Similarity

18 Pages Posted: 29 Jun 2022

See all articles by Xiaoyuan Guo

Xiaoyuan Guo

Emory University

Jiali Duan

University of Southern California

Judy Gichoya

Emory University

Hari Trivedi

Emory University - School of Medicine

Saptarshi Purkayastha

Indian Institute of Management (IIM), Calcutta

Ashish Sharma

Emory University

Imon Banerjee

Mayo Clinic

Abstract

Introduction: Multi-label image retrieval is a challenging problem in the medical area. First, compared to natural images, labels in the medical domain exhibit higher class-imbalance and much nuanced variations. Second, pair-based sampling for positives and negatives during similarity optimization are ambiguous in the multi-label setting, as samples with the same set of labels are limited. 
Methods: To address the aforementioned challenges, we propose a proxy-based multi-class similarity (PMS) framework, which compares and contrasts samples by comparing their similarities with the discovered proxies. In this way, samples of different sets of label attributes can be utilized and compared indirectly, without the need for complicated sampling. PMS learns a class-wise feature decomposition and maintains a memory bank for positive features from each class. The memory bank keeps track of the latest features, used to compute the class proxies. We compare samples based on their similarity distributions against the proxies, which provide a more stable mean against noise.
Results: We benchmark over 10 popular metric learning baselines on two public chest X-ray datasets and experiments show consistent stability of our approach under both exact and non-exact match settings.
Conclusions: We proposed a methodology for multi-label medical image retrieval and design a proxy-based multi-class similarity metric, which compares and contrasts samples based on their similarity distributions with respect to the class proxies. With no perquisites, the metrics can be applied to various multi-label medical image applications. The implementation code repository will be publicly available after acceptance.

Keywords: Multi-label Medical Image Retrieval, Multi-class Similarity, Radiology Image

Suggested Citation

Guo, Xiaoyuan and Duan, Jiali and Gichoya, Judy and Trivedi, Hari and Purkayastha, Saptarshi and Sharma, Ashish and Banerjee, Imon, Multi-Label Medical Image Retrieval Via Learning Multi-Class Similarity. Available at SSRN: https://ssrn.com/abstract=4149616 or http://dx.doi.org/10.2139/ssrn.4149616

Xiaoyuan Guo (Contact Author)

Emory University ( email )

201 Dowman Drive
Atlanta, GA 30322
United States

Jiali Duan

University of Southern California ( email )

2250 Alcazar Street
Los Angeles, CA 90089
United States

Judy Gichoya

Emory University ( email )

201 Dowman Drive
Atlanta, GA 30322
United States

Hari Trivedi

Emory University - School of Medicine

1364 Clifton Rd. NE
Atlanta, GA 30322
United States

Saptarshi Purkayastha

Indian Institute of Management (IIM), Calcutta

Ashish Sharma

Emory University ( email )

201 Dowman Drive
Atlanta, GA 30322
United States

Imon Banerjee

Mayo Clinic ( email )

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
64
Abstract Views
298
Rank
627,771
PlumX Metrics