Improving SAM model for medical image segmentation

Abstract

Early detection of polyps in the colon is crucial for preventing colorectal cancer, the second leading cause of cancer-related deaths globally. However, accurate identification of polyps can be challenging due to factors like subtle visual cues, variable lighting, and human fatigue. This work aims to adapt the Segment Anything Model (SAM) to segment colonoscopy polyp by replacing its encoder with a lightweight convolutional neural net- work. Additionally, we strive to enhance the model’s accuracy and automation through the implementation of zero-shot learning. This approach involves utilizing a pre-trained object detection model with K-means clustering algorithm to extract the bounding box prompt, which serves as auxiliary information for SAM, thereby improving its performance on un- seen polyp data without the need for fine-tuning or manual prompt design. The proposed method reduces the number of SAM encoder parameters from 91M to 3M. It demonstrates superior performance compared to some existing approaches that work to fine-tune SAM with large number of parameters.This work offers a contribution to computer-aided polyp detection. It paves the way for more efficient and accurate polyp segmentation systems, ultimately improving early cancer diagnosis and patient care.