Treffer: Learning few-shot semantic segmentation with error-filtered segment anything model.

In computer graphics and vision, segmentation is an important foundation task. Owing to the lack of annotated data, few-shot semantic segmentation (FSS) leverages a limited number of annotated images to segment new objects. Lacking of annotated data makes FSS perform poorly in predicting masks with precise contours. This limits the usage of FSS in a lot of downstream tasks such as medical image analysis, object tracking, and image editing. Recently, we have noticed that the large foundation model Segment Anything Model (SAM) has good generalization ability in processing image details and textures. As a preliminary attempt, we propose vanilla SAM to boost FSS methods to address the issue of inaccurate contour. The proposed vanilla SAM works as a training-free post-processing tool for FSS methods to improve the accuracy of predicting masks with the assistance of SAM. Specifically, we use predicted masks from pretrained FSS methods to generate prompts and then input the prompts and original images to SAM to get new masks. We propose single-object-prompt and multiple-object-prompt methods. However, inaccurate original masks or inaccurately generated prompts may lead to wrong predictions by SAM. Therefore, we further propose error-filtered SAM (EF-SAM) based on vanilla SAM. It can exclude wrong masks remarkably with the error-filtering (EF) algorithm. Extensive experiment results on PASCAL- 5 i and COCO- 20 i show that our EF-SAM outperforms both the original FSS method and vanilla SAM method in both qualitative and quantitative aspects. The experiment results on polyp image set Kvasir-SEG demonstrate the effectiveness of EF-SAN in downstream tasks. The codes are available at https://spsspsdoi.org/sps10.5281/spszenodo.13093953 and https://github.com/fcbfcb1998/EFSAM. [ABSTRACT FROM AUTHOR]