Segment Anything Model without foreground points or bounding box

Hi,

I'm using the new R2024a "Segment Anything" model with good results.

However, it seems like the original Segment Anything Model by Meta has also the 'everything' option, where it somehow segments all the objects in the image with no user input.

Is there a way to use the Matlab version for a similar result? Just specifying a grid of 'foreground points' does not help, as the Matlab function considers them all to belong to the same object.

thanks in advance

Answer 1

[Garmit Pant]

Hello tsvi lev

From what I understand, you are using Segment Anything model in MATLAB using “segmentAnythingModel” and can segment single objects in images. You are looking to use the Segment Everything functionality of the original SAM model in MATLAB.

As of R2024a, MATLAB supports only the SAM foundational model. The Segment Everything functionality is not available in MATLAB. However, given that it is a downstream task, it can be implemented using the model available.

As you have mentioned, just specifying a grid of foreground points will not solve the task since the SAM model differentiates between the foreground and the background based on prompts. In the workaround that you have mentioned, the prompt is a grid of foreground points. Thus, just specifying the entire grid will make the model consider each point as a point referring to the foreground.

Please refer to the following general workflow to implement Segment Everything functionality:

1. Generate image embeddings using “extractEmbeddings” object function.
2. Generate a grid of foreground points to pass to the function.
3. Batch the points together and generate masks for the batches of points using the “segmentObjectsFromEmbeddings” function.
4. Compare the masks produced for the batches and remove the unreliable masks based on some thresholds set for the confidence score, size etc.
5. Remove overlapping masks.
6. Remove duplicate detections.

By following the above steps, you will be able to replicate the 'Segment Everything' functionality with appropriate pre- and post-processing".

For further understanding, I suggest you refer to the following MathWorks Documentation and resources:

1. “segmentObjectsFromEmbeddings” function: https://www.mathworks.com/help/images/ref/segmentanythingmodel.segmentobjectsfromembeddings.html
2. “segmentAnythingModel” object: https://www.mathworks.com/help/images/ref/segmentanythingmodel.html

Hope you find the above explanation and suggestions useful!