how can I use t-sne to visualize network in image segmentation

How can I get my semantic segmentation model to output the similar picture

S. Gou, Z. Yue, K. Di, C. Zhao, R. Bugiolacchi, J. Xiao, Z. Cai, S. Jin, Transverse aeolian ridges in the landing area of the Tianwen-1 Zhurong rover on Utopia Planitia, Mars, Earth and Planetary Science Letters, 595 (2022) 117764.

---------------------------------

In the paper, t-sne is used for visualization the network, I have read the introduction of t-sne in matlab, and now I understand the application of t-sne in picture classification. In the case of handwritten digit recognition, a row represents an image, and the pixels of the image are the number of columns, that is, the number of dimensions, and then t-sne is used for dimension reduction.

So how does t-sne work in semantic segmentation?

Answer 1

[Shantanu Dixit]

Hi Yan, t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used in semantic segmentation to visualize the high-dimensional features corresponding to each class in a lower-dimensional space. Below is a high-level overview of how to apply t-SNE in this context:

• Obtain the feature maps: Perform semantic segmentation on your input image (H x W x C). This outputs a feature map of size H x W x numClasses (in its penultimate layer), where each layer corresponds to the predicted probabilities or scores for a specific class.
• Extract class-specific features: Before applying the argmax across the classes to obtain the final segmentation map, extract the feature map corresponding to each class. Each of these feature maps will have a size of H x W x 1 (or simply H x W).
• Flatten the feature maps: For each class-specific feature map, flatten it into a vector of size H*W x 1. This allows to reshape the spatial information into a format suitable for t-SNE.
• Apply t-SNE: Use the 'tsne' function in MATLAB to reduce the dimensionality of each flattened vector. This outputs low-dimensional embeddings for each class, which can be plotted to visualize the feature distribution.
• Visualize the results: After running t-SNE, plotting the low-dimensional embeddings gives information on how the features for each class are distributed in the reduced space.

Additionaly refer to the MathWorks documentation for more information:

• tsne: www.mathworks.com/help/stats/tsne.html
• semantic segmentation: www.mathworks.com/help/vision/ug/semantic-segmentation-using-deep-learning.html

[Yan]

Thanks for your answer