The points are overlapping because of the marker size and the data resolution set by the axis limits. If you zoom in or use smaller marker size, those points will not overlap. You need to set a minimal distance threshold that defines what "overlapping" or "near" means.
You can use pdist or apply the distance formula to all pairs of points to get the distance between all pairs of points and then determine which paired-distances fall below your threshold.
This demo creates a set of (x,y) coordinates and uses pdist to compute the paired distances. It then plots a distribution of distances that you can use to choose a threshold that defines a distance considered to be 'overlapping'. Lastly, it marks the paired coordinates whose distances are below the threshold.
- xy is an nx2 matrix of [x,y] coordinates
- threshold is a value you choose to define 'near'
- nearIdx is a vector of indices showing which values in xy are 'near'
plot(ax1, xy(:,1), xy(:,2), 'bo', 'MarkerSize', 10)
pairedDist = squareform(pdist(xy)) + triu(inf(size(xy,[1,1])));
title(ax2, 'Paired distances')
xline(ax2, threshold, 'r-', 'Theshold')
[xIdx,yIdx] = find(pairedDist < threshold);
plot(ax1, xy(nearIdx,1), xy(nearIdx,2), 'rx')