How to get the location of an object using computer vision?

I need to detect objects in a scene using a top fixed camera and then send their metric locations to a robotic arm to pick them up. I am using MATLAB for color detection of the objects and I got their locations but in pixel coordinate system. How to map them to real world x-y metric coordinates?

This is the code I use:

    % Capture the video frames using the videoinput function
    % You have to replace the resolution & your installed adaptor name.
    vid = videoinput('winvideo',1, 'YUY2_1280x720');

    % Set the properties of the video object
    set(vid, 'FramesPerTrigger', Inf);
    set(vid, 'ReturnedColorspace', 'rgb')
    vid.FrameGrabInterval = 5;

    %start the video aquisition here
    start(vid)

    % Set a loop that stop after 100 frames of aquisition
    while(vid.FramesAcquired<=200)

        % Get the snapshot of the current frame
        data = getsnapshot(vid);
        hsv = rgb2hsv (data);
        hue = hsv (:,:,1);
        sat = hsv (:,:,2);
        val = hsv (:,:,3);
        hThresholds = [0, 0.4];
        sThresholds = [0, 0.3];
        vThresholds = [0.7, 1];

        binaryH = hue >= hThresholds(1) & hue <= hThresholds(2);
        binaryS = sat >= sThresholds(1) & sat <= sThresholds(2);
        binaryV = val >= vThresholds(1) & val <= vThresholds(2);

        diff_im = binaryH & binaryS & binaryV; % get the common region

        %Use a median filter to filter out noise
        %out = medfilt2(out, [3 3]);
        % Convert the resulting grayscale image into a binary image.
        %out = im2bw(out,0.18);

        % Remove all those pixels less than 300px
        diff_im = bwareaopen(diff_im,500);

        % Fill holes
        diff_im = imfill(diff_im, 'holes');

        % Label all the connected components in the image.
        bw = bwlabel(diff_im, 8);

        % Here we do the image blob analysis.
        % We get a set of properties for each labeled region.
        stats = regionprops(bw, 'BoundingBox', 'Centroid');

        % Display the image
        imshow(data)

        hold on

        %This is a loop to bound the red objects in a rectangular box.
        for object = 1:length(stats)
            bb = stats(object).BoundingBox;
            bc = stats(object).Centroid;
            rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
            plot(bc(1),bc(2), '-m+')
            a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), '    Y: ', num2str(round(bc(2)))));
            set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
        end

        hold off
    end
    % Both the loops end here.

    % Stop the video aquisition.
    stop(vid);

    % Flush all the image data stored in the memory buffer.
    flushdata(vid);

    % Clear all variables
    clear all

Answer 1

[Image Analyst]

A camera on top of your rig could see lots and lots of objects. Imagine a video camera going down the street. You need to segment your scene and that means precisely defining exactly what attributes define the object you want out of all the other possible background, objects, and clutter in your image. Have you done that? Because we can't answer (yet) because there is no one set of rules that defines a colored object for all possibles scenes in the universe.

In the meantime, see the color segmentation demos in my File Exchange https://www.mathworks.com/matlabcentral/fileexchange/?term=authorid%3A31862&sort=downloads_desc