Virtual Mouse Project

Hey everyone, I'm doing a virtual mouse project by controlling the mouse pointer on the computer screen using a webcam to track movement of my hand n some mouse clicking event. I have to learn matlab from scratch n I've tried doing as possible as I can but still can't produce the right codes or thru simulink. Can anyone really help me on this bcuz I'm kinda desperate right now n I'm really don't know what I'm suppose to do next. If anyone can give me the codes or simulink models, that would be really helpful, thank u..

[Jan]

It is impossoble to "give you the codes" after this vague description of your project. It is more likely to get help, if your question concerns a specific problem. See: http://www.mathworks.com/matlabcentral/answers/6200-tutorial-how-to-ask-a-question-on-answers-and-get-a-fast-answer

[Syahrul Fikri]

okay, besides that, ive managed to construct a simulink model, which is to track my hand movement by drawing a bounding box around it, but then, do u or anyone know how to connect it to the mouse input so that i can control the mouse pointer??

[Syahrul Fikri]

thanks for all ur suggestions, ive search all over the intrnet, n the only solution is thru java, but anyone can help me on how to connect tracking codes to java robot, let say, im using image acquisition to capture video of my hand, n use image processing to draw a bounding box around my hand, n then how to connect this code to java robot?? i have nearly zero knowledge on matlab, i have to learn ervythng on myself, n frankly im learning on how to this tracking method thru codes since simulink method is consider a dead end

[Matt Kindig]

I don't know Simulink very well, but it might be worthwhile for us to post a screenshot of your Simulink model. That way we can get an idea of what approach you took, and can give some pointers for how to implement your code in Matlab. In Simulink, click Edit->Copy Model to Clipboard to get an image of your model, and upload it somewhere (pointing us to a link to it).

[Syahrul Fikri]

oh, that will truly grateful, thanx man, i used the simulink correlation demo as a guide, which is available in the matlab demos, here is the link to the picture, http://www.flickr.com/photos/43454132@N04/7050008053/in/photostream thanx again, anyone who can give any ideas are very much appreciated

[Syahrul Fikri]

final step people!! how do i convert my virtual mouse m.file to exe.file?? i tried using the deployment project method - file/new/deployment project.. , all i got is error - >> coder('-build', 'Virtual Mouse (Red Object Tracking).prj') ??? This text contains non-empty top-level expressions. It appears to be a script. Error in ==> tracking_red Line: 1 Column: 1 Code generation failed: Open error report. - my codes are below for checking

[Walter Roberson]

You can only compile function .m, not script .m . Add a "function" line to your .m file.

Answer 1

[Matt Kindig]

Controlling the mouse pointer is actually a bit difficult, as you need to use the Java Robot class (it's not a Matlab function per se). I'm not even sure how you would access this via Simulink.

See below: http://www.mathworks.com/support/solutions/en/data/1-2X10AT/index.html?solution=1-2X10AT

[Walter Roberson]

Example Java Robot use of keyboard: http://www.mathworks.com/matlabcentral/answers/5259-how-to-simulate-keyboard-key-inputs

The Solution that Matt shows above has an example of mouse control.

You could use a MATLAB Function block to access this kind of code; you would create the robot for the initialization phase and give commands to the robot based upon the run-time input signals.

Answer 2

[Teja Muppirala]

MATLAB can move the mouse around:

x = 500;
y = 600;

set(0,'PointerLocation',[x y]);

But, I think you do need the Java Robot to make a click.

[Walter Roberson]

Unless you can figure out which object you are over and locate its callback function and activate it, passing in a suitable "event" structure to represent right-click, shift-click etc.

Answer 3

[Syahrul Fikri]

hye all, ive found this one m file code from the matlab central file exchange, this code is to track red object in real time image acquisition, my question is, how do u connect the tracking bounding box to the mouse input from the java robot library?? the following are the codes

% Capture the video frames using the videoinput function % You have to replace the resolution & your installed adaptor name.

vid = videoinput('winvideo',1,'RGB24_320x240');

% Set the properties of the video object

set(vid, 'FramesPerTrigger', Inf); set(vid, 'ReturnedColorspace', 'rgb') vid.FrameGrabInterval = 5;

%start the video aquisition here

start(vid)

% Set a loop that stop after 100 frames of aquisition

while(vid.FramesAcquired<=200)

    % Get the snapshot of the current frame
    data = getsnapshot(vid);

    % Now to track red objects in real time
    % we have to subtract the red component 
    % from the grayscale image to extract the red components in the image.
    diff_im = imsubtract(data(:,:,1), rgb2gray(data));
    %Use a median filter to filter out noise
    diff_im = medfilt2(diff_im, [3 3]);
    % Convert the resulting grayscale image into a binary image.
    diff_im = im2bw(diff_im,0.18);

    % Remove all those pixels less than 300px
    diff_im = bwareaopen(diff_im,300);

    % Label all the connected components in the image.
    bw = bwlabel(diff_im, 8);

    % Here we do the image blob analysis.
    % We get a set of properties for each labeled region.
    stats = regionprops(bw, 'BoundingBox', 'Centroid');

    % Display the image
    imshow(data)

    hold on

    %This is a loop to bound the red objects in a rectangular box.
    for object = 1:length(stats)
        bb = stats(object).BoundingBox;
        bc = stats(object).Centroid;
        rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
        plot(bc(1),bc(2), '-m+')
        a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), '    Y: ', num2str(round(bc(2)))));
        set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
    end

    hold off
end

[Walter Roberson]

x = bc(1);

y = bc(2);

and then use the technical solution Matt referred to.

[Syahrul Fikri]

the code works!!! thank u so much for all your help walter, im truly grateful, thanx a zillion times

Answer 4

[vaibhav mathur]

put the value of red object's centroid in the [x y] mat. and set(0,'PointerLocation',[x y]); and the pointer moves according to centroid may be it work good luck!

Answer 5

[Syahrul Fikri]

hye all, i would like to say thank u so much for all your help n suggestions, finally, my code works by using the red tracking code n connect it to the java robot, thanx again all, if i encounter further problems, ill post it here, :D

Answer 6

[Syahrul Fikri]

Hye all, from the code above, I've successfully execute it with the java robot command, I hv a question, can anyone explain to me how can a mouse clicking event being execute when the program detects 2 tracking red object, the code can detect 2 red object, my plan is the first red object is to move the mouse pointer, but when it detects the second red object, a mouse clicking event is execure

[Walter Roberson]

mousePress

http://docs.oracle.com/javase/1.4.2/docs/api/java/awt/Robot.html#mousePress%28int%29

[Syahrul Fikri]

ok.., i know about the mouse press n release java code, ive implemented it, but the mouse press n release is always on whenever i move my pointer, do u know how to write the command of the mouse press n release code being execute only when my code track the second red object?? im still learning the red tracking codes by the way, :D, here is my code so far:

% Capture the video frames using the videoinput function

% You have to replace the resolution & your installed adaptor name.

vid = videoinput('winvideo',1,'RGB24_640x480');

% Set the properties of the video object

set(vid, 'FramesPerTrigger', Inf);

set(vid, 'ReturnedColorspace', 'rgb')

vid.FrameGrabInterval = 5;

%start the video aquisition here

start(vid)

% Set a loop that stop after 100 frames of aquisition

while(vid.FramesAcquired<=20000)

% Get the snapshot of the current frame

data = getsnapshot(vid);

% Now to track red objects in real time

% we have to subtract the red component

% from the grayscale image to extract the red components in the image.

diff_im = imsubtract(data(:,:,1), rgb2gray(data));

%Use a median filter to filter out noise

diff_im = medfilt2(diff_im, [3 3]);

% Convert the resulting grayscale image into a binary image.

diff_im = im2bw(diff_im,0.18);

% Remove all those pixels less than 300px

diff_im = bwareaopen(diff_im,300);

% Label all the connected components in the image.

bw = bwlabel(diff_im, 8);

% Here we do the image blob analysis.

% We get a set of properties for each labeled region.

stats = regionprops(bw, 'BoundingBox', 'Centroid');

% Display the image

imshow(data)

hold on

%This is a loop to bound the red objects in a rectangular box.

for object = 1:length(stats)

bb = stats(object).BoundingBox;

bc = stats(object).Centroid;

rectangle('Position',bb,'EdgeColor','r','LineWidth',2)

plot(bc(1),bc(2), '-m+')

a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), ' Y: ', num2str(round(bc(2)))));

set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');

import java.awt.Robot;

import java.awt.event.*;

mouse = Robot;

mouse.mouseMove(bc(1), bc(2));

mouse.mousePress(InputEvent.BUTTON2_MASK);

mouse.mouseRelease(InputEvent.BUTTON2_MASK);

end

hold off

end

[vidit virmani]

hey.... i feel lucky to find this page and the above discussion.... because i am too working on the same project of controlling mouse by color detectin methods.... i used the same code to detect a color.. i used blue instead of red.. i planned to execute the clicking events by manipulating the distance calculated between my finger and the thumb(both covered with a blue tape).. how can i do that....?? . . %% import java.awt.Robot; import java.awt.event.*; mouse = Robot; mouse.mouseMove(bc(1), bc(2)); mouse.mousePress(InputEvent.BUTTON2_MASK); mouse.mouseRelease(InputEvent.BUTTON2_MASK); %% do i have to use this code directly with the color detection code(as u have stated in above comment) or some prerequisites are to b done to interface java with MATLAB..?? please guide me for i am a new matlab user...

[Aakash]

Hi, could you please tell me how the clicking is done here...please...!! thanks..!

Answer 7

[vidit virmani]

@syahrul fikri : i used your code and its working... the mouse pointer moves and also clicks, but the problems i faced are..... 1) i dont know how the clicking operation is being executed.... does it executes when pointer is stable on a folder for some specified time or when another color is detected by the system?...... 2) if i move my hand to right , the pointer moves left and vice versa..... how to reverse that....... 3)if the camera resolution is less than the resolution of the screen, whole screen is not covered with this system.... how to eliminate this?..... 4)if i want to detect two different colors and execute clicking operations by manipulating the distance between the colors, how can i do that..?? please help....

[Rajshekar]

@vidit: 2) this is normal mirror concept: hence the pointer position shall have the formula of [1024-x, 780-Y] where x and y are the original positions in your code. the numbers 1024 AND 780 SORT OF RESOLUTION OF YOUR SCREEN.It depends on your screen resolution.

Answer 8

[Rajshekar]

Hey i tried to check this interesting thing: i used java robot.was able to move pointer and click as well. But there is this thing : when i right clicked a block in simulink(ofcourse by java robot), menu drop down appeared as normal; in the next lines of codes, i meant to position cursor on one of the menu item so i tried to reposition and use right click to select the menu item. But by the time these lines were to execute, the menu was not visible and hence the menu item cannot be selected

i think the menu items disappears when next lines of codes starts execution because the script window activates.

Do you know how to avoid this disappearance of the menu.

Answer 9

[cpe111]

Hey can you help me, please ? my project is how to control mouse by face , I have done face with nose and eyes detection , but I don't know how to control the cursor movements and clicking ! any help please ! because my matlab knowledge is so weak !

Thank you .