Image acquisition in simulink (Framerate problem)

Hello,

I'm using simulink to process live images. But I have problems with the speed/framerate of the model.

I reduced the model to only this blocks:

From Video Device
Frame Rate Display 
To Video Display

Even in this model the Framerate is below 3fps. If I also remove the "To Video Display" Block the fps climbs to 8fps. Also to slow.

I made the settings of the shutter and so. In the Preview in the "From Video Device" settings the video is displayed smoothly.

I tryed out the sampling time attribute of the "From Video Device" set it to 1/30 and 1/120. But no change. I'm not really comfortable with the solver and sampling time.

How can I get to a faster processing? Can you give me a clue?

Thank you for your help!

I will appreciate it.

Yours sincerely, Timo

I'm using:

Matlab 2010b
Simulink
Image Acquisition Toolbox
Video and Image Processing Blockset

[Shankar Subramanian]

Hi Timo,

Changing the sample time attribute from 1/30 to 1/120 affects only the timing of execution of the block in relation to the simulation time.

1. What camera are you using? Do you have vendor application in which you are able to acquire faster? What is the rate that you are expecting from the camera?

2. Some cameras do have a "Frame Rate" property when you get to the source properties. Can you double click on "From Video Device" block in your model and click the "Edit properties..." button to check if it exists. If so, what is the value that it is showing?

3. Also, what is the ROI that you have for this particular acquisition. Higher the ROI, the lower the frame rate. Can you also try lowering the ROI and see if your acquisition rate increases?

Thanks

Shankar

[Timo]

Hallo Shankar,

thank you for your answer.

1. An ICube Camera, which model I don't know. Yes in a vendor application the framerate is high.

The max Framerate of the Cam is 30fps. For my tracking application a higher framerate is an increase of precision. I would be lucky with a framerate above 15fps. I want to embed this tracking in a closed-loop control.

2. No this "Framerate Property" doesn't exist.

3.It is a RGB24_1280x1024 Image. In the Preview it works well(framerate is high). But with the 3 blocks model, which doesn't do anything but displaying the video and the framerate, the framerate is low (3 fps with/8fps without - displaying the video).

I also reduced the ROI: framerate increased with displaying the video to 5fps for an 8*640*480. A lower ROI is a loose of precision, I will probably need.

Thanks & Greeting

Timo

Answer 1

[Lukas]

We've probably found a simple solution for Simulink - to create your own simulink block based on m-functions from Image acquisition toolbox. Bellow you can find the code for Simulink->User Defined Functions->S-function block. It works well at 30 FPS. Just one known lack is to overflow the memory after tens seconds of run, so it is suitable just for short experiments or you need to make periodical flush of the image buffer (e.g. function getdata(), but it causes a few frames delay).

    function camera_pike_s_function(block)
        setup(block);
    %end camera_pike_s_function

    function setup(block)
        block.SetPreCompPortInfoToDefaults;

        block.NumInputPorts  = 0;
        block.NumOutputPorts = 1;
        block.OutputPort(1).Dimensions  = [480 640]; % TODO - automatic setting from camera properties
        block.OutputPort(1).DatatypeID  = 3;         % uint8
        block.OutputPort(1).Complexity  = 'Real';

        block.NumDialogPrms = 0;
        block.SampleTimes = [-1 0];                   %Inherited sample time
        block.SimStateCompliance = 'DefaultSimState';

        block.RegBlockMethod('PostPropagationSetup',    @DoPostPropSetup);
        block.RegBlockMethod('Start', @Start);
        block.RegBlockMethod('Outputs', @Outputs);    
        block.RegBlockMethod('Update', @Update);
        block.RegBlockMethod('Terminate', @Terminate);
    %end setup

    function DoPostPropSetup(block)
        % TODO - get camera properties form block parametres
        device =  videoinput('avtmatlabadaptor_r2009b', 1);
        set_param(block.BlockHandle,'UserData',device);
        src = getselectedsource(device);
        set(src, 'FrameRate', '30');

        block.NumDworks = 1;
        block.Dwork(1).name = 'image';  
        block.Dwork(1).Dimensions  = block.OutputPort(1).Dimensions(1) * block.OutputPort(1).Dimensions(2);
        block.Dwork(1).DatatypeID  = 3;  %uint8
        block.Dwork(1).Complexity  = 'Real';
    %end DoPostPropSetup

    function Start(block)
        device = get_param(block.BlockHandle,'UserData');
        set(device, 'FramesPerTrigger', inf)
        start(device);
    %end Start

    function Outputs(block)
        block.OutputPort(1).Data = reshape(block.Dwork(1).data,block.OutputPort(1).Dimensions);
    %end Outputs

    function Update(block)
        device = get_param(block.BlockHandle,'UserData');
        %disp(device.FramesAvailable);
        numAvailable = device.FramesAvailable;
        if (numAvailable > 0)
            %snapshot = getdata(device, 1); %remove it from buffer is too slow, don't use it
            snapshot = peekdata(device, 1);
            snapshot = snapshot(:);
            if (length(snapshot) == length(block.Dwork(1).data))
                block.Dwork(1).data = snapshot;
            end
        end
    %end Update

    function Terminate(block)
        device = get_param(block.BlockHandle,'UserData');
        stop(device);
        %get recorded imageData and export it into workspace (if needed)
            %imageData = getdata(device, device.FramesAvailable);
            %evalin('base','imageData = 0;');
            %assignin('base','imageData',imageData);
        %destroy video object
        delete(device);
        clear device
    %end Terminate

Answer 2

[Shankar Subramanian]

Hi Timo,

I do not know which model of ICube Camera you have but I had checked their product pages where they specify frame rate along with ROI. I might not be seeing same camera, but some cameras of ICube specify 30fps or 15fps at lower ROIs and it also changes depending on whether you have USB2.0 or lower.

The output of From Video Device block is single by default. Can you change it to uint8 and determine the frame rate? Also do check the frame rate without the Display block in the model.

Thanks.

Answer 3

[Shankar Subramanian]

Hi Timo,

Can you run the following code in MATLAB and check the frames rate (fps) value. This calculates the rate at which images are acquired in MATLAB.

% Select the right adaptor, deviceID and format.
vid = videoinput('winvideo', 1, 'RGB24_1280x1024'); 
vid.FramesPerTrigger = 20;
start( vid );
wait( vid, Inf );
[d t] = getdata( vid, vid.FramesAvailable );
fps =  1 / mean( diff( t ) );

Shankar

Answer 4

[Timo]

Hello Shankar,

Thank you again for your reply. I actual get higher framerates outside of the Simulink model. I have USB 2. I removed also the display block and got up to 7fps, still to slow. I will try your uint8 tip tomorrow and post the result.

Thanks Timo

Answer 5

[Timo]

Hallo Shankar,

I tried uint8 but it didn't change anything. Perhaps the performance will get better if I exported the model with the real time workshop.

Thanks Timo

Answer 6

[Timo]

Hallo Shankar,

thanks for continuing helping me.

I executed your code and got 7.1 fps, but the light isn't so good. So I adjusted the exposure like I did it in Simulink.

If I excecute the code below I get fps=21.4

vid = videoinput('winvideo', 1, 'RGB24_1280x1024'); 
set(vid,'FramesPerTrigger',20);
src = getselectedsource(vid)
set(src, 'ExposureMode', 'manual');
set(src, 'Exposure', -8);
start( vid );
wait( vid, Inf );
[d t] = getdata( vid, vid.FramesAvailable );
fps =  1 / mean( diff( t ) );

Thanks again, Timo

Answer 7

[Gilmore]

Hello Everyone, I am trying to modify the exposure rate of my webcam for maximum fps. I cannot find it listed to the edit properties of my video device block. Any suggestions pls. I am using the 2010A

Answer 8

[Lukas]

Hello everyone,

we got the exact problem as Timo described. We just use another camera, namely Pike Camera made by Allied Vision Technologies. Everything works well with it except the simulation run.

 Win XP SP3 32bit
 Matlab R2010b
 AVT Adaptor for Matlab Image Acquisition v 1.0.3.
 Driver: Intek
 Smartview (vendor software): OK (30 FPS)
 Imaqtool: OK (30 FPS, mode F0M5)
 Simulink - block properties - preview: OK (30 FPS, mode F0M5)
 Simulink - simulation - mode F0M5: just 1 FPS (30 FPS set in Edit properties)
 ROI settings: no effect

We have contacted the Allied Vision technical support, but they suppose that the problem is in Matlab or Image Acquisition toolbox.

Please, did anyone of you find any solution? It would be really helpful. We've tried possibly all combinations of drivers and setting, but with no result.

Thank you very much in advance.

Lukas

Answer 9

[Timo]

Hello,

we changed to C# and OpenCV, because we didn't find a solution.

Timo

Answer 10

[Shankar Subramanian]

Hi Lukas,

This is an issue with Allied Vision Tech adaptor and they have an updated version for the fix as well. Please download the latest from the following link. Please copy the library into your adaptor directory path for AVT.

Also, do update if this fixed your performance problem.

Thanks

Shankar