Why doesn't the frame rate drop with extra processing time per frame?

I'm processing images inside a for loop with the trigger set to immediate. In the following code:

start(vid);
frame = getdata(vid);
t_total_Start = tic;
for iframe = 1:Nframe
    [frame,lT(iframe)] = getdata(vid);
    snapshotbuffer(:,:,iframe) = frame;
    hvpc.step(frame);
end

release(hvpc);
elapsedTime = toc(t_total_Start);
timePerFrame = elapsedTime / iframe
effectiveFrameRate = 1 / timePerFrame

The effective frame rate approximately ~450 FPS.

After adding a few functions into the for loop:

start(vid);
frame = getdata(vid);
t_total_Start = tic;
for iframe = 1:Nframe
    [frame,lT(iframe)] = getdata(vid);
    snapshotbuffer(:,:,iframe) = frame;
    % Two functions used to process the frame
    function1(frame);
    function2(frame);
    
    hvpc.step(frame);
end
release(hvpc);
elapsedTime = toc(t_total_Start);
timePerFrame = elapsedTime / iframe
effectiveFrameRate = 1 / timePerFrame

The effective frame rate is still around 450 FPS, but I expected it to drop because function1 and function2 are now inside the loop. I measured their execution time using tictoc, and they take about 0.5 ms together, so the frame rate should drop to around 370 FPS (1/(1/450 + 0.5e-3) = 370 FPS).

I also tested by replacing function1 and function2 with pause(0.0005), but the frame rate still didn’t drop. I'm wondering why the frame rate isn’t decreasing as expected.

I have another question about timing. Is getdata providing the correct timestamps after adding functions inside the loop? Should I use tic and toc to measure the time for each frame? The timestamps from getdata don’t seem to change, even when I add a pause in the loop.

Thanks!

Answer 1

[Umar]

Hi @Yujie Luo ,

The observation that your effective frame rate remains around 450 FPS despite adding functions is intriguing. Here are several factors that could explain this phenomenon:

Processing Overhead vs. Frame Capture Time: The frame capture time may be significantly less than the time taken by function1 and function2. If getdata(vid) is the bottleneck (i.e., it takes longer than the processing functions), then even if you add processing time, it won't affect the overall frame rate. Since you mentioned that both functions together take about 0.5 ms, if getdata(vid) is executing faster than this or if it is optimized in a way that compensates for additional processing, you may not see a drop in FPS.

Multithreading or Buffering: Some video processing libraries utilize multithreading or buffering techniques to handle frames efficiently. If your video input system has buffering capabilities, it may be capturing frames ahead of time while processing previous ones, thereby maintaining a high frame rate.

Measurement Timing: Ensure that you measure the total execution time accurately. If you are measuring just the time taken for function1 and function2, ensure that you're not inadvertently including other operations that might skew results.

To gain better insights into your performance metrics:

Use tic and toc for each function call: Wrap each function call with tic and toc to get precise measurements of their execution times.

tic;
function1(frame);
elapsedFunction1 = toc;

tic;
function2(frame);
elapsedFunction2 = toc;

Measure total loop execution time: You can also measure the total execution time for each iteration of your loop to see how it varies with different functions:

t_total_Frame = tic;
% Your existing code...
elapsedFrame = toc(t_total_Frame);

Regarding your concern about timestamps from getdata, it's important to note that:

• The timestamps returned by getdata should reflect the moment each frame was captured from the video stream.

• If you observe that timestamps remain constant even after introducing pauses or additional processing, consider Frame Buffering, timestamps might not update if frames are being buffered; check if your video object has such behavior. Also, if your timing mechanism lacks precision (e.g., milliseconds), subtle delays might not be captured.

Utilize MATLAB's built-in profiler (profile on; ...; profile viewer;) to analyze where time is being spent within your loop. This will help identify any unexpected performance. If feasible, explore asynchronous methods to process frames without blocking the main loop. This could allow capturing new frames while processing previous ones. Test with varying complexities in function1 and function2 to see how they impact performance, helping establish a clearer relationship between computational load and frame rate.

By following these suggestions and understanding potential underlying mechanisms in your video processing setup, you can better analyze and optimize your effective frame rates while ensuring accurate timestamp management.

Hope this helps.

Please let me know if you have any further questions.

[Yujie]

Hi Umar,

Thank you for your informative reply. I have a few questions/comments regarding the following:

Processing Overhead vs. Frame Capture Time: Since the FPS is around 450 without image processing, the getdata function takes approximately 1/450 seconds (~2.2 ms), which is comparable to 0.5 ms. This made me question whether this could be the case.

Multithreading or Buffering: I’m performing continuous acquisition with the frame per trigger set to 1. Would buffering or multithreading help in my case? I would expect the video input system to only retain the newest frame.

Measurement Timing: I appreciate your suggestion for the timing measurement method. I’ve already implemented what you recommended, but I’m confused by the results I’m getting.

For the tic-toc timing method, the time measured for each loop (t_loop_time) is significantly longer or shorter than the expected acquisition time based on the camera's frame rate. The t_loop_time ranges from 0.7 ms to as long as 27 ms. On average, it's about 2.2 ms per loop. While I understand that the camera can sometimes be slow, I'm skeptical about the 0.7 ms loop time. How can the camera appear to be three times faster (0.7 ms compared to 2.2 ms) than its physical limit of 450 FPS? From the timestamps I'm getting using getdata, the time difference is almost negligible. The code I used for timing is shown below.

for iframe = 1:Nframe
    t_loop_start = tic;
    [frame,lT(iframe)] = getdata(vid)
    t_loop_time = toc(t_loop_start)
end

I agree with you that the timestamp returned by getdata reflects the moment each frame was captured in the video stream relative to the first trigger. If I use the getdata function to retrieve timestamps, it should give me the timestamps lT(iframe) for each frame, including any extra processing time with respect to when the first trigger occurs. Is that correct?

for iframe = 1:Nframe
    [frame,lT(iframe)] = getdata(vid);
    snapshotbuffer(:,:,iframe) = frame;
    % Two functions used to process the frame
    function1(frame);
    function2(frame);
    
    hvpc.step(frame);
end

Sorry for asking so many questions, and I really appreciate your help.

[Umar]

Hi @Yujie Luo ,

Please see my response to your comments below.

Comment#1: Processing Overhead vs. Frame Capture Time Since the FPS is around 450 without image processing, the getdata function takes approximately 1/450 seconds (~2.2 ms), which is comparable to 0.5 ms. This made me question whether this could be the case.

My response: The FPS of around 450 suggests that, ideally, each frame should be captured in approximately 2.2 ms (1/450 seconds). However, your observations indicate variability in loop timing (`t_loop_time`) ranging from 0.7 ms to 27 ms, with an average around 2.2 ms. This inconsistency can arise from several factors:

Processing Time: The functions function1(frame) and function2(frame) might introduce additional processing delays that vary based on the complexity of operations performed on each frame.

Camera Performance: Although the camera is rated for 450 FPS, environmental conditions, settings, or the camera's internal processing capabilities can affect actual performance.

System Load: If your system is handling other processes simultaneously, it may impact timing accuracy.

Comment#2: Multithreading or Buffering: I’m performing continuous acquisition with the frame per trigger set to 1. Would buffering or multithreading help in my case? I would expect the video input system to only retain the newest frame.

My response: When performing continuous acquisition with `FramesPerTrigger` set to 1, you are indeed capturing one frame at a time. Here’s how multithreading and buffering could enhance performance:

Multithreading: By offloading processing tasks to separate threads, you can maintain a steady flow of data acquisition while simultaneously processing frames. This approach can help minimize delays caused by computational overhead.

Buffering: Implementing a buffer to store incoming frames allows for smoother processing. Instead of processing each frame immediately after acquisition, you could retrieve multiple frames into a buffer and process them concurrently or sequentially without halting acquisition.

_Comment#3 Measurement Timing I appreciate your suggestion for the timing measurement method. I’ve already implemented what you recommended, but I’m confused by the results I’m getting.

For the tic-toc timing method, the time measured for each loop (t_loop_time) is significantly longer or shorter than the expected acquisition time based on the camera's frame rate. The t_loop_time ranges from 0.7 ms to as long as 27 ms. On average, it's about 2.2 ms per loop. While I understand that the camera can sometimes be slow, I'm skeptical about the 0.7 ms loop time. How can the camera appear to be three times faster (0.7 ms compared to 2.2 ms) than its physical limit of 450 FPS? From the timestamps I'm getting using getdata, the time difference is almost negligible. The code I used for timing is shown below._

for iframe = 1:Nframe
  t_loop_start = tic;
  [frame,lT(iframe)] = getdata(vid)
  t_loop_time = toc(t_loop_start)
end

I agree with you that the timestamp returned by getdata reflects the moment each frame was captured in the video stream relative to the first trigger. If I use the getdata function to retrieve timestamps, it should give me the timestamps lT(iframe) for each frame, including any extra processing time with respect to when the first trigger occurs. Is that correct?

for iframe = 1:Nframe
  [frame,lT(iframe)] = getdata(vid);
  snapshotbuffer(:,:,iframe) = frame;
  % Two functions used to process the frame
  function1(frame);
  function2(frame);

    hvpc.step(frame);
  end

My response: Your implementation of the tic-toc method appears valid for measuring loop execution times. However, consider these aspects:

Measurement Overhead: The tic-toc method itself incurs a small overhead that may slightly skew your results.

Frame Retrieval Timing: If getdata is blocking (which it is), it waits until enough frames are available before returning control to MATLAB. This behavior can lead to discrepancies if the camera's frame rate fluctuates.

To troubleshoot further:

• Ensure that any processing functions do not exceed expected execution times.

• Consider logging timestamps before and after significant operations within your loop to identify where delays occur.

Clarification on Timestamps from getdata

You are correct that using getdata returns timestamps (lT(iframe) that indicate when each frame was captured relative to the first trigger. It’s essential to understand that these timestamps reflect the actual capture time and may include delays from both acquisition and any subsequent processing.

To further improve performance and clarify timing issues:

Experiment with Buffer Sizes: Adjust buffer sizes in your acquisition setup to see how it affects performance.

Profiling: Use MATLAB's built-in profiler (profile on, profile viewer) to analyze where time is being spent within your loop.

Check Camera Settings: Ensure that your camera’s settings (like resolution and exposure time) are optimized for high-speed capture.

Hope this helps.

[Yujie]

Hi Umar,

I really appreciate your reply. I learned a lot from your response, and it helped me better understand the imaging system and code.

Thanks again!

[Umar]

Hi @ Yujie Luo,

Thank you for your kind words. I am pleased to hear that my response was helpful and contributed to your understanding of the imaging system and code. If you have any further questions or need additional clarification, please do not hesitate to reach out. I am here to assist you.