save loop data only if 'if statement' is true

Hey,

I need to make a list of a for loop output if the if statement is true, but feel like I'm missing something basic...

Let's say I have a variable 'Teapot_names' = {'Teapot-1', 'Teapot-2', 'Teapot-3'}

And a list of files I want to iterate through (fileList):

*edited the code to reflect more info

threshold = 0.35
fileList = dir(fullfile(somepath, '*_table.tsv'));
Teapot_names = dir(fullfile(some_path, 'Teapot-*'));
idx = [Teapot_names.isdir]'; 
Teapot_names = {Teapot_names(idx).name};
for i = 1:numel(fileList) 
    fullFileName = fullfile(fileList(i).folder, fileList(i).name);             
    thisTable = readtable(fullFileName,'TreatAsEmpty','n/a','FileType', 'text', 'Delimiter', 'tab');  
    
    % here I extract some variables and calculate the variable percent
    percent = sum_of_some_variables_thisTable / total_thisTable
    
    if percent > threshold
        warning('exceeded threshold by %s', Teapot_names{i}); 
        x{i} = Teapot_names{i};
    end
end 

that gives the Error: Unable to perform assignment with 0 elements on the right-hand side.

It prints a warning for those iterations for which the if statement is true, but does not print the Teapot_names.

I went through the iterations for each of the files and I know that some if statements are true.

I thought maybe the problem is my understanding of cell arrays, so I tried something that worked in previous scripts (in shorter version):

x = zeros(size(fileList)); 
for i = numel(fileList) 
    some_calculations
    if some_calculation
        x(i) = 1;
    end
end

However, that doesnt work either, the variable x stays all zero, while I would expect some of them to have changed to 1.

Can somebody point me in the right direction please =)?

[Matt J]

All the necessary trouble-shooting info is hidden from us. We have to see what condition the if statement is testing.

[Alix]

Sorry, the if statement does test whether a certain threshold has been exceeded.

I added some more information, but left the calcualtions of the variables to calculate 'percent' out (tripple checked, it works fine for each iteration).

Hope it makes it more clear, please let me know if you need more info!

(side note: the lenght of fileList is always the same as the lenght of Teapot_names)

threshold = 0.35
fileList = dir(fullfile(somepath, '*_table.tsv'));

Teapot_names = dir(fullfile(some_path, 'Teapot-*'));
idx = [Teapot_names.isdir]'; 
Teapot_names = {Teapot_names(idx).name};


for i = 1:numel(fileList) 
    fullFileName = fullfile(fileList(i).folder, fileList(i).name);             
    thisTable = readtable(fullFileName,'TreatAsEmpty','n/a','FileType', 'text', 'Delimiter', 'tab');  
    
    % here I extract some variables and calculate the variable percent
    percent = sum_of_some_variables_thisTable / total_thisTable
    
    if percent > threshold
        warning('exceeded threshold by %s', Teapot_names{i}); 
        x{i} = Teapot_names{i};
    end
end 

[Voss]

One thing to fix:

for i = numel(fileList)

iterates the loop one time with the value of i equal to numel(fileList). You probably want to say:

for i = 1:numel(fileList)

I'm not sure that is the cause of the problem though. Hard to say without seeing the actual code you are running. You use k to index fileList inside the i loop, for instance. Is this an error in your real code or an error when copy/pasting here? Or what if the calculation for sum_of_some_variables_from_current_table modifies the value of i (or k)? We cannot know.

[Alix]

sorry, it was a copy/paste mistake, I edited my post so it reflects more info

[Alix]

And I don't believe any of the calculations modify the value of i (how could that happen?), I ran the script for the different tables seperately (by adapting e.g. for i = 1:1(fileList)), and it spits out the correct values for 'percent' each time. It also spits out the warning message when the if statement is met and when running it over all files. The only thing it doesn't do is that it doesn't insert the name (Teapot_name(i)) inside the message or add it to the variable x.

[Voss]

How modifying the value of i inside a for loop iterating over i can happen (very easily, it turns out):

for i = 1:3
    disp(i);
    i = 99;
    disp(i);
end

     1

    99

     2

    99

     3

    99

So in your case, if sum_of_some_variables_from_current_table is a script that sets the value of i, then that i is the same as in your loop, so if you use it later in the same iteration of the loop, you're using the value set by the script, which is likely incorrect. (This type of bug can be particularly difficult to track down, and that's one reason I don't use scripts at all - except for prototyping/testing the logic of an idea and of course here on MATLAB Answers.) Note that the variable x (or any other variable) could also be set by a script, but if there are no scripts called within your loop, then this shouldn't be a problem.

[Alix]

ah okay like that, thx.

I am not using a script, and i'm not calling an empty array i=[] anywhere.

the sum_of_some_variables_from_current_table is just the sum of two columns in the table (thisTable).

The total_thisTable is the number of observations in the table (hight(thisTable)

[Voss]

Whelp, i = [] was my best guess, since it was consistent with all the observations. Of course, I had to speculate that it was happening within a script, since I can't see the whole code.

If you would like us to solve this problem, please share the complete code (and ideally attach the files too and describe the relevant directory structure). Evidently something non-obvious is going on, or else someone would've pointed it out by now.

Answer 1

[dpb]

As @Benjamin pointed out, for starters fix the loop construct and indexing...

...
for i=1:numel(fileList) 
  fullFileName = fullfile(fileList(i).folder, fileList(i).name);
  ...

That will at least run the loop and address the files in the returned directory structure, assuming there were some that matched the wildcard string.

W/o knowing something about the file structure and seeing the actual rest of the code, we can do no more than guess otherwise, but the above lines as posted would definitely not do what you're expecting them to do...

[Alix]

as I replied and changed in the code i posted, it was a typo.

the rest of the code is a bit long, but as I just replied to @Benjamin, the variable 'percent' is calculated correctly for each iteration, and the if statement works fine I believe as the warning message is being spit out for the iterations that meat the statement. The only thing that doesn't happen is that the Teapot_name is not inserted at the end of the warning message, or added to x.

What other information would you need?

[dpb]

Enough to be able to reproduce the problem, simply put. We can't see your terminal from here, nor can we reproduce what we cannot see--and we don't have data nor the actual complete code to be able to know what happens...and the Crystal Ball Toolbox is still to be released.

Execute

dbstop on error in WHATEVERISYOURCODENAME

and then look at what are the variables at the time. As the error message shows, you DO have an empty element somewhow; how/why we can't tell because of the above.

Substitute your function for "WHATEVERISYOURCODENAME" above, of course.

Answer 2

[Voss]

Note that if i gets set to the empty array [] somehow (e.g., in a script called from the loop), we get the error:

Teapot_names = {'tp1' 'tp2' 'tp3'};
x = {};

i = [];

warning('exceeded threshold by %s', Teapot_names{i}); 

Warning: exceeded threshold by %s

try
    x{i} = Teapot_names{i};
catch ME
    disp(ME.message);
end

Unable to perform assignment with 0 elements on the right-hand side.

So maybe that's what's happening.

And in the other test case, x remains all zeros, like you saw:

x = zeros(1,3);
i = [];
x(i) = 1;
disp(x)

     0     0     0

[Alix]

thank you for the examples!

I am not using a script, and i'm not calling an empty array i=[] anywhere.

the sum_of_some_variables_from_current_table is just the sum of some columns in the table (thisTable).

The total_thisTable is the number of observations in the table (hight(thisTable).

I did add your example to the for loop

try
    x{i} = Teapot_names{i};
catch ME
    disp(ME.message);
end

just to see what happenes. and I do get:

Unable to perform assignment with 0 elements on the right-hand side.

[Voss]

For what it's worth I added the try/catch in there just to get the code further down to execute, since execution will stop when an error is encountered. But having the try/catch there allows the code to continue to the next iteration of the loop, which may be useful, e.g., does the same error happen next time through the loop? etc.

[Alix]

Thank you so much for all your time and help @Benjamin @dpb. I have no idea why but after restarting my laptop this morning and clearing out all the workspace in matlab it worked. ...Really sorry to have put you through all this, I did learn a lot from your replies and examples though! Thank you for being so kind and responsive =)

[dpb]

"WhY" is undoubedly because it is a script and not encapsulated in a function or you're using global variables so that whatever is hanging around in the workspace is there still instead of having a clean workspace inside a function.

Which illustrates why it is so important to post something that is actually reproducible...and a complete example that does reproduce the problem.