Loop to replace outliers with NaN extremely slow
11 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
TL
el 21 de En. de 2022
Comentada: Star Strider
el 22 de En. de 2022
I want to replace outliers with NaN in a large table (> 3 standard deviations from each column's mean) and my code works in principle but is incredibly slow, i.e. still not done after 10 minutes. The table size is about 2000x150. Is there a faster way, maybe without the loop, and could someone tell me what is wrong with my version?
%Version 1: loop through column names
var_list = mytable.Properties.VariableNames(4:140)
for i = 1:length(var_list)
mytable.(var_list{i}) = filloutliers(mytable.(var_list{i}),nan,'mean','ThresholdFactor', 3)
end
%Version 2: loop through column indices
for i = 4:140
mytable(:,i) = filloutliers(mytable(:,i),nan,'mean','ThresholdFactor', 3)
end
2 comentarios
Mathieu NOE
el 21 de En. de 2022
hello Tanja
just a question : is removing the outliers the "real" need or a smoothing approach would also fit your needs ?
Respuesta aceptada
Star Strider
el 21 de En. de 2022
The way the table addressing is coded is likely the problem.
I’m not certain, however using parentheses () addresses the table (or variables as individual table arrays), while curly braces {} address the variable contents themselves.
So for example
mytable(:,i) =
creates a new table as ‘mytable’ while
mytable{:,i} =
addresses only the contents of the variable.
Again, I’m not certain wht the problem is, however experimenting with changing the addressing method could provide a solution.
Also, I’m not certain if the loop is even necessary, since filloutliers appears to work on arrays as well as vectors, and operates on each column separately, according to the documentation.
.
4 comentarios
Más respuestas (0)
Ver también
Categorías
Más información sobre Loops and Conditional Statements en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!