How to Extract ROI on a signal with additional rows before and after threshold

I have data with the following details :

a) Three columns (Time, Signal 1, Signal 2) and 950000 rows.

b) My Threshold (Th) = 0.5

c) RT (Raising Threshold) = FT (Falling Threshold) = Th

I would like to extract the ROI between RT and FT for both signals simultaneously, but I also would like to include 300 rows Before RT (BRT) and 300 rows After FT (AFT).

Until now, I could extract only the values that are greater than my Threshold for S1 (Signal 1), but it was not enough, as it does not include S2 (Signal 2) and BRT and AFT either.

If someone could help, I would greatly appreciate it.

Thank you in advance.

Answer 1

[dpb]

It's gotten too convoluted -- yes, the original answer does, as noted in the last comment, only return the first/last -- your original post/illustration didn't mention that there are multiple transients...as it went on to say, one has two ways to then approach it; I took the latter here...

load A A
figure
hL=plot(A(:,2:3)); xlim([0 1E4]); yline(th)         % just to see what's we got...
tmp=int8([0; diff(A(:,2)>th)]);                     % the engine -- find the transition points
hold on
hT=plot(tmp);                                       % show the flag variable, too...
ix1=find(tmp==1);                                   % now find +ive transitions...
plot(ix1,th,'*r')                                   % show them on the plot, too...
ix2=find(tmp==-1);                                  % then the negative
plot(ix2,th,'xr')                                           
ix1=ix1-PreTrigNo;                                  % adjust to pre-, post-trigger ranges
ix2=ix2+PostTrigNo;
S=arrayfun(@(i1,i2) A(i1:i2,:),ix1,ix2,'uniform',0);    % and pick up the data in those sections
ROI=cellfun(@(c,i)i*ones(size(c,1),1),S,num2cell([1:numel(S)].'),'UniformOutput',0);    % create ROI grouping variable
tS=array2table([cell2mat(ROI),cell2mat(S)],'VariableNames',{'ROI','Time','S1','S2'});   % and put into a table

The above results in

>> [head(tS);tail(tS)]
ans =
  16×4 table
    ROI     Time        S1          S2  
    ___    ______    _________    ______
     1     3.1488     0.010937    1.3819
     1     3.1488     0.009375    1.3775
     1     3.1488     0.032813    1.3813
     1     3.1489       0.0125    1.3794
     1      3.149     0.009375     1.375
     1      3.149      0.00625    1.3725
     1      3.149    0.0078125     1.375
     1     3.1491      0.00625    1.3781
    10     4.6906    0.0078125    1.2581
    10     4.6907     0.010937    1.2628
    10     4.6908        0.025    1.2631
    10     4.6908     0.009375     1.265
    10     4.6909     0.015625    1.2641
    10     4.6909     0.010937    1.2691
    10     4.6909     0.017188    1.2744
    10      4.691     0.010937    1.2737
>> 

NB: However, the above also shows your time data don't contain enough significant digits to be unique; you'll need to either record those with more precision originally or regenerate new values knowing a fixed sample rate. They're perhaps immaterial, anyways...??? But, they're not useful as recorded here.

NB Second: It'll be far easier to handle using grouping variables and the splitapply work flow or rowfun and friends with the table rather than the suggested cell array horizontally

It's not a difficult task conceptually, one just has to approach it in logical manner thinking about how to identify the transitions above/below the threshold. Converting to a logical variable to make the values fixed and known to be 0|1 then lets you also know what the value of the difference is when the state changes, precisely. This is an easy case because the signals are so clean; you don't have to worry about noise or large real fluctuations that make the segmentation much more difficult -- the second channel would be more of a challenge, for example.

[Mino.G]

Yes, I forget to mention in my Q that the signal repeats sorry for that.

But what do you mean by the time data don't contain enough significant digits to be unique? immaterial? can you elaborate more on this point please?

I will investigate more about your proposed variables and the splitapply work flow or rowfun etc.. thanks

Regarding the first signal, S1, you are right it does look so clean, but this is just the beginning of the process; as I mentioned previously, the signal will get messier later at the end of the last phase, and this is the reason why I'm applying a flexible/adjustable ROI cursor of expansion.

I do agree with you regarding the simplicity of the signal S1 and that the second channel would be more of a challenge. But as those two signals were recorded simultaneously, my approach was to focus only on the simplest signal, in this case, S1, and then, based on the rules and the fixed logic used to extract S1 we can project these same points to extract S2 no matter the complexity of this last, as it's located within the ROI. What do you think? As this was the purpose of this Q is'n it?

[dpb]

It turns out I fell in trap of just looking at display time, not actual values...

>> all(diff(tS.Time))
ans =
  logical
   1
>> 

shows they are all unique-enough to not actually have a zero difference; it just looks like it when looking at the table output format with format short doesn't have sufficient precision to show a difference.

As for the last, it's all up to what you need -- the second trace follows the first if you find sufficient posttrigger data to include as much of it as is needed for whatever is to be gleaned from the data, that would appear to be adequate.

[Mino.G]

I got your point, I'm afraid I need to review the sampling parameters for the next experiment.

All I applied is the basic rule:

oscilloscope real-time sample rate = 3 to 4 times oscilloscope's bandwidth

If I may ask, what do you recommend for improving the quality of my data in terms of sampling parameters, for example, or in other words, what will give us the precision that you are looking for?

Thank you so much dpb for you help and for the interesting discussion that we had to solve this question.

[dpb]

Since you're apparently capturing the whole transient and post-processing, it's not a critical decision a priori; you can always redo the selection with different values for the retained number of samples.

Given that there apparently are a lot of these in any one test and the tests are apparently being run repetitively, with the extremely large amount of data that will produce you may well want to look into a technique to ensure the second trace has reached equilibrium so you can tell whether the sample size was, indeed, large enough or not.

That all depends on just what it is that is of significance here which, of course, we've no idea about.

[dpb]

I have no way to comment/know anything at all about the data collection parameters since, to quote Sgt Schultz "I know NOTHINK!" about the process under study.

From the traces you've shown here the sampling rate of 20 kHz seems reasonable -- if you blow up the time axis to expand one transient, the rise and fall occurs within just 2-3 sample periods, but there are one or two samples in that time -- unless the fine structure inside the transient (which I presume must be an input of some sort to the DUT) is significant, you could probably safely cut the sampling rate in half or at least by 1/4 or 1/3 without sacrificing much since the response trace has a much slower frequency response.

The only real advantage to this would be the size of the output files but if your systems can handle the ones you're getting now, then it wouldn't seem to matter; you could always choose to decimate the files before processing, but you can't get back data you didn't collect in the first place.

If, indeed, the whole secondary transient until it reaches essentially steady-state isn't of real interest and your 'scope supports it, you could alternatively set it up to trigger on the transient with a set pretrigger hold time and also set a fixed capture/record time and essentially do the same thing as doing now except only collect the data around the impulses.

That all just depends on the capabilities of your instrumentation and how drastic/expensive the loss would be if you didn't capture the whole transient. Those are decisions that only you/your organization has the ability to make.

Answer 2

[dpb]

Revised to match the figure and new understanding of intent to isolate one transient and select ROI around it -- the second trace is immaterial in setting the ROI limits; however, one will have to somehow ensure the ROI pre- and post-trigger intervals are wide enough to capture the entire ROI for the second trace as well -- in the later figure there appears to be a transient before S2 settles down to its final SS value that isn't captured in the shown ROI after S1 has returned to its baseline. This is a different kind of search than thresholding...

But, for the Q? as asked, it's pretty simple

PreTrigNo=300;
PostTrigNo=300;
tmp=int8(A(:,2)>Th);                % convert to logical, save memory with int8
ix1=find(tmp,1)-PreTrigNo;          % +ive crossing (first "1"); subtract pretrigger 
ix2=find(tmp,1,'last')+PostTrigNo;  % -ive crossing (+1, -1) add posttrigger
S=A(ix1:ix2,:);                     % select all the array between
clear tmp

To be completely robust, one should check that the result of the find operations are >=1 or <=size(A,1) BEFORE trying to address the S array with them; that's where the previous failed because there were no elements that matched for the second trace if it looked like the example figure.

[Mino.G]

First, thank you for your answer, but it's not working.

I'm receiving the following Error:

Error using max

Option must be 'omitnan' or 'includenan'.

In addition, looking at my workspace I'm having ix1= - 299 . (Just a reminder my dataset A= 950000x3 double) (Time, Signal 1, Signal 2)

for the ROI Extraction I'm not having the whole points that I want between (RT + 300 rows before) and (FT + 300 rows) I'm having only somthinmg like index points. such as ix1.

The final output S shuold include the ROI (all data points between the two threshold RT and FT plus 300 datapoint f )

Let me add some more information:

% Threshold (Th)
Th=0.5;
% Threshold BRT (Befor Raising Threshold)  that equal to 300 rows
PreTrigNo=300;
% Threshold AFT (After Falling Threshold) that equal to 300 rows
PostTrigNo=300;

% Finding logocal index of BRT (ix1 in this case) and AFT (ix2) ?
ix1=min(find(A(:,2)>Th,1),find(A(:,3)>Th,1))-PreTrigNo;
ix2=max(find(A(:,2)<Th,1),find(A(:,3)<Th,1),'last')+PostTrigNo;

% Extract the ROI? 
S= A(ix1:ix2,:);

I will apreciate if you can add some explanation within the code, in order to to facilitate the understanding of your approach.

Thank you again

[dpb]

There's a misplaced parenthesis in the second find to look at the end -- and you caught the missing comma in the indexing expression in the first...

% first location >Th in either column 2 or 3 less the pre-trigger count
ix1=min(find(A(:,2)>Th,1),find(A(:,3)>Th,1))-PreTrigNo;             
% last location >Th in either column 2 or 3 less the pre-trigger count
ix2=max(find(A(:,2)<Th,1),find(A(:,3)<Th,1),'last')+PostTrigNo;

But, on reflection the last won't do what you're wanting -- it'll pick the very last point in the trace <Th, not the first that drops below it after the first.

Needs some more refinement to the logic. Before just trying something willy-nilly, it would help a whole lot to have an idea of what the signal looks like -- is it uniformly going to be above the threshold once it triggers and then drop again and stay down below the threshold once does?

[Mino.G]

I hope this attached figure will helps to bring more clarification and refinement to the logic.As I said previousely, I want to extract the whole ROI between curseur A and B, (which means including BRT and ART intervales ) for both S1 and S2 simultaneously for the whole series.

Note: Regarding the Error, I'm still getting the same Error even after correction.

[dpb]

Yeah, knowing that general shape does help -- shouldn't need terribly complicated logic.

See the updated Answer -- it's sill "air code" which means untested, but the idea should be clear...

I'll be unavailable rest of the day today, so you'll have to debug anything yourself...shouldn't be tough to do presuming you have sufficient memory (the only reason for the int8() for the logic variable to save memory)

[Mino.G]

Thanks dpb for your kind support. It's seems that shouldn't need terribly complicated logic, but the issue still persisting.

I'm getting the following error:

>> ix1=min(find(tmp(:,2)== 2,1),find(tmp(:,3)== 3,1))-PreTrigNo;

Index in position 2 exceeds array bounds. Index must not exceed 2.

While trying to look into the solution, I've come across the following function (see link below) do you think we can use it to solve this ROI problem?

Extend signal regions of interest to left and right - MATLAB extendsigroi - MathWorks United Kingdom

Thank you

[dpb]

I was very rushed the other day and didn't look at the figure carefully enough -- in fact, the second trace in the above doesn't cross the threshold and the ROI is based ONLY on the first signal -- I thought the intent/need was to isolate two transients at different triggering times -- remove the tests for the second channel entirely and all should be resolved.

I edited the Answer to match the problem...

Answer 3

[Mino.G]

Thank you for your update and feedback.

You are right; the second trace in the above doesn't cross the threshold, and the ROI is based ONLY on the first signal.

But I need to Extract S2 the same as S1 only between the cursor A and B. The idea here is to use S1 as a reference to also extract S2 at the same time.

These two signals represent two different sensors that operate simultaneously. I want to apply this ROI to improve the classification of my machine learning model.

The signals you see vary from "Normal" to "Anomalous" at the end of the manufacturing process.

Thus, the idea is to clean up the noise, minimize the data point, and extract only the informative zone (in this case, the area between the cursor AB) and label them correctly.

In addition, the threshold represents the crucial section that my learning algorithm will be focused on. However, when the manufacturing process approaches the end /"Anomalous" additional fluctuations appear in the signals from both sides, which I believe need consideration and that I want to include in the ML learning process. It is the reason why I want to include additional rows/points from both sides, what you called them (PreTrigNo and PostTrigNo)

What I want to obtain as the final result or what I want S to look like is the following attached figure.

[dpb]

"I need to Extract S2 the same as S1 only between the cursor A and B. ..."

Well, yes. That's EXACTLY what the revised Answer code will do -- excepting it was based on the original figure and Q? that didn't point out that the signal repeats.

Instead of the above with the 'first' and 'last' options, you'll need to either

1. apply the above iteratively with the starting point for each successive iteration after the end point for the previous, or
2. revert to the previous use of sign and find all crossings from -ive to +ive and then iterate over that collection.

The basic idea works as in the above code...if you want actual debugged code, attach a .mat file with the first few cycles of your data.

[Mino.G]

Thanks again, dpb for taking the time to help with this problem.

I don't have an error now, but the S output prints all without distinction.

In addition, I've tried to change the value of PreTrigNo and PostTrigNo to see if it is possible to extend or shrink the cursor area, but it seems working only on the left side.

As final result if S result is successful, I want to add a final line of code that built a vertical matrix where each row will hold only one ROI at a time (it will be very helpful for labeling and classification purposes)