Minimize error between data distribution and expected distribution

Hi all,

I have a 3 set of data which are expected to:

1) 1st data-block to approach a Gaussian distribution with mu = 0 and sigma = 1;

2) 2nd data-block to approach a Gaussian distribution with mu = 0 and sigma = .8;

3) 3rd data-block to approach a Gaussian distribution with mu = 0 and sigma = .5;

Each data-block has only a limited number of representations (generally between 2048 and 8192) and because of some filter effects drawn by the specific code I use, they will not exactly match the corresponding expected distribution.

The point is that, although what it implies in terms of manipulation, I want each data-block to minimize the discrepancy between actual and expected distribution. It's to be remarked that I won't increase the number of representations, due to some need I will not explain in detail.

Generally, the first data-block, respect to the normal Gaussian distribution, looks like the followinf figure:

I was thinking to use lsqcurvefit for this purpose.

What would you suggest?

Answer 1

[Wouter]

Do you know this function:

histfit

[PEF]

sure I know it, and it's not the answer: histfit only fit a certain distribution to your data: my problem is reversed.

Given the distribution, my data (which is already quite well-represetative of the expected Gaussian) has to be manipulated in order to better fit the desired distribution.

As you can see from the snapshot above, my data (blue-bars) are not exactly fitting the expected distribution (red line): hence, the data must be non-linearly handled in order to better match the desired bell-shape.

Is it clearer now?

[Wouter]

Still not very clear to me; it sounds a bit counterintuitive.

 1) you create your own data
 2)your data has noise
 3) you data does not meet your expectations (i.e. not a perfect Gaussian  distribution)
 4) you want to change your data?

And regarding the blue bars not matching the red line exactly; this is always the case with noisy data right? You could try to make bigger bins in your histogram to visually filter it out, but still they will be there.

PS: Sorry if I misunderstand your question :) It is rather confusing.

[PEF]

I try to reformulate the question once again:

1) I generate random numbers with randn(), generally around 4096 samples;

2) I treat this numbers with a piece of code;

3) because of point 2) and because of little number of samples, the above random numbers are somehow filtered;

4) Point 3) implies a loss of variance.

Therefore, if I expect my data to approach unitary standard deviation, this will not happen because of the above discussed filter smoothing effects. Hence, when plotting the expected distribution function (Gaussian with mu = 0 and sigma = 1) and the actual distribution, they will not match, cause the area under the two different distribution is not the same (as the figure clearly shows).

Therefore and in the end, I want to manipulate the data in order to better match the expected distribution and I think that is doable by using lsqcurvefit.

Hope it got clearer and thanks for the support ;)

[Shashank Prasanna]

I agree with Wouter, it does seem counter-intuitive, to modify your data to fit an ideal distribution.

In most situations the reason you fit distributions is so that you can generate random samples from it, and the fitted distribution is the closest Gaussian you can get.

I guess if you can provide some details about what kind of filtering you are doing that is inducing non normality, there may be a solutions that is better suited.

[Wouter]

You could try to change individual datapoints after your filteringset in order to update your datapoints; this will change the blue bars. For example; find a blue bar that is too high; change one of those datapoints into a value which lies in a blue bar that too low (compared to the red line). This does however changes your data and will render step 2)treat_with_piece_of_code useless.

However it makes more sense to find a better fit to the histogram; i.e. change the red line. Lsqcurvefit would only be useful if you would like to update the red line (fit)

[PEF]

I think that you started to get the point :)

The major concern is that I don't want to find the best fit to the data, but the best data fitting the standard normal distribution: for some reasons I need my data to fit gaussian distribution with mean 0 and sigma 1.

At the moment I'm proceeding this way:

 data = randn(4096,1);

 [f_p,m_p] = hist(data,128);
 f_p = f_p/trapz(m_p,f_p);

 x_th = min(data):.001:max(data);
 y_th = normpdf(x_th,0,1);

 f_p_th = interp1(x_th,y_th,m_p,'spline','extrap');

 figure(1)
 bar(m_p,f_p)
 hold on
 plot(x_th,y_th,'r','LineWidth',2.5)
 grid on
 hold off

 figure(2)
 bar(m_p,f_p_th)
 hold on
 plot(x_th,y_th,'r','LineWidth',2.5)
 grid on
 hold off

Now, I would proceed with calculating a scaling factor

 sf = abs(f_p_th,f_p);

and I consequently scale the data in accordance to the scale factor of the corresponding bin; for example:

if data(1) falls within bin(1) --> scale with sf(1) and so on.

I do think that my question is no counter-intuitive, it's only reversing the standard procedure of fitting a distribution to a given set of data.