How to test if values are close togather using p-value (ttest)

I am testing the output of a low-power cell battery and then I measured the output after a while. I did multiple runs and the output was slightly different every time. I would like to calculate the P-value to show that my output measurements are reliable and consistent. The standard is to show a P-value of P < 0.05.

My approach (MATLAB):

mu = 2.366; % Population mean

sample=[2.180213,2.178298 ,2.310851 ,2.114255 ,3.012553 ,2.69234 ,2.079787];

n = numel(sample);

[h,ptest] = ttest(sample,mu,0.05,'right')

My p-vlue is always high. I think I am doing it wrong. I want to show that the numbers are "close together" and not "far apart". How do I do this?

Answer 1

[Adam Danz]

The ttest assumes that data come from a normal distribution and that a reasonable large sample size is used. Your sample data appear to violate both assumptions. Instead of using a parametric test, I highly urge you to use bootstrap confidence intervals that are free from these assumptions (but still requires a few samples, of course).

This demo uses bootci() from the Stats and Machine Learning toolbox to compute the 95% confidence interval of the sample based on the percentile method. It increases your sample size to 1000 by sampling with replacement to compute the 95% CI. If the expected mean (mu) is outside that interval, you can assume that the data come from a different distribution than the distrubtuion used to calculate the expected mean.

mu = 2.366;     % Population mean 
sample=[2.180213,2.178298   ,2.310851   ,2.114255   ,3.012553   ,2.69234    ,2.079787];

bci = bootci(1000, {@mean, sample}, 'Alpha', 0.05, 'Type', 'per')

bci = 2×1

2.1512 2.6599

isdiff = ~all(sign(mu-bci)==[1;-1]) % 0/false means mu is within the 95% CI of the sample

isdiff = logical
   0

If you do not have the Stats and ML toolbox or if you'd like to see what's going on in bootci, see this answer that computes 90% CIs using bootci and by computing the bootstrapped means and CI directly without any toolboxes.

[Danial Waleed]

does bootci represent the result in p-value?

[Danial Waleed]

Also, I was thinking of using [h,p,stats] = chi2gof(___)

[Adam Danz]

> Does bootci represent the result in p-value?

No, it's better than a p-value. It produces a 95% confidence interval which is similar to a p-value but without the baggage. Some well-respected scientific journals have even started banning the use of p-values without additional statistics or descriptions of error that do not rely on significance testing.

> Also, I was thinking of using ....

You're p-hacking. You're looking for a test that gives you the results you want rather than choosing a test based on the question you're asking and the test's null-hypothesis and assumptions. P-hacking is really bad (see this explanation which also explains why bootstrap CIs are much better).

[Adam Danz]

@Danial Waleed, if any of the answers to your questions helped solve the problems, please consider accepting the answers to indicate that they were successfull. This will help future visitors find viable solutions quickly.

Your question:

https://www.mathworks.com/matlabcentral/answers/?term=asked_by_id:237840

Answer 2

[Jeff Miller]

A small p value means that the data are incompatible with some underlying assumption/model. For example, you could potentially show that the variability among your measurements is small enough to rule out the assumption that the population standard deviation of those measurements is more than (say) 0.5 units.

A better approach might be to compute a confidence interval for the population variance of your measurements, based on the observed sample that you have. With your data, this calculator gives a 95% confidence interval for the variance among your measurements as ranging from 0.1331 to 0.2316. Taking the square roots of those values, the confidence interval for the population standard deviation of measurements is 0.265 to 0.481. So, you can say with p<.05 that the standard deviation among your measurements is less than .48 or so.