paired t-test result interpretation

hi all,

need a refresher course on t-test, I seek your help.

I have 2 algorithms to solve the same problem, the output of each algorithm produce a single number. This single number indicate the efficacy of the algorithm (lower the better).

In a simulation, I ran both algorithms for, say, 10000 times. At each time, inputs to both algorithms are the same; their ourputs are different, of course.

How do I know which algorithm gives a better results on average, and that their outputs are indeed different with significance? The means(std) of both output vectors are small, and the t-test results are:

[h,p]=ttest(x1, x2)

h =

1

p =

0.002082123196735

Questions: - am I using ttest the right way for paired t-test, and

- how do I interpret these results?

regards,

Answer 1

[Samayochita]

Hi Elvis,

I understand that you want to know if the “ttest” function has been used correctly and how to interpret the t-test results (h and p values).

Since both the algorithms were run on the same inputs each time, the outputs (x1 and x2) are paired, thus the use of “ttest” is correct.

The MATLAB function “ttest(x1, x2)” performs a paired t-test by default, testing if the mean difference between x1 and x2 is zero.

The t-test results could be interpreted as follows:

• h = 1 means the test rejects the null hypothesis at the default 5% significance level (p < 0.05).
• p = 0.00208 is the p-value, which is much less than 0.05.
• This means that there is a statistically significant difference between the outputs of the two algorithms.
• Since "lower is better" I would suggest checking the means of x1 and x2:

If mean(x1) < mean(x2): Algorithm 1 is better.

If mean(x2) < mean(x1): Algorithm 2 is better.

For more information on the “ttest” function, please refer to the documentation link given below:

https://in.mathworks.com/help/stats/ttest.html