How can I test whether one classifier is significantly better?

I have a dataset consisting out of two classes. I try to classify the dataset using different classifiers. I want to test whether the difference in the results of the classifier are significant or not. For classifier A=0.7420 and B=0.7210. Anybody knows an appropriate test and how to code it in matlab?

Answer 1

[Akshat]

Hi @Stef,

To test whether the difference in performance between two classifiers is statistically significant, you can use statistical tests designed for comparing classification results. One common approach is to use McNemar's test, which is suitable for paired nominal data, such as classification results.

Here's how you can implement McNemar's test in MATLAB:

b = 30; % Instances where A is correct, B is not
c = 40; % Instances where B is correct, A is not

% Calculate McNemar's test statistic
chi2 = (b - c)^2 / (b + c);

% Calculate p-value
p = 1 - chi2cdf(chi2, 1);

A low p-value (typically < 0.05) indicates a significant difference between the classifiers.

For more advanced comparison, consider using cross-validation and paired t-tests on the accuracy scores across folds.

Hope this helps!