How to evaluate the performance metrics Accuracy, precision, recall, f1Score on the tuned FIS?

In the example contained in the Fuzzy logic user guide documentation by mathworks, Tune Fuzzy Inference System at the Command Line, page 225, is it possible to use the confusion matrix to calculate Accuracy, precision, recall, f1Score? if not possible, how can I evaluate it using accuracy, specificity and sensitivity? I tried so many times to write a function that will calculate those performance metrics. Below is the sample of the Tune Fuzzy Inference System at the Command Line in page 225 of Fuzzy logic user guide documentation by mathworks. I tried something like this:

 [data,name] = loadGasData; 
    X = data(:,1:6); 
    Y = data(:,7);
    trnX = X(1:2:end,:); % Training input data set 
    trnY = Y(1:2:end,:); % Training output data set 
    vldX = X(2:2:end,:); % Validation input data set
    vldY = Y(2:2:end,:); % Validation output data set
    dataRange = [min(data)' max(data)'];
    fisin = mamfis;
    for i = 1:6
        fisin = addInput(fisin,dataRange(i,:),'Name',name(i),'NumMFs',2); 
    end 
    fisin = addOutput(fisin,dataRange(7,:),'Name',name(7),'NumMFs',64);
    figure 
    plotfis(fisin)
    options = tunefisOptions('Method','particleswarm',... 
    'OptimizationType','learning', ... 
    'NumMaxRules',64);
    options.MethodOptions.MaxIterations = 20; 
    rng('default') 
    runtunefis = false;
    % 
    if runtunefis 
        fisout1 = tunefis(fisin,[],trnX,trnY,options); %#ok
    else 
        tunedfis = load('tunedfismpgprediction.mat');
        fisout1 = tunedfis.fisout1; 
        fprintf('Training RMSE = %.3f MPG\n',calculateRMSE(fisout1,trnX,trnY));
    %% this is where I want to call and print the calculateMetrics function.   
    end
    plotfis(fisout1) 
    
    
   

I tried something like this:

function [accuracy, precision, recall, f1Score] = calculateMetrics(confmat)
%%confmat

% Extract TP, TN, FP, FN from the confusion matrix
TP = confmat(2, 2);
TN = confmat(1, 1);
FP = confmat(1, 2);
FN = confmat(2, 1);    
      % Calculate metrics
accuracy = (TP + TN) / (TP + TN + FP + FN);
precision = TP / (TP + FP);
recall = TP / (TP + FN);
f1Score = 2 * (precision * recall) / (precision + recall);
end

[Ahmad]

@Sam Chak !

@Walter Roberson !

[Walter Roberson]

https://datascience.stackexchange.com/questions/20157/how-do-you-calculate-precision-and-recall-using-a-confusion-matrix-in-matlab

Answer 1

[Sam Chak]

Hi @Ahmad

Most special performance metrics are user-defined, and only designers understand what they really mean. A simple test below shows that your performance metrics work in the sense that you coded them, as long as you supply a proper Confusion Matrix as the argument of the function.

Since the metric function works, the next question is how exactly you produce the Confusion Matrix (and plot it if you want)? I'm not an expert, but a simple search shows that you can produce the matrix using either the confusionmat() or evaluate() functions.

https://www.mathworks.com/help/stats/confusionmat.html

https://www.mathworks.com/help/vision/ref/imagecategoryclassifier.evaluate.html

%% a Matrix
confmat = magic(4)

confmat = 4×4

16 2 3 13 5 11 10 8 9 7 6 12 4 14 15 1

%% Call calculateMetrics() function to print accuracy, precision, recall, f1Score
[accuracy, precision, recall, f1Score] = calculateMetrics(confmat)

accuracy = 0.7941

precision = 0.8462

recall = 0.6875

f1Score = 0.7586

function [accuracy, precision, recall, f1Score] = calculateMetrics(confmat)
    %% confmat
    % Extract TP, TN, FP, FN from the confusion matrix
    TP = confmat(2, 2);
    TN = confmat(1, 1);
    FP = confmat(1, 2);
    FN = confmat(2, 1);    
    % Calculate metrics
    accuracy  = (TP + TN) / (TP + TN + FP + FN);
    precision = TP / (TP + FP);
    recall    = TP / (TP + FN);
    f1Score   = 2*(precision*recall)/(precision + recall);
end