Different results from ranova and JASP

Dear all, I am struggling to make sense of ranova with multcompare vs JASP (a statistical program). They are providing different results.

What I have is a table where rows correspond to subjects, and columns correspond to conditions. I want to compare each condition to each other condition. Therefore I am using ranova and multcompare.

load('data.mat','resultsAll')
rTable = array2table(resultsAll); 

varNames = strsplit(string(num2str(1:size(resultsAll,2)))); 

withinDesign = table(varNames','VariableNames',{'State'});
withinDesign.State = categorical(withinDesign.State);

conditionString = sprintf('%s-%s ~ 1',rTable.Properties.VariableNames{1},rTable.Properties.VariableNames{end}); 

rm = fitrm(rTable,conditionString,'WithinDesign',withinDesign);
fullTable = ranova(rm,'WithinModel','State');
disp(fullTable)

                          SumSq      DF     MeanSq        F         pValue      pValueGG      pValueHF     pValueLB 
                         ________    __    _________    ______    __________    _________    __________    _________

    (Intercept)           0.43612     1      0.43612    26.656     0.0066842    0.0066842     0.0066842    0.0066842
    Error                0.065443     4     0.016361                                                                
    (Intercept):State     0.43911     5     0.087821    20.282    3.2733e-07    0.0019177    0.00014406     0.010793
    Error(State)         0.086601    20    0.0043301                                                                

c = multcompare(rm,'State');
disp(c); 

    State_1    State_2    Difference     StdErr       pValue       Lower         Upper  
    _______    _______    __________    _________    ________    __________    _________

       1          2           0.3161     0.056406     0.02833      0.048612      0.58359
       1          3          0.25331     0.059127    0.069544     -0.027085       0.5337
       1          4          0.33228     0.056094    0.023317      0.066273      0.59829
       1          5           0.3353     0.067885    0.043651       0.01338      0.65723
       1          6          0.33858     0.068912    0.044434      0.011784      0.66537
       2          1          -0.3161     0.056406     0.02833      -0.58359    -0.048612
       2          3        -0.062793     0.040148     0.65386      -0.25318      0.12759
       2          4         0.016182    0.0072343     0.37993     -0.018125     0.050488
       2          5         0.019203      0.01907     0.89461      -0.07123      0.10964
       2          6         0.022477     0.019813     0.84709     -0.071481      0.11643
       3          1         -0.25331     0.059127    0.069544       -0.5337     0.027085
       3          2         0.062793     0.040148     0.65386      -0.12759      0.25318
       3          4         0.078975     0.037339     0.42181     -0.098094      0.25604
       3          5         0.081996     0.035042      0.3474      -0.08418      0.24817
       3          6         0.085269     0.034084     0.30158     -0.076361       0.2469
       4          1         -0.33228     0.056094    0.023317      -0.59829    -0.066273
       4          2        -0.016182    0.0072343     0.37993     -0.050488     0.018125
       4          3        -0.078975     0.037339     0.42181      -0.25604     0.098094
       4          5        0.0030213     0.016544     0.99994     -0.075434     0.081477
       4          6        0.0062946     0.017727     0.99861     -0.077772     0.090361
       5          1          -0.3353     0.067885    0.043651      -0.65723     -0.01338
       5          2        -0.019203      0.01907     0.89461      -0.10964      0.07123
       5          3        -0.081996     0.035042      0.3474      -0.24817      0.08418
       5          4       -0.0030213     0.016544     0.99994     -0.081477     0.075434
       5          6        0.0032733    0.0025707     0.78816    -0.0089175     0.015464
       6          1         -0.33858     0.068912    0.044434      -0.66537    -0.011784
       6          2        -0.022477     0.019813     0.84709      -0.11643     0.071481
       6          3        -0.085269     0.034084     0.30158       -0.2469     0.076361
       6          4       -0.0062946     0.017727     0.99861     -0.090361     0.077772
       6          5       -0.0032733    0.0025707     0.78816     -0.015464    0.0089175

On the other hand, JASP gives very different results for a repeated measures ANOVA.

I get the same F value of 20.282.

But the p-values in multiple comparisons is very different.

I am using all the default parameters of JASP.

Answer 1

[Shivansh]

Hi Joshua!

It looks like you are comparing the results from MATLAB and JASP. As you have mentioned, the "f-value" and "df" are similar in both cases.

For the pairwise comparison using "multcompare", I can observe that the mean difference is also consistent across the cases. The difference in standard error will lead to a difference in p-values and as a result, lower and upper bounds will also be different. There might be the case that the implementations in MATLAB and JASP use slightly different approaches for pooling variance or handling within-subject correlations, especially in the context of a repeated measure, affecting "StdErr", which in turn affects p-values and confidence intervals.

You can refer to the following link to know more about the "ranova" and "multcompare" functions in MATLAB:

https://www.mathworks.com/help/stats/repeatedmeasuresmodel.ranova.html.

https://www.mathworks.com/help/stats/multcompare.html.

I hope it helps!