Is there a way to create a custom cvpartition?

7 visualizaciones (últimos 30 días)
Christian el 14 de Jun. de 2021
Respondida: Giovanni el 9 de En. de 2024
I'd like you use crossvalidation in a model with a custom set of training/test sample vectors.
To use some of the built-in functionalities of Matlab I'd like to pass the input via the CVPartition Name/Value input parameter. Is there a way to define a custom set of train/test indices using cvpartition?

Respuesta aceptada

Shraddha Jain
Shraddha Jain el 25 de Jun. de 2021
Hi Christian,
As of now, the functionality to use custom indices in the cvpartition object is not available in the Statistics and Machine Learning toolbox. It might be considered in a future release.
  3 comentarios
Giovanni Attolico
Giovanni Attolico el 13 de Oct. de 2021
It seems to me that the indices are stored inside the cvpartition variable but they are read-only using the functions "training" and "test". While waiting for the "future release", it is not possible to define two functions "settraining" and "settest" that write indices in the variable? That would be really useful in many situation to allow the use of the resulting variable in all the situations where the current random partition can be used ...
Christian el 13 de Oct. de 2021
I had the same thoughts on this. However, I stopped trying to alter the code in cvpartition after a few attempts and ended up in writing a complete custom approach for crossvalidation procedures. In addition to custom indices you're also more flexible with the used metrics and and and...

Iniciar sesión para comentar.

Más respuestas (1)

Giovanni el 9 de En. de 2024
Custom partition has been recently introduced in MATLAB r2023B, but I suspect it does not work as expected since cross validation results from 2 neural netwroks differ even if trained and validated with the same folds (cv partition using the custom partition options with indices generated by crossvalind) and the networks are trained with the same parameters.
clear variables
close all
tab = readtable("three_selected_feature.xlsx"); %features dataset
featuresTab = tab(:,2:end-1);
feature = table2array(tab(:,2:end-1));
y = table2array(tab(:,end));
for i = 1:6 %repeat for different training test splitting.
cTest = cvpartition(y,'HoldOut',0.2,'Stratify',true);
train = training(cTest);
y_train = y(train);
testing = test(cTest);
cvIndices = crossvalind('Kfold',size(y_train,1),5);
cv = cvpartition("CustomPartition",cvIndices);
% cv = cvpartition(y(train),"Resubstitution");
features_scaled = normalize(feature,'zscore');
modelNET1 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
modelNET2 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
%models trained with the same parameters and features and crossvalidated
%using custom partitions. cv is based on cvIndices generated by the
%crossvalind function. For the same "i" (for loop) results of the two
%models should be the same, but they differ.
% cvmodelLDA1 = crossval(modelLDA1,'CVPartition',cv);
lossNET1(i) = kfoldLoss(modelNET1);
accNET1(i) = 1 - lossNET1(i);
% cvmodelLDA2= crossval(modelLDA2,'CVPartition',cv);
lossNET2(i) = kfoldLoss(modelNET2);
accNET2(i) = 1 - lossNET2(i);





Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by