Visibility flag on Keypoint Detector Object

Hello everyone,

I was wondering if anyone could help me understanding how to set up the training data for KeyPoint Detector Object (trainHRNETObjectKeypointDetector).

The help says:

"Keypoint locations, defined in spatial coordinates as an N-by-2 or N-by-3 numeric matrix with rows of the form [x y] or [x y v], respectively, where:

• N is the number of keypoint classes.
• x and y specify the spatial coordinates of a keypoint.
• v specifies the visibility of a keypoint."

I was wondering how to setup the visibility flag v .

Is it 1 when the key point is visible and annotated and 0 when it is not. Or same as Yolo 0 = key point out of view and not labelled; 1 = key point present but not visible; 2 = key point visible ?

Thank you very much for your help

Answer 1

[Vivek Akkala]

As mentioned in the doc, A value of 1 (true) indicates a valid keypoint and 0 (false) indicates an invalid keypoint.

[Marc]

Hi Vivek,

Thanks a lot for your help.

I indeed saw the documentation for the Detect function and tried implenting the same logic to the trainHRNetObjectKeypointDetector. Unfortunately I received the following error message. Also see below the code used.

Error using trainHRNetObjectKeypointDetector>iValidateData (line 222)

Expected input to be nonzero.

Error in

trainHRNetObjectKeypointDetector (line 27)

iValidateData(keypointDetector, trainingData);

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

%Code given in the trainHRNetObjectKeypointDetector function documentation

downloadFolder = tempdir;

dataset = helperDownloadHandPoseDataset(downloadFolder); %with helperDownloadHandPoseDataset as defined in the documentation

data = load(dataset);

handPoseDataset = data.handPoseDataset(1:100,:);

handPoseDataset.imageFilename = fullfile(downloadFolder,"2DHandPoseDataAndGroundTruth","2DHandPoseImages",handPoseDataset.imageFilename);

handPoseImds = imageDatastore(handPoseDataset.imageFilename);

%---Modification to add random visibility flag 0 or 1 to the original training data

trainingData = handPoseDataset(:,2);

keypoints = table2cell(trainingData);

for i = 1:100;

TD = keypoints{i,1};

extraDat = [];

for j = 1:21;

if rem(j,2) == 1;

extraDat(j,1) = 0;

else;

extraDat(j,1) = 1;

end;

end;

keypoints{i,1} = [TD extraDat];

end;

keypoints = table(keypoints);

%---

handPoseArrds = arrayDatastore(keypoints);

handPoseBlds = boxLabelDatastore(handPoseDataset(:,3));

trainingData = combine(handPoseImds,handPoseArrds,handPoseBlds);

keypointClasses = ["forefinger3","forefinger4","forefinger2","forefinger1", ...

"middleFinger3","middleFinger4","middleFinger2","middleFinger1", ...

"pinkyFinger3","pinkyFinger4","pinkyFinger2","pinkyFinger1", ...

"ringFinger3","ringFinger4","ringFinger2","ringFinger1", ...

"thumb3","thumb4","thumb2","thumb1","wrist"]';

handKeypointDetector = hrnetObjectKeypointDetector("human-full-body-w32",keypointClasses);

options = trainingOptions("adam", ...

MaxEpochs=20, ...

InitialLearnRate=0.001, ...

MiniBatchSize=16, ...

LearnRateSchedule="piecewise", ...

LearnRateDropFactor=0.1, ...

LearnRateDropPeriod=12, ...

VerboseFrequency=25, ...

BatchNormalizationStatistics="moving", ...

ResetInputNormalization=false);

[trainedHandKeypointDetector,info] = trainHRNetObjectKeypointDetector(trainingData,handKeypointDetector,options);