Visibility flag on Keypoint Detector Object

Hello everyone,

I was wondering if anyone could help me understanding how to set up the training data for KeyPoint Detector Object (trainHRNETObjectKeypointDetector).

The help says:

"Keypoint locations, defined in spatial coordinates as an N-by-2 or N-by-3 numeric matrix with rows of the form [x y] or [x y v], respectively, where:

• N is the number of keypoint classes.
• x and y specify the spatial coordinates of a keypoint.
• v specifies the visibility of a keypoint."

I was wondering how to setup the visibility flag v .

Is it 1 when the key point is visible and annotated and 0 when it is not. Or same as Yolo 0 = key point out of view and not labelled; 1 = key point present but not visible; 2 = key point visible ?

Thank you very much for your help

Answer 1

[Vivek Akkala]

As mentioned in the doc, A value of 1 (true) indicates a valid keypoint and 0 (false) indicates an invalid keypoint.

[Marc]

Hi Vivek,

Thanks a lot for your help.

I indeed saw the documentation for the Detect function and tried implenting the same logic to the trainHRNetObjectKeypointDetector. Unfortunately I received the following error message. Also see below the code used.

Error using trainHRNetObjectKeypointDetector>iValidateData (line 222)

Expected input to be nonzero.

Error in

trainHRNetObjectKeypointDetector (line 27)

iValidateData(keypointDetector, trainingData);

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

%Code given in the trainHRNetObjectKeypointDetector function documentation
downloadFolder = tempdir;
dataset = helperDownloadHandPoseDataset(downloadFolder); %with helperDownloadHandPoseDataset as defined in the documentation
data = load(dataset);
handPoseDataset = data.handPoseDataset(1:100,:);
handPoseDataset.imageFilename = fullfile(downloadFolder,"2DHandPoseDataAndGroundTruth","2DHandPoseImages",handPoseDataset.imageFilename);
handPoseImds = imageDatastore(handPoseDataset.imageFilename);

%---Modification to add random visibility flag 0 or 1 to the original training data
trainingData = handPoseDataset(:,2);
keypoints = table2cell(trainingData);
for i = 1:100;
    TD = keypoints{i,1};
    extraDat = [];
    for j = 1:21;
        if rem(j,2) == 1;
            extraDat(j,1) = 0;
        else;
            extraDat(j,1) = 1;
        end;
    end;
    keypoints{i,1} = [TD extraDat];
end;
keypoints = table(keypoints);
%---

handPoseArrds = arrayDatastore(keypoints);
handPoseBlds = boxLabelDatastore(handPoseDataset(:,3));
trainingData = combine(handPoseImds,handPoseArrds,handPoseBlds);
keypointClasses = ["forefinger3","forefinger4","forefinger2","forefinger1", ...
    "middleFinger3","middleFinger4","middleFinger2","middleFinger1", ...
    "pinkyFinger3","pinkyFinger4","pinkyFinger2","pinkyFinger1", ...
    "ringFinger3","ringFinger4","ringFinger2","ringFinger1", ...
    "thumb3","thumb4","thumb2","thumb1","wrist"]';
handKeypointDetector = hrnetObjectKeypointDetector("human-full-body-w32",keypointClasses);
options = trainingOptions("adam", ...
    MaxEpochs=20, ...
    InitialLearnRate=0.001, ...
    MiniBatchSize=16, ...
    LearnRateSchedule="piecewise", ...
    LearnRateDropFactor=0.1, ...
    LearnRateDropPeriod=12, ...
    VerboseFrequency=25, ...
    BatchNormalizationStatistics="moving", ...
    ResetInputNormalization=false);
[trainedHandKeypointDetector,info] = trainHRNetObjectKeypointDetector(trainingData,handKeypointDetector,options);

[Andreas]

Hey Marc, I hope all is well. For the past week I've also ben trying to make the visibility work and I keep getting errors. Did you end up figuring out how to solve it? Or did you use a different method all together? Thanks!

[Vivek Akkala]

Marc, you don’t need to modify the training data to include a visibility flag. Currently, trainHRNetObjectKeypointDetector does not expect a visibility flag, which is why you encounter an error when a keypoint value of 0 is present in the training data. You can train the network directly using the available keypoints without the visibility information. During inference, the network applies a confidence threshold internally. Keypoints with scores below this threshold are discarded, and the remaining keypoints are treated as the visible ones.

[Vivek Akkala]

@Andreas, Could you please share more details about the issues you’re currently facing? Specifically, what do you mean by "making the visibility work"?

[Andreas]

I want to use HRNet to identify keypoints on aircraft, similar to how it is used in the available example with the spacecraft. However, in my dataset not all images have all keypoints labelled, as I want to have a detector that is trained in a variety of angles. Thus I need a way to handle the fact that in some images some keypoints are not visible.

The "trainHRNetObjectKeypointDetector" documentation states the following:

"Keypoint locations, defined in spatial coordinates as an N-by-2 or N-by-3 numeric matrix with rows of the form [x y] or [x y v], respectively, where:

• N is the number of keypoint classes.
• x and y specify the spatial coordinates of a keypoint.
• v specifies the visibility of a keypoint."

Thus, I assumed that I indeed can include images where not all keypoints are visible, however when I try to have a 7x3 matrix in my case, where visible points are assigned a 1 and not visible points a 0, I get a very similar error to Marc above.

Is this just a case of me defining visibility wrong, or is the documentation wrong and you can't train the detector unless all of your images have all of the keypoints labelled?

Thanks!