How does the OverlapThreshold argument in balanceBoxLabels get passed along to the resulting blockLocationSet?

If balanceBoxLabels simply creates a blockLocationSet ("bSet"), how exactly does the OverlapThreshold argument/value specified in balanceBoxLabels get 'passed along' to the subsequent step of creating a boxLabelDatastore ("blds") based on the blockLocationSet, given that a blockLocationSet only contains the properties (1) Image number(s), (2) Block origin(s), (3) Block size and (4) Resolution level(s)?

For example, if I do:

bSet = balanceBoxLabels(boxLabels,blockedImages,blockSize,numObservations,OverlapThreshold=0.5)

This is supposed to specify that any bounding box that partially overlaps a block by >50% of the bounding box's size gets cropped to the block boundary, while any bounding box that partially overlaps a block by <50% of the bounding box's size gets discarded. So far, so good. But then when I do:

blds = boxLabelDatastore(boxLabels,bSet)

How does the OverlapThreshold value of 0.5 specified when creating "bSet" using BalanceBoxLabels get passed along to the creation of "blds" using boxLabelDatastore?

[Dominique]

As a follow-up, after running some tests, as far as I can tell the OverlapThreshold value is not passed on in any way to the resulting block location set and subsequent box label datastore; meaning that as far as I can tell it's a pointless feature that doesn't actually do what it's supposed to.

For example, even if set the OverlapThreshold to 1--which is supposed to result in discarding any and all bounding boxes that overlap block boundaries rather than cropping them--when I create a box label datastore with the resulting block location set, the box label datastore still contains cropped bounding boxes even though it is expected to contain only whole bounding boxes.

So either it's an impotent feature, or I'm somehow not utilizing it correctly.

Answer 1

[Aneela]

Hi Dominique,

The “boxLocationSet”, “bSet”, resulting from the “balanceBoxLabels” does not explicitly carry the “OvelapThreshold” as one of its properties.

When the “OverlapThreshold” in “balanceBoxLabels” is specified, the function uses this threshold to decide which bounding boxes to include or exclude in each block.

bSet = balanceBoxLabels(boxLabels,blockedImages,blockSize, numObservations,OverlapThreshold=0.5);

• This decision-making and adjustment process happens during the execution of “balanceBoxLabels”.
• The outcome is a “boxLocationSet” object, “bSet”, that contains information about Image numbers, block origins, block sizes, and the information about which bounding boxes to include.

When a “boxLabelDatastore” is created using:

blds = boxLabelDatastore(boxLabels,bSet);

• The “bSet” already incorporates the effects of the “OverlapThreshold” through its content, which blocks were created and which bounding boxes they include.
• The “boxLabelDatastore” function then uses this information to create a datastore of the cropped images and their corresponding adjusted bounding boxes.

For more information on “balanceBoxLabels”, refer to the following MathWorks documentation: https://www.mathworks.com/help/vision/ref/balanceboxlabels.html

[Dominique]

Hi Aneela,

Thank you very much for taking the time to respond to my question.

Here is the part I'm skeptical about:

"The outcome is a “boxLocationSet” object, “bSet”, that contains information about Image numbers, block origins, block sizes, and the information about which bounding boxes to include."

According to the MATLAB documentation, and as far as I can tell from manually inspecting the properties of box location sets generated using the balanceBoxLabels function, it does not in fact contain information about which bounding boxes to include; only the image number(s), block origins, block size, and resolution level(s). Consequently, and as mentioned in the follow-up comment to my initial question, I've even tested setting OverlapThreshold to 1 (i.e. this is supposed to specify no cropping of bounding boxes that overlap block boundaries), yet in the box label datastore subsequently created from the block location set, I'm still clearly seeing numerous cropped bounding boxes.

Since posting this question I've had the chance to perform quite a bit of further testing to try to figure what OverlapThreshold is actually doing (if anything), and this is the conclusion I've come to:

In the balanceBoxLabels documentation, under Algorithms, the key part seems to be: "The [randomly sampled] blocks without any objects are discarded."

Based on testing, it seems the only thing the OverlapThreshold is impacting is this very specific part of the process: if you set OverlapThreshold to 1, it will discard any randomly sampled block that doesn't contain at least one entire bounding box; and conversely keep any block that contains at least one entire bounding box, regardless of whether it contains additional bounding box(es) that overlap(s) the block boundaries--hence why you end up with cropped bounding boxes in the subsequent box label datastore despite setting the OverlapThreshold to 1. Decreasing the OverlapThreshold below 1 simply makes it keep (rather than discard) randomly sampled blocks that contain a single bounding box even it overlaps the block boundary, as long as the amount of overlap is greater than the specified OverlapThreshold.

So it does appear to do something, just not nearly as much as the documentation insinuates. At the end of the day, there seems to be no way to avoid ending up with cropped bounding boxes in the subsequent box label datastore, regardless of the OverlapThreshold value specified in balanceBoxLabels.

[Corey Kurowski]

Hey Dominique,

What you stated here is correct. The algorithm for balanceBoxLabels uses a weighted sampling based on the class's overall representation in the dataset. So for a series of blockedImages, with a specified block size, the algorithm will prioritize grabbing image blocks (with minor x/y shifts) that contain the least represented class. It will do this until the precalculated quota is met [based on total number of observations] - taking into account the overlap threshold. The algorithm only cares if at least one object from the given class is contained within the image block [and checks if the overlap threshold is met with that object]. All other bounding boxes in that image block are lower priority and may be cropped if they fall on the image block boundary. This process is repeated with the second least represented class and so forth.

The intent of the function is to bring forth as many under-represented classes as possible to "balance" the dataset, but it is not a perfect solution since very under-represented classes will be balanced to merely be slightly shifted versions of the original tiles. This algorithm serves us well when then is a slight under-representation, but when it is ~1:10 or so, it really starts to fail a bit.

When it comes to blockedImage processing, it can be beneficial to have partial objects on the image block bounds, as unless all images are uniform in size and careful calculation is made to include only full objects, a detector is likely to encounter similar partial objects at inference time. This is one of the reasons why we do not automatically discard those cropped boxes.

With that being said, it would be great to understand what parts of the documentation are unclear to you regarding this function, and we can use that feedback to improve clarity.

As an offshoot, please see the following example for a more custom design of detecting objects in an image block: https://www.mathworks.com/help/vision/ug/object-detection-in-large-satellite-imagery-using-deep-learning.html

[Dominique]

Hi Corey,

Thank you very much for chiming in and confirming my observations.

I agree with you that it's generally good for a model to be trained on partial objects because it's likely to subsequently encounter some during inference.

Regarding the documentation, I would say it's definitely not clearly stated that OverlapThreshold specifically impacts the process of deciding which randomly sampled blocks to keep vs. discard, and moreover that it solely impacts that specific process. As currently stated, you really get the impression that it gives full control over the extent of clipping of all boxes that end up in your eventual box label datastore:

"When the overlap between a bounding box and a cropping window is greater than the threshold, boxes in the boxLabels input are clipped to the image block window border. When the overlap is less than the threshold, the boxes are discarded."

As a suggestion, I think revising it as follows would make it clearer:

"When the overlap between at least one bounding box and a cropping window is greater than the threshold, all boxes overlapping the cropping window are clipped to the image block window border. When no box overlaps the cropping window by more than the threshold, the block is discarded."

Beyond that, since I've got your attention, I would add that I think the balanceBoxLabels function has the potential to be further developed and presented as a more versatile and powerful function. For example, despite its name and implicit focus on balancing imbalanced training sets, I've actually found it to be a generally useful function to sample network-input-size blocks from larger annotated images even in cases where there's just a single object class.

Furthermore, what would make it truly powerful is if it could be refashioned to work like a sort of object detection version of randomPatchExtractionDatastore (which is used for semantic segmentation tasks); whereby instead of randomly extracting blocks prior to model training and then being stuck with that finite set of blocks throughout training, it extracts an all new random sampling of blocks at each epoch, during training. I find this to be a great approach to counteract overfitting, which even in a sense redefines the very meaning of an "epoch", since the model virtually never sees any given randomly extracted image more than once throughout training.

[Corey Kurowski]

Hey Dominique,

Thanks for this feedback. Both of your noted additional functionalities are definitely something for us to consider in near future releases to help with a more robust deep learning training workflow. While we will likely keep balanceBoxLabels as a dedicated function, we may very well introduce additional functions that can assist in each of these cases.