Blocked images larger than a certain size will not render in Image Labeler

I am endeavouring to train a deep learning object detection network. For training data, I have large photomosaics that I would like to label using the Image Labeler app, and then ultimately create a Blocked Image Datastore with a Block Location Set argument to sample blocks from the large training images that are the appropriate input size for the network.

I appreciate that the Image Labeler app can import large images like these as blocked images, but I've found that above a certain size it seems to be unable to render them, and consequently I am unable to work with them in the Image Labeler.

Specifically, when things are working properly, if I import a blocked image into the lmage Labeler and proceed to zoom in from its initial zoom level, after several seconds at most the resolution of the image updates to the new zoom level. But above a certain image size, when I zoom in it just remains 'pixelated' from the previous zoom level's resolution, the resolution never updates to the new zoom level, and eventually after some time (several or more minutes) the MATLAB command window reports a graphics timeout error.

As it stands, my largest training image to not have this issue in Image Labeler is 15887 x 33197 pixels (≈ 527 million pixels), but as of my next size up image (18682 x 34701 pixels ≈ 648 million pixels), it will not render at higher zoom levels.

I'm not sure precisely which hardware resources are used to render images in Image Labeler, but my PC has 64 GB of RAM as well as an NVIDIA GeForce RTX 3060 with 12 GB of VRAM (and yes, I have the Parallel Computing Toolbox).

Any advice/insight?

[Matt J]

15887 x 33197 is an usually large image. Is there a reason you don't resize?

[Dominique Chabot]

Hi Matt,

Thanks for your response.

I can't downsample because these are images of massive cliffs captured from a helicopter in which I'm labeling/detecting relatively tiny birds nesting on the cliffs—I really do need the full resolution.

I could of course break the images up into smaller images but that would make my workflow less efficient; and aside from that I'm just puzzled by this apparent limitation, because for years I've been working with even much more obscenely large images in GIS software (i.e. ArcGIS, QGIS)—which have functionalities equivalent to MATLAB's blocked images for working with very large images—and have never had a problem zooming in and viewing them at full resolution, no matter how large the image. For example, I just opened a 349,141 x 135,260 pixel image in QGIS and am zooming way in—no problem, really snappy rendering.

[Matt J]

Have you given the images to the Labeler app in the form of a blockedImageDataStore? If so, what is the block size? If not, in what form have you given them, and what is the numerical format of the pixel values?

If they are not uint8, it is very easy to see a 648 million pixel RGB image straining the resources of a 12 GB card.

[Matt J]

Maybe you could show us the details of the blockedImage representation, e.g.,

blockedImage with properties:

Read only properties

Source: [128x128x27 uint8]

Adapter: [1x1 images.blocked.InMemory]

Size: [128 128 27]

SizeInBlocks: [1 1 1]

ClassUnderlying: "uint8"

[Dominique Chabot]

Thank you for your additional replies.

Regarding how I've given the images to the Image Labeler, it doesn't seem to be able to import a blockedImageDatastore from the workspace, only a regular imageDatastore (and when I put the images into a regular imds and import them into the Labeler, it doesn't even suggest converting them to blocked images; just goes ahead and imports them as regular images, so that's a non-starter), so I've been importing them from file (in which case it does suggest converting them to blocked images, though you don't get to specify a block size; my understanding has been that it automatically sets the block size based on the size of the image and maybe the zoom level).

As for the image properties, here are those of the smallest image I have that gives me rendering issues in the Image Labeler, after converting it to a blocked image in the workspace:

blockedImage with properties:

Read only properties

Source: "G:\Imagery for processing\2022 CWS seabird cliff surveys\MATLAB\Training images\DSC02051 Panorama.jpg"

Adapter: [1×1 images.blocked.GenericImage]

Size: [34701 18682 3]

SizeInBlocks: [1 1 1]

ClassUnderlying: "uint8"

In any case, I thought the whole point of blocked images--as described by the Image Labeler app itself--is to "load only the data being looked at" so as not to strain the memory in the PC or graphics card...

Answer 1

[Ashish Uthama]

Thanks for sharing the blockedImage representation - This is a JPG file. Which means there is really no way to do blocked IO (i.e read only parts of the file). Since its a compressed format, the whole file needs to be uncompressed to access any small region of the image.

You could convert this file (one time hit) using something like: (choose a blocksize the same as the image size you want to train your network on, that will help IO performance down the line)

makeMultiLevel2D(bim, BlockSize=[1024 1024], Scales=[1 .1], OutputLocation="withOverview.tif", Adapter=images.blocked.TIFF)

Loading this TIFF file into the app should hopefully give you better performance. It will also help when you get to the blockedImageDatastore workflow.

[Dominique Chabot]

Thank you, and very interesting. So instead of trying your code, what I did was I went back into the image stitching software used to create these large photomosaics in the first place, and re-exported that 18682 x 34701-pixel one as a TIFF instead of a JPEG, resulting in a 731 MB file (compared to 158 MB for the JPEG version); and though it takes appreciably longer to import into Image Labeler, once it's loaded it works perfectly, no rendering issues, pretty snappy navigation (note that it does have LZW compression though; is that a problem in some way?). Even my largest image (28308 x 57914 pixels, 1.1 GB TIFF), which takes quite a long time to import, works.

You'd think this JPEG limitation would be mentioned in the blocked image documentation somewhere? Or that Image Labeler wouldn't dumbly suggest converting a JPEG into a blocked image if it won't work properly? Are there certain blocked image functionalities that work with JPEGs while others don't? What about simply running the 'detect' function on blocked JPEG images with the block size set to the input size of my eventual trained network?

The only other thing I'm wondering now is I have 25 of these very large photomosaic training images, and given how huge they collectively are and the amount of time they seem to take to import into Image Labeler, is it wise to try to put them all in a single labeling project? Or would it be wiser to create a separate labeling project for each image, separately export a groundTruth object from each project to the workspace when I'm finished labeling, and somehow combine the boxLabelDatastores extracted from each groundTruth object into a single boxLabelDatastore? Maybe like this?

combinedBlds = combine(blds1,...,blds25,ReadOrder="sequential")

Will this combinedBlds be able to be properly 'mapped back' to my array of blocked training images if I number the blds's (e.g. blds1, blds2, ..., blds25) in the same order as the filenames of their corresponding images, i.e. the same order as the images in the array?

[Ashish Uthama]

Many things to unpack :)

TIFF vs JPEG - JPEG is 'lossy' compression, details are lost. Which is why it can compress much better than LZW in TIFF (lossless). Depending on your experiment, the loss in quality may not matter. There is an option to have each tile in TIFF comrpessed with JPEG which gives you the best of both worlds. Not sure if your image stitching software has than option to expore. You can create that with blockedImage/write though - this should reduce file size significantly.

adapter = images.blocked.TIFF;
adapter.Compression = Tiff.Compression.JPEG;
write(bim,'out.tif', BlockSize = [1024, 1024],Adapter=adapter)

About long time to import - if your generating software has a capability to create pyramid versions (which is what the makeMultiLevel2D function does) that would help significantly. TIFFs support this format, JPG do not. Having a low resolution version that can be quickly loaded helps the labeller app quickly show you the overiew and only load the finer details when you get down to the block level.

Good point about the suggestion from Image Labeler. That catch is that the boundary of when a JPEG file is 'too large' is highly dependent on a user's machine, ability to change format and a user's patience :) We can definitely add more color to that suggestion.

Not familiar, but using a single labelling project should be the right thing to do.

Answer 2

[Matt J]

I could of course break the images up into smaller images but that would make my workflow less efficient

Breaking them up for the purposes of network training shouldn't impact your workflow. After the network is trained, you can use the original image size for inference.

[Dominique Chabot]

Yes, I suppose that would work, though I'm still curious whether this is a known limitation for blocked images in the Image Labeler app (no image size limitation appears to be mentioned in the app's documentation) or whether it may have something to do with my PC's hardware being insufficient.

Answer 3

[Matt J]

my understanding has been that it automatically sets the block size based on the size of the image and maybe the zoom level

Perhaps it should, but my guess would be that it chooses it no differently from how the blockedImage constructor chooses it and, as you can see from the object properties, the default SizeInBlocks that it has chosen is [1,1,1]. In other words, the software is assuming that a [34701 18682 3] chunk of data is just fine for the memory resources of your computer. No need to split it up.

Are you able to import a blockedImage object to the app? If so, you should probably import to the workspace as a blockedImage object with the blockSize chosen by you directly, e.g.,

bim = blockedImage(Source, blockSize=[1000,1000]);

[Matt J]

and when I put the images into a regular imds and import them into the Labeler, it doesn't even suggest converting them to blocked images

What if you use the ReadFcn option of the imds to convert the image into a blockedImage (of a size chosen by you)?

imds=imageDataStore(location, ReadFcn=@(fn)blockedImage(fn,[1000,1000]) );

[Dominique Chabot]

Unfortunately there seems to be no way to know the blockedImage properties set by the Image Labeler when you import an image from file as a blocked image--it simply asks whether you want to import it as a blocked image, you click 'yes', and it gets imported with unspecified properties.

And unfortunately there also seems to be no way to first create a blocked image (or blocked image datastore) with specific properties in the workspace and then import it directly in that form from the workspace to the Image Labeler.

As for your crafty suggestion to try to make it work through a regular image datastore's ReadFcn property, I'm able to successfully create the imds in the workspace, and Image Labeler recognizes it as an importable object, but upon trying to import it I get the following error:

Expected Image to be one of these types:

double, single, int16, uint16, uint8, logical

Instead its type was blockedImage

So I'm not sure if we've exhausted all ideas at this point, but in any case I do very much appreciate your time and effort on this question--I'm getting the sense that it's a fairly tricky one!

Worst case, as previously discussed, I'll just break up the large images.

[Matt J]

I think we may have exhausted all avenues, but you could always try Tech Support, to see if there's something we've overlooked. If nothing else, they should probably be informed about these limitations.

Answer 4

[Matt J]

One other possible workaround is to convert the image to grayscale, e.g., using the ReadFcn property of the imageDataStore, for the purposes of labeling only. I don't know if the loss of color affects your ability to recognize targets for labeling purposes. If not, it will cut down memory requirements by a factor of 3.

I would still recommend you chop up the images in any case. I would expect it to make training go much faster.