Issue with ncread when _FillValue, add_offset, and scale_factor are present

31 views (last 30 days)
I have run into an issue when trying to concatenate multiple netcdf files with the same variables into one file, combining these variables.
First, I create a new netcdf file using the header information from one of the files I wish to concatenate using ncinfo and ncwriteschema, changing the length of the variable along which I wish to concatenate to unlimited.
When I read a variable that does not have the attributes _FillValue, add_offset, and scale_factor using ncread, the variable reads into my workspace as I would expect, and writes as expected into the new file.
However, when I read a variable that does have those attributes, the variable reads into my workspace seemingly without applying those attributes, contrary to ncread's documentation. I checked this using netcdf.getVar, and the results are the same between the two methods, which seems to indicate that ncread is failing to apply the attributes.
Strangely, however, when I use ncwrite to store the variable data in the new netcdf file, and then read it back in with either ncread or netcdf.getVar, I get something different! The variable data now oscillates between the _FillValue and -1*_FillValue.
Hypothesis: To me, this seems to indicate that ncwrite is trying to apply the attributes, but since ncread or netcdf.getVar did not apply them inversely, it's now applying them a second time! This causes the values to be larger in magnitude than the _FillValue, and confuses the netCDF file.
An easy workaround would be to apply the attributes myself using ncreadatt and ncwriteatt, but I'm not sure my hypothesis is correct, so I'm hesitant to brute force the process in that way.
Working on attaching code and small enough netCDF files, will update when those are ready.
UPDATE: netCDF files are uploaded as "netcdf_files.zip", matlab code uploaded as "forumpost_ncread.m".

Accepted Answer

meghannmarie
meghannmarie on 2 Dec 2020
Edited: meghannmarie on 2 Dec 2020
Ncread is working, put a breakpoint on line 108. When you use ncread and have a scale factor/add offset, it returns a double. In your code, you are taking your double variable and recasting it an int16 which is turning everything into integers. You need to delete lines 108 and 109.
vname = info.Variables(kn).Name;
var = ncread(fget,vname); %variable that is double precision when there is a scale factor/add offset
dtype = info.Variables(kn).Datatype;
var = cast(var,dtype); %You are now recasting double to int16, delete this
  4 Comments
meghannmarie
meghannmarie on 3 Dec 2020
If you add the lines below to your code, you will see ncread is working just fine. Apply the offset and scale factor and then look at your comparison pictures. Without applying those, you are comparing apples to oranges (int16 numbers to doubles).
ncid = netcdf.open(flist(1).name); % open original netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway = double(u_longway) * scale_factor + add_offset;
netcdf.close(ncid);
ncid = netcdf.open(fname); % open new netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway_new = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway_new = double(u_longway_new) * scale_factor + add_offset;
netcdf.close(ncid);

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by