How to import 3GB netcdf file into matlab

I know, how to read netcdf data into matlab and due to the size of this file (3gb) maltab is saying out of limit (array exceeds maximum array size preference. Creation of arraysgreater than this limit may take a long time and cause MATLAB to become unresponsive.)

I wan to read variables sparately and I did some to them by using simple ncread but the size of main variable is about 3gb and that evil is making problem here.

How Could I deal it ?

thank you in advance

Answer 1

[KSSV]

YOu are trying to load the enitre data from netCDF into MATLAB..this is not the right option..first get familiar with the data..if the data is 3D, you can read matrix one by one in a loop and process or plot. If your matrix is of long dimension and only one matrix..you can read chunk or part of it by providing indices. Read about ncread, you have maultiple options out there.

[Shakir Hussain]

I want to read the variables separately and I did some of them but the size of mian varaible is about 3GB which create a problem here.

[KSSV]

Think do you need the main variable for enitre domain and for all lats, lons? Do you need in the same resolution?

[Shakir Hussain]

Yes, I need the main vaiable in same resolution and the dimension of that variable is something like 10902x6818x846 and size is more than 3 GB.

or

can we break down the chunk of netcfd into small parts? I used a default method of 7zip but the output was not accurate.

[KSSV]

As I said you have a 3D matrix..you can read it matrix by matrix in loop. Check the below code:

down = 10 ;
file = 'myfile.nc' ;
lon = ncread(file,'lon') ; nx = length(lon) ;  % take your x/ longitude variable
lat = ncread(file,'lat') ; ny = length(lat) ; % take your y/ latitude variable

t = ncread(file,'time') ; % your thirs dimension variable 
nt = length(t) ;
% in your case nx = 10902, ny = 6818, nt = 846
%% using p color to plot 
for i = 1:1:length(t)
    Zi = ncread(file,'Z',[1,1,i],[nx,ny,1]) ;   % your 10902x6818 variable     
    pcolor(lon(1:down:end),lat(1:down:end),Zi(1:down:end,1:down:end)')    
    shading interp ;           
    drawnow ;     
end

[Shakir Hussain]

can we take the out of main variable into small parts instead of plot? e.g 10902*6818*100 .....and so on in sequence from top to bottom or bottom top.

[KSSV]

We have a plot it means..we have the data in hand....I am reading a specific time stamps matrix and plotting it. If you want to save it..then again you will get out of memory problem if you save entire matrix with same resolution. If you down scale, if the matrix is not huge..then you can have it as a 3D matrix. But the question again is do I need a 3D matrix in workspace? What for?

[Shakir Hussain]

Off course, we need 3d matrix (i.e. lat*lon*precip) but I was asking about the possibility to get output and save in 5-10 small parts.

e.g 10902x6818x846 split into

10902x6818x100

10902x6818x100

10902x6818x100...... and so on and save each file separately but this execution must be carry put in sequence

[Shakir Hussain]

How could we can regrid it directly from nc file and take result of regrid as output?