How to open a file in the remote server, read the contents, and save it to work space

Hi,

I have some large size NetCDF files store on a remote server. Those files are at least 100GB, and that is just the data for 1 day. I would need data for multiple days for some analysis. I certainly do not want to download each file to the local.

Is there any way that I can connect to the remote server from local Matlab, open these NetCDF files, grab the variables I need, and load them to the local workspace?

I have tried using

unix('ssh [e-mail]')

It does connect me to the server. However, I do not know how to open the NetCDF files and grab the variables to my local workspace. Is this even doable?

Thank you very much for your help!

[Walter Roberson]

netcdf libraries in MATLAB support accessing at least some URI; see https://www.mathworks.com/help/matlab/import_export/importing-network-common-data-form-netcdf-files-and-opendap-data.html . See for example https://www.mathworks.com/matlabcentral/answers/74947-i-have-a-problem-with-ncread-opendap-and-4-dim-data

[Ching An Yang]

Hi Walter, thanks for the reply. I am not sure if each remote server has OpenDap? I have tried http://[username:password@]host[:port], but how do I navigate to other directory? For example, someone else's directory? Thanks a lot.

[Walter Roberson]

When you are working with cdf files stored on remote systems, there are three possibilities:

1. That the remote system might only support downloading the file; if this is your situation then you would need to copy each of the 100 gigabyte files onto your system even though you only want to access a portion of the data;
2. That the remote system might support basic reading (and possibly seeking) within the file; if this is your situation then there would end up being a lot of back and forth traffic but it might not be necessary to download the entire file;
3. That the file might support OpenDap operations; if this is your situation then the cdf library would be able to tell the remote system exactly which subset of the data you wanted to be able to access, and it would access the data on your behalf and send you only what you need. This is the best case scenario

Are you able to tell us which server you are accessing from? If it is a well-known system then we might be able to investigate.

[Ching An Yang]

Hi Walter, thank you! I learn a lot from your comments. I am not sure which case is the server, is there a way to tell? I asked some collegues and one told me that server that requires Duo when log in would not work.

[Walter Roberson]

I am not sure what Duo is: is that the Single Sign-On security software?

Your reference to xxxx.gov suggests a US government site. US government sites tend to support OpenDap for science data. If, however, you happen to be working with data that is restricted access (possibly classified) then I do not have any information about what various US government sites support.

Answer 1

[Aditya]

Hi Ching,

I understand that you want to retrieve some data from the NetCDF files present on the remote server.

For this you can follow these steps:

1. Connect to the remote server using ‘ssh’ command via the ‘system’ function.
2. Navigate to the directory where the NetCDF files are located using the ‘cd’ command via the ‘system’ function.
3. Launch MATLAB command line interface on the remote server using the ‘matlab -nodesktop’ command via the ‘system’ function.
4. Use NetCDF related commands like ‘ncread’ and ‘ncinfo’ to read and retrieve information from the files.
5. Save the extracted variables to a file using MATLAB’s ‘save’ function.
6. Exit the remote MATLAB session using ‘exit’ command.
7. Download the extracted data file to your local maching using the ‘scp’ command via the ‘system’ function.

You can refer to the below links for the above-mentioned functions:

System Function: https://www.mathworks.com/help/releases/R2023a/matlab/ref/system.html

NetCDF commands: https://www.mathworks.com/help/releases/R2023a/matlab/import_export/importing-network-common-data-form-netcdf-files-and-opendap-data.html

SCP command: https://stackoverflow.com/questions/11304895/how-do-i-copy-a-folder-from-remote-to-local-using-scp#:~:text=6054-,scp%20%2Dr%20user%40your.server.example.com%3A/path/to/foo%20/home/user/Desktop/,-By%20not%20including

Hope this will resolve the issue!