Download file using "websave" from a website that requires login

I am trying to download the following file from NASA's Archive of Space Geodesy Data:

https://cddis.nasa.gov/archive/gnss/data/daily/2021/143/21h/MAR700SWE_R_20211430000_01D_SN.rnx.gz,

but I am having a hard time downloading the file to my computer using websave because downloading the file requires login information.

When the file download link is clicked, it first redirects to a login page, and when the correct credentials are entered, it then redirects back to the original link (which is the one above) and the file is downloaded. I was wondering whether there is a way to download the file despite the login information and redirection being required. Currently, my code returns an html file of the login page.

Here is the code I am using if it is of any help:

filename = 'RINEXFile.rnx.gz';
fileurl = 'https://cddis.nasa.gov/archive/gnss/data/daily/2021/143/21h/MAR700SWE_R_20211430000_01D_SN.rnx.gz'

websave(filename,fileurl)

[Cris LaPierre]

I'm no expert, but have you explored weboptions, which does allow you set a username and password? I would not be overly optimistic if it is redirecting you to a login page, though.

outfilename = websave(___,options)

[Abhinav Pandey]

I have tried weboptions, but the redirect page is still getting in the way.

[Jan]

Did you ask the admins of the service for a REST interface?

[Abhinav Pandey]

I have not done that yet. I will try asking the adminds for a REST interface.

[Irem Köz]

Hi Abhinav Pandey,

I wonder whether there is any progress about this problem or not? I also have the same problem with the same website.

Answer 1

[Abhiram]

Hi @Abhinav Pandey,

In order to to download the file, you can follow the given steps:

• The login information must be specified using a 'POST' request to the login form.
• The web cookies will be provided in the response message.
• These web cookies should then be used in a 'GET' request to download the necessary file.

The code given below shows an example on how to implement 'POST' and 'GET' requests:

import matlab.net.*; 
import matlab.net.http.*; 
import matlab.net.http.field.*; 

% POST Request
uriPost = URI('https://www.httpbin.org/post'); 
data = struct ('msg', 'hello world', 'date', datetime('now')); 
acceptHeader = AcceptField('application/json'); 
contentType = ContentTypeField('application/json'); 
headers = [contentType, acceptHeader]; 
httpOpts = HTTPOptions; 
r = RequestMessage ('POST', headers, data); 
[resp, req, hist] = r.send(uriPost, httpOpts); 
out = resp.Body.Data; 

% GET Request
uriGet = URI('https://www.httpbin.org/get'); 
acceptHeader = AcceptField('application/json'); 
headers = acceptHeader; 
httpOpts = HTTPOptions; 
r = RequestMessage('GET',headers); 
[resp, req, hist] = r.send(uriGet, httpOpts); 
out = resp.Body.Data; 

For additional guidance, you can refer to the MATLAB documentation for “RequestMessage”, “Cookie”, and “FormProvider” classes:

https://www.mathworks.com/help/matlab/ref/matlab.net.http.requestmessage-class.html

https://www.mathworks.com/help/matlab/ref/matlab.net.http.cookie-class.html

https://www.mathworks.com/help/matlab/ref/matlab.net.http.io.formprovider-class.html