urlwrite with http authentication

Hi,

I got one problem , that with urlwrite function,can it work for http authentication also? I have checked early some people said: function has a "params" argument, but these are CGI-style parameters that get encoded in the URL. Authentication is done with lower-level HTTP Request parameters.

Herｅ　Ｉ　ｆｏｕｎｄ　ｔｈａｔ　Authentication option in urlwrite.m is ONLY available since Matlab release 2013a. You can check your matlab version by typing ver at the Matlab prompt.　With the example

 URL = 'http://csa.esac.esa.int/csa/aio/product-action'; 
 fileName = tempname; 
 gzFileName = [fileName '.gz']; 

 [gzFileName,st] = urlwrite(URL,gzFileName, 'Authentication', 'Basic', ...
    'Get', {'Username', '<username>', 'password', '<password>', ...
    'DATASET_ID', 'C1_CP_WHI_ELECTRON_DENSITY', 'START_DATE', ...
    '2011-11-10T18:00:00Z', 'END_DATE', '2011-11-10T21:00:00Z', ...
    'NON_BROWSER', '1'}); 

 gunzip(gzFileName); 
 fileNames = untar(fileName); 
 for iFile = 1:numel(fileNames)
    disp(fileNames{iFile});
 end

this gives me error all the time. Is this correct parameters?

Answer 1

[Guillaume]

I don't think it's possible using urlwrite, but the weboptions of the newish websave allows you to specify username / password.

[buer]

Thanks for your reply, unfortunatelly I do not have websave in Matlab 2013. But how about with Name-Value Pair Arguments? in the documentation it mentioned 'Basic' authentication...

[Guillaume]

Actually, you are right, there are authentication properties for urlwrite as well. Doesn't the server you connect to support basic authentication?

Looking at the source code of urlwrite it uses java, just like the code in the link provided by Cedric. And just like urlwrite, that code only supports basic authentication (i.e. username and password are sent in plain text, so you'd better do that over an HTTPS connection).

So, have you tried?

urlwrite('https://somewhere/on/the/web/', 'somefile.txt', 'Authentication', 'Basic', 'Username', 'someusername', 'Password', 'somepassword');
assert(exist('somefile.txt', 'file'), 'Didn't work!')

[buer]

Hi,

I did tried...in fact it returns 0 and also it deleted the file I downloaded before.I do not understand what happened here, it in fact delete the file I download from the. By the way can you explain little more what is teh difference between URLREAD/WRITE_AUTH and urlwrite? I am totally confused now...Thanks again

[Guillaume]

Ok, I've justed looked at the implementation of urlwrite (in r2014b) and it is strictly a superset of the urlwrite_auth found in Cedric's link. It uses exactly the same java functions and provides more functionality (support for proxy, post variables, etc.). So, in 2014b, there's no reason to use urlread/write_auth.

That may not be the case in R2013. You'll have to find out for yourself.

edit urlwrite.m

to look at the code and compare (in 2014b, most of it is actually implemented in urlreadwrite.m.

Perhaps, what you should do is put a breakpoint at the beginning of urlwrite.m and follow what happens step by step when you call the function. That way you'll see where it fails and hopefully why.

Note that you actually get less information out of urlwrite if you request status information. status has only two values, 0 for failure, 1 for success. If you don't request a status, you instead get an exception which details the cause of the failure. So call urlwrite with only 0 or 1 output arguments.

[buer]

Hi, Thanks for your reply. In fact I tried that also...it gives error :

Error using urlreadwrite (line 91)
The IP address of "HostName" could not be determined.

Here in Matlab 2013 urlwrite mostly is implemented in urlreadwrite as well.The error happens when reading data from the connection. I am in a rush with deadline, please do give some advice. Thanks

[Guillaume]

Well, here you go, the problem appears to have nothing to do with authentication, but is a failure to connect to the server.

You could possibly get more details about the error by putting a breakpoint on line 91 and looking at e.message when it's hit.

[buer]

Here is the error, it came just after I press step. SO it is still in line 91. This is funny, I checked the url, I can access it. what is the meaning that can not connect to server....

 91          error(mm(fcn,'UnknownHost',host{1}));
  Error using urlreadwrite (line 91)
  The IP address of "HostName" could not be determined.

Error in urlwrite (line 38)
[f,status] = urlreadwrite(mfilename,catchErrors,url,filename,varargin{:});

[Guillaume]

As I said, when you it the breakpoint on line 91, look at e.message (don't press step). This is will give you the full text of the error as it was raised by java, before it gets translated by matlab.

Note that urlwrite uses the system proxy if one is configured. Depending on your browser, it may use something different.

[buer]

Hi, here I found one link which says :the Authentication option in urlwrite.m is ONLY available since Matlab release 2013a. http://csa.esac.esa.int/csa/aio/html/other_clients.shtml

Looks like good.

[Guillaume]

Yes, at some point in the past, it obviously didn't have authentication since somebody felt the need to write urlwrite_auth. Since you're on R2013(a|b) you're fine with that.

Do you get the same error (Unknown Host) if you try to connect without authentication or do you get an authentication error?

[buer]

It is the same errors....The IP address of "HostName" could not be determined. Even this authentication should work for Matlab 2013b, but the result shows it did not work....

[Guillaume]

I don't think it has anything to do with authentication. It's not even tried to authenticate because matlab / java couldn't find the server.

[buer]

So what do you think create the problem? The url I am trying to download from, is just a data container...so I want to download from there all the data...and the url might be not online to public...is it possible it is because of this?

and it is the same problem, when I am trying to use urlread...

[buer]

I there something wrong with my IP? Because even I try to urlread from a public place...it still gives the same error...

[Guillaume]

Does

urlread('https://www.google.com')

work for you?

Can you share the url you're trying to connect to, assuming it's not on a LAN? Even if we can't authenticate, we can see if we can initiate a connection to the server.

[buer]

This one does not work also...same error...

I think it might have something to do with the Matlab license. Because I am using network licensed Matlab, which might make difference...

[Guillaume]

One thing for sure, this has nothing to do with your matlab license.

Have you looked at the Web settings in matlab preferences / Web ? Are they what they should be?

If you can't connect to even google.com, then the issue is obviously nothing to do with authentication. I would suggest you start a new question trying to resolve this connection issue, because I'm afraid I can't help you there.

[Guillaume]

Just seen your edit to the question. The syntax of your urlwrite call is wrong. It should be

[gzFileName,st] = urlwrite(URL,gzFileName, 'Authentication', 'Basic', ...
  'Username', '<username>', 'password', '<password>', ...
  'Get', {'DATASET_ID', 'C1_CP_WHI_ELECTRON_DENSITY', 'START_DATE', ...
  '2011-11-10T18:00:00Z', 'END_DATE', '2011-11-10T21:00:00Z', ...
  'NON_BROWSER', '1'});

The reply I get (without the username / password)

LOGIN_REQUESTED http://csa.esac.esa.int/csa/aio/product-action?DATASET_ID=C1_CP_WHI_ELECTRON_DENSITY&START_DATE=2011-11-10T18%3A00%3A00Z&END_DATE=2011-11-10T21%3A00%3A00Z&NON_BROWSER=1

which is expected since I didn't supply the username/password

This won't solve the problem that you can't connect to anything at all, though.

[buer]

You hit the spot. The settings in my preference/Web is wrong, it was using Proxy server..Now after I took it away, it works with this.

urlread('https://www.google.com')

What is the Get parameter, also the time etc. I want to download the file which is linked in the url ... This might be the problem? I do not think I need start date and end data etc...

[gzFileName,st] = urlwrite(url,wanted_h5, 'Authentication', 'Basic', ...
  'Username', '<uuuu>', 'password', '<ooo>');

But when I tried this it did not download also.

Thanks a lot , I will try to figure it out

Answer 2

[Cedric]

I have been using the URLREAD/WRITE_AUTH provided here, or similar, for years now.

[buer]

In fact I am also using this function you mentioned. But I want to know is it possible when you login once you want to download many files, but only with one time authentication....so the username and password is in memeory...that is why I am thinking if Matlab have the already working function would be nice. Because I do not know how to handle with this ...How do you handle with this? Thanks