Using parfor to Make Many Web Requests
My colleague asked me to access all the pages on a web server in order to populate its cache. I plan to use parfor
to get through the more than 250k pages in a timely manner. I will also need to not go too fast and overload the server.
I’ve used parfor
before for web page access and found that it is one of the rare situations where you can use more MATLAB® workers than available physical or even logical processors, which is not normally recommended. It works because web requests require a lot of waiting, and often the processing I need to do in MATLAB is little.
I’ll admit this video gets a little boring near the end as I try out different numbers of workers. Remember that you can increase the playback speed of the video in the lower right corner of the player.
Originally posted on Stuart’s MATLAB Videos blog.
Recorded: 26 May 2021