[status, msg] = system('/usr/bin/vlc test.y4m &')
Note the single quotes on either end of the command to be sent to the prompt. In your statement of the problem you have single & double quotes. In your above answer you have no quotes at all.
Highlight, copy, and paste what I wrote above into MATLAB and see if that works. If it doesn't, highlight and copy everything inside of the single quotes (not including those quotes) and paste that to a terminal. If that doesn't work, the problem is not in MATLAB but in the UNIX command.