LD_LIBRARY_PATH is a standard UNIX (POSIX) environment variable that affects the directories searched by the linker at run-time to locate shared objects (.so) that were coded with a relative path instead of with an absolute path. It's behaviour is not specific to MATLAB, and MATLAB cannot control its behaviour (other than to change the contents of the environment variable.)
MATLAB has one or more directories that contains the shared objects that it uses, and it sets up LD_LIBRARY_PATH to refer to those directories, so that the linker knows where to look for the objects.
It is the run-time equivalent of the linker -L flag . At link time, the linker looks for shared objects in the -L directory list, and examines their symbol tables to create a translation list, but the list is not filled in with actual addresses. Then, at run time, the linker (exec) looks for libraries with the same names in the directories determined by the hard coded paths and by LD_LIBRARY_PATH, finds the shared objects with the matching name, looks up the implementation offset for each of the symbols, and fills in the translation table with the base virtual address for the library plus the symbol offset. Once that is done, the executable can read or write to those virtual addresses (for data) or call into the functions at those addresses. All standard operation of the way that code is executed on Unix-type systems.
When you system() out you might find that LD_LIBRARY_PATH has been set to refer to MATLAB directories. But sometimes the executables you are trying to invoke need a different version of a library with the same name -- for example MATLAB might have a libC.so on its LD_LIBRARY_PATH and the executable might need a different (newer) libC.so . In such a case, you might need to setenv() the LD_LIBRARY_PATH before the system(), or you might need to have the command being system()'d change the LD_LIBRARY_PATH