Function precedence in multiple nested Matlab projects

Matlab projects can add other projects as references. I started to use this to modularize my Matlab libraries. In fact, every git repo belongs to one Matlab package/namespace and gets consumed by an application. I started to use buildinspace/peru: a generic package manager, for including other people's code in your projects (github.com) for source code based dependency management (no, I do not want to use submodules).

That means, it ends up with a folder structure like

myApp

- src

-- myApp.m % needs my.namespace.lib1.m and my.othernamespace.lib2

- myAppPrj.prj % references lib-ml-lib1.prj and lib-ml-lib2.prj, does not set anything manually to the path

- resources % ML project resources

- lib

-- lib-ml-lib1

--- lib-ml-lib1.prj % adds src folder of lib1 to path

--- resources % ML project resources lib1

--- src/

---- +my/+othernamespace/lib1.m

-- lib-ml-lib2

--- lib-ml-lib2.prj % adds src folder of lib2 to path

--- resources % ML project resources lib2

--- src/

---- +my/+namespace/lib2.m

I hope the structure is clear; myApp.m depends on lib1.m and lib2.m which are functions or classes in nested packages.

This worked fine until lib1 and lib2 started to depend on lib3, but in potentially different versions. peru successfully copies the requested versions of lib3 into the lib folders in lib1 and lib2, but it seems to be random which version is chosen and I don't understand what's going on.

New situation:

myApp

- src

-- myApp.m % needs my.namespace.lib1.m and my.othernamespace.lib2

- myAppPrj.prj % references lib-ml-lib1.prj and lib-ml-lib2.prj, does not set anything manually to the path

- resources % ML project resources

- lib

-- lib-ml-lib1

--- lib-ml-lib1.prj % adds src folder of lib1 to path

--- resources % ML project resources lib1

--- src/

---- +my/+othernamespace/lib1.m

--- lib

---- lib-ml-lib3 % lib 3 in version 1.x

----- lib-ml-lib3.prj % adds src folder of lib3 to path

----- src/

------ +my/+otherothernamespace/lib3.m

-- lib-ml-lib2

--- lib-ml-lib2.prj % adds src folder of lib2 to path

--- resources % ML project resources lib2

--- src/

---- +my/+namespace/lib2.m

--- lib

---- lib-ml-lib3 % lib 3 in version 2.x

----- lib-ml-lib3.prj % adds src folder of lib3 to path

----- src/

------ +my/+otherothernamespace/lib3.m

Some questions

• Is the first found function in the pathtool used or the last one?
• how do referenced projects work, does every project have its own path or does everything go into a global one?
• how to explain that which("my.otherothernamespace.lib3", "-all") from the myApp (which does not directly depend on lib3) level finds the lib in the lib-ml-lib2 folder, i.e. in version 2.x but the profiler shows that lib-ml-lib2 uses the lib3 in lib-ml-lib1 folder (!), i.e. in version 1.x although its local lib folder contains the lib3 in version 2.x?
• What whould you do to solve that? It's not possible to avoid nested libs if I want to do proper dependency control

Thanks!

Jan

Answer 1

[Rishi]

Hi Jan,

I understand from your question that you want explanations regarding function precedence in MATLAB search path. Below are the answer to your questions:

1. The first file or function found in the pathtool is used. To verify this, you can use the 'which' function. In the MATLAB Command Winodw, type the following:

which filename

This returns the file which will be used. You can learn more about it from the given documentation:

https://www.mathworks.com/help/matlab/ref/which.html#:~:text=If%20item%20is%20an%20overloaded%20function%20or%20method%2C%20then%20which%20item%20returns%20only%20the%20path%20of%20the%20first%20function%20or%20method%20found.

2. Referenced projects in MATLAB add their paths to the global MATLAB path. This path is shared across the entire MATLAB environment, which means that all projects contribute to and use the same global path.

3. The 'which' function, when used with '-all' flag, prints out all of the paths of all item with the requested name. Hence, the below line will show all instances of the function 'my.otherothernamespace.lib3' currently on path:

which("my.otherothernamespace.lib3", "-all")

More about it can be read from the below documentation:

https://www.mathworks.com/help/matlab/ref/which.html#:~:text=___-,%2Dall,-displays%20the%20paths

The Profiler shows that 'lib-ml-lib2' uses the 'lib3' in 'lib-ml-lib1' folder because this version of the file is higher up on the MATLAB path than the 'lib-ml-lib2' folder.

4. Here are a couple of possible solutions to this:

Version Control within Namespaces: Ensure that each version of lib3 is placed in a different namespace corresponding to its version. This way, lib1 and lib2 can explicitly reference the correct version of lib3.

Project Initialization Scripts: Write custom initialization scripts for each project that set up the path correctly by removing any conflicting paths and then adding the required version of lib3. This ensures that the correct version is used during the project's execution. You can do this using the functions 'addpath' and 'rmpath'. You can learn more about these functions from the below documentations:

https://www.mathworks.com/help/matlab/ref/addpath.html

https://www.mathworks.com/help/matlab/ref/rmpath.html

Hope this helps!

[Jan Kappen]

Dear @Rishi,

thanks a lot for your answer. I think that confirms and explains the behavior I'm seeing.

Let's please check the possible options

1. Putting the libs in different namespaces would shift the version control from peru/exactly one config file per ML project to every place in the code where it's used - upgrading to a new version would require to change the code everywhere the namespaces are used - that's in a lot of locations - that does not sound feasable to me, or am I missing something? Could import be used so make that transparent somehow?
2. I do have project init scripts. They refresh the libraries defined in the project's peru configuration file, i.e. peru runs and clones all libraries into the ./lib sub folder of every project. The paths are automatically set by the projects, I suppose they use addpath.

Since the path is always global, it looks like there's no option to use two identically named (and in the same namespace) classes in different versions (different git SHAs or tags, and the class has a version property) in the top-level project and its sub projects? Note, I'm not trying to run them both from the top-level project, but from the different sub-projects.

What about publishing it as toolboxes? Most of the low-level libs do not change frequently, I could pack them as toolboxes, but I think that does not solve any of the issues, because the toolboxes use a global path, too?

Thanks!

[Rishi]

You can create a wrapper function in which you can pass in the namespace as an argument, and then use the 'eval' function to evaluate the function. You can learn more about the 'eval' function from the below documentation:

https://www.mathworks.com/help/matlab/ref/eval.html

I have implemented the code below:

function output = libraryWrapper(version,inputArg1,inputArg2)
evalString = ['my.', version, '.lib3(', num2str(inputArg1),',', num2str(inputArg2), ');'];
output = eval(evalString);
end

My folders are as follows:

Main

- Sub1

-- +my

--- +ns

---- lib3.m

- Sub2

-- +my

--- +ns1

---- lib3.m

In this sample implementation, the 'lib3' function in 'ns' adds two numbers, whereas 'lib3' in 'ns1' subtracts them.

I have attached the output for both of the cases.

While publishing your libraries as MATLAB toolbox will directly solve your issue as toolboxes are added to the global path, but it can be helpful in packaging, versioning and ease of distribution.

[Jan Kappen]

Dear @Rishi,

thanks for your answer but I'm afraid that's not the right approach. eval is definetely not recommended and I can't wrap everything into different namespaces. That won't work :(

An ideal solution would be that every ML project only sees files that belong to its path, not to the parent's project path.

Does anyone know if MathWorks is working on something like that? A global path breaks everything here.

Or am I tackling the problem from a wrong side? I just do not want to update all ML projects that depent on a certain library when that one gets updated.

Simple example:

• lib_io depends on lib_system v1.1
• my_app depends on lib_io and lib_system v1.1

All dependency management happens on code level using peru (similar, but imo better than git submodules), i.e. every lib_io and my_app both contain a lib folder with the correctly checked out version v1.1. Now I make an update to lib_system and tag it v1.2, because my_app needs a new functionality.

New situation:

• lib_io depends on lib_system v1.1
• my_app depends on lib_io and lib_system v1.2

Now, it's not clear if v1 or v1.2 is used, because both locations are in the global path when the corresponding projects are loaded.

Of course I could update lib_io to use lib_system v1.2, too. But that requires to check out the lib_io's code, update the manifest file where I declare the dependency to lib_systems to use lib_systems v1.2, check that change in, increase version of lib_io and ultimately update my_app to use the latest lib_io.

I need to do that for loooots of libraries - in lockstep. Just to have a new functionality in my_app. For sure this could be scripted, but it just works as long as there are minor and backward compatible changes. What if there are incompatible changes?

Ultimately, I think everything points to the direction that MATLAB needs a proper built-in dependency manager like pip, but right now there's none coming and I wonder how to fix it on my own.

Any ideas?

Thanks,

Jan