Closing the loop here. The main problem was the initial condition setting at different joint targets. You can check if a system initialized in the desired target by going to the Variable Viewer:
And then checking if the system could initialize correctly:
Usually relaxing some of the target leads to the system initalizing correctly.