The reason for this happening is a InitFcn Callback, which simulates a different model. If I remove that callback, it works as expected.
I did this because I am a newbie and wanted to have a nice but easy to make persistent GUI for the configuration of my large model. So I made a different model, which has all the info with nice radio buttons and everything and send that to the workspace by simulating it. I also wanted the big model to always have the most current settings in the workspace, so I simulated it in that callback. I am pretty sure there is a better way to do this, but that was the quickest for me.