Intellij keeps adding Maven modules as external libraries to the project libraries

Hi everyone, 

I have a Maven project consisting of several modules. For what ever reason Intellij keeps adding some of the modules's compiled libraries to the project libraries. When I then run my application and try to debug some code, the debugger does not open the source files, but jumps into the decompiled sources of the libraries. That's super annoying. 

When I then remove the libraries manually from the project's libraries, everything works like expected. After some time, I haven't figured out why, the libraries are again added to the project's libraries. 

Any idea what might lead to that problem? I'm switching branches (Git) quite often, and newer branches might have more Maven modules than others, could it be somehow related to auto import of Maven modules? 

Best Regards,

Flo