IntelliJ Idea not removing (external) libraries

Expected:
1) add dependency to maven
2) run 'reload all maven projects' (maven toolwindow) > the dependency is added to 'external libraries' (project tool window)
3) remove the dependency or change version
4) run 'reload all maven projects' (maven toolwindow) > the dependency is removed from 'external libraries' and in case of version change the new version is added as library
5) searching in the project folder I can only find it back under the .idea/libraries folder

Actual:
4) run 'reload all maven projects' (maven toolwindow) > the dependency is not removed from 'external libraries' and in case of version change the new version is added as library => 2 versions of the same library
5) searching in the project folder it also/still appears in the .iml file under <orderEntry/>

The removed versions keep coming back. To fix this I tried:
- manually deleting those libraries in IntelliJ: this works until the next 'reload' which load all the versions you ever entered
- manually deleting the .iml file: as soon as this file is recreated, the problem returns
- manually deleting the .idea folder: solves the problem of the returning bad versions until you change a dependency version or remove the dependency. So this fix is temporary and only useful as a workaround. The better workaround is roll back to a previous IntelliJ version (the Jetbrain toolbox is very useful for this).

Tested versions that have this problem:
- 2020.2.4
- 2020.3.3
- 2021.1
Tested version that doesn't have this problem:
- 2020.1.4

Screenshots of the problem:

I work on several projects. I didn't test them all but I only noticed this on my latest one. For testing purposes I picked a random other project and there I don't have that problem.

And another strange side-effect of this problem is some plugin entries keep duplicating every time I click reload in the maven toolwindow. This happens for: ear, ejb, jar and war (the list was too long to screenshot everything). No idea if this is related to the problem.