Tool window of my plugin doesn't hide when other plugin tool windows are opened

Clicking another tool window button while my plugin is open just opens the new tool window behind my plugin's tool window. Similarly, when another plugin's tool window is opened and I click on my plugin's tool window button, it just shows it on top of the existing ones without closing it

I set secondary=false in plugin.xml but I'm not sure the change is getting reflected. The plugin is in local development and I install it by copying and extracting the latest zip file (from buildPlugin task) to the plugins directory

<toolWindow id="<id>"
            anchor="right"
            secondary="false"
            icon="<icon>"
            factoryClass="<factory_class>"/>