Git -> Update project fails

I'm a framework developer and my project setup is as follows:

Main project is the framework (https://github.com/Elgg/Elgg) the framework can be extended by plugins/modules so I also have a lot of those (50+) in the same project as attached projects.

Now when ever I try Git -> Update project in order to update everything eventually I get the error:

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

I have no problem when doing a git pull on each separate attached project, so it's not a permission issue, just when I pull everything.

Is there a GitHub pull limit I'm not aware of (couldn't find anything online).

I've also seen https://intellij-support.jetbrains.com/hc/en-us/community/posts/360004124959-IntelliJ-2019-1-3-Can-t-access-git-using-SSH-keys and this is configured.

Any suggestions?