Open the modal window by double clicking on the custom Project View tree item

Does "custom project view" means that you have own AbstractProjectViewPSIPane with own nodes?

Then IDEA executes the following method to process every selected node that implement Navigatable

OpenSourceUtil#navigate(boolean, boolean, Iterable<? extends Navigatable>) 

At this moment I use a class which extends ProjectViewPane and create my own nodes in getChildren() method.

I do not understand how OpenSourceUtil#navigate can help me.

I found EditSourceOnDoubleClickHandler and use its method

install(JTree, Runnable)

for implements custom behavior on double click. Is this a normal way?

No, it is applicable for your custom tree. The Project View tree has already installed a double click handler. And your handler may affect it.

Note, that your Runnable is called right after processing selected nodes, but it does not know about selection: 

OpenSourceUtil.openSourcesFrom(dataContext, true);
if (myWhenPerformed != null) myWhenPerformed.run();

I recommend to implement the following methods in your ProjectViewNode:

boolean expandOnDoubleClick(); return false (needed for non-leaf nodes)
boolean canNavigate(); // return true (will be processed by openSourcesFrom)
boolean canNavigateToSource(); // return false
void navigate(boolean requestFocus); // open dialog

 Thanks, this method works fine.