How to develop on top of the IDEA?

I'm not sure if I got the question correctly.

Do you want to use intellij features from some external Qt-like GUI framework?
Or do you want to create a plugin, that would use Swing and intellij features - but ignore the rest of existing UI (starting from scratch) ?

I want to create GUI tool that is some kind of file manager , 
left side it files tree , right side is the content of the file formatted to text boxs and check box's , that mean the right side frame will load the file parse it and build simple GUI out of it . 
and i realy what to use the search options the original intellij have .

In this case you might want to use `com.intellij.openapi.application.ApplicationStarter` (or ApplicationStarterEx, to reuse existing IDE instance to reduce startup time).
You can take a look at `com.intellij.diff.applications.DiffApplication` or `com.intellij.formatting.commandLine.FormatterStarter` as an example.
https://github.com/JetBrains/intellij-community/blob/master/platform/lang-impl/src/com/intellij/formatting/commandLine/FormatterStarter.java
https://github.com/JetBrains/intellij-community/blob/master/platform/diff-impl/src/com/intellij/diff/applications/DiffApplication.java

Also, you might be interested in Commander plugin. 
https://github.com/JetBrains/intellij-community/blob/master/plugins/commander/src/com/intellij/ide/commander/Commander.java
https://github.com/JetBrains/intellij-community/blob/master/platform/lang-impl/src/com/intellij/ide/commander/CommanderPanel.java

Though, intellij features are heavily limited without a Project (and its indexes). So search might be dramatically slower for unindexed part of file system.
Actually, it is better to avoid using "Virtual File System" for non-project files at all (and it is mandatory for most of IDE features).
https://www.jetbrains.org/intellij/sdk/docs/basics/virtual_file_system.html

Overall, intellij might not be the right toolkit to build a general-purpose File Manager.

Thanks allot for the info indid this is good starting point . 

about the "Project" what do you mean "without project"? 

there will be project even several project in the tree view . but the entris in this tree will not be real files , this is what you mean "virtual file system'?

in the backend i will save it as 1 json or xml project file that contains all the data 

The `Project` mentioned above refers to `com.intellij.openapi.project.Project`, and `virtual file system` refers to the file system based on `com.intellij.openapi.vfs.VirtualFile`.

 Also, data inside IntelliJ IDEA is stored in a `org.picocontainer.PicoContainer`, and we use xmls to persist data.

You need "Git Gud Scrub" and start looking at programming stuffs, Meiry.