Invalidation of all PSI when there's a filesystem change?

We are seeing this with any UI built on top of BasicDomElementComponent. All of our test users have reported it as soon as they upgraded to 2017.3 

It seems to me that invalidating all of the Psi in an open project would be an expensive operation with far-reaching consequences.I might be able to work around this, if someone can tell me how to get things that use BasicDomElementComponent to refresh themselves when they've been invalidated. I worry about performance, though.

This looks like it was done to fix a problem in PyCharm - PY-25892 - and I am really surprised I am the first person to report issues related to it. 

Thanks for reporting and sorry for the breakage! I've filed https://youtrack.jetbrains.com/v2/issue/IDEA-185043 and will try to fix it ASAP.

There's at least one bug that `createStableCopy` doesn't work correctly for top-level elements. But I'm not sure it's the only one causing this issue in your editors. Which API classes are you using to create editors and controls?

That's a difficult question for me to answer, Peter -- this part of the plugin is unfamiliar to me.  It looks like in some cases we just have a JTree, for which we build a TreeModel from a Supplier<XMLFIle>. 

In other cases we seem to be using DomFileEditor, or a class someone wrote called "BetterDomFileEditor" which is attempting to imitate it while adding some other behaviors. These are definitely using CompositeCommitable and DOMUIControl.

We are probably dependent on version 2017 at this point (2018 will contain too many changes for us to integrate in a short period) so I am hoping whatever changes you and we end up making can be supported on 2017.3.#

In the cases where we have a JTree, I can detect invalidation pretty easily and have the JTree rebuild its internal state. I'm having most of the trouble with the things that are using the DomFileEditor or BetterDomFileEditor. 

Can your plugin work with IntelliJ Community Edition? If yes, I'd appreciate if you checked whether a change I've made (currently in master, but can be cherry-picked to 173) fixes your issue: https://github.com/JetBrains/intellij-community/commit/7188958adbf94d43d86eeddecdf78204b8d8d1fb

Yes, we deliberately develop on Community because that's what we redistribute to customers. We currently run with 2017.3.1 -- I don't know whether we can run successfully with later versions.

How would I test this change?

You can either build community (173 branch) from source, with this commit applied. Or, I could build that for you and upload somewhere, if you like.

I was able to compile the 173 branch using IntelliJ, but I don't see how I would package it or run my plugins with it.

Please see the last section at https://www.jetbrains.org/pages/viewpage.action?pageId=983225. Does this help?

     [java] BUILD FAILED

     [java] /Users/rberlin/idea-2017.3.1/build/gant.xml:69: Compilation failed with exception: Compilation failed

     [java]

     [java] Total time: 2 minutes 50 seconds

     [java] Compilation errors (java):

     [java] Cannot find JDK '1.8.0_144' for module 'updater'

 [java] Cannot find JDK '1.8.0_144' for module 'updater'

It looks like updater.iml file is modified in your copy of intellij-community project.

Thanks. Actually it looks like something in the .idea directory got changed as I was trying to set up the IntelliJ build.  I put all files back to their repo values, and now I get the problem below. I think perhaps the instructions on how to build on the command line are incomplete...?

-- Rich

     [java] Compilation errors (java):

     [java] Fatal Error: Unable to find package java.lang in classpath or bootclasspath

     [java]

     [java] BUILD FAILED

     [java] /Users/rberlin/idea-2017.3.1/build/gant.xml:69: Compilation failed with exception: Compilation failed

     [java]

     [java] Total time: 57 seconds

Ah. OK, it's environment-dependent. I tried in another shell window and it succeeded. Sorry for the trouble.

OK, I cherry-picked that change but I am not seeing any difference in behavior: after making a filesystem change my editor still goes blank.

Tomorrow I plan to run in the debugger to see whether/when the library classes I am using call DomFileElement.createStableCopy

Most of the UI implementations use DomWrapper. I don't see any calls to createStableCopy() in our editors, and there are only a couple of examples in IntelliJ.

I think maybe I need to find a reference manual or tutorial for how DOM-based editors are expected to be constructed.

OK, found this: https://www.jetbrains.org/intellij/sdk/docs/reference_guide/frameworks_and_external_apis/xml_dom_api.html

Just a little bit of light reading. :-) 

Please try to pass the result of `createStableCopy` instead of the original root DOM element when you're creating the file editor. Then all inner DOM controls should also be working over stable copies, and survive on root invalidation. If this doesn't help, you could try putting a breakpoint into StableInvocationHandler#revalidate and trace why DOM can't be restored and which instance of `myProvider` still returns invalid one.

Interesting. Debugger shows that I am calling DomInvocationHandler.createStableCopy(), but I am not hitting the breakpoints in StableInvocationHandler.invalidate() or StableInvocationHandler.revalidate()

Is com.intellij.util.xml.impl.StableInvocationHandler#invoke being hit at all?

You could also put a breakpoint into com.intellij.util.xml.impl.DomInvocationHandler#invoke and see how it's called. Theoretically all calls from UI should go via some stable copy. If it's not the case, then `createStableCopy` should be added somewhere around the creation of those UI components.

I might try this with a different editor -- the one I have been using is pretty complicated -- although I will eventually need to get it to work.

I could use a suggestion, because it's been hard for me to find examples of how this stuff is used. How do you suggest I construct things like JList and JComboBox so that the underlying model is using a stable copy of the collection?

Normally you create stable copy of the root object, then call children getter on it (which also return stable objects), and create UI controls bound to them. For JList, that'd be DomCollectionControl, for check box — BooleanControl. Each control has getComponent method, which returns something you can put in your Swing hierarchy. They should also be added to some CompositeCommittable, which is a bit described in the documentation you linked above.

OK. the simpler editors all seem to work OK with your fix and small changes to use stable copies. So I think I just have the most complex one to do.

I guess I will have to sit and try to figure out how to get DomCollectionControl to behave like a JList.

There's actually a table inside, but you can pass in a single ColumnInfo to make it look more like a list (but the model will still be one of a table).

Any updates? The fix is now backported to 2017.3.4.

Thanks for that, Peter. We've picked up 2017.3.4 for our testing. I've traced through the remaining problematic editor and still am convinced it will need a complete rewrite, which is going to be a big job.

I guess I'm actually having trouble figuring out how I actually need to put this together.

The XML that this editor handles is a complicated list-within-list.

1. I have a JComboBox which selects the main element from the top-level list
2. There's a JList which shows an abbreviated view (ID and name) of the second-level list
3. The rest of the editor shows an editable, expanded view of the item that is selected by (2) above
4. There is also one additional field that edits something at the very top level

If the Supplier is set up to create a stable copy, most of the screen (i.e. 3, 4) tolerates invalidation just fine. 1 and 2 do not, though, because the JComboBox and JList models don't tolerate the invalidation.

One adidtional complication is that there are also (add/remove) buttons which can change the contents of (1) and (2).

I assumed a ComboControl would work for (1) but I'm scratching my head because it's actually needing to display information derived from the list at the *top* level of the XML. How do I get it to show only one field from the XmlElement, and then return the entire DomElement to the thing that needs it?

> I assumed a ComboControl would work for (1) but I'm scratching my head because it's actually needing to display information derived from the list at the *top* level of the XML. How do I get it to show only one field from the XmlElement, and then return the entire DomElement to the thing that needs it?

Sorry, I don't understand this. Maybe you could illustrate it with an example of XML parts being used and displayed?

Got it working.

The dom contains a list of large-ish objects of type Destination. We have a ComboBox which is used to select one of the Destinations, and then the rest of the editor is filled in to edit that section of the DOM.

The problem I was having was that the ComboBox doesn't have a model which supports DOM invalidation...and it is just a selector, not a control that is itself modifying a field in the DOM.

I ended up changing the ComboBox to contain a unique key for the Destinations, and then wrote explicit code that pulls out the corresponding element.  For the second-level (which selects an individual port in a Destination) I was able to use the DomCollectionControl with no problem.

Once I made sure I was using 2017.3.4 it all started to work properly. 

Thanks for your help!

Interestingly, I'm now seeing another source of invalidations. I switch focus away from IntelliJ, come back, and click a couple of times in the Table component that is associated with my DomCollectionControl -- and then resulting selection is associated with an invalid object. A call to PsiInvalidElementAccessException.getInvalidationTrace() tells me "file type change"