How Can I prevent FragmentElementType from affect native HTML parsing?

Hi, I'm develop a plugin for supporting Regularjs. Which is a template engine basing on HTML. I have already finish lexer and parser.

But there is some problem.


Picture 1 shows that,when not using my plugin, native html parser deals with `on-mouseleave={this.toggleSubmenu($event,false)} ` well, it treat the content after `=` as PsiElement(XML_ATTRIBUTE_VALUE)。

But once I apply my plugin to this code, as picture 2 shows, it will treat the content after `=` as PsiElement(RegularFragmentElementType), so that the parser will not work well.

And what is worse, in the code `class="menu-item menu-item-withsub {item.className||''}"`, since the {item.className||''} will be lexer as PsiElement(RegularFragmentElementType), this will broken the parser rule(string) of native html.

Could anyone tell me how to fix this problem?


You're using TemplateDataElementType and PsiElement(RegularFragmentElementType) is an OuterLanguageElement, right?

If not, it should be so. If yes, then you have an opportunity to customize the text that HTML lexer/parser see (TemplateDataElementType#createTemplateText, TemplateDataElementType#appendCurrentTemplateToken). There, you can analyze the template data tokens you have and possibly insert additional characters in place of template language fragments, to make the text well-formed enough.

It seems that in your case it should be enough to insert some identifier after "=" or a space.


Thanks for reply. Your solution seems fantastic.

But how can I use TemplateDataElementType?  The only way I can figure out is the extents a new class base on  TemplateDataElementType, and override the code of TemplateDataElementType#appendCurrentTemplateToken, change the code as below:

protected void appendCurrentTemplateToken(StringBuilder result, CharSequence buf, Lexer lexer) {

Is this right?


I don't see the code below, but from your description it seems to be the approach I had in mind.


Hi, Peter

I have updated my code above. Could you see it?



Yes, now I see it. With this implementation, you'll get empty HTML text. Please call super.appendCurrentTemplateToken to ensure the text in the file is appended, and after that you can also append something else, to make HTML parser happier.


Hi, Peter

Thanks a lot. Your idea really works! 

There are still a problem, though it may not a big deal. 

As the picture shows below, the code `class="menu-item {item.className || '''}"` should be parse as

"menu-item" , "RegularFragmentElementType", but as you can see, here it's parsed as "menu-i", "RegularFragmentElementType", "term".  In here, I just replace {item.className || '''} as space as you said. Is this a bug of PsiViewer or I have do something wrong?

Thanks again for your help!


Now I have found a critical problem.

Here is my code.  

And now I have found that I couldn't type in '=', and when I type in other word, it will be block. I thought result is just for creating a virtual psifile for native html parser, without affecting the original file, am I wrong?


It seems that I was wrong, and it's much more complicated. Changing the text is not enough, it also requires some quite complicated PSI patching, for which there's no API yet. Most other template language supports of ours have the same issue, unfortunately. With attributes, the only viable strategy so far seems to add quotes to the template text. For cases like the one with CSS classes, the highlighting errors are just suppressed.

I'll investigate and see if we can extract some API.


Do you means that I should change the grammar of template language, changing




If so, I'm afraid that I couldn't do that, since we have a lot code writing in this way. 

Thanks anyway.

And please tell us the result of investigating, that's will be helpful for someone who has the same problem.


Hi, Peter.

Is there any progress?


Not in this release, unfortunately. I've filed an issue that you can watch:


Sad. Does it means this feature will not work until next year?


We plan to release 2017.2 in summer. Note that while we'll strive to fix this request until then, I can't absolutely guarantee that.


Hi, Peter,

It seems that it would be too long to wait this feature. I thought whether I could do this in another way.  Is there any specific way  to achieve this rather than a general api?


There's only a very hacky way so far:

1. remember the inserted strings and their offsets in a thread-local field in overridden "createTemplateText" or "appendCurrentTemplateToken",

2. use this information in overridden "prepareParsedTemplateFile" and remove the inserted strings from template data file leaves, so that its text corresponds to the original one. Removing is done by replacing leaves with their copies with a new text (LeafElement#replaceWithText).


Thanks a lot! I will try it soon.


Hi Peter,

This is my implementation of  "prepareParsedTemplateFile": 

protected void prepareParsedTemplateFile(@NotNull FileElement root) {
ArrayList<Pair<Integer, String>> pairs = hackyStringMapThreadLocal.get();
int size = pairs.size();
for(int i = size -1; i >= 0; i--){
Pair<Integer, String> entry = pairs.get(i);
LeafElement oldLeaf = root.findLeafElementAt(entry.getFirst());
LeafElement newLeaf = oldLeaf.replaceWithText(oldLeaf.getText().replace(entry.getSecond(), ""));
root.replaceChild(oldLeaf, newLeaf);

As you said, I need remove all my inserted strings in "appendCurrentTemplateToken", but I tried PsiElement#replaceChild, this method will throw some Exception, breaking out the while loop, so I wander if there another way to replace leaf node. Could you have any suggestion to me?




You don't need replaceChild, because rawReplaceWithText/replaceWithText already does that. Please use rawReplaceWithText. If the exception persists, please post it here.


It seems works very well!  Peter, thank you very much!


Please sign in to leave a comment.