Intellij Grammar Kit - indent based language
I'm creating a language using the Intellij IDEA Grammar-Kit plugin, and would like the blocks to be indent-based, rather than brace based. The only way I've found to do this so far is to use the <<indent ...>>
function from https://github.com/kandeshvari/idea-nim/blob/master/src/org/dmitrigb/ideanim/parser/ParserUtil.java.
I have minimized the code from that link to:
package org.intellij.sdk.language;
import com.intellij.lang.LighterASTNode;
import com.intellij.lang.PsiBuilder;
import com.intellij.lang.PsiParser;
import com.intellij.lang.impl.PsiBuilderImpl;
import com.intellij.lang.parser.GeneratedParserUtilBase;
import com.intellij.openapi.util.Key;
import com.intellij.psi.TokenType;
import com.intellij.psi.tree.IElementType;
import com.intellij.psi.tree.TokenSet;
import com.intellij.util.containers.IntIntHashMap;
// https://github.com/kandeshvari/idea-nim/blob/master/src/org/dmitrigb/ideanim/parser/ParserUtil.java
public class SPPParserUtil extends GeneratedParserUtilBase {
private static class ParserState {
enum PrimaryMode {
NORMAL, TYPE_DEF, TYPE_DESC
}
private PsiBuilder builder;
private int currentIndent = 0;
private int pragmaCount = 0;
private PrimaryMode primaryMode = PrimaryMode.NORMAL;
private int semiStmtListCount = 0;
private IntIntHashMap tokIndentCache = new IntIntHashMap();
ParserState(PsiBuilder builder) {
this.builder = builder;
}
private String getPrecedingWhiteSpace() {
int wsOffset = 0;
while (builder.rawLookup(wsOffset - 1) == TokenType.WHITE_SPACE)
--wsOffset;
int wsStart = builder.rawTokenTypeStart(wsOffset);
return builder.getOriginalText().subSequence(wsStart, builder.getCurrentOffset()).toString();
}
int getTokenIndent() {
int tokStart = builder.getCurrentOffset();
if (tokIndentCache.containsKey(tokStart))
return tokIndentCache.get(tokStart);
int indent = -1;
String ws = getPrecedingWhiteSpace();
int nlPos = ws.lastIndexOf('\n');
if (nlPos != -1)
indent = ws.length() - nlPos - 1;
tokIndentCache.put(tokStart, indent);
return indent;
}
}
private static Key<ParserState> parserStateKey = new Key<>("parser-state");
private static ParserState getParserState(PsiBuilder builder) {
return builder.getUserData(parserStateKey);
}
public static boolean indented(PsiBuilder builder, int level, Parser parser) {
ParserState state = getParserState(builder);
int tokIndent = state.getTokenIndent();
if (tokIndent > state.currentIndent) {
int prevIndent = state.currentIndent;
state.currentIndent = tokIndent;
boolean result = parser.parse(builder, level + 1);
state.currentIndent = prevIndent;
return result;
}
return false;
}
}
saved as SPPParserUtil.java
This is my minimal BNF so far:
{
parserUtilClass='org.intellij.sdk.language.SPPParserUtil'
tokens=[
TK_DOT = '.'
TK_ARROW = '->'
TK_STAR = '*'
TK_COLON = ':'
KW_FILE = 'file'
KW_IMPORT = 'import'
TK_EOL = 'regexp:[\n|\r]*'
TK_IDENTIFIERS = 'regexp:[a-zA-Z_][a-zA-Z0-9_]*(,\s+[a-zA-Z_][a-zA-Z0-9_]*)*'
TK_SCOPED_IDENTIFIER = 'regexp:[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)*'
TK_SPACE = 'regexp:\s+'
]
}
root ::= Program
Program ::=
ProgramContents <<eof>>
ProgramContents ::=
FileDefinition ImportBlock? TK_EOL
FileDefinition ::=
KW_FILE TK_SPACE ScopedIdentifier TK_EOL
ImportLocation ::=
TK_DOT* ScopedIdentifier
ImportDefinition ::=
ImportLocation TK_SPACE TK_ARROW TK_SPACE (Identifiers | TK_STAR) TK_EOL
ImportBlock ::=
KW_IMPORT TK_COLON <<indented ImportDefinition+>>
ScopedIdentifier ::=
TK_SCOPED_IDENTIFIER
Identifiers ::=
TK_IDENTIFIERS
Here is some example code that should pass:
file src.classes.class1
import:
src.classes.class2 -> class2, e2
src.classes.class3 -> class3, e3
src.classes.class4 -> *
However, whenever I try to use <<indent ...>>
in my BNF there is always an error: The error described is ':' expected, got ':'
, on the colon following "import". Basically whatever is before the <<indent ...>>
seems to be consumed by the <<indent ...>>
. How would I change the BNF / use of the ParseUtil
to fix this?
---
EDIT
I changed the indented
function to always return true, and not use any of the other Java code and the error persists; this implies that the issue isn't with the Java code but maybe how I'm calling it in the BNF file?
Please sign in to leave a comment.
Could you please tell what language is this?
Hi, I'm not sure exactly what you mean by this? The grammar is in a .bnf file, and the external rule code is written in a Java file, as the `GeneratedParserUtilBase` class needs to be extended. The language I'm designing doesn't exist already; I'm creating a new language.
However, I think the issue isn't specifically with the `indented` function's code, because when I replace the body of the function with just `return True` or copy the original `<<eof>>` external rule code into the function body I still have the same error. Maybe I'm missing a static member or constructor, I'm not sure.
Thanks
It's working now. I had to add the `adapt_builder_` function into the ParseUtil class, and then even though in live preview it wasn't working, after generating the lexer and parser code and running the plugin in Gradle it works fine.
It would be nice to look at language grammar/spec first to decide a proper approach for implementing IDE support.
> I had to add the `adapt_builder_` function into the ParseUtil class
can you please post the missing code snippets you added? It would help me to understand your solution