Formatter NoneIndent


I'm trying to build a Formatter, but can't figure out Indents.

When I simply try to build blocks, giving every single one Indent: NONE, it still formats to 8 spaces.
I suppose, that it has something to do with ASTree. 

The tree in screenshoot has WhiteSpace character to those 8 spaces, before it is NewLine WhiteSpace and Colon, after is "Get" token... but bnf grammar is little different, new line is actually a separate token, including an IndentToken, this has size 0 and is not visible there, there screen of parsed Tokens before AST:

I'm not sure, why those 2 tokens are not in resulting AST, but I suspect, they are the reason for wrong indentation.

Simplified grammar for it looks like:

varDeclaration ::= VAR IDENTIFIER setgetDecl
getDecl ::= GET COLON stmtOrSuite

Does anyone have an idea what I might be doing wrong, so formatter is this broken?

Or is it even ok, for a grammar to check newLines & indents like this?



Are the whitespaces significant in your language? Is there any need for NEW_LINE or INDENT tokens? Usually, they are unnecessary and only needed in languages like Python, where indents matter.


Yes, unfortunately this is Python like language, where whitespace indents specify code blocks.

Valid if/else is like:

if true:

I've tried to mess with it a little bit, but still no success.

All empty token (WhiteSpace, NewLine, Indent, Dedent) are ignored when building child blocks.

I've also changed Lexer, because I thought that the reason can be, that there was IndentToken, then WhiteSpace and only after it GetToken, so now the order is reversed - first WhiteSpace, then Indent:

But still it adds 12 instead of 4 spaces even when (well it seems that 8 spaces there always, while last 4 differ on settings -> so if I set indent to 2, there's 10 spaces total, even when in BlockTree is only one block having a NormalIndent, while all the others have NoneIndent.


Does anyone have at least an idea, why formatter would force 8 spaces, even when all blocks have NoneIndent?


Ok, I've found the culprit I guess:

This SETGET_DECL node does contain both colon at the end of the line and definition on the next line -> so those 8 spaces where just where this block started.

But I still don't know how to solve this issue -> is it completly wrong, that 1 block is on multiple lines or is there some way to say to ignore this?

Problem with splitting is, that when there is not that "get: ..." line, there should not even be that colon. :/ That's why it's a single block


Finally solved it today, 
All I had to do, was to skip that element over multiple lines, when building child blocks, and instead build it from it's children.