JetBrains: What we're busy with right now. Follow

>> I was looking through the source of the Javascript language plug-in -
>> it uses a lexer generated by JFlex. I don't know much about
>> lexing/parsing, but does IDEA reparse the entire file every time the
>> file is changed? Or does JFlex lexer provide a way to incrementally
>> re-lex?
>>

We wrap lexers provided with one able to re-lex incrementally. That's
why getState() and getLastState() methods need to be exposed from the
lexer.

So you need special JFlex config file or something? Will you release yours
that you used for JavascriptLanguage?

Thanks for questions, Keith. Hope together we'll be able to build
something really useful out of this idea.

Me too, I'm excited about it.

If you meant template file - it's there, attached.

-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

>>> I was looking through the source of the Javascript language plug-in
>>> - it uses a lexer generated by JFlex. I don't know much about
>>> lexing/parsing, but does IDEA reparse the entire file every time the
>>> file is changed? Or does JFlex lexer provide a way to incrementally
>>> re-lex?
>>>
>> We wrap lexers provided with one able to re-lex incrementally. That's
>> why getState() and getLastState() methods need to be exposed from the
>> lexer.
>>

So you need special JFlex config file or something? Will you release
yours that you used for JavascriptLanguage?

>> Thanks for questions, Keith. Hope together we'll be able to build
>> something really useful out of this idea.
>>

Me too, I'm excited about it.

begin 777 idea-skeleton
M("`O*BH@:6YI=&EA;"!S:7IE(&]F('1H92!L;V]K86AE860@8G5F9F5R("HO
M"BTM+2!P<FEV871E('-T871I8R!F:6YA;"!I;G0@6EI?0E5&1D524TE:12`]
M("XNCL*"B`@RHJ(&QE>&EC86P@<W1A=&5S("HO"BTM+2`@;&5X:6-A;"!S
M=&%T97,L(&-H87)M87`"B`@+RH@97)R;W(@8V]D97,@B\*("!P<FEV871E
M('-T871I8R!F:6YA;"!I;G0@6EI?54Y+3D]73E]%4E)/4B`](#`["B`@<')I
M=F%T92!S=&%T:6,@9FEN86P@:6YT(%I:7TY/7TU!5$-((#T@,3L*("!P<FEV
M871E('-T871I8R!F:6YA;"!I;G0@6EI?4%532$)!0TM?,D))1R`](#(["B`@
M<')I=F%T92!S=&%T:6,@9FEN86P@8VAA<EM=($5-4%197T)51D9%4B`](&YE
M=R!C:&%R6S!=.PH@('!R:79A=&4@<W1A=&EC(&9I;F%L(&EN="!9645/1B`]
M("TQ.PH("`OB!E<G)O<B!M97-S86=E<R!F;W(@=&AE(&-O9&5S(&%B;W9E
M("HO"B`@<')I=F%T92!S=&%T:6,@9FEN86P@4W1R:6YG(%I:7T524D]27TU3
M1UM=(#T@>PH@("`@(E5N:V]W;B!I;G1E<FYA;"!S8V%N;F5R(&5R<F]R(BP*
M("`@(")%<G)O<CH@8V]U;&0@;F]T(&UA=&-H(&EN<'5T(BP*("`@(")%<G)O
M<CH@<'5S:&)A8VL@=F%L=64@=V%S('1O;R!L87)G92(("!].PH+2TM(&ES
M1FEN86P@;&ES=`H@("\J*B!T:&4@8W5R<F5N="!S=&%T92!O9B!T:&4@1$9!
M("HO"B`@<')I=F%T92!I;G0@>GI3=&%T93L*"B`@+RHJ('1H92!C=7)R96YT
M(&QE>&EC86P@<W1A=&4@B\("!P<FEV871E(&EN="!Z>DQE>&EC86Q3=&%T
M92`](%E924Y)5$E!3#L*"B`@+RHJ('1H:7,@8G5F9F5R(&-O;G1A:6YS('1H
M92!C=7)R96YT('1E>'0@=&\@8F4@;6%T8VAE9"!A;F0@:7,*("`@("`@=&AE
M('-O=7)C92!O9B!T:&4@>7ET97AT*"D@<W1R:6YG("HO"B`@<')I=F%T92!C
M:&%R('IZ0G5F9F5R6UT@/2!%35!465]"549&15(["@H@("\J*B!T:&4@=&5X
M='!O<VET:6]N(&%T('1H92!L87-T(&%C8V5P=&EN9R!S=&%T92`J+PH@('!R
M:79A=&4@:6YT('IZ36%R:V5D4&]S.PH("`OBH@=&AE('1E>'1P;W-I=&EO
M;B!A="!T:&4@;&%S="!S=&%T92!T;R!B92!I;F-L=61E9"!I;B!Y>71E>'0@
MB\("!P<FEV871E(&EN="!Z>E!U<VAB86-K4&]S.PH("`OBH@=&AE(&-U
M<G)E;G0@=&5X="!P;W-I=&EO;B!I;B!T:&4@8G5F9F5R("HO"B`@<')I=F%T
M92!I;G0@>GI#=7)R96YT4&]S.PH("`OBH@<W1A<G1296%D(&UA<FMS('1H
M92!B96=I;FYI;F<@;V8@=&AE('EY=&5X="@I('-T<FEN9R!I;B!T:&4@8G5F
M9F5R("HO"B`@<')I=F%T92!I;G0@>GI3=&%R=%)E860["@H@("\J*B!E;F12
M96%D(&UA<FMS('1H92!L87-T(&-H87)A8W1E<B!I;B!T:&4@8G5F9F5R+"!T
M:&%T(&AA<R!B965N(')E860*("`@("`@9G)O;2!I;G!U="`J+PH@('!R:79A
M=&4@:6YT('IZ16YD4F5A9#L*"B`@+RHJ"B`@("H@>GI!=$)/3"`]/2!T<G5E
M(#P]/B!T:&4@<V-A;FYE<B!I<R!C=7)R96YT;'D@870@=&AE(&)E9VEN;FEN
M9R!O9B!A(&QI;F4("`@B\*("!P<FEV871E(&)O;VQE86X@>GI!=$)/3"`]
M('1R=64["@H@("\J*B!Z>D%T14]&(#T]('1R=64@/#T^('1H92!S8V%N;F5R
M(&ES(&%T('1H92!%3T8@B\("!P<FEV871E(&)O;VQE86X@>GI!=$5/1CL*
M"BTM2!U<V5R(&-L87-S(&-O9&4*"B`@RHJ"B`@("H@0W)E871E<R!A(&YE
M=R!S8V%N;F5R"B`@("H@5&AE<F4@:7,@86QS;R!A(&IA=F$N:6\N26YP=713
M=')E86T@=F5R<VEO;B!O9B!T:&ES(&-O;G-T<G5C=&]R+@H@("`J"B`@("H@
M0'!A<F%M("`@:6X@('1H92!J879AFEOE)E861E<B!T;R!R96%D(&EN<'5T
M(&9R;VTN"B`@("HO"BTM+2!C;VYS=')U8W1O<B!D96-L87)A=&EO;@H*"B`@
MRHJ"B`@("H@4F5F:6QL<R!T:&4@:6YP=70@8G5F9F5R@H@("`J"B`@("H@
M0')E='5R;B`@("`@(#QC;V1E/F9A;'-E/"]C;V1E/BP@:69F('1H97)E('=A
M<R!N97<@:6YP=70N"B`@("H@"B`@("H@0&5X8V5P=&EO;B`@(&IA=F$N:6\N
M24]%>&-E<'1I;VX@(&EF(&%N>2!)+T\M17)R;W(@;V-C=7)S"B`@("HO"B`@
M<')I=F%T92!B;V]L96%N('IZ4F5F:6QL*"D@=&AR;W=S(&IA=F$N:6\N24]%
M>&-E<'1I;VX@>PH@("`@<F5T=7)N('1R=64["B`@?0H("`@(`H@("\J@H@
M("`J($-L;W-E<R!T:&4@:6YP=70@<W1R96%M@H@("`JPH@('!U8FQI8R!F
M:6YA;"!V;VED('EY8VQO<V4H*2!T:')O=W,@:F%V82YI;RY)3T5X8V5P=&EO
M;B!["B`@("!Z>D%T14]&(#T@=')U93L@("`@("`@("`@("`O*B!I;F1I8V%T
M92!E;F0@;V8@9FEL92`J+PH@("`@>GI%;F1296%D(#T@>GI3=&%R=%)E860[
M("`OB!I;G9A;&ED871E(&)U9F9E<B`@("`J+PH@('T"@H@("\J*@H@("`J
M(%)E<V5T<R!T:&4@<V-A;FYE<B!T;R!R96%D(&9R;VT@82!N97<@:6YP=70@
M<W1R96%M@H@("`J($1O97,@;F]T(&-L;W-E('1H92!O;&0@<F5A9&5R@H@
M("`J"B`@("H@06QL(&EN=&5R;F%L('9A<FEA8FQE<R!A<F4@<F5S970L('1H
M92!O;&0@:6YP=70@<W1R96%M(`H@("`J(#QB/F-A;FYO=#PO8CX@8F4@<F5U
M<V5D("AI;G1E<FYA;"!B=69F97(@:7,@9&ES8V%R9&5D(&%N9"!L;W-T2X
M("`@*B!,97AI8V%L('-T871E(&ES('-E="!T;R`\='0^6EI?24Y)5$E!3#PO
M='0^+@H@("`J"B`@("H@0'!A<F%M(')E861E<B`@('1H92!N97<@:6YP=70@
M<W1R96%M(`H@("`J+PH@('!U8FQI8R!F:6YA;"!V;VED('EY<F5S970H:F%V
M82YI;RY296%D97(@<F5A9&5R*2!["B`@("!Z>D%T0D],("`]('1R=64["B`@
M("!Z>D%T14]&("`](&9A;'-E.PH@("`@>GI%;F1296%D(#T@>GI3=&%R=%)E
M860@/2`P.PH@("`@>GI#=7)R96YT4&]S(#T@>GI-87)K9610;W,@/2!Z>E!U
M<VAB86-K4&]S(#T@,#L*("`@('IZ3&5X:6-A;%-T871E(#T@65E)3DE424%,
M.PH@('T"@H@("\J@H@("`J(%)E='5R;G,@=&AE(&-U<G)E;G0@;&5X:6-A
M;"!S=&%T92X("`@B\*("!P=6)L:6,@9FEN86P@:6YT('EY<W1A=&4H*2![
M"B`@("!R971U<FX@>GI,97AI8V%L4W1A=&4["B`@?0H*"B`@+RHJ"B`@("H@
M16YT97)S(&$@;F5W(&QE>&EC86P@<W1A=&4("`@@H@("`J($!P87)A;2!N
M97=3=&%T92!T:&4@;F5W(&QE>&EC86P@<W1A=&4("`@B\*("!P=6)L:6,@
M9FEN86P@=F]I9"!Y>6)E9VEN*&EN="!N97=3=&%T92D@>PH@("`@>GI,97AI
M8V%L4W1A=&4@/2!N97=3=&%T93L("!]"@H("`OBH("`@*B!2971U]]>'!R
M97-S:6]N@H@("`JPH@('!U8FQI8R!F:6YA;"!3=')I;F<@>7ET97AT*"D@
M>PH@("`@<F5T=7)N(&YE=R!3=')I;F<H('IZ0G5F9F5R+"!Z>E-T87)T4F5A
M9"P@>GI-87)K9610;W,M>GI3=&%R=%)E860@3L("!]"@H("`OBH*("`@
M*B!2971U71E>]]>'0H2YC:&%R070H<&]S2P@8G5T
M(&9A<W1E<@H@("`J"B`@("H@0'!A<F%M('!O<R!T:&4@<&]S:71I;VX@;V8@
M=&AE(&-H87)A8W1E<B!T;R!F971C:"X@"B`@("H@("`@("`@("`@("!!('9A
M;'5E(&9R;VT@,"!T;R!Y>6QE;F=T:"@I+3$N"B`@("H("`@B!`<F5T=7)N
M('1H92!C:&%R86-T97(@870@<&]S:71I;VX@<&]S"B`@("HO"B`@<'5B;&EC
M(&9I;F%L(&-H87(@>7EC:&%R870H:6YT('!O<RD@>PH@("`@<F5T=7)N('IZ
M0G5F9F5R6WIZ4W1A<G1296%DW!O<UT["B`@?0H"B`@+RHJ"B`@("H@4F5T
M=7)N<R!T:&4@;&5N9W1H(&]F('1H92!M871C:&5D('1E>'0@<F5G:6]N+@H@
M("`J+PH@('!U8FQI8R!F:6YA;"!I;G0@>7EL96YG=&@H*2!["B`@("!R971U
M<FX@>GI-87)K9610;W,M>GI3=&%R=%)E860["B`@?0H*"B`@+RHJ"B`@("H@
M4F5P;W)T<R!A;B!E<G)O<B!T:&%T(&]C8W5R960@=VAI;&4@<V-A;FYI;F<N
M"B`@("H("`@B!);B!A('=E;&QF;W)M960@<V-A;FYE<B`H;F\@;W(@;VYL
M>2!C;W)R96-T('5S86=E(&]F(`H@("`J('EY<'5S:&)A8VLH:6YT*2!A;F0@
M82!M871C:"UA;&P@9F%L;&)A8VL@<G5L92D@=&AI<R!M971H;V0@"B`@("H@
M=VEL;"!O;FQY(&)E(&-A;&QE9"!W:71H('1H:6YG<R!T:&%T(")#86XG="!0
M;W-S:6)L>2!(87!P96XB+@H@("`J($EF('1H:7,@;65T:&]D(&ES(&-A;&QE
M9"P@<V]M971H:6YG(&ES('-E<FEO=7-L>2!W<F]N9PH@("`J("AE+F<N(&$@
M2D9L97@@8G5G('!R;V1U8VEN9R!A(&9A=6QT>2!S8V%N;F5R(&5T8RXI+@H@
M("`J"B`@("H@57-U86P@<WEN=&%X+W-C86YN97(@;&5V96P@97)R;W(@:&%N
M9&QI;F<@<VAO=6QD(&)E(&1O;F4("`@B!I;B!E<G)O<B!F86QL8F%C:R!R
M=6QE<RX("`@@H@("`J($!P87)A;2`@(&5R<F]R0V]D92`@=&AE(&-O9&4@
M;V8@=&AE(&5R<F]R;65S<V%G92!T;R!D:7-P;&%Y"B`@("HO"BTM+2!Z>E-C
M86Y%<G)O<B!D96-L87)A=&EO;@H@("`@4W1R:6YG(&UE<W-A9V4["B`@("!T
M<GD@>PH@("`@("!M97-S86=E(#T@6EI?15)23U)?35-'6V5R<F]R0V]D95T[
M"B`@("!]"B`@("!C871C:"`H07)R87E);F1E>$]U=$]F0F]U;F1S17AC97!T
M:6]N(&4I('L*("`@("`@;65S<V%G92`](%I:7T524D]27TU31UM:6E]53DM.
M3U=.7T524D]273L("`@('T"BTM+2!T:')O=W,@8VQA=7-E"B`@?2`*"@H@
M("\J*@H@("`J(%!U<VAE<R!T:&4@<W!E8VEF:65D(&%M;W5N="!O9B!C:&%R
M86-T97)S(&)A8VL@:6YT;R!T:&4@:6YP=70@<W1R96%M+@H@("`J"B`@("H@
M5&AE>2!W:6QL(&)E(')E860@86=A:6X@8GD@=&AE;B!N97AT(&-A;&P@;V8@
M=&AE('-C86YN:6YG(&UE=&AO9`H@("`J"B`@("H@0'!A<F%M(&YU;6)E<B`@
M=&AE(&YU;6)E<B!O9B!C:&%R86-T97)S('1O(&)E(')E860@86=A:6XN"B`@
M("H@("`@("`@("`@("`@("`@5&AI<R!N=6UB97(@;75S="!N;W0@8F4@9W)E
M871E<B!T:&%N('EY;&5N9W1H*"DA"B`@("HO"BTM+2!Y>7!U<VAB86-K(&1E
M8VP@&-O;G1A:6YS('IZ4V-A;D5R<F]R(&5X8V5P=&EO;BD("`@(&EF("@@
M;G5M8F5R(#X@>7EL96YG=&@H2`I"B`@("`@('IZ4V-A;D5R<F]R%I:7U!5
M4TA"04-+7S)"24<I.PH("`@('IZ36%R:V5D4&]S("T](&YU;6)E<CL("!]
M"@H+2TM('IZ1&]%3T8("`OBH("`@*B!297-U;65S('-C86YN:6YG('5N
M=&EL('1H92!N97AT(')E9W5L87(@97AP<F5S<VEO;B!I<R!M871C:&5D+`H@
M("`J('1H92!E;F0@;V8@:6YP=70@:7,@96YC;W5N=&5R960@;W(@86X@22]/
M+45R<F]R(&]C8W5R<RX("`@@H@("`J($!R971U<FX@("`@("!T:&4@;F5X
M="!T;VME;@H@("`J($!E>&-E<'1I;VX@("!J879AFEODE/17AC97!T:6]N
M("!I9B!A;GD@22]/45R<F]R(&]C8W5R<PH@("`JPHM+2T@>7EL97@@9&5C
M;&%R871I;VX("`@(&EN="!Z>DEN<'5T.PH@("`@:6YT('IZ06-T:6]N.PH
M("`@("\O(&-A8VAE9"!F:65L9',Z"B`@("!I;G0@>GI#=7)R96YT4&]S3#L*
M("`@(&EN="!Z>DUA<FME9%!O<TP["B`@("!I;G0@>GI%;F1296%D3"`]('IZ
M16YD4F5A9#L*("`@(&-H87(@6UT@>GI"=69F97),(#T@>GI"=69F97(["B`@
M("!C:&%R(%M=('IZ0TUA<$P@/2!:6E]#34%0.PH*+2TM(&QO8V%L(&1E8VQA
M<F%T:6]N<PH("`@('=H:6QE("AT<G5E2!["B`@("`@('IZ36%R:V5D4&]S
M3"`]('IZ36%R:V5D4&]S.PH*2TM('-T87)T(&%D;6EN("AL:6YE"!C:&%R
M"!C;VP@8V]U;G0I"B`@("`@('IZ06-T:6]N(#T@3$["@H@("`@("!Z>D-U
M<G)E;G10;W-,(#T@>GI#=7)R96YT4&]S(#T@>GI3=&%R=%)E860@/2!Z>DUA
M<FME9%!O<TP["B`@"BTM+2!S=&%R="!A9&UI;B`H;&5X<W1A=&4@971C0H
M("`@("`@>GI&;W)!8W1I;VXZ('L*("`@("`@("!W:&EL92`H=')U92D@>PH@
M("`@"BTM2!N97AT(&EN<'5T"!L:6YE+"!C;VPL(&-H87(@8V]U;G0L(&YE
M>'0@=')A;G-I=&EO;BP@:7-&:6YA;"!A8W1I;VX*("`@("`@("`@("`@>GI!
M8W1I;VX@/2!Z>E-T871E.PH@("`@("`@("`@("!Z>DUA<FME9%!O<TP@/2!Z
M>D-U<G)E;G10;W-,.PHM+2T@;&EN92!C;W5N="!U<&1A=&4*("`@("`@("`@
M('T*"B`@("`@("`@?0H@("`@("!]"@H@("`@("`O+R!S=&]R92!B86-K(&-A
M8VAE9"!P;W-I=&EO;@H@("`@("!Z>DUA<FME9%!O<R`]('IZ36%R:V5D4&]S
M3#L*2TM(&-H87(@8V]U;G0@=7!D871E"@HM2T@86-T:6]N<PH@("`@("`@
M(&1E9F%U;'0Z(`H@("`@("`@("`@:68@*'IZ26YP=70@/3T@65E%3T8@)B8@
M>GI3=&%R=%)E860@/3T@>GI#=7)R96YT4&]S*2!["B`@("`@("`@("`@('IZ
M071%3T8@/2!T<G5E.PHM+2T@96]F=F%L=64*("`@("`@("`@('T@"B`@("`@
M("`@("!E;'-E('L+2TM(&YO(&UA=&-H"B`@("`@("`@("!]"B`@("`@('T
7("`@('T("!]"@HM+2T@;6%I;@H?0IH
`
end

Hi Maxim,

1. Language API. This stuff begin implemented finally
will allow a 3rd party
plugin writers to embed particulary any language
support to IDEA and benefit
from it's features.

is the new Language API a first step towards your Metaprogramming System?

Thanks.

I wonder, why not use a generated scanner/parser as well? Your Javascript
parser is not trivial, over 1,000 lines, all appears to be hand-written,
and it doesn't appear to do anything too complicated which some grammar based
parser generator could not do, don't you think? Manual parser may be okay
for Javascript, but other languages (like Groovy, for example) may have more
complicated syntax.

Also - most parts of JavascriptLanguage plugin are clear, but what is the
purpose of WordsScanner?

We do build search caches for find usages (as well as refactorings) with
help of WordsScanner. Basically we need to know what words certain file
contains.

-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Also - most parts of JavascriptLanguage plugin are clear, but what is
the purpose of WordsScanner?

Thought so too. FWIW, a Groovy integration is in the works. The current ANTLR grammar for Groovy is modelled after J2SE5. Are you interested in joining the effort?

Regards,

Franck

FWIW, a Groovy integration is in the works.

Great news!

As the matter of fact we do not insist parsers to be written in recursive
down method we've done for javascript. The only thing we need is input token
stream to be marked with PsiBuilder.Marker calls to build a syntax tree that
maps to original text. ANTLR grammar (probably?) could do the same in its
semantic parts. In the worst case one can build its own whatever format tree
and then translate it to IDEA's format by means of traversing and calling
PsiBuilder's methods while traversing.

-

Maxim Shafirov
http://www.jetbrains.com
"Develop with pleasure!"

Thought so too. FWIW, a Groovy integration is in the works. The
current ANTLR grammar for Groovy is modelled after J2SE5. Are you
interested in joining the effort?

Regards,

Franck

Thought so too. FWIW, a Groovy integration is in the works. The
current ANTLR grammar for Groovy is modelled after J2SE5. Are you
interested in joining the effort?

No, it was only an example, if I develop a language plugin, I think it will
be for Ruby.

late reply, I know, but +1 for ruby. ;o)