Plugin with only a single ExternalAnnotator

I want to create an external annotator plugin for a language, but I got stuck at getting it to work even for java. When I add a Java ExternalAnnotator and put a print (or try to break on collectInformation), it never gets executed:

    <externalAnnotator language="JAVA" implementationClass="Annot" />
class Annot: ExternalAnnotator<Int, Int>() {
    override fun collectInformation(file: PsiFile, editor: Editor, hasErrors: Boolean): Int? {
        println("a")
        return null
    }
}

Am I missing some idea-plugin configuration or is this behavior expected? What can I do about it?