Disable annotation processors for intellij import from gradle buildscript

I'm trying to make importing a gradle project into intellij idea work the way I need (because right now I have to redo a manual part of configutation on every refresh, which appears to happen automatically very often).

In gradle 4.x and older, annotation processors just "magically" worked in gradle when I had them in any of my dependencies.
An accidental consequence of that, was that when intellij importee the project, it didn't enable any annotation processors at all. So annotation processors ran in gradle build, but not in intellij build.
This just happened to be the behavior I wanted.

Now with gradle 5.x I have to explicitly add annotation processors in gradle buildscript. This works fine for gradle build, but now intellij also imports that. This means that now, when I import my project - intellij always enabled annotation processors.

And here is the problem - the annotation processor I have require specifying some special parameters to not throw a lot of errors. And on top of them, even when configured, it doesn't properly support being run inside intellij (there are some invalid assumptions it makes that breaks incremental build in intellij, I don't remember the details). So I have to manually disable the annotation processor every time I import intellij project.

While it's possible to disable annotation processor entirely in gradle, it is actually required for the build to produce a valid jar file for running in production as it generates some metadata that isn't needed when running from IDE.

I tried to use the idea gradle plugin to configure it but that option doesn't seem to be exposed in the plugin. Not only that, the option is actually on "compiler.xml" file, and that file isn't exposed in the idea plugin at all.

Is there any way to have the old import behavior without downgrading gradle version?