Maven integration unusable for larger projects

we also have the same issue. i hope jetbrains clears up these performance issues surrounding maven integration and re-indexing files. if idea was developed using maven, the jetbrains developers would have hit this and fixed the perf issues eons ago. you don't hit these issues with a tiny 5 module project.

a few outspoken members of our team are using these performance issues as a reason to move to eclipse. when you spend more time waiting for intellij than coding, development is not a pleasure.

i agree with the prior post that the maven sync can be made a lot more intelligent and only update the module that has changed. frequent freezes (with idea taking exactly 50% of my dual-core cpu) has made me move back to build 7962 (selena)

-Trevor

I agree with the above. We only have a couple of small projects linked together to around 30 modules, and the parsing of the poms is slow. This cannot be done in the background, which makes it worse. The dependency resolution is can however be done in the background, which is great, but it would be better to only reimport the changed modules, thus saving a lot of time.

Another issue I encounter is the lack of feedback. If a typo was made in the dependencies (for example, a wrong version number), an 'empty' dependency is added to the project, instead of an error message or something. That would also be a very nice enhancement, cause now, I have to run every maven target manually first, to see if my dependencies are correct....

same feelings here. Maven itself is painfully slow and can be major PITA, adding intellij reparsing/reloading stuff, it becomes totally unusable. Now, I also cannot see myself moving to Eclipse or Netbeans, but there were moments I had those bad thoughts..Lets hope this gets fixed.

I completely agree. Maven performance is terrible in 8.0 EAP

Same problem.

One way to increase performance is to uncheck the Maven Integration
setting "Automatically generate sources and update folders on import".
When I need to update folders (not very often), I manually click the
"Update folders" icon in the Maven panel instead.


What would be great is to have a button "Re-import only this pom.xml" in
addition to "Re-import" for the "The pom.xml has been changed." promt (the
yellow promt at the top of the editor). The most common use case is that I
have added a dependency, and it shouldn't take me 5-10 minutes to be able
to use that dependency.

Hi,
Could anyone take a CPU snapshot during opening/re-importing and send it to Anton.Makeev@jetbrains.com?

Thanks,
Anton Makeev

Do you know where my "create CPU snapshot" button has gone? It no longer appears on the toolbar as of several EAPs. This is on Linux (Fedora 9).

Please check if yout idea.properties file has "-agentlib:yjpagent" line and there is "yjpagent.dll" file in your idea.home/bin directory.

According to several snapshots I've got, the main problem is with connection to the remote repository. The central repository (repo1.maven.org) is very slow, so please consider using a public mirror, or creating a local repository mirror (using the Nexus manager nexus.sonatype.org).

Thanks,
Anton Makeev

Hi Anton,
this may be true, however, to give you an example:

1. let say I have a pom with artifactA and artifactB in it.
2. I import this into intellij and everything is ok: sources downloaded and all sits in /home/user/.m2/repo directory
3. now I remove artifactA. What I expect is that Idea is smart enough to just remove libraries/dependencies that belong to that artifact and that it doesn't need to go to http://maven.repo.xxx.com

For my part, take whatever it needs to parse re-parse stuff first time, I will wait.
What happens now is that for every simple change it seems whole pom is reparsed and all dependencies checked again and again.

btw, we run archiva here but it's still just to slow..
(http://archiva.apache.org/)

For me, downloading artifacts got significantly faster in the latest builds - thanks. I even notice the difference in speed using Nexus. Would it be possible to download multiple artifacts at once if the remote repository is recognized to be a local one? This could speed up things possibly even more.

The Maven performance needs improvements, I agree. However, honest to God, if your project has 50-100 modules(!), and they are all coupled in the way that you can't separate them into separate projects where each project uses most dependencies as jars (while other developers are working on later versions), then you have much bigger problems to worry about then Idea's performance... Something must be terribly wrong with your package structure and dependencies if you can't split things up.

I see the problem, after the 8.0 will be released I'll try to improve the importing logic so that it won't read all project's poms if only one has been changed.

Thanks everyone for feedback,
Anton Makeev

We have 100 modules because:
1. it's a big project
2. we use OSGi and put much emphasis on building a modular system (e.g. we sometimes put API, implementation and SPI into different bundles/modules)

We are following an agile process which means that over time, every developer gets to touch many parts of the system and needs to be able to do refactorings spanning multiple modules. Jar dependencies on our own code can quickly get in the way here. If you have read my first post, one feature I'd like to see is the ability to quickly define/change a "working set" of modules, where modules within the working set are linked within IDEA whereas all other dependencies (including modules) are taken from the Maven repository. Also it's worth noting that we haven't experienced any performance problems under Eclipse (but we'd prefer to stay with IDEA if possible).

Cheers,
Peter

We host our own repository (Artifactory) and therefore aren't affected by performance problems of repo1.maven.org. "mvn -U eclipse:clean eclipse:eclipse" is at least 10 times faster than opening the project in IDEA, so I suspect there is something wrong about the way IDEA integrates with Maven. Unfortunately we won't be able to use (and therefore order) IDEA 8 until performance improves significantly. Hate to go back to Eclipse but it's the only way out for us. However I'll keep on using IDEA at home. :)

Cheers,
Peter

It is, of course, too late to suggest anything for a project of that scale that is well under way...but, for all it's worth, here are my thoughts on packaging. I am with you on modularity. Everyone talks about SOA, but most people continue to build application-centric architectures. In such architectures, services are not reusable for one obvious reason: people bundle things by their type rather then their relevance to a particular functionality. As the result, the implementation of a single functional piece is often sprinkled through a bunch of modules that have names like "model", "domain", "services", "daos", "configuration", etc. There are even worse examples: "exceptions" and "constants". Needless to say, such packaging makes everything tightly coupled with everything. So, if one developer is working on some kind of "User" functionality, he/she is simultaneously messing with your "model.user", "services.user", etc. This is a very common approach that many projects follow to one degree or another. Even code examples in books and on-line articles advocate that. No wonder, so many projects become so convoluted and dependent on themselves. Why people continue to do so amazes me! As if no one has ever come across a situation when they needed to re-use the same functionality they had just implemented in a web application - for some end-of-day batch process... Except, they can't take it out of their web application because it is totally dependent on the web app's configuration, domain model, daos, services, constants, and exceptions! D'oh!

The simple - and most obvious, I would think - solution is to bundle classes in modules according to their functional/problem area. Each module will provide the problem-specific Domain Model and API. It may also provide default implementations of the service and DAOs that applications may re-use if they can. Or, each client application can provide their own implementation of the service that conforms to the API, if necessary. DAOs often will be application-specific if different apps talk to different data sources. For example, you will have modules like this:

com.mycompany.core (for all your core/common classes, utilities, and, perhaps some marker interfaces, etc.)

com.mycompany.user (User Domain Model, User API, user DAO API+ user svc reference implementation that may be re-used by client apps)
com.mycompany.someservicearea (Domain Model, API, etc.)

Then, you will have your applications that will have your re-usable "platform" components as dependencies. You may have web apps, schedulers, batch applications, etc. Each may provide some app-specific domain objects, DAO implementations, presentation-specific packages, etc. But everything that is reusable will live outside - in your "platform".

com.mycompany.myapplication.etc.

So, your applications should live parallel to your reusable functionality-specific modules.

For example, take a look at how the Spring modules are packaged. There is Spring Core, and each individual Spring piece you want to use requires you to include only the minimum dependencies, not the whole Spring stack.


Agile is good - as long as it is not understood as completely chaotic and messy. In fact, the approach I have just described really helps in agile development because it makes it easier to view modules as complete "black boxes" between the iterations.

Cheers,
Constantine

P.S. Sorry for the off-topic post!

if you add something like this to your settings.xml, you can proxy all request through yuor local maven repository (artifactory, archive, etc). set up your repo to reach out to the outside world if it is not already cached in your repo. Therefore, you rarely will go out to the internet, you'll stay on yuor intranet.

i'm pretty happy with the maven speed right now in v8, although we can still optimize to only update the module whose pom has changed.

my-maven-repo * http://repo-url/artifactory/repo Artifactory ]]>

I just moved to Intellij (EAP #9880) and am hitting the same "re-import triggered for all modules" issue. Has there been any progress in this area since this thread was active last October?

Thanks,

e

Hello Eric,

I just moved to Intellij (EAP #9880) and am hitting the same
"re-import triggered for all modules" issue. Has there been any
progress in this area since this thread was active last October?

Sure. You can find greatly improved Maven integration in Maia EAP builds:
http://www.jetbrains.net/confluence/display/IDEADEV/Maia+EAP

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"

Thanks for the tip.  I'm already experiencing a bit of pain using the 8.1.3 EAP (I'm the one who submitted issues IDEA-23359, 23360, and 23361 for example).  I don't think I want to jump into using a full revisions EAP as my primary dev environment just yet.  Have you all put up any blog entries or other docs that talk about the maven enhancements in Maia?

Thanks,

e

Hi Eric,
I am using Maia and maven implementation is greatly improved.
However, it is EAP and I have some "keyboard not responding" issues with Maia form time to time (at least on linux)

Hi, Eric,

You can try Maia 9.0 M1 and next 9.0 eaps and give some feedback so that I could improve the Maven Integration before 9.0 release.
I would appreciate it very much )

Thanks,
Anton Makeev

Hi MJ,

I've experienced the keyboard not responding issues waay too often and I don't believe they are related to Maia or even IntelliJ. I 've heard of a long standing bug in the AWT toolkit that causes this behavior in Swing apps. The problem has gotten worse for me of the years and I sometimes fall back into Windows. A potential (but not absolute) band-aid would be to put something like this:
AWT_TOOLKIT=XToolkit
#AWT_TOOLKIT=MToolkit
at the top of idea.sh and toggle between the toolkits to see which one locks up the least. Beware as one (and I can't remember which one) works better with Beryl/Compiz-Fusion special effects but may cause the keyboard issue to recurr.

Hi Anton,

Long time Idea and Maven fan... I'm thinking of upgrading to 9.0M1 as well. I have one other suggestion. Could you change the "Before Launch" in Run/Debug configurations to use sensable defaults in the case of a Maven project? I'd like it to automatically know to use Maven and run test-compile for the module I'm working in rather than defaulting to Idea's "Make" command. the problem I keep stubbing my toe on is certain unit test resources never get pushed out to the test output folder prior to running the tests with Idea's "Make". Maybe adding a custom command line with some sort of macro language like in Eclipse (I shudder to mention or suggest Eclipse features) would help here. Consider this. Projects A, B, and C. I'm working in A and set the default in the run config to Run Maven goal 'test-compile' from A. I then pop over to B and create some unit tests there. If I click the class nameand Ctrl+Shift+F10 to run all tests it picks up the defaults and runs goal 'test-compile' from A leaving my new test un-built and resulting in an error class not found. I have to now either set the run config for B or change my default as I work in B. Then we wash, rinse, repeat when we move on to C.

-Cliff

Hi, Clifton,

I see your problem. I believe there is a problem with eigher configuration of test resources or Maven integration: IDEA should automatically pick up test resources and configure project's source folders accordingly.
Could you please attach a sample project so I could fix the problem?

Thanks,
Anton Makeev

I am a long time IDEA user.  The slowness of the Maven integration is killing me too.  Perhaps there are some tricks that could be applied beyond more efficient parsing.

• cache in the workspace file, or somewhere, the dependency relationships (parent/child & import) and modification dates of the poms.
• if a pom changes, only review poms that depend on that pom or are children.  Do not rescan them all unless explicitly told to do so.
• Do not rescan all modules at start up if the poms have not changed.
• When scanning all the modules scan them in the order of the module that has focus and work through its dependencies before modules that are unrelated.  Again, do all the parsing in the background.
• populate data structures that reflect the maven import and parent child relationships and put a facade around them.  That way if a set of dependencies (.jars) are updated, a modules library list is retrieved by traversing the structure.
• try to multithread the parsing.  We are all using multi-core systems.  If the datastructures containing the dependencies are separate and linked, then there will be low contention as they are being parsed.

Our team is trying to figure out how to break our multi-pom project apart because it is taking so long.  It is killing our productivity.  We are also doing SOA, OSGi, git, and GWT, so IDEA is a great fit, it is just getting too slow.

I did try 9.0M1, but could not get it to load the Maven project, so I had to give up and go back to 8.1.

A nice feature would be to be able to "New" a new child pom off of a parent module and have it automatically added to the parent and show up as a module.  Trying to get IDEA to recognize the new module is painful doing a rescan, and sometimes does not work.

Fred

Hi, Frederick,

All that you mentioned has already been implemented in 9.0 branch.
Please tell why couldn't you open your project in 9.0M1, or provide a sample project and I'll try to fix this problem.

Thanks,
Anton Makeev

I can confirm that Maven support is much faster in 9 EAP. One problem we have though is the treatment of Maven's resource folders. On import, IDEA automatically classifies them as source folders. But resource folders may contain just about anything - invalid source code, source code for the classes in the project, etc. In our case we put some of the project's source code in (test) resource folders so that it's available at runtime. As a result, IDEA complains that duplicate classes exist (compile error!). This means that after every Maven import (which I do about once a day), I have to manually remove the source folder tag from resource folders before I can continue to work.

Cheers,
Peter

PS: On second thought, can I use Settings->Maven->Ignored Files to solve my problem? Do I have to enter a path for each affected module?

Hi, Peter,

The problem is that IDEA doesn't current support plain resources folders and in order code-insight features to work Maven have to set such folders as sourse folders.
To temporarily workaround the problem you can exclude such folders from compilation (Settings->Compiler).

Thanks,
Anton Makeev