Continuous compilation

Hi.

Can someone tell me why we need continuous compilation?

Thanks,
Amnon

0
34 comments
Avatar
Permanently deleted user

:) In fact I think we need a faster compilation. And if the compilation is fast enough, why not to have it continious?=)

0

:) In fact I think we need a faster compilation. And
if the compilation is fast enough, why not to have it
continious?=)


We have a project with 1280 java source files (exact reading as of today =)). The built in incremental build system keeps compile times (Make project) very low, in the range of 1..10 seconds. I very much prefer to spend any extra processor time with continuos inspection than with continuous compile.

IntelliJ runs quite smoothly even with a lot of inspections enabled. BTW: The system is lowly laptop with 1.6GHz mobile pentium but 1024Mb of memory with windows swapfile disabled to keep the GCs in memory.

I find it hard to imagine a situation where a system equipped with modern dual core processor and at least 1GB of ram suffers from long compile times?

Even the whole customer release build with ANT (clean all) together with CVS update, some obfuscations, library buildings, directory formations, help file indexing, +600 *.png resources, installer creation, etc, etc.. takes less than 30 seconds on an old Athlon server with 1.6GHz Athlon/Duron? and 512Mb of memory.

I'm guessing that the release build would take ~10 seconds with AMD X2 3800+ equipped with modern, fast hard disk?

What kinds of compile times others get?

0

The cynical answer is "because Eclipse has it".

The slightly less cynical answer is that I'll soon be developing on a dual core system, and expect IDEA to be able and willing to wring every last cycle out of both cores. Continuous compilation is probably necessary to achieve that.

I'm not so much interested in continuous compilation, per se, as I am in continuous global inspection, continuous testing, and continuous code coverage calculation. Thanks to the "red box" and programming by intention, I see compile errors approximately once a month, mostly due to bad merges. It's shocking, but true. That means that continuous compilation would at best save me a bit of time on test launch, and with modern Java compilers being so fast there's just not much payoff. On the other hand, I run at a very high level of inspections, and have a boatload of unit tests, and live and die by coverage numbers (at least in my day job). Having those inspections/test/coverage checked continuously in the background would allow me to code with considerably more confidence , and save me considerable time.

--Dave Griffith

0

Lucky you!
IDEA project takes ~10min to clean build and production build with obfuscation, installer, documentation generation etc. takes ~ 35 min on a top notch machine.

0

I find it hard to imagine a situation where a system
equipped with modern dual core processor and at least
1GB of ram suffers from long compile times?


Unfortunately, I don't think most programmers out there actually work on a machine like this, specially here on the other side of the equator.

Sure, development companies probably value programmer productivity, and try to provide the best environment possible. Companies where software development is done only for internal use, however, show a different story: managers don't understand how important having a top-notch machine is. Right now I'm working on a 2.4Ghz Pentium 4 (non-HT) sporting the luxurious amount of 512M RAM.

That being said, I agree with you: I don't really want a faster compiler, but faster running inspections.

0

Hi.

Where do you find this option? Does it exist in 5.1 (4121)?

Cheers
Oliver

0

Each time I try out Eclipse, that is one of the first things I turn off. It is quite
annoying. Not having this, at least by default, is a selling point for IDEA in my opinion.

I would greatly prefer seeing faster parsing and inspections than continuous compilation!

Please don't do this just because Eclipse does it.

JetBrains is a leader, don't turn into a follower.

0

Each time I try out Eclipse, that is one of the first things I turn
off. It is quite annoying. Not having this, at least by default, is a
selling point for IDEA in my opinion.


I could not agree any more. Not that I need the instant compilation, but
for me it is not annoying. Much more annoying with IDEA in the last weeks
is, that even after closing and reopening a project, it does not pick up new
updated files correctly. That's why I work with Eclipse at the moment.

Tom

0

One of the benefits of continuous compilation is having error markers in the package tree.

This makes for a more intuitive fix-and-test experience after big refactorings. Per-package (and sub-package) solving of compilation errors works much better than the experience provided by IDEA: Compile and get the first 100 errors (which often are not the best spot to start working)

The ability to run the test suite while having the project in a state of great flux (again, after big refactorings) is another benefit.

0
Avatar
Permanently deleted user

It would be great if you could file a detailed request with these problems to JIRA. I promise to do my best to work on it before 5.1 release.

Eugene.

0

Maybe you just hit on something that can be of benefit. Why not allow
it to be enabled for files that have changed or something like that? If
I do a refactor that changes 5 files, I want to know if they have
errors? This can also be very helpful after a CVS merge of several files.

Taras Tielkes wrote:

One of the benefits of continuous compilation is having error markers in the package tree.

>

This makes for a more intuitive fix-and-test experience after big refactorings. Per-package (and sub-package) solving of compilation errors works much better than the experience provided by IDEA: Compile and get the first 100 errors (which often are not the best spot to start working)

>

The ability to run the test suite while having the project in a state of great flux (again, after big refactorings) is another benefit.

0

Eugene, if I would have detailed information, I would have posted it
already, but it just happens once in a while, not reliable reproducible.

Tom

0

Yes, I think that the that Eclipse presents this information is good:
-overlay icons in tree views (project, package)
-issues 'tab' on bottom of screen (with filtering etc)

Of course it's quite easy produce an information overload to the developer.

0

The truth is that it enables several other useful features, many of which have already been mentioned. Faster feedback, essentially. I'm sure they'll make it nicely configurable so it doesn't get in the way if your project doesn't suit it.

0

In article <29885998.1138074545485.JavaMail.itn@is.intellij.net>,
dukie_banderjee@hotmail.com says...

The truth is that it enables several other useful features, many of which have already been mentioned. Faster feedback, essentially. I'm sure they'll make it nicely configurable so it doesn't get in the way if your project doesn't suit it.


It would be nice to have a the window on the bottom pane display compile
problems as you refactor. Just today, I moved a static util class into a
module that introduced a compilation problem in several files in
complete different package, so the red ticks in the gutter weren't seen
until I tried to debug and had the compile problems.
--
-


David H. McCoy


-


0

Well, whenever I work with Eclipse, its also the first thing I turn off...
What I like very much about IDEA is that it doesn't depend on the
compiler for error-feedback, that was a BIG problem with JBuilder and
still is annoying with Eclipse, somehow a compiler isn't suitable for
this b/c it was created to compile correct code and reject (and report)
bad one.
In the IDE, I want to have feedback even for classes that don't compile,
for the parts which are correct.
Of course, it depends on the way it is implemented but since VAJ/Eclipse
advertised this as a "selling point" I'd think that they put quite some
effort into it... and it still doesn't work very well (e.g. in Eclipse I
have to turn auto-compilation off every now and then and rebuild all
projects... on a stupidly-structured project, admittedly).
So b/c I don't want it I'd turn it off nonetheless, so I'd also rather
see improvements in other areas, but if you (JB) think its necessary,
ok, but PLEASE don't rely on compiled code (bytecode) for any feedback.

kind regards,

Messi

Rob Harwood wrote:

The truth is that it enables several other useful features, many of which have already been mentioned. Faster feedback, essentially. I'm sure they'll make it nicely configurable so it doesn't get in the way if your project doesn't suit it.

0

In article <dr5nf2$spj$1@is.intellij.net>, bmesserer_NO_SPAM@gmx.at
says...

Well, whenever I work with Eclipse, its also the first thing I turn off...
What I like very much about IDEA is that it doesn't depend on the
compiler for error-feedback, that was a BIG problem with JBuilder and
still is annoying with Eclipse, somehow a compiler isn't suitable for
this b/c it was created to compile correct code and reject (and report)
bad one.
In the IDE, I want to have feedback even for classes that don't compile,
for the parts which are correct.
Of course, it depends on the way it is implemented but since VAJ/Eclipse
advertised this as a "selling point" I'd think that they put quite some
effort into it... and it still doesn't work very well (e.g. in Eclipse I
have to turn auto-compilation off every now and then and rebuild all
projects... on a stupidly-structured project, admittedly).
So b/c I don't want it I'd turn it off nonetheless, so I'd also rather
see improvements in other areas, but if you (JB) think its necessary,
ok, but PLEASE don't rely on compiled code (bytecode) for any feedback.

kind regards,

Messi

Rob Harwood wrote:

The truth is that it enables several other useful features, many of which have already been mentioned. Faster feedback, essentially. I'm sure they'll make it nicely configurable so it doesn't get in the way if your project doesn't suit it.

Exactly. Provide feedback, perhaps, about files that won't compile, but
don't make actions depedent on successful compilations.
--
-


David H. McCoy


-


0

It seems like continuous compilation is a step backwards for JetBrains.

AFAIK so far, all (or at least most) of the excellent productivity enhancing features of
IDEA have been built upon the intelligent parsing and PSI developed by JetBrains. This
works so much better than the continuous compilation feedback in Eclipse. I would hate
to see JetBrains start building features upon continuous compilation instead of enhancing
and extending the parsing and analysis of the PSI.

I would like to see (configurable) feedback in the project toolbar in the form of an icon
overlay for classes and files with errors in them. It seems to me that this could be
implemented by analyzing files that are not currently open in an editor window using the
same techniques that IDEA uses on open files. This should definitely be done in
low-priority threads so that it does not impact the user editing experience. Using the
compiler to do this would be the lazy way to get it done quickly and would introduce a whole
host of new issues in the long run.

Tim

0

It seems this topic finally requires a clarification.
No way we're going to use continuous compilation instead of PSI or depend
on up to date class files in IDEA's analyses or whatsoever.
Continuous compilation is just what it says: compile continuously in background
so you'll be able to run your tests faster etc.
Additionally, we'll use results of the compilation for the project view highlighting.
It doesn't seem possible to use IDEA editor highlighting for this purpose
since it just too slow for such kind of batch operation and chances low it
will react fast enough on changes made to be useable.

-


Maxim Shafirov
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


0

This is already in plan for Demetra, and I'm pretty sure that they won't be trying to do it by retrofitting hundreds of analyses into a continuous compiler.

--Dave Griffith

0

Yet another very annoying feature that I'll have to turn off right away UNLESS of course the code can be written such that the editor actually WORKS. Right now with all the stupid inspections, my semi-new T42 with a 1.6Ghz Pentium M struggles when IDEA decides to reparse .java files each and everytime I visit them. Same goes for large XML-files.

0

Maxim Shafirov (JetBrains) wrote:

It doesn't seem possible to use IDEA editor highlighting
for this purpose since it just too slow for such kind of batch operation
and chances low it will react fast enough on changes made to be useable.

To me this is part of a bigger issue - that of 'IDEA doesn't make good
use of my idle time'. If I go off for a tea break, or sit reading a web
page for half an hour, why can't IDEA be inspecting my files in the
background as well as compiling them? It doesn't really matter how slow
it is, just the fact that it's doing something with my idle CPU time
would be good.

Also, it does seem to be jumping the gun somewhat to be talking about
doing less important things in the background when IDEA won't even carry
on inspecting the file you have open in your editor if it doesn't have
focus! I left it analysing a class while I wrote this and it could have
finished some time ago, but no, I still must wait for it when I return.
To me this is a fundamental flaw in IDEA's workflow that was brought
up versions ago and should have been remedied by now. I hope it is
considered along with this subject.

N.

0

I'm not sure this is necessarily desireable. Often during development, when
my editor is not in focus, it is because I'm using some other development
tool or doing testing, and I don't want CPU cycles consumed by IDEA. Granted,
thread priorities can mitigate the impact idea would have on other processes,
but not entirely. Take a look at the myriad desktop search tools and they're
convoluted schemes for getting indexing done without impact the user experience.
I've yet to see a scheme that correctly defines "idle" as both "when I'm
not using the keyboard and mouse" AND "when some other process isn't consuming
CPU and/or IO resources".

Even the best of these schemes doesn't seem to know to make way for an IO
hungry task.

Hello Nathan,

Also, it does seem to be jumping the gun somewhat to be talking about
doing less important things in the background when IDEA won't even
carry
on inspecting the file you have open in your editor if it doesn't
have
focus! I left it analysing a class while I wrote this and it could
have
finished some time ago, but no, I still must wait for it when I
return.
To me this is a fundamental flaw in IDEA's workflow that was brought
up versions ago and should have been remedied by now. I hope it is
considered along with this subject.

N.



0

Russ Egan wrote:

Even the best of these schemes doesn't seem to know to make way for an
IO hungry task.

Those in the know correct me if I'm wrong, but I don't think this is an
IO hungry task... it is simply analysing the code in memory and
performing checks and inspections on it, all of which is cpu intensive,
not io intensive. And even if it is extended to operate across your
entire codebase in the background, the bulk of the work will still be
processing, with io operations just limited to file access.

As for doing work in the background, well there are plenty of examples
of distributed computing apps out there designed to use your spare
cycles without interrupting your real work, why would this be any
different? At worst different levels of background features should be
able to be turned off or temporarily suspended if you use other tools
and feel their performance is compromised by a low_pri thread. But in
the case where you don't use other tools or you don't mind the odd
percent here and there, I'd love for IDEA to intelligently work alone
like this and decrease my idle time.

N.

0

Those in the know correct me if I'm wrong, but I don't think this is an IO hungry task... it is simply analysing the code in memory and performing checks and inspections on it, all of which is cpu intensive, not io intensive. And even if it is extended to operate across your entire codebase in the background, the bulk of the work will still be processing, with io operations just limited to file access.]]>

Maxim may have better information, but the short answer is, you're wrong. For all but a handful of inspections, processing time is not the issue. The bulk of time spent by inspections is either the pure I/O involved in parsing the files into internal PSI trees, or resolving external references, which also seems to be mostly I/O against disk caches and indices. The tricky bits of inspection, walking trees and checking for possible error conditions, are otherwise essentially free. The cost of an inspection or error annotation is almost entirely related to how much external reference resolution it has to do, and whether those resolutions hit the internal caches or not.

That's true for Java, at least. I have no idea why checking large XML files can be so problematic (other than that large XML files are much more common than large Java files).

--Dave Griffith

0

"It seems like continuous compilation is a step backwards for JetBrains.

AFAIK so far, all (or at least most) of the excellent productivity enhancing features of
IDEA have been built upon the intelligent parsing and PSI developed by JetBrains. This
works so much better than the continuous compilation feedback in Eclipse. I would hate
to see JetBrains start building features upon continuous compilation instead of enhancing
and extending the parsing and analysis of the PSI."

I completely agree. Continuous compilation, imho, is a waste of developer time for JetBrains programmers. The only reason to have it is to make the eclipse crowd feel more comfortable trying IDEA. Many of them freak out without continious compilation and they simply can't deal with it. Other than that, there really isn't any logical argument to having it. Like everyone else is saying, there's a million other things I'd like IDEA to be doing with my CPU time.

0

So does that mean you are in favor of it inspecting in the background
when it does not have focus?

Dave Griffith wrote:

<i>Those in the know correct me if I'm wrong, but I don't think this is an
IO hungry task... it is simply analysing the code in memory and
performing checks and inspections on it, all of which is cpu intensive,
not io intensive. And even if it is extended to operate across your
entire codebase in the background, the bulk of the work will still be
processing, with io operations just limited to file access.</i>

>

Maxim may have better information, but the short answer is, you're wrong. For all but a handful of inspections, processing time is not the issue. The bulk of time spent by inspections is either the pure I/O involved in parsing the files into internal PSI trees, or resolving external references, which also seems to be mostly I/O against disk caches and indices. The tricky bits of inspection, walking trees and checking for possible error conditions, are otherwise essentially free. The cost of an inspection or error annotation is almost entirely related to how much external reference resolution it has to do, and whether those resolutions hit the internal caches or not.

>

That's true for Java, at least. I have no idea why checking large XML files can be so problematic (other than that large XML files are much more common than large Java files).

>

--Dave Griffith

>

0


I certainly think it would be a great option, particularly for those developing on dual-core systems with fast disks. I'm also in favor of having IDEA continuously running unit tests and collect code coverage information in the background, given a perfect world. Those opinions are pretty much independent of whether inspections are processor or IO bound.

--Dave Griffith

0

I agree. I have a dual CPU + ht machine, and I hate to see Idea stop
just because I am doing something else. I am doing something else
because I don't want to have to sit idle while the program does
something that may take a while.

Dave Griffith wrote:

I certainly think it would be a great option, particularly for those developing on dual-core systems with fast disks. I'm also in favor of having IDEA continuously running unit tests and collect code coverage information in the background, given a perfect world. Those opinions are pretty much independent of whether inspections are processor or IO bound.

>

--Dave Griffith

0

Dave Griffith wrote:

Maxim may have better information, but the short answer is, you're
wrong. For all but a handful of inspections, processing time is not
the issue. The bulk of time spent by inspections is either the pure
I/O involved in parsing the files into internal PSI trees, or
resolving external references, which also seems to be mostly I/O
against disk caches and indices. The tricky bits of inspection,
walking trees and checking for possible error conditions, are
otherwise essentially free. The cost of an inspection or error
annotation is almost entirely related to how much external reference
resolution it has to do, and whether those resolutions hit the
internal caches or not.


In this case I think you have another definition of "I/O-intensive" and
"CPU-intensive" than I have. When I'm waiting for all the inspections
to finish analyzing the file I'm working with, the CPU meter is
generally pegged at 100% and the hard disk LED is completely dark,
because everything is already in RAM. To me this would mean that the
process is CPU-intensive, even though it may be external method calls
for reference resolution that uses all that CPU time, rather than the
inspection code itself. Or have I misunderstood something in your
argument? It's late and I'm a bit tired...

I still regularly curse IDEA for not continuing with inspections when it
loses focus, though. It's quite annoying having to wait a full minute
in some cases without being able to do anything else like having
Thunderbird in the foreground, reading a few newsgroup posts (which
consumes maybe all of 0.1% of the CPU on average, given the idle time
when I'm just reading something, leaving 99.9% for IDEA to waste).

0

Please sign in to leave a comment.