JetBrains doesn't test its builds ?

Folks, why do I have an impression that JetBrains developers don't even run some minimal set of regression tests for a new build they make avilable for download ?
They don't even try to install it to notice the icon is wrong ? Are we supposed to be their testers reporting about most trivial bugs ? Honestly, I was expecting more attention and care to the crowd of EAP users. Sorry for the language, I really appreciate JetBrains and its product. Really. But the fact they don't test their builds surprises me a lot

55 comments
Comment actions Permalink

I was one of the people who crashed with every keystroke, even after clearing the cache. So what did I do? Went back to an earlier version.

The idea that our JetBrains heros don't test before releasing the builds is pretty nutty as far as I'm concerned. I do suspect that they didn't test it on every possible machine configuration, with every size of project. I don't know what the problem is there. What would it be? No more than about 200,000 different machine configurations. Hell, they could do that easy with only about $4,000,000 and about 50,000 testers. What's the problem?

I also suspect that the JetBrains people count much like the rest of us do...1...2...3. So there were, oh, I don't know, maybe 11 builds that they considered not ready for public consumption before they finally released 908.

Somehow I suspect we'll see a new build, perhaps 909, 910, maybe even a 911 or 912 within the next few days. I doesn't help their efforts in the least to have a build that most EAP users can't use. I'm sure fixing the problems are very close to the top of their todo list. So let's cut them some slack, shall we?

0
Comment actions Permalink

The idea that our JetBrains heros don't test before
releasing the builds is pretty nutty as far as I'm
concerned. I do suspect that they didn't test it on
every possible machine configuration, with every size
of project. I don't know what the problem is there.
What would it be? No more than about 200,000
different machine configurations. Hell, they could
do that easy with only about $4,000,000 and about
50,000 testers. What's the problem?



Java is not "write once, run anywhere" ? I used to think so a long time ago. May be, it should be updated in "write once, try to run it anywhere". :)

0
Comment actions Permalink

Oh come on, it's one thing to complain about EAP not working, or having serious problems, or wondering about the future direction of the project.

It's quite another to start blaming the developers and whining about how many testcases they have. That's an implementation detail that is simply none of our business, and frankly it's insulting and rude to the talented, patient, and incredibly helpful and attentive IDEA developers.

To any dev folk that might be listening, we all love you and the work that you do. The frustration that you hear is out of love and concern. IDEA is one of the few IDE's that people have come to love wholeheartedly, and the prospect of having a lot to dislike/not love is somewhat depressing.

There have always been 'useless' EAP builds, it happens. In the past couple of eap cycles though, we've generally had a new build very shortly thereafter to make the build 'usable', or a workaround mentioned somewhere visible (eg, edit some xml file to enable the toolbar so that 908 can start!).

People get upset when they see a 'bad' build persist and remain the current latest for weeks, I don't know what's happened in this EAP, but for a variety of factors, it looks like there's less goodwill overall, and more people upset at the direction IDEA is heading in. Will this mean any customers will be lost? No, of course not. IDEA users don't even compare it to anything else, they compare it to how they used to feel about IDEA and past releases and degrees of improvement.

0
Comment actions Permalink

Hah! How do you think us OSX users feel? We had a nice custom icon for 70x, but the EAP's have had the generic icon every since build 800! It's too trivial to complain about now, but I'll kick up a fuss once I see a production release being promised within 2-3 months ;)

0
Comment actions Permalink

+if we do a full production testing for each EAP build
you'll get not more than one build in 1.5-2 month. This is not what EAP is for.+

Funny, that's an answer to my original question I've started the thread with. Maxim, believe me I had no intention to fight with you. I'll repeat it the 3-rd time - I'm grateful for your work and will always use IDEA for my work because it's such a beauty.

I just tried to rise an issue of "let's care a bit more about stable builds than addind new features" over here. May be you didn't like the way I said it, so I apologized.

But you're right, let's finish this discussion. Besides, I've got the answers to my questions although it pissed of a couple of people over here.

0
Comment actions Permalink

If I understand Install Anywhere right, it should be fixed in the next EAP
build :)

"Hani Suleiman" <hani@formicary.net> wrote in message
news:3104269.1062079042380.JavaMail.itn@is.intellij.net...

Hah! How do you think us OSX users feel? We had a nice custom icon for

70x, but the EAP's have had the generic icon every since build 800! It's too
trivial to complain about now, but I'll kick up a fuss once I see a
production release being promised within 2-3 months ;)


0
Comment actions Permalink

Evgeny Goldin wrote:

Why do I always have to hear the same excuse for unstable and buggy
builds - "It's an EAP". Because it is, you say ? As a developer -
don't you feel responsible for allowing people to download something
that works and not crashes every time they type a letter or trying to
use a new feature ?


Why do I always have to hear the same excuse for not accepting that this
is a preview -- "But I want to use it for my real work". Because you do
want to use it for real work, you say? As a beta tester - don't you
feel responsible for allowing the IDEA developers to recieve good bug
reports and suggestions to improve the future release of IDEA and not
complain every time you used the product in a way it was not intended?

0
Comment actions Permalink

"Write once, run anywhere" works in theory, but it has been my experience that in practice it's more like "write once, try to run anywhere, tweak, test, tweat, test, tweak, test, run anywhere" At least for projects of any substancial size. If this weren't the case, OS would not be a field when reporting bugs ;)

0
Comment actions Permalink

haven't you heard? That phrase was refactored to
write once debug everywhere ;)

0
Comment actions Permalink

Someone stole Hanis account, this can't be him. ;)

0
Comment actions Permalink

Evgeny:

I feel you are getting out of line here. Its a "EARLY ACCESS PROGRAM".
If you need to work, then clearly using bleeding edge development
version is NOT the way to go. I don't understand why you are trying to
turn the EAP into release-candidates. They aren't.

Would you ever use the development release of NetBeans? God help you if
you do; would you use the development (NOT milestone) release of
Eclipse? Same thing, bugs everywhere.

Jesus calm down and use the release version if your work is so critical.
If they didn't have the EAP program you wouldn't even be complaining, so
it seems to me you've created a problem for yourself and are now whining
about it.

I might as well walk outside and let the air out of my tires and then
call Discount Tire and complain to them about the tires they sold me.

-Riyad

Evgeny Goldin wrote:

I like yours, too. Seems like when lot's of people are screaming "We can't work !" your answer is "Whops, sorry, bad luck." Why not taking some more responsibility for something not working the way it should ?


0
Comment actions Permalink

First, it's not the first time people claim that "using EAP for the real work is not the way it was intended for". Really ? So what was it intended for ? Playing at home ? Developing open-source ? So, let all of us agree that yes, EAP is intended to be used on real projects, because otherwise there's no much left for most of us.

Second, I'm tired of saying that I'm not complaining and never did. The whole point of starting this thread was to make a point that in my opinion Aurora builds are not properly tested before making them available for download. The answer to my question was "We can't, it'll take months". Ok, a legitimate one. I really didn't want an Aurora developers to take my words to the heart.

0
Comment actions Permalink

["Evgeny Goldin"]
First, it's not the first time people claim that "using EAP for the real

work is not the way it was intended for".

Really ? So what was it intended for ? Playing at home ? Developing

open-source ? So, let all of us agree that yes,

EAP is intended to be used on real projects, because otherwise

there's no much left for most of us.

EAP isnt intended for production use. It is intended for people who have
time to test and evaluate an alpha version of the software. Now - if that
software happens to be stable enough for people (like me) to use in
production. If it is having an influencially derrogative effect on your
work - then you are obviously using the wrong build. Try using Andrina which
has alot of current features but is in much more of a stable form - as its
has been through the cutting edge EAP process and has been proven to work by
a mass audience.

(feel free to correct me if Im wrong)

Second, I'm tired of saying that I'm not complaining and never did.


Whether your intention was complaint, or not - you attitude certainly was.

The whole point of starting this thread was to make a point that *in my

opinion* Aurora builds are not properly

tested before making them available for download.


That, however you look at it, is a complaint (even if in the form of a
suggestion).

The answer to my question was "We can't, it'll take months".
Ok, a legitimate one. I really didn't want an Aurora developers to take my

words to the heart.

Unfortunatly there is no way to approach this issue without much heated
feelings being involved ... its happened before, and Im sure it will happen
again.


0
Comment actions Permalink

Well, I had my words misunderstood another time (must be my lucky day :). When I said "We can't work" I didn't talk about myself. So far, EAP builds were never a showstopper for me, there were always some workaround or prevous build to use. I talked about other people - those that start "Build XYZ is unusable", "Build XYZ is way too slow" and the like threads .. Because I felt their pain. And made a correct conclusion that no proper testing is done with those builds. Which was approved by IDEA people - they can't do it because they want to make a build available every week and not every two months.

For some reason, I thought running some regression test won't take more than a day. If it appears to take months - be it but I was still correct saying that "Aurora builds are not tested". Which made some JetBrains people very angry although it's true.

0
Comment actions Permalink

Evgeny Goldin wrote:

First, it's not the first time people claim that "using EAP for the real work is not the way it was intended for". Really ? So what was it intended for ? Playing at home ? Developing open-source ? So, let all of us agree that yes, EAP is intended to be used on real projects, because otherwise there's no much left for most of us.

Second, I'm tired of saying that I'm not complaining and never did. The whole point of starting this thread was to make a point that in my opinion Aurora builds are not properly tested before making them available for download. The answer to my question was "We can't, it'll take months". Ok, a legitimate one. I really didn't want an Aurora developers to take my words to the heart.


Dude I hope JetBrains tosses you off EAP. Maybe then you'll just use
what you're supposed to be using, and we don't have to blow 60 messages
over a lame discussion.

EAP, Alpha code, hear it, learn it, breathe it, and please uninstall the
build and go do the work you're getting paid to do. As far as I can
tell you're not working for JetBrains, and therefore your company pays
your salary to write your code, using a RELEASE BUILD... not provide
free testing and complain about how you can't do things, and how the
icon doesn't look right.

Sheesh man.

R

0
Comment actions Permalink

Hani Suleiman wrote:

Oh come on, it's one thing to complain about EAP not working, or having serious problems, or wondering about the future direction of the project.

It's quite another to start blaming the developers and whining about how many testcases they have. That's an implementation detail that is simply none of our business, and frankly it's insulting and rude to the talented, patient, and incredibly helpful and attentive IDEA developers.

To any dev folk that might be listening, we all love you and the work that you do. The frustration that you hear is out of love and concern. IDEA is one of the few IDE's that people have come to love wholeheartedly, and the prospect of having a lot to dislike/not love is somewhat depressing.

There have always been 'useless' EAP builds, it happens. In the past couple of eap cycles though, we've generally had a new build very shortly thereafter to make the build 'usable', or a workaround mentioned somewhere visible (eg, edit some xml file to enable the toolbar so that 908 can start!).

People get upset when they see a 'bad' build persist and remain the current latest for weeks, I don't know what's happened in this EAP, but for a variety of factors, it looks like there's less goodwill overall, and more people upset at the direction IDEA is heading in. Will this mean any customers will be lost? No, of course not. IDEA users don't even compare it to anything else, they compare it to how they used to feel about IDEA and past releases and degrees of improvement.


+100

0
Comment actions Permalink

People, come on, I really don't complain, please,
read my post below. All the noise is about whether
JetBarins whishes their EAP builds to be more stable
or not.


They need them just stable enough that people like you and me can provide feedback on those features that are newest and most likely to break. That does not mean that every developer should be able to run every EAP on his or her project.

I went 3 months without being able to run the EAP, and in the end I had to spend 100 hours of my own development time (at least) to change how we structure our packages so it would work with the EAP.

I've been here for quite a while, and I have to say that I miss the old days where folks such as yourself didn't exist (except for Hani's compaints about Mac :)

While the statement "It's EAP" doesn't solve your problem, and probabaly leaves a bad taste in your mouth.

Think about this from the developer's perspective. They open themselves up to criticism at an early stage. A stage where in any other situation they wouldn't have to deal with this type of comment. I think they expect a certain amount of restraint. Too much "open criticism" that isn't aligned to their goals is just going to shut down the EAP process, and reduce it's value to JetBrains.

Remember. This is a two way street. You get new software, they get valuable, focused feedback. You are violating that part of the agreement. If we keep this up, we will get exactly what we ask for. Must more infrequent (and shorter) EAP periods with less features (because JetBrains needs to start a formal testing group).

This may be what you want, but it is not what I want. I'm willing to shoulder the testing burden.

Mike

0
Comment actions Permalink

Very well stated. This is another good reason to have moderated forums, even if the community has to do the moderation.

0
Comment actions Permalink

Thank you, totally agree.

0
Comment actions Permalink

Jesus, people, what happened to you ? Why do you make me say the same things over and over again ? Please, stop playing "let's throw this moron out" game and attack me personally. I do report bugs, like all of you. And not only for IDEA, but for many other products I'm using and always downloading every beta I can.

Let's say this in bold so that somebody lazy enough to read the posts before dropping the note will notice:

*All I was trying to figure out whether EAP builds are tested in any way or not. The answer is "no". The formal testing will take too much so the only testing is "using Aurora ourselves". That's it, end of story, bye-bye. *

Did I say I can't work ? No, I can work and do it with pleasure. Did I say I hate when Aurora throws an exception ? No, I report them when I have a time for it. Did I whine about wrong icon ? No, it wasn't the icon that made me scratch my head - I expalined it and apologized for making the wrong assumption. What else do you need ?

Common, guys, don't make me look like a real moron and miss the old days when people like me didn't exist. I'm with all of you being grateful for EAP and reporting bugs. You're taking this way too far and I can feel somehow enjoy doing it.

0
Comment actions Permalink

In article <3202268.1062073642003.JavaMail.itn@is.intellij.net>,
Evgeny Goldin <no_mail@jetbrains.com> wrote:

First, my point about the icon was misunderstood. Did you really think I care
much about how does the EAP icon look like ? Of course no ! My point was the
following "Did anybody at JetBrains install the new version and saw the wrong
icon ?" Hell, it's not that tiny icon that was important to me. The fact that
nobody seemed to notice it before the EAP crowd - that what surpised me. What
made me think that may be nobody even installed the new build ... I
hope my point is clear now.


For what it is worth, a missing icon is the kind of thing that automated
tests often miss. I suspect they do quite a bit of regression testing,
as the EAP builds do seem, in the main, to work more often than they do
not. Getting this kind of stability even in the early stages just
screams automated tests to me.

GUI testing is a lot harder to implement, as many GUI problems can best
be detected by putting a human in front of it and seeing what they
notice.

Having done that, I know that it takes a very special person to check
every detail before shipping. That is why I like JUnit for my code -
the tester can then spend their time looking for cases that my unit
tests miss. Unfortunately, it is a lot easier to spec a JUnit test for
source than for a gui.

Scott

0
Comment actions Permalink

Evgeny Goldin wrote:

*All I was trying to figure out whether EAP builds are tested in any way or not. The answer is "no". The formal testing will take too much so the only testing is "using Aurora ourselves". That's it, end of story, bye-bye. *


I think you may have misunderstood what Maxim said. What I -think- he meant
was that the EAP builds, although they are tested, just don't have the same
level of testing as a final released version. I've no doubt they do have a
large suite of regression tests that they run before and after making each
build. Regression tests just don't catch some kinds of bug though, which is
why we sometimes get unstable EAPs. Hopefully that makes things a bit
clearer, anyway.

Cheerio,
Vil.
--
Vilya Harvey
vilya.harvey@digitalsteps.com / digital steps /
(W) +44 (0)1483 469 480
(M) +44 (0)7816 678 457 http://www.digitalsteps.com/

Disclaimer

This e-mail and any attachments may be confidential and/or legally
privileged. If you have received this email and you are not a named
addressee, please inform the sender at Digital Steps Ltd by phone on
+44 (0)1483 469 480 or by reply email and then delete the email from
your system. If you are not a named addressee you must not use,
disclose, distribute, copy, print or rely on this email. Although
Digital Steps Ltd routinely screens for viruses, addressees should
check this email and any attachments for viruses. Digital Steps Ltd
makes no representation or warranty as to the absence of viruses in this
email or any attachments.

0
Comment actions Permalink

Let's say this in bold so that somebody lazy enough
to read the posts before dropping the note will
notice:

*All I was trying to figure out whether EAP builds
are tested in any way or not. The answer is "no". The
formal testing will take too much so the only testing
is "using Aurora ourselves". That's it, end of story,
bye-bye.*


From your original post:
+"Folks, why do I have an impression that JetBrains developers don't even run some minimal set of regression tests for a new build they make avilable for download ?
They don't even try to install it to notice the icon is wrong ? Are we supposed to be their testers reporting about most trivial bugs ? Honestly, I was expecting more attention and care to the crowd of EAP users. Sorry for the language, I really appreciate JetBrains and its product. Really. *But the fact they don't test their builds surprises me a lot*"+

Now, an extract from your most recent post:

All I was trying to figure out whether EAP builds
are tested in any way or not. The answer is "no". The
formal testing will take too much so the only testing
is "using Aurora ourselves". That's it, end of story,
bye-bye.


Now read your original post and tell us if you really were asking. Looks to me as you were making a statement about the IntelliJ team not spending enough time to please your needs (although your message makes it look like your needs are the needs of all of those participating in EAP).

Common, guys, don't make me look like a real moron
and miss the old days when people like me didn't
exist. I'm with all of you being grateful for EAP and
reporting bugs. You're taking this way too far and I
can feel somehow enjoy doing it.


You're not a moron, but you need to make sure that you spend some time contemplating your posts to the list so they don't cause such a misunderstanding. I think that your poorly thought out original post has now taken up much more time from the EAP-ers than the bugs that are in the EAP have done today.

I think the reason for the strong reaction is that people have seen too many cool things like the EAP disappear because users started to make unfair expectations to the people behind them.

We all love the EAP and we don't want to see IDEA 5.0 developed the "JBuilder way".

Take care.

0
Comment actions Permalink

I agree, my first post wasn't accurate, I realized it and apologized for being impolite very quickly (the post about a beer down the page).
What didn't stop many others from endless attacks saying basically the same things (guess what, known to me from the very beginning).

So .. I think we understand each other now and I apologize again for being rude and impolite to JetBrains people.

0
Comment actions Permalink

You have got to be kidding me.

I think the IDEA guys took to heart the complaints of build quality. I
think their last 3 EAP releases are fantastic. I use it for 8 hours a day.
This latest is a little buggy, but it is EAP. So I close/open idea maybe
1-3 times a day. I don't lose anything and I have access to the new
features that make me more than productive to account for the restarts (that
are quick anyway).



"Evgeny Goldin" <no_mail@jetbrains.com> wrote in message
news:29479915.1062062087908.JavaMail.itn@is.intellij.net...

Folks, why do I have an impression that JetBrains developers don't even

run some minimal set of regression tests for a new build they make avilable
for download ?

They don't even try to install it to notice the icon is wrong ? Are we

supposed to be their testers reporting about most trivial bugs ? Honestly, I
was expecting more attention and care to the crowd of EAP users. Sorry for
the language, I really appreciate JetBrains and its product. Really. But the
fact they don't test their builds surprises me a lot
>


0

Please sign in to leave a comment.