Regressions, regressions, regressions

My company has been using Rubymine and overall people have been pretty upset over the regressions that occur with every "release". Basic things, such as file renaming were broken with the last release. A number of people are switching to Textmate.

Do you have automated unit tests, acceptance tests? What is your QA process like? It seems like many issues get through.

Thanks,
Brian

7 comments
Comment actions Permalink

Hello Brian,

We're sorry for the inconvenience, but please understand that the product
is currently in preview phase, and cannot yet be considered even beta quality.
We're frequently overhauling low-level parts of the system, which sometimes
causes regressions.

We do have a suite of more than 2000 automated functional tests running on
our TeamCity server, but they don't catch all problems. In particular, they
do not check performance, and they don't check the UI (and the renaming problem
was purely a UI issue - the underlying rename logic worked correctly, but
an NPE occurred while showing the dialog). We're constantly extending the
test suite, and we plan to add automated performance tests to it in order
to avoid perfromance regressions.

We also have a QA engineer doing manual testing and fix verification.

Note that our issue tracker is entirely public, so you can watch all the
activity going on there if you're interested.

My company has been using Rubymine and overall people have been pretty
upset over the regressions that occur with every "release". Basic
things, such as file renaming were broken with the last release. A
number of people are switching to Textmate.

Do you have automated unit tests, acceptance tests? What is your QA
process like? It seems like many issues get through.


--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0
Comment actions Permalink

Thanks for the response. I think the expectations regarding Ruby/Rails is that the HEAD of the SCM should be working without many regressions.

I'm curious also if you have any plans for automated end to end integration testing to make sure UI changes don't negatively affect the fundamental usability of the app.

Overall we're big fans of RubyMine here and looking forward to a beta release.

0
Comment actions Permalink

Hello Brian,

I'm curious also if you have any plans for automated end to end
integration testing to make sure UI changes don't negatively affect
the fundamental usability of the app.


We've never been doing this for IDEA, and so far don't plan to start doing
so for RubyMine - in our opinion, the effort spent on writing the tests and
keeping them up to date doesn't quite pay off.

The regressions themselves are not as bad as long as they're fixed promptly
- and here's where we definitely plan to improve. Fixing the regression next
day is much better than leaving the build with a regression up for a week
and a half.

Overall we're big fans of RubyMine here and looking forward to a beta
release.


Thanks for the encouragement! :)

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0
Comment actions Permalink

I'm really happy with your latest release (PR1). Its starting to look good. :-)

0
Comment actions Permalink

I wish I could say the same as Brian about PR1 but it's not usable for me due to 100% CPU.  I've been happy with builds up to 638 but the last two builds include 715 kills the computer.  I have tried to bump up the vm options, rebuild the cache, etc., but nothing works so far.  I'm on OS X, ruby 1.8.6, gems 1.3.1.  I have reported some crushes using blame core but not sure if that's even related.  Help?

0
Comment actions Permalink

Hello zhao,

I wish I could say the same as Brian about PR1 but it's not usable for
me due to 100% CPU. I've been happy with builds up to 638 but the
last two builds include 715 kills the computer. I have tried to bump
up the vm options, rebuild the cache, etc., but nothing works so far.
I'm on OS X, ruby 1.8

6, gems 1.3.1. I have reported some crushes using blame core but not
sure if that's even related. Help?


Please take a CPU snapshot as described in http://jetbrains.net/devnet/docs/DOC-192
and file a JIRA issue at http://www.jetbrains.net/jira with the snapshot
attached.

--
Dmitry Jemerov
Development Lead
JetBrains, Inc.
http://www.jetbrains.com/
"Develop with Pleasure!"


0

Please sign in to leave a comment.