IDEA to IDEA developers..

Do you use standard collection classes to store lots of indexing data or other similar stuff?

If you do you should take a look at fastutil http://freshmeat.net/projects/fastutil/ package. The primitive container classes can sometimes save as much as 200..800% of memory.

As an example I need to store and retrieve integer values from a set and the fastutil's IntOpenHashSet consumes 8x less memory than regular HashSet.. Plus it's much faster too =)

0
55 comments
Avatar
Permanently deleted user

sounds great! but is this possible?! why does the java developers don't implement the class into the jdk when they are so great?

@jetbrains: please test it!

0
Avatar
Permanently deleted user

Hezekiel wrote:

Do you use standard collection classes to store lots of indexing data or
other similar stuff?

If you do you should take a look at fastutil
http://freshmeat.net/projects/fastutil/ package. The primitive container
classes can sometimes save as much as 200..800% of memory.


We use a (code derived from) Trove4j: http://trove4j.sourceforge.net/

Friendly,
Dmitry
--
Dmitry Lomov
IntelliJ Labs / JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Avatar
Permanently deleted user

are these collections faster than fastutil?

0
Avatar
Permanently deleted user

Dmitry Lomov (JetBrains) wrote:

>We use a (code derived from) Trove4j: http://trove4j.sourceforge.net/
>

>


Could you let us use it in plugins?

Alain

0
Avatar
Permanently deleted user

sounds great! but is this possible?! why does the
java developers don't implement the class into the
jdk when they are so great?


Sun's view of java is simplicity above all. Total size of fastutil package is >8MB. You can guess that for many different implementations and for many different containers with many different primitives there are lot of permutations: e.g. just some interfaces

Int2ObjectMap
Int2IntMap
Byte2ShortMap
Long2BooleanMap
BooleanList
ObjectSet
ShortSortedSet
LongSortedSet
.
.
etc..

If java would have generics support for primitives this mess could be avoided, but in this case the guest for simplicity leads to much greater complexity. The java source files for fastutil are generated by C preprocessor...

Standard containers store everything as objects which is a huge overkill if all you need is just numbers.

Go ahead and experiment with IntOpenHashSet, store 1000000 ints to it and then do the same with HashSet. Measure memory consumption..

0

It's LGPL, so shouldn't you grant access to your derived code to all of us?

0
Avatar
Permanently deleted user

Stephen Kelvin wrote:

It's LGPL, so shouldn't you grant access to your derived code to all of
us?


I guess so, but the actual derived code is just some stuff moved to
different packages, with some defaults tweaked probably (I am not sure now
to what extent).

We can give it easily to anyone, but you won't be interested :)

Friendly,
Dmitry
--
Dmitry Lomov
IntelliJ Labs / JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Avatar
Permanently deleted user

It's LGPL, so shouldn't you grant access to your
derived code to all of us?


Now this is interesting... =)

0
Avatar
Permanently deleted user

Dmitry Lomov (JetBrains) wrote:

>We can give it easily to anyone, but you won't be interested :)
>

>

Please put it in the OpenAPI.

In my plugin test code, I create a lot of hashTables with a wrapped int
as a key.
I wouldn't mind writing less code, and my app. running faster, at no cost.


Alain

0
Avatar
Permanently deleted user

Actually, I think only modifications they make to the actual Trove software would have to be released. The LGPL doesn't have the GPL's derived work clause, ie, if I link, I have to release, or the "viral" effect.

Correct me if I'm wrong...

--pete

0
Avatar
Permanently deleted user

Peter Morelli wrote:

Actually, I think only modifications they make to the actual Trove
software would have to be released. The LGPL doesn't have the GPL's
derived work clause, ie, if I link, I have to release, or the "viral"
effect.

Correct me if I'm wrong...


You're wrong ;)

... or rather, this is more complex and depends on what jetbrains means
about "code derived from".

Without any further clarification, my understanding of "code derived
from" is that some source files or source snippets where copied,
changed (modified) and are now part of the idea code base.

If this is correct, Jetbrains has to either release the copied/changed
code in lgpl form or remove it from idea code base.


I'll not enter here into any copyright discussion about what should be
done if they reimplemented the code while looking at the trove source
(without any copying of source files/source snippets).


As to granting access (as in enabling calls by plugins) that's not
mandatory, only the derived source has to be released.


Just to finish, IANAL, don't whish the horrible fate of becoming one
and these are only my .02? (last checked worth 1.22 more than .02$ :)

0
Avatar
Permanently deleted user

What is IANAL? "I am not a liar"? "I'm anal"? "I am not a lawyer"? Oh,
they're all the same:)

"Carlos Costa e Silva" <carlos@keysoft.pt> wrote in message
news:br7u8c$ub1$1@is.intellij.net...

Peter Morelli wrote:

>

Actually, I think only modifications they make to the actual Trove
software would have to be released. The LGPL doesn't have the GPL's
derived work clause, ie, if I link, I have to release, or the "viral"
effect.

>

Correct me if I'm wrong...

>

You're wrong ;)

>

... or rather, this is more complex and depends on what jetbrains means
about "code derived from".

>

Without any further clarification, my understanding of "code derived
from" is that some source files or source snippets where copied,
changed (modified) and are now part of the idea code base.

>

If this is correct, Jetbrains has to either release the copied/changed
code in lgpl form or remove it from idea code base.

>
>

I'll not enter here into any copyright discussion about what should be
done if they reimplemented the code while looking at the trove source
(without any copying of source files/source snippets).

>
>

As to granting access (as in enabling calls by plugins) that's not
mandatory, only the derived source has to be released.

>
>

Just to finish, IANAL, don't whish the horrible fate of becoming one
and these are only my .02? (last checked worth 1.22 more than .02$ :)



0
Avatar
Permanently deleted user

Brad wrote:

What is IANAL?



Acronym database: http://www.acronymfinder.com/

or

Search google for: define ianal


Results:

IANAL: I Am Not A Lawyer

0
Avatar
Permanently deleted user

Carlos Costa e Silva wrote:

Peter Morelli wrote:

>> Actually, I think only modifications they make to the actual Trove
>> software would have to be released. The LGPL doesn't have the GPL's
>> derived work clause, ie, if I link, I have to release, or the "viral"
>> effect.
>>
>> Correct me if I'm wrong...


You're wrong ;)

... or rather, this is more complex and depends on what jetbrains means
about "code derived from".

Without any further clarification, my understanding of "code derived
from" is that some source files or source snippets where copied,
changed (modified) and are now part of the idea code base.


Let me clear this once and for all.
IANAL, but LGPL allows us to
1. Use Trove4J in commercial product as a library
2. Modify Trove4J as we see fit, provided we release modified library
under LGPL. We are free to use modified version as a library
in commercial product.

Our modifications to Trove4J are extremely unimportant (minor repackaging
and defaults tweaking).
We are ready to give it away to whoever needs it, but there were none such
yet.

Friendly,
Dmitry
--
Dmitry Lomov
IntelliJ Labs / JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Avatar
Permanently deleted user

Dmitry Lomov (JetBrains) wrote:

>We are ready to give it away to whoever needs it, but there were none such
>yet.

>


I don't need you to "give" it to me, but I'd like you to "let us use it
through the openAPI".

Alain

0
Avatar
Permanently deleted user

I was being facetious.

"Carlos Costa e Silva" <carlos@keysoft.pt> wrote in message
news:br8d7p$8f0$1@is.intellij.net...

Brad wrote:

>

What is IANAL?

>
>

Acronym database: http://www.acronymfinder.com/

>

or

>

Search google for: define ianal

>
>

Results:

>

IANAL: I Am Not A Lawyer



0
Avatar
Permanently deleted user

Brad wrote:

I was being facetious.

Somebody told me once that there are only two words in the English
language which contain each vowel, once only, in alphabetical order.
Anybody know what the other one is?

0
Avatar
Permanently deleted user

isn't 'y' a vowel in english ? as far as I remember my distant school days, it is in french

0
Avatar
Permanently deleted user

'y' is a semivowel in english :)

N.

Thibaut wrote:

isn't 'y' a vowel in english ? as far as I remember my distant school days, it is in french

0
Avatar
Permanently deleted user

Thibaut wrote:

isn't 'y' a vowel in english ? as far as I remember my distant school days, it is in french

It's a kind-of vowel. Anyway, you can include it pretty easily (said he
facetiously!)
A+,
Robbie

0
Avatar
Permanently deleted user

Wearing a L hat:

2. Modify Trove4J as we see fit, provided we release modified library
under LGPL. We are free to use modified version as a library
in commercial product.


I was just noting that Jetbrains has not done that (release a modified
library). This means that you are violating the LGPL license.

Our modifications to Trove4J are extremely unimportant (minor
repackaging and defaults tweaking).
We are ready to give it away to whoever needs it, but there were none
such yet.


Not an excuse. Where has Jetbrains posted a notice that there is trove
derived code in Idea, and where is the link for the download?

[btw, I just noticed that there isn't a license for trove under the
idea/license dir]


My main point being that Jetbrains has to be aware that a LGPL
violation is being done.


Removing hat (phewww, what a relief :)


1. I prefer extending libs (override constructor/methods to obtain the
desired result) to changing them, just so that this doesn't happen.
This has the (hoped for) advantage that new lib versions can be
integrated without changes.

2. In the few cases where I made small changes to lgpl libs, I
submit/mail them back to the authors and don't bother releasing
modified source. Anyone wanting the modification can go to the original
site and get the patch.

3. Releasing a modified lib just for small changes seems a bit silly
(even it it's what the lib authors said they want done).

0
Avatar
Permanently deleted user


I you needed Trove functionality in a plugin (which is way unlikely, IMHO, as Trove doesn't buy you anything over the Collections API except under fairly extreme performance circumstances), why wouldn't you just ship it in as part of the plugin dir?

That said, JetBrains should probably Do The Right Thing and drop their modified Trove into the lib directory, and the license into the license dir. It's not like these things are all that onerous.

--Dave

0
Avatar
Permanently deleted user

Dave Griffith wrote:

>I you needed Trove functionality in a plugin (which is way unlikely, IMHO, as Trove doesn't buy you anything over the Collections API except under fairly extreme performance circumstances), why wouldn't you just ship it in as part of the plugin dir?

>


As I wrote earlier, I just need hashMaps with int keys. Wrapping and
unwrapping the int key works fine, but I could use the simpler code, and
better performance.

If their - IDEA's - derived code is general enough to be useful to us,
moving it to some OpenAPI util package is a no-brainer. They could do
that simple thing for us.. if their derived code is general enough to be
useful to us.



Alain

0
Avatar
Permanently deleted user


If you're doing less than hundreds of thousands of int mappings, going to Trove isn't going to buy you enough speed increase to notice. You might get some decreased memory footprint, but things would have to be pretty tight to justify going to a non-standard API. I did it once in a performance critical piece of code (as in,
"triple the speed and half the footprint, or lose a multi-million dollar deal"), and it was worth it, but I certainly wouldn't do it as a standard practice. Just a bit of advice.

--Dave

0
Avatar
Permanently deleted user


"Robert Gibson" <robbie_usenet@yahoo.co.uk> a écrit dans le message de
news:bracc6$ek6$1@is.intellij.net...

Thibaut wrote:

isn't 'y' a vowel in english ? as far as I remember my distant school

days, it is in french

It's a kind-of vowel. Anyway, you can include it pretty easily (said he
facetiously!)


There are many words lists on the web.
http://www.und.edu/org/crypto/crypto/words/english_word_list/

Take a file, look at all the words in that file, and use a regexp to see
which word has all the vowels in the correct order.

Guillaume Laforge


0
Avatar
Permanently deleted user

a, e, i, o, u... and also y

abstemious
abstemiously
abstemiousness
abstentious
adventitious
adventitiously
adventitiousness
antireligious
arsenious
facetious
facetiously
facetiousness
sacrilegious
sacrilegiously
sacrilegiousness

So there is quite a few words with those vowels in the right order !

--
Guillaume Laforge
http://glaforge.free.fr/weblog/index.php?catid=2&blogid=1


0
Avatar
Permanently deleted user

J'arrive pas a croire qu'un francais me fait des lecons sur l'anglais!
Et en fait, seul les mot "abstemious" et "arsenious" rentrent dans mes
criteres, car il faut que chaque voyelle apparaisse une seule fois :P

A+,
Robbie

0
Avatar
Permanently deleted user

was soll das? ich schreib doch auch nix in deutsch ins forum(auser jetzt, aber das hat sein müssen :))!

0
Avatar
Permanently deleted user

Und dann noch mit Rechtschreibfehlern... :P

Tom

0
Avatar
Permanently deleted user

Well I said it was OT!

0

Please sign in to leave a comment.