Large XML files

Hi,

I was just wondering if anyone has tried opening a large xml file?

Well, i've just opened a 11MB xml file with IDEA and it took ages to load. Idea is now consuming 372 MB of RAM.

Furthermore clicking on the to close a branch doesn't work.

Is it just me, or are there other people experiencing this issue and is identified. I haven't anything within the forum yet on this point. I'm using build 896.

Cedric

9 comments

Hi Cedric,

Loading a large XML document completely in memory with all its structure and metadata consumes a lot of memory. IDEA seems to parallel DOM as far as document structure goes and it seems to use JDOM/Xerces for XML parsing. Just plain DOM parsing of such a large XML document would require lots of memory and time so it's really not surprising that IDEA RAM consumption inflates when loading XML files.

Hope this helps,
Chris

0

Hi Chris,

I understand this, however i am surprised that for a 12 MB document, IDEA requires 300MB and is hardly responsive.
The memory consumption is a pain because i only have 512MB of RAM, but it really is the speed that bothers me.

I think that it can be improved because other editors don't have such a long load time, what do you guys think?

a++ Cedric

0

1. Do other editors provide xml-specific functionality there? Or do they
just treat XML as plain text?
2. What was the pupropse of opening 12MB XML in IDEA? Is it supposed to be
edited?

--

Best regards,
Maxim Shafirov
JetBrains, Inc / IntelliJ Software
http://www.intellij.com
"Develop with pleasure!"


"Cedric ROUVRAIS" <no_mail@jetbrains.com> wrote in message
news:13307844.1061474714544.JavaMail.itn@is.intellij.net...

Hi Chris,

>

I understand this, however i am surprised that for a 12 MB document, IDEA

requires 300MB and is hardly responsive.

The memory consumption is a pain because i only have 512MB of RAM, but it

really is the speed that bothers me.
>

I think that it can be improved because other editors don't have such a

long load time, what do you guys think?
>

a++ Cedric



0

JDOM is not the speediest nor the leanest Java API to access XML information. Sure, it is easier to use than direct DOM or SAX but it has an associated overhead which may explain the perceived lack of performance. Maybe if the Jetbrains guys were using a lower-level API, performance would be better (I myself use dom4j which is leanier than JDOM while still being rather easy to work with)?

On the other hand, IDEA's XML editor is not meant to be a really powerful XML editor. There are better tools to work with XML files out there. It sure would be nice if the XML ability of IDEA was better though...

a+
Chris

0

We do not use JDOM there. Instead we use PSI. The same for java files.

--

Best regards,
Maxim Shafirov
JetBrains, Inc / IntelliJ Software
http://www.intellij.com
"Develop with pleasure!"


"Chris Laprun" <chris.laprun@nist.gov> wrote in message
news:5745316.1061476481231.JavaMail.itn@is.intellij.net...

JDOM is not the speediest nor the leanest Java API to access XML

information. Sure, it is easier to use than direct DOM or SAX but it has an
associated overhead which may explain the perceived lack of performance.
Maybe if the Jetbrains guys were using a lower-level API, performance would
be better (I myself use dom4j which is leanier than JDOM while still being
rather easy to work with)?
>

On the other hand, IDEA's XML editor is not meant to be a really powerful

XML editor. There are better tools to work with XML files out there. It sure
would be nice if the XML ability of IDEA was better though...
>

a+
Chris



0

Hi Maxim,

Nice! I just (wrongly obviously) assumed that since there was a jdom.jar in the lib directory, you guys used it for the XML part of the editor. _;;
It did seem weird though... I'll shut up now!

Best,
Chris

0

1/ Yes they provide XML functionnality.
2/ Yes i wanted to edit this XML file.

I'll be sending to you the editors i'm using in a second.

0

I've seen this as well. I have some xslt generated monster files, and I usually end up editting them in vi or nedit. Mine tend to be 9-15 megs in size.

Also, I have another problem, maybe related. Of course, the bug was closed...

http://www.intellij.net/tracker/idea/viewSCR?publicId=6756

Still happens in the latest builds. Should I refile this? I think the problem in the bug case above is that one schema links to hundreds of others. I guess intellij tries to load them all once a doc is opened that references the "master" schema...

Same behavior, hang, goes up to max ram...

--pete

0

The JVM exaggerates its memory usage a great deal. If you began running lots of other applications, you would see the amount of memory allocated to the JVM decrease (it gives memory back to the OS when it's needed). I believe it just allocates as much as it can / as much as it thinks might be needed and generally doesn't use much of it.

0

Please sign in to leave a comment.