Issue with DataGrip data export to CSV on big data volume.
Hi Team,
DataGrip has an issue with data export to CSV on big data volume. What I mean by saying issue? On 2.5 GHz Intel Core i7 with 16 Gb memory a data extract of 182 million rows failed. Total size of the result CSV created by psql \copy was 36 Gb. The main point - 100% CPU consuming plus a good hit on memory, and the notebook was sounding like a Super Hornet taking off.
I could assume DataGrip tries to get all data for a result set before to spool data to disk. Though I have no evidence, except one - file was empty for 20 minutes before I killed DataGrip process. As result I was forced to use psql \copy command.
Could you please take a look at the case, and remediate DataGrip.
With best regards,
Kostyantyn
DataGrip 2017.1.4
Build #DB-171.4694.3, built on May 16, 2017
Licensed to Kostyantyn Krakovych
Subscription is active until September 24, 2017
JRE: 1.8.0_112-release-736-b21 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.12.4
Please sign in to leave a comment.
Hi,
As a temporary workaround, you can use pg_dump utility. DataGrip can be integrated with it:
Thank you.
I can concur I have the exact same problem on a 15G table.
Thanks for your post!
@ Philippe Lavoie-mongrain Hi,
Try to use native tool to dump big tables.
Thank you.
DataGrip still chokes on even moderately large exports. Is there an issue in the tracker for this? If so I have been unable to locate it.
Yep, it's a real shame that when trying to dump large tables DataGrip just silently fails and outputs 0-sized file. It would be nice to be able to rely on the tools we use every day.
Using pg_dump is often out of the question because it doesn't support outputting TSV/CSV/JSON files, which are handy for moving data between different kinds of systems reliably.