Referencing a CSV file

Irina Rapoport

Created September 22, 2016 15:31

I am trying to do something very similar to Simple Plugin. Namely, I have strings in my DSL that refer to rows in a CSV file, that I need to refer to.

There are some differences.

I could not find any functioning plugin for CSV files (strangely, because there is such an abundance of plugins for everything else!), so my CSV at first was a plain text file. Using plain text files approach, it is my understanding that I would need to fetch all CSV files in the project, then parse all of them manually until I found the file(s) with the key I am looking for.

That sounds inefficient, especially given that I could not find a way to look up files by extension, and my project has >2000 plain text files (with various extensions) and growing. This way, I am not taking advantage of IntelliJ indexing capabilities, right? What do I do if one of these files change?

So in my second attempt, I started writing my own plugin for CSV. Problem is, now that CSV is a file type, it no longer opens in "Table Mode", and I kind of liked the Table Mode. Is there still a way to do it?

While at it, a tangential question: is there an API for Table Mode? I.e. could I open a CSV in table mode programmatically, with specific settings, and programmatically select the row I need?

Спасибо!

Ирина

9 comments

Imants Cekusins

Created September 22, 2016 15:51

why there may be few CSV plugins:

bulk CSV data is often imported to spreadsheet software or SQL database, where it is stored and processed.

text editor is usually used for ad hoc viewing / editing CSV files.

Irina Rapoport

Created September 23, 2016 17:51

Hi Imants,

Yes, it makes sense.

I just realized that the situation is more complex than I thought: apparently, only the source of a reference may have a TextRange, but not a destination. Bummer, considering my destination is a large plain text file. How do I refer to a range in it? Perhaps by adding dummy children nodes to a PsiPlainText node?

My other questions still stand, too.

Would greatly appreciate help with this.

Imants Cekusins

Created September 23, 2016 20:19

Ирина,

maybe this could help: CSVReader

if your plugin depends on many large CSV files, maybe it is worth to consider adding a dependency on SQL database and interact with it via SQL driver.

If I were to work with structured data of more than a few short files, I would use a database.

Irina Rapoport

Created September 23, 2016 20:36

Hi Imants,

Thank you for a quick response. I already have a CSV reader, but yours might be better :-)

We don't have enough CSV files to warrant a database. I don't want an ORM, object/relational impedance mismatch and all the other good things that happen when you interact with a relational database from Java. I also have not done any SQL stuff from Intellij. However, from the purely presentational standpoint, if you say running an SQL driver over a bunch of CSV files will make the data more user-friendly and is easy to setup on a user's machine, I'd be interested in exploring it. Especially if these records can be selected programmatically and be easily used for autocomplete and such, without writing SQL in Java.

Still eagerly awaiting an answer to my OP.

Marcin Rembała

Created September 25, 2016 12:35

About Table Mode, if You use latest IntelliJ IDEA, You can do as following to create table view:

Add dependency in Your plugin descriptor:

<depends>com.intellij.database</depends>

Add extension:

<extensions defaultExtensionNs="com.intellij">
        <fileEditorProvider implementation="csv.CsvFileEditorProvider"/>
    </extensions>

Implement provider:


public class CsvFileEditorProvider extends CsvTableFileEditorProvider {

    public static final String CSV_SEPARATOR = ";";
    public static final CsvFormat CSV_FORMAT;

    static {
        CsvRecordFormat csvRecordFormat = new CsvRecordFormat("", "", "",
                Collections.<CsvRecordFormat.Quotes>emptyList(), CsvRecordFormat.QuotationPolicy.AS_NEEDED,
                CSV_SEPARATOR, "\n", false);
        CSV_FORMAT = new CsvFormat("My CSV format", csvRecordFormat, csvRecordFormat, false);
    }

    public boolean accept(@NotNull Project project, @NotNull VirtualFile file) {
        //implement your condition
        return true;
    }

    @NotNull
    @Override
    public String getEditorTypeId() {
        return "my-csv-editor";
    }

    @NotNull
    public CsvTableFileEditor createEditor(@NotNull Project project, @NotNull VirtualFile file) {
        return new CsvTableFileEditor(project, file, CSV_FORMAT);
    }
}

Marcin Rembała

Created September 25, 2016 12:45

About referencing CSV file values - You probably need to implement custom PSI for yours CSV files, look here for details

Irina Rapoport

Created September 26, 2016 21:40

Hi Marcin,

Thank you for your reply.

<depends>com.intellij.database</depends>

not working for me under IU-162.2032.8

Same under the freshly pulled master branch of intelllij-community.

Marcin Rembała

Created September 27, 2016 07:12

Example above is from IU-145, for newer version API may be different.

I forgot to add - You need to extend IntelliJ SDK classpath adding this library:

$IDEA_HOME/plugins/DatabaseSupport/lib/database-openapi.jar

Irina Rapoport

Created September 27, 2016 19:11

Thank you Marcin!

database-openapi.jar was already in my classpath. database-impl.jar was not. After adding it to classpath (under File > Project Structure > SDKs > classpath, your snippet above compiles. However, when trying to run, I am getting the following error. Looks like some classloader did not get the classpath. How to satisfy that?

As an aside, is there any way to view a complete error stack, without the "... 63 more"?

Thank you so much! We are getting closer.

Irina

com/intellij/database/editor/CsvTableFileEditorProvider
com.intellij.openapi.extensions.impl.PicoPluginExtensionInitializationException: com/intellij/database/editor/CsvTableFileEditorProvider
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getComponentInstance(ExtensionComponentAdapter.java:96)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getExtension(ExtensionComponentAdapter.java:119)
at com.intellij.openapi.extensions.impl.ExtensionPointImpl.processAdapters(ExtensionPointImpl.java:249)
at com.intellij.openapi.extensions.impl.ExtensionPointImpl.addExtensionPointListener(ExtensionPointImpl.java:363)
at com.intellij.openapi.fileEditor.impl.FileEditorProviderManagerImpl.<init>(FileEditorProviderManagerImpl.java:60)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.picocontainer.defaults.InstantiatingComponentAdapter.newInstance(InstantiatingComponentAdapter.java:193)
at org.picocontainer.defaults.ConstructorInjectionComponentAdapter$1.run(ConstructorInjectionComponentAdapter.java:220)
at org.picocontainer.defaults.ThreadLocalCyclicDependencyGuard.observe(ThreadLocalCyclicDependencyGuard.java:53)
at org.picocontainer.defaults.ConstructorInjectionComponentAdapter.getComponentInstance(ConstructorInjectionComponentAdapter.java:248)
at com.intellij.util.pico.CachingConstructorInjectionComponentAdapter.getComponentInstance(CachingConstructorInjectionComponentAdapter.java:58)
at com.intellij.openapi.components.impl.ServiceManagerImpl$MyComponentAdapter.getComponentInstance(ServiceManagerImpl.java:232)
at com.intellij.util.pico.DefaultPicoContainer.getLocalInstance(DefaultPicoContainer.java:246)
at com.intellij.util.pico.DefaultPicoContainer.getComponentInstance(DefaultPicoContainer.java:212)
at com.intellij.openapi.components.ServiceManager.doGetService(ServiceManager.java:46)
at com.intellij.openapi.components.ServiceManager.getService(ServiceManager.java:37)
at com.intellij.openapi.fileEditor.ex.FileEditorProviderManager.getInstance(FileEditorProviderManager.java:31)
at com.intellij.openapi.fileEditor.impl.HistoryEntry.a(HistoryEntry.java:206)
at com.intellij.openapi.fileEditor.impl.HistoryEntry.createLight(HistoryEntry.java:85)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$UIBuilder.processFiles(EditorsSplitters.java:850)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$UIBuilder.processFiles(EditorsSplitters.java:816)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$ConfigTreeReader.process(EditorsSplitters.java:807)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters.openFiles(EditorsSplitters.java:235)
at com.intellij.openapi.fileEditor.impl.OpenFilesActivity.runActivity(OpenFilesActivity.java:35)
at com.intellij.ide.startup.impl.StartupManagerImpl.a(StartupManagerImpl.java:146)
at com.intellij.ide.startup.impl.StartupManagerImpl.c(StartupManagerImpl.java:342)
at com.intellij.ide.startup.impl.StartupManagerImpl.runPostStartupActivitiesFromExtensions(StartupManagerImpl.java:150)
at com.intellij.openapi.project.impl.ProjectManagerImpl.a(ProjectManagerImpl.java:339)
at com.intellij.openapi.project.impl.ProjectManagerImpl.openProject(ProjectManagerImpl.java:353)
at com.intellij.openapi.project.impl.ProjectManagerImpl$2.compute(ProjectManagerImpl.java:413)
at com.intellij.openapi.project.impl.ProjectManagerImpl$2.compute(ProjectManagerImpl.java:402)
at com.intellij.openapi.progress.Task$WithResult.run(Task.java:307)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:635)
at com.intellij.openapi.progress.impl.CoreProgressManager$9.run(CoreProgressManager.java:384)
at com.intellij.openapi.progress.impl.CoreProgressManager$3.run(CoreProgressManager.java:170)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:494)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:443)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:155)
at com.intellij.openapi.application.impl.ApplicationImpl.a(ApplicationImpl.java:556)
at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:308)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/intellij/database/editor/CsvTableFileEditorProvider
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at com.intellij.util.lang.UrlClassLoader._defineClass(UrlClassLoader.java:268)
at com.intellij.util.lang.UrlClassLoader.defineClass(UrlClassLoader.java:264)
at com.intellij.util.lang.UrlClassLoader._findClass(UrlClassLoader.java:233)
at com.intellij.ide.plugins.cl.PluginClassLoader.b(PluginClassLoader.java:119)
at com.intellij.ide.plugins.cl.PluginClassLoader.a(PluginClassLoader.java:73)
at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.loadImplementationClass(ExtensionComponentAdapter.java:157)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getComponentImplementation(ExtensionComponentAdapter.java:66)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getComponentInstance(ExtensionComponentAdapter.java:73)
... 48 more
Caused by: java.lang.ClassNotFoundException: com.intellij.database.editor.CsvTableFileEditorProvider PluginClassLoader[com.mycompany.myscript.plugin.id, 1.4.0 with external annotator]
at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 63 more
Caused by:

java.lang.NoClassDefFoundError: com/intellij/database/editor/CsvTableFileEditorProvider
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at com.intellij.util.lang.UrlClassLoader._defineClass(UrlClassLoader.java:268)
at com.intellij.util.lang.UrlClassLoader.defineClass(UrlClassLoader.java:264)
at com.intellij.util.lang.UrlClassLoader._findClass(UrlClassLoader.java:233)
at com.intellij.ide.plugins.cl.PluginClassLoader.b(PluginClassLoader.java:119)
at com.intellij.ide.plugins.cl.PluginClassLoader.a(PluginClassLoader.java:73)
at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:62)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.loadImplementationClass(ExtensionComponentAdapter.java:157)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getComponentImplementation(ExtensionComponentAdapter.java:66)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getComponentInstance(ExtensionComponentAdapter.java:73)
at com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.getExtension(ExtensionComponentAdapter.java:119)
at com.intellij.openapi.extensions.impl.ExtensionPointImpl.processAdapters(ExtensionPointImpl.java:249)
at com.intellij.openapi.extensions.impl.ExtensionPointImpl.addExtensionPointListener(ExtensionPointImpl.java:363)
at com.intellij.openapi.fileEditor.impl.FileEditorProviderManagerImpl.<init>(FileEditorProviderManagerImpl.java:60)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.picocontainer.defaults.InstantiatingComponentAdapter.newInstance(InstantiatingComponentAdapter.java:193)
at org.picocontainer.defaults.ConstructorInjectionComponentAdapter$1.run(ConstructorInjectionComponentAdapter.java:220)
at org.picocontainer.defaults.ThreadLocalCyclicDependencyGuard.observe(ThreadLocalCyclicDependencyGuard.java:53)
at org.picocontainer.defaults.ConstructorInjectionComponentAdapter.getComponentInstance(ConstructorInjectionComponentAdapter.java:248)
at com.intellij.util.pico.CachingConstructorInjectionComponentAdapter.getComponentInstance(CachingConstructorInjectionComponentAdapter.java:58)
at com.intellij.openapi.components.impl.ServiceManagerImpl$MyComponentAdapter.getComponentInstance(ServiceManagerImpl.java:232)
at com.intellij.util.pico.DefaultPicoContainer.getLocalInstance(DefaultPicoContainer.java:246)
at com.intellij.util.pico.DefaultPicoContainer.getComponentInstance(DefaultPicoContainer.java:212)
at com.intellij.openapi.components.ServiceManager.doGetService(ServiceManager.java:46)
at com.intellij.openapi.components.ServiceManager.getService(ServiceManager.java:37)
at com.intellij.openapi.fileEditor.ex.FileEditorProviderManager.getInstance(FileEditorProviderManager.java:31)
at com.intellij.openapi.fileEditor.impl.HistoryEntry.a(HistoryEntry.java:206)
at com.intellij.openapi.fileEditor.impl.HistoryEntry.createLight(HistoryEntry.java:85)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$UIBuilder.processFiles(EditorsSplitters.java:850)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$UIBuilder.processFiles(EditorsSplitters.java:816)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters$ConfigTreeReader.process(EditorsSplitters.java:807)
at com.intellij.openapi.fileEditor.impl.EditorsSplitters.openFiles(EditorsSplitters.java:235)
at com.intellij.openapi.fileEditor.impl.OpenFilesActivity.runActivity(OpenFilesActivity.java:35)
at com.intellij.ide.startup.impl.StartupManagerImpl.a(StartupManagerImpl.java:146)
at com.intellij.ide.startup.impl.StartupManagerImpl.c(StartupManagerImpl.java:342)
at com.intellij.ide.startup.impl.StartupManagerImpl.runPostStartupActivitiesFromExtensions(StartupManagerImpl.java:150)
at com.intellij.openapi.project.impl.ProjectManagerImpl.a(ProjectManagerImpl.java:339)
at com.intellij.openapi.project.impl.ProjectManagerImpl.openProject(ProjectManagerImpl.java:353)
at com.intellij.openapi.project.impl.ProjectManagerImpl$2.compute(ProjectManagerImpl.java:413)
at com.intellij.openapi.project.impl.ProjectManagerImpl$2.compute(ProjectManagerImpl.java:402)
at com.intellij.openapi.progress.Task$WithResult.run(Task.java:307)
at com.intellij.openapi.progress.impl.CoreProgressManager$TaskRunnable.run(CoreProgressManager.java:635)
at com.intellij.openapi.progress.impl.CoreProgressManager$9.run(CoreProgressManager.java:384)
at com.intellij.openapi.progress.impl.CoreProgressManager$3.run(CoreProgressManager.java:170)
at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:494)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:443)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:155)
at com.intellij.openapi.application.impl.ApplicationImpl.a(ApplicationImpl.java:556)
at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:308)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.intellij.database.editor.CsvTableFileEditorProvider PluginClassLoader[com.mycompany.lmyscript.plugin.id, 1.4.0 with external annotator]
at com.intellij.ide.plugins.cl.PluginClassLoader.loadClass(PluginClassLoader.java:64)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 63 more

Please sign in to leave a comment.