How Is The Maven Provided Scope Supposed To Work When Run From IDEA?

I don't think that is how the Maven scope=provided works.  The dependency with that name is pulled from your local mvn repository, just like normal.  The dependency will not be packaged with the other dependencies.  Which means that your tests, etc will use the version from your local maven repository.  It sounds like you are using the groupId/artifactId of one thing, but trying to use something else.  Maybe give it it's own groupId/artifactId and deploy it to your maven repositories?

You say you are running it from IDEA.  What is it running as?  An application, web app or something else???

sheltonn wrote:

I don't think that is how the Maven scope=provided works.  The dependency with that name is pulled from your local mvn repository, just like normal.  The dependency will not be packaged with the other dependencies.  Which means that your tests, etc will use the version from your local maven repository.

I agree with what you are saying here.
That is what I am expecting, however, it isn't what I am seeing.

Try this pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>ProvidedIssue</groupId>
    <artifactId>ProvidedIssue</artifactId>
    <version>1.0</version>
  <dependencies>
    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.3</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>

</project>

package com.jetbrains.youtrack.issues;

import org.apache.commons.io.FileUtils;

public class Demo {
  public static void main(String[] args) {
    System.out.println(FileUtils.getTempDirectory());
  }
}

C:\tools\Java\jdk1.6.0_29\bin\java -Didea.launcher.port=7533 "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 117.747\bin" -Dfile.encoding=UTF-8 -classpath "C:\tools\Java\jdk1.6.0_29\jre\lib\alt-rt.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\alt-string.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\charsets.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\deploy.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\javaws.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\jce.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\jsse.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\management-agent.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\plugin.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\resources.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\rt.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\ext\dnsns.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\ext\localedata.jar;C:\tools\Java\jdk1.6.0_29\jre\lib\ext\sunjce_provider.jar;D:\home\ietester\Projects\Delete\ProvidedIssue\target\classes;C:\Program Files (x86)\JetBrains\IntelliJ IDEA 117.747\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain com.jetbrains.youtrack.issues.Demo
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/io/FileUtils
     at com.jetbrains.youtrack.issues.Demo.main(Demo.java:7)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils
     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
     ... 6 more

Process finished with exit code 1

According to Maven docs 'provided' scope 'indicates you expect the JDK or a container to provide the dependency at runtime'. So IDEA doesn't include
such dependencies to the classpath when you run the application. It looks correct for me.

--
Nikolay Chashnikov
Software Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

No I don't agree that Intellij is correct here.

If I use the exec plugin to run my app then the classpath will include provided scope jars.
At minimum Intellij's run configuration should give me a choice to choose which scope to run as.

This is proper behavior for Maven.  IDEA can't fix it.  

I use provided a lot in web containers.  I develop against the JDBC drivers that are common to the machine and use the Servlet API, etc that is provided by the container.  I use scope=provided to tell IJ that it can use this library for compiling, testing, etc, but DO NOT package it into the distribution (.war file in this case) that it produces.

In your case, commons is not in the classpath of your application because you told Maven that you would provide it.  You did not.  In order for it to work the way you have done it here, you would need to add the library to the -classpath manually.

http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope

If you truly have a patched version of the commons-io jar, wouldn't you want to compile against it?  The real solution seems to be that the patched version of commons-io needs to be added to your local repository with a unique groupId/artifactId/version.

Either way, if you tell it provided, you will be responsible for ensuring that the jar is on the classpath.  

How are you running mvn:exec?
If I do it with the example above the following happens:

>mvn exec:java -Dexec.mainClass="com.jetbrains.youtrack.issues.Demo"
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'exec'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - ProvidedIssue:ProvidedIssue:jar:1.0
[INFO]    task-segment: [exec:java]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing exec:java
[INFO] No goals needed for project - skipping
[INFO] [exec:java {execution: default-cli}]
[WARNING]
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/io/FileUtils
        at com.jetbrains.youtrack.issues.Demo.main(Demo.java:7)
        ... 6 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 7 more
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] An exception occured while executing the Java class. null

org.apache.commons.io.FileUtils
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2 seconds
[INFO] Finished at: Thu Aug 30 12:45:17 EDT 2012
[INFO] Final Memory: 7M/111M
[INFO] ------------------------------------------------------------------------

IDEA uses it for compiling and running test classes, but not for running production classes.

I was incorrectly assuming that IDEA was the container.

Now I use the compile scope and use the maven-assembly-plugin to remove it from my distribution.
Using the provided scope and the -classpath option is probably more appropriate but it means I have to maintain the classpath in the run config, which is a bit of a pain.

BTW. It isn't commons-io that is patched, it is a commercial application that wont let you connect to it unless the binaries are the same, even if the APIs haven't changed between versions.

Hi

I'm currently investigating if we could switch from Eclipse to IntelliJ and this issue popped up for me as well. We use the provided scope in our projects to tell maven not to include certain libraries in the jar that we deploy on our servers. However, during development we often run small test applications from within IntelliJ and in this case we need the respective libraries (and its dependencies) to be provided by IntelliJ.

Our usecase is that we run a Storm topology using LocalCluster on our development machines in which case the Storm jar needs to be in the classpath. For deployment we need the Storm libraries to be excluded from the jar, as Storm requires us to deploy the topolgy without the Storm dependencies.

Adding the dependencies manually as a library seems cumbersome as it basically defies the purpose of using a dependency management tool. Putting "compile" scope and removing the dependencies in the assembly plugin also seems unelegant.

This feature is available in Eclipse. Are there any plans of adding this feature to IntelliJ?

Best,
Lorenz

IntelliJ 12.1.4 (community edition)

Maven does have a test scope.  That sounds like what you are looking for.

http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope

Hi Norris

Thanks for the quick reply! I'm impressed :-)

I quickly tested with the "test" scope, but this results in the dependency only being available in the "test" phase during build. In our case I guess we're really expecting the runtime environment of the IDE to act as the "container", the application would be running in in production.

Best,
Lorenz

Don't think of it from the IDE's perspective.  That gets Eclipse or IDEA in the mix.  Think of it from Maven.  Configure the pom correctly and the IDE will follow.  It sounds like you have a library that you need for compilation, testing and running, but the library will be provided by whatever container it runs in.  That is <scope>provided</scope>.  IDEA will make it available to your code, but will not package the file in the artifact.

Exactly, this part works fine. However, when we run a main method from within the IDE the libary is also not available, which results in a NoClassDefFoundError. The feature would be to provide "provided" dependencies when running the main method of a class in the IDE, but to keep excluding the dependency when building the jar to deploy on the server.

BINGO.  That is a common problem.  You said that you would provide the library at runtime.  You didn't.  You expected IDEA to do it, but you already told IDEA not to worry about it.  So, you would have to add that library to the classpath of your run command.  

Actually, the "provided" scope is telling maven something, not IntelliJ. But that is not the point. If you work on a project in which you deploy an application on a server that provides certain libraries that you're not supposed to put into your jar, this feature makes your life a lot easier. Without it you need to manually add stuff to classpaths or find other workarounds to run the application locally.

I was just asking if and when this feature will be available.

Best,
Lorenz

It is telling Maven and IDEA builds the same thing.  Eclipse cheats and provides some functionality beyond Maven.  It's good and bad.  Provided means that the container will provide it.  If you deploy to an EJB server, then that means that it will be in a common library path, etc.  This gets messy when you are talking about a jar artifact.  Then you are the container.  Not the editor.  Think of it like this.  You have a script or some command that you run to execute the jar.  That is the container.  That is what must provide the jar.  

Also, you do have the manifest at your disposal.  It sounds like this is similar to what you need.

http://www.mkyong.com/maven/how-to-create-a-manifest-file-with-maven/

Hi Norris

Thank you very much for your help and sorry for not being able to explain the situation properly. Looking at your forum profile you seem to have been working with IntelliJ for a while. I'm relatively new to Intellij, I have been using Eclipse in various probjects in the past and I'm now looking into the possibility of transitioning.

Are you a JetBrains dev? If you want I can try and describe the usecase in more detail as I think many people will run into this if they are working on projects that use Hadoop, Storm, Enterprise Java, or other applications that run on a Java based server.

Regarding the manifest: I'm not trying to run the jar from within the IDE. I'm trying to run the main method of a class within the project for debug purposes. I only build the jar before deploying to to the server.

Best,
Lorenz

Sorry, I'm not a DEV, only a long time zealot.  lol.  Provided works perfectly when you use a JEE container.  That is what I normally create.  

I think IDEA could be better when the scope is provided for an executable jar.  I have seen that problem a lot.  

If you make a ticket with your issue and post the link back here, I will be glad to add my .02c  Unlike the Eclipse community, the DEVs are really good about working through issues.  They have created tons of changes for me over the years.

http://youtrack.jetbrains.com/issue/IDEA-85457#newissue=yes

There's is an issue for this already: http://youtrack.jetbrains.com/issue/IDEA-107048

Thanks for your help!

Best,
Lorenz

If I use the provided scope things like maven-assembly-plugin work as expected (but IDEA can't run it).
If I use the compile scope to make the run/debug config work in IDEA I have to do extra filtering and manage my dependencies manually in assembly.xml.

This manual managing of the dependencies makes me think that using the compile scope is wrong.

I tried using the Maven Run Configuration instead of a normal application run configuration
http://www.jetbrains.com/idea/webhelp/creating-maven-run-debug-configuration.html

I added the following to the command line:

exec:java -Dexec.classpathScope=compile -Dexec.args="--arg1 val1 --arg2 val2"

as per
http://mojo.codehaus.org/exec-maven-plugin/examples/example-exec-for-java-programs.html
Interestingly IDEA moves the first quote to:

exec:java -Dexec.classpathScope=compile "-Dexec.args=--arg1 val1 --arg2 val2"

Regardless, I get an error saying that one of my modules in my multi-module build has not been installed.

[INFO] Failed to resolve artifact.

Missing:
----------
1) com.jetbrains.youtrack.issues:ProvidedIssue:jar:1.0-SNAPSHOT

  Try downloading the file manually from the project website.

  Then, install it using the command:
      mvn install:install-file -DgroupId=com.jetbrains.youtrack.issues -DartifactId=ProvidedIssue -Dversion=1.0-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file

Shouldn't IDEA act as the container and provide the jars on the classpath when you are running as a java application (seperate case that when you are running as a web application)?

The error above could be that I am running from the working directory of the sub-module as I don't get that error if I run from the working directory of the parent pom.
However, I get a ClassNotFoundException when I run from the working directory of the parent pom. :(

One solution is using scope variables where the 'provided' scope variable is 'provided' for the default profile as expected but is overridden to 'compile' scope for a custom IDE build profile.

If you have 2 modules one for application and one for test, ususally you have provided library in application, but in test they are seen like compiled.

Now you can create a Run configuration for your application and specify that you want use classpath of test module. For me this resolve the problem.

bye

@NorrisShelton Contrary to your assumptions, this is in fact an IDE problem.

For application containers, the provided dependencies available (usually), because the IDE has a Jetty/Tomcat/whatever plugin that modifies application startup into a sequence of starting the container if necessary, uploading the app, and connecting to the debug port of the container.
The provided dependency was shown to the Java compiler, but is not part of the deployed war. The idea is that the container will have compatible classes somewhere on the classpath; if that does not hold, you don't get good diagnostics, just ClassNotFoundExceptions and/or incompatibility-induced misbehaviour.

For applications, the debugger does not start a container and uploads the war, it starts the JVM with the jar (with an option that makes the JVM stop and wait before loading the first class from the classpath). The debugger then connects to the debug port.

Now since the IDE knows about the differences in start-up behaviour, it is also reasonable to make it responsible for providing provided dependencies.
For standalone applications, this is easy: Just put the provided dependencies into the classpath.
For container applications, it might be a good idea to check that all provided dependencies are indeed available. Bonus points if a version check happens, too (this might be difficult or awkward to do). In theory there might be situations where a provided dependency needs to come from the IDE anyway - it would have to be a scenario where the prod container has e.g. a license file and the test container does not, but the developer has, and while that's conceivable I don't know whether that's really happening anywhere.
The third scenario is the container application for which there's not plugin (yet). The above-mentioned Hadoop applications might be a case.
Fourth scenario: Stuff on the bootstrap classpath on the prod server. Using the bootstrap classpath is generally a bad idea, but it's there and people have been using it. I dimly recall having seen bootstrap mentioned in Maven docs, so may that's already covered - I never used that stuff in anger.

I am on latest version 2018.2.3(Ultimate Edition) and there is an option in the Run configuration that says 'Include dependencies with "Provided" scope'. That did the trick for me.