Intellij + sbt + unmanagedBase doesn't appear to resolve dependencies properly
Hi,
I have a project that's running in EMR, so there's a number of provided dependencies for hadoop/spark. For cross cloud reasons, we're not depending on the components provided by AWS for builds. Because the AWS components differ slightly from some of the public jars we're depending on with `provided` we have a few minor tweaked jars that we place in sbt `unmanagedBase`. So to summarize it looks like like:
build.sbt:
...
"org.apache.hadoop" % "hadoop-aws" % "2.8.3" % Provided,
...
unmanagedBase := baseDirectory.value / "scala_lib",
$ ./coursier resolve --tree org.apache.hadoop:hadoop-aws:2.8.3
Result:
└─ org.apache.hadoop:hadoop-aws:2.8.3
├─ com.amazonaws:aws-java-sdk-s3:1.10.6
$ ls scala_lib/
aws-java-sdk-s3-1.10.51.jar
Okay, so the unmanaged version of aws-java-sdk is > the version included in the provided dependency, and the default resolution for this is to go with the higher version. From command line sbt builds all work this way. However in intellij, it looks like the unmanaged dependencies are not being resolved.
I went to project structure -> modules -> dependencies and I checked the list, I do see my scala_lib directory added in the dependencies list under unmanaged dependencies.
So I think that the issue is that conflicting dependency versions here are not being resolved in an expected way.
Can anyone offer me any help here on how to fix this?
Please sign in to leave a comment.
>However in intellij, it looks like the unmanaged dependencies are not being resolved.
What exact issue do you see in IDE? Could you describe your steps and the result you get? it would be greatly help to have a sample project for the investigation.
> What exact issue do you see in IDE?
When I use intellij to run a unit test, it underlines a method call and gives a build error that this method does not exist. When I click to source on the object, it takes me to com.amazonaws:aws-java-sdk-s3:1.10.6 source code which lacks the method. The method is present in aws-java-sdk-s3-1.10.51.jar which is in the unmanaged directory.
Every now and then I can run tests, it can feel a little intermittent.
> Could you describe your steps and the result you get?
I pick a scala unit test and right click -> run, it kicks off a build which fails. If I run the same test from the command line with sbt it succeeds.
> it would be greatly help to have a sample project for the investigation.
I'll see what I can whip up.