Error downloading org.apache.spark:spark-core_3:3.5.1
I have spark 3.5.1 installed , path is added to system path variable.
I try to build in Intellij a simple build.sbt containing :
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.5.1"
and receive the following error:
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_3:3.5.1
When I try to use spark from the powershell terminal in Intellij, I can import org.apache.spark.sql without any problems:
Using Scala version 2.12.18 (Java HotSpot(TM) 64-Bit Server VM, Java 11.0.23)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession
scala> var spark = SparkSession.builder().appName("DataFramesExercise").getOrCreate()
24/06/20 16:53:51 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
spark: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@76705ce1
请先登录再写评论。
My Intellij IDEA version is Ultimate 2024.1.2
Hello,
Is it possible to share idea.log after restarting IDE and reproducing the issue to check IDE configuration and recorded events? You may share files via https://uploads.jetbrains.com/ form. Don't forget to provide uploadID afterwards.
Do you use any antivirus software or firewall/proxy/VPN?
Thank you for the answer. I've uploaded idea.log file:
Upload id: 2024_06_23_9MHHHdwbvWrgc5dsXdtjag (file: idea.log)
Yes, I am using VPN and there is some antivirus on the computer.
Probably that's the issue. I see several network errors in log. Is it possible to temporarily disable AV?
I am without VPN now, on a different computer - getting the same error:
C:\Program Files\Java\jdk-22\bin\java.exe -Djline.terminal=jline.UnsupportedTerminal -Dsbt.log.noformat=true -Dfile.encoding=UTF-8 -Dgrouping.with.qualified.names.enabled=true -Didea.managed=true -Dfile.encoding=UTF-8 "-Didea.installation.dir=C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2024.1.4" -jar C:/Users/yuliaa/AppData/Roaming/JetBrains/IdeaIC2024.1/plugins/Scala/launcher/sbt-launch.jar
[info] welcome to sbt 1.10.0 (Oracle Corporation Java 22.0.1)
[info] loading settings for project start1-build from plugins.sbt ...
[info] loading project definition from C:\Work\scala.projects\start1\project
[info] loading settings for project root from build.sbt ...
[info] set current project to start1 (in build file:/C:/Work/scala.projects/start1/)
[warn] there's a key that's not used by any other settings/tasks:
[warn]
[warn] * root / idePackagePrefix
[warn] +- C:\Work\scala.projects\start1\build.sbt:8
[warn]
[warn] note: a setting might still be used by a command; to exclude a key from this `lintUnused` check
[warn] either append it to `Global / excludeLintKeys` or call .withRank(KeyRanks.Invisible) on the key
[info]
[info] Here are some highlights of sbt 1.10.0:
[info] - SIP-51 support for Scala 2.13 evolution
[info] - Various Zinc fixes
[info] - ConsistentAnalysisFormat: new Zinc Analysis serialization
[info] - CommandProgress API
[info] See https://eed3si9n.com/sbt-1.10.0 for full release notes.
[info] Hide the banner for this release by running `skipBanner`.
[info] sbt server started at local:sbt-server-c90f28fdf9ebc30b27e4
[info] started sbt server
sbt:start1>
[info] Defining Global / sbtStructureOptions, Global / sbtStructureOutputFile and 1 others.
[info] The new values will be used by cleanKeepGlobs
[info] Run `last` for details.
[info] Reapplying settings...
[info] set current project to start1 (in build file:/C:/Work/scala.projects/start1/)
[info] Applying State transformations org.jetbrains.sbt.CreateTasks, sbt.jetbrains.LogDownloadArtifacts from C:/Users/yuliaa/AppData/Roaming/JetBrains/IdeaIC2024.1/plugins/Scala/repo/org/jetbrains/scala/sbt-structure-extractor_2.12_1.3/2024.1.2/sbt-structure-extractor-2024.1.2.jar
[info] Reapplying settings...
[info] set current project to start1 (in build file:/C:/Work/scala.projects/start1/)
[info] Reapplying settings...
[info] set current project to start1 (in build file:/C:/Work/scala.projects/start1/)
[warn] sbt 0.13 shell syntax is deprecated; use slash syntax instead: Global / dumpStructure
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom
[info] downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom
[info] downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom.sha1
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom.sha1
[info] downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom.sha1
[info] downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom.sha1
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_3:3.5.1
[error] Not found
[error] Not found
[error] not found: C:\Users\yuliaa\.ivy2\localorg.apache.spark\spark-core_3\3.5.1\ivys\ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom
[error] Error downloading org.apache.spark:spark-sql_3:3.5.1
[error] Not found
[error] Not found
[error] not found: C:\Users\yuliaa\.ivy2\localorg.apache.spark\spark-sql_3\3.5.1\ivys\ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom
[error] (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_3:3.5.1
[error] Not found
[error] Not found
[error] not found: C:\Users\yuliaa\.ivy2\localorg.apache.spark\spark-core_3\3.5.1\ivys\ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-core_3/3.5.1/spark-core_3-3.5.1.pom
[error] Error downloading org.apache.spark:spark-sql_3:3.5.1
[error] Not found
[error] Not found
[error] not found: C:\Users\yuliaa\.ivy2\localorg.apache.spark\spark-sql_3\3.5.1\ivys\ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-sql_3/3.5.1/spark-sql_3-3.5.1.pom
[error] Total time: 3 s, completed 23 ביוני 2024, 21:37:03
[info] shutting down sbt server
When I do import and run command from the Terminal of Intellij, it works:
====
Using Scala version 2.12.18 (Java HotSpot(TM) 64-Bit Server VM, Java 22.0.1)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apa24/06/23 21:43:36 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession ^
scala> val spark = SparkSession.builder.appName("start1").config("spark.master","local").getOrCreate()
24/06/23 21:50:31 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
spark: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@6c6cde22
====
Please try to select Scala version 2.12.18 for your project in IDEA. With this version spark library is resolved correctly.