dl4j-spark 1.0.0-beta5 depends on datavec-spark 1.0.0-SNAPSHOT, not 1.0.0-beta5, causing sbt's resolution to fail
envgap__deeplearning4j__deeplearning4j-8230
01 / FAILURE SIGNATURE
As reported upstream
[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
78bca543a8109a6d2132122c90da1678f96b91af- Manifest
deeplearning4j/deeplearning4j-scaleout/spark/pom.xml- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
deeplearning4j/deeplearning4j #8230 · read the original issue
#### Issue Description
https://mvnrepository.com/artifact/org.deeplearning4j/dl4j-spark_2.12/1.0.0-beta5 indicates that the dl4j-spark `1.0.0-beta5` artefact depends on datavec-spark `1.0.0-SNAPSHOT`, instead of the `1.0.0-beta5` which also seems to be released. This is can be confirmed by looking at the pom files directly:
```xml
$ curl -s https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom | grep -C 1 datavec
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-spark_2.12</artifactId>
<version>${datavec.spark.version}</version>
</dependency>
$ # parent pom:
$ curl -s https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom | grep datavec
<datavec.spark.version>1.0.0-SNAPSHOT</datavec.spark.version>
```
Interestingly, maven depending on this works by default, e.g. `mvn package` with the following `pom.xml` (just the pom, code or other folders) is successful:
```xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>example</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>dl4j-spark_2.12</artifactId>
<version>1.0.0-beta5</version>
</dependency>
</dependencies>
</project>
```
```
$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------< com.example:example >-------------------------
[INFO] Building example 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom (3.8 kB at 3.6 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom (10 kB at 32 kB/s)
...
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-20190911.215825-33.jar
Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-hadoop/1.0.0-SNAPSHOT/datavec-hadoop-1.0.0-20190911.215715-3218.jar
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-20190911.215825-33.jar (134 kB at 93 kB/s)
Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-hadoop/1.0.0-SNAPSHOT/datavec-hadoop-1.0.0-20190911.215715-3218.jar (34 kB at 22 kB/s)
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/wil9dw/test-scala/dl4j-beta5/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/wil9dw/test-scala/dl4j-beta5/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ example ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/wil9dw/test-scala/dl4j-beta5/target/example-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:15 min
[INFO] Finished at: 2019-09-12T16:39:46+10:00
[INFO] ------------------------------------------------------------------------
```
However, `sbt` (version 1.2.8) fails:
```scala
// build.sbt
ThisBuild / scalaVersion := "2.12.7"
ThisBuild / organization := "com.example"
lazy val test = (project in file("."))
.settings(
libraryDependencies += "org.deeplearning4j" %% "dl4j-spark" % "1.0.0-beta5",
//dependencyOverrides += "org.datavec" %% "datavec-spark" % "1.0.0-beta5",
name := "Test"
)
```
```
$ sbt compile
[info] Loading settings for project global-plugins from idea.sbt ...
[info] Loading global plugins from /Users/wil9dw/.sbt/1.0/plugins
[info] Updating ProjectRef(uri("file:/Users/wil9dw/.sbt/1.0/plugins/"), "global-plugins")...
[info] Done updating.
[info] Loading project definition from /Users/wil9dw/test-scala/dl4j-beta5/project
[info] Updating ProjectRef(uri("file:/Users/wil9dw/test-scala/dl4j-beta5/project/"), "dl4j-beta5-build")...
[info] Done updating.
[info] Loading settings for project test from build.sbt ...
[info] Set current project to Test (in build file:/Users/wil9dw/test-scala/dl4j-beta5/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] Updating ...
[warn] module not found: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT
[warn] ==== local: tried
[warn] /Users/wil9dw/.ivy2/local/org.datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-SNAPSHOT.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /Users/wil9dw/.sbt/preloaded/org.datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////Users/wil9dw/.sbt/preloaded/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-SNAPSHOT.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.datavec:datavec-spark_2.12:1.0.0-SNAPSHOT
[warn] +- org.deeplearning4j:dl4j-spark_2.12:1.0.0-beta5 (/Users/wil9dw/test-scala/dl4j-beta5/build.sbt#L6)
[warn] +- com.example:test_2.12:0.1.0-SNAPSHOT
[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found
[error] at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:332)
[error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:208)
...
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found
[error] Total time: 5 s, completed 12/09/2019 4:46:12 PM
```
Uncommenting the dependency override section means resolution and compilation works:
```
$ sbt compile
[info] Loading settings for project global-plugins from idea.sbt ...
[info] Loading global plugins from /Users/wil9dw/.sbt/1.0/plugins
[info] Loading project definition from /Users/wil9dw/test-scala/dl4j-beta5/project
[info] Loading settings for project test from build.sbt ...
[info] Set current project to Test (in build file:/Users/wil9dw/test-scala/dl4j-beta5/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] Updating ...
[info] downloading https://repo1.maven.org/maven2/org/datavec/datavec-spark_2.12/1.0.0-beta5/datavec-spark_2.12-1.0.0-beta5.jar ...
[info] downloading https://repo1.maven.org/maven2/org/datavec/datavec-hadoop/1.0.0-beta5/datavec-hadoop-1.0.0-beta5.jar ...
[info] [SUCCESSFUL ] org.datavec#datavec-spark_2.12;1.0.0-beta5!datavec-spark_2.12.jar (1022ms)
[info] [SUCCESSFUL ] org.datavec#datavec-hadoop;1.0.0-beta5!datavec-hadoop.jar (1317ms)
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[success] Total time: 12 s, completed 12/09/2019 4:34:07 PM
```
#### Version Information
Please indicate relevant versions, including, if relevant:
* Deeplearning4j version: 1.0.0-beta5
* Platform information (OS, etc): macOS
04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]