← All tasks
javadeeplearning4j/deeplearning4j #8230Not a task: not reproduced

dl4j-spark 1.0.0-beta5 depends on datavec-spark 1.0.0-SNAPSHOT, not 1.0.0-beta5, causing sbt's resolution to fail

envgap__deeplearning4j__deeplearning4j-8230

01 / FAILURE SIGNATURE

As reported upstream

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found
Not a benchmark task.
  • In a clean container the reported failure did not reproduce, or the known fix did not make the project run.

02 / ENVIRONMENT RECIPE

Base commit
78bca543a8109a6d2132122c90da1678f96b91af
Manifest
deeplearning4j/deeplearning4j-scaleout/spark/pom.xml
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

deeplearning4j/deeplearning4j #8230 · read the original issue
#### Issue Description



https://mvnrepository.com/artifact/org.deeplearning4j/dl4j-spark_2.12/1.0.0-beta5 indicates that the dl4j-spark `1.0.0-beta5` artefact depends on datavec-spark `1.0.0-SNAPSHOT`, instead of the `1.0.0-beta5` which also seems to be released. This is can be confirmed by looking at the pom files directly:



```xml

$ curl -s https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom | grep -C 1 datavec

        <dependency>

            <groupId>org.datavec</groupId>

            <artifactId>datavec-spark_2.12</artifactId>

            <version>${datavec.spark.version}</version>

        </dependency>



$ # parent pom:

$ curl -s https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom | grep datavec

        <datavec.spark.version>1.0.0-SNAPSHOT</datavec.spark.version>

```



Interestingly, maven depending on this works by default, e.g. `mvn package` with the following `pom.xml` (just the pom, code or other folders) is successful:



```xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>



  <groupId>com.example</groupId>

  <artifactId>example</artifactId>

  <version>1.0-SNAPSHOT</version>



  <dependencies>

    <dependency>

      <groupId>org.deeplearning4j</groupId>

      <artifactId>dl4j-spark_2.12</artifactId>

      <version>1.0.0-beta5</version>

    </dependency>

  </dependencies>

</project>

```



```

$ mvn package

[INFO] Scanning for projects...

[INFO] 

[INFO] ------------------------< com.example:example >-------------------------

[INFO] Building example 1.0-SNAPSHOT

[INFO] --------------------------------[ jar ]---------------------------------

Downloading from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom

Downloaded from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/dl4j-spark_2.12/1.0.0-beta5/dl4j-spark_2.12-1.0.0-beta5.pom (3.8 kB at 3.6 kB/s)

Downloading from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom

Downloaded from central: https://repo.maven.apache.org/maven2/org/deeplearning4j/spark_2.12/1.0.0-beta5/spark_2.12-1.0.0-beta5.pom (10 kB at 32 kB/s)



...



Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-20190911.215825-33.jar

Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-hadoop/1.0.0-SNAPSHOT/datavec-hadoop-1.0.0-20190911.215715-3218.jar

Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-20190911.215825-33.jar (134 kB at 93 kB/s)

Downloaded from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/org/datavec/datavec-hadoop/1.0.0-SNAPSHOT/datavec-hadoop-1.0.0-20190911.215715-3218.jar (34 kB at 22 kB/s)

[INFO] 

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] skip non existing resourceDirectory /Users/wil9dw/test-scala/dl4j-beta5/src/main/resources

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ example ---

[INFO] No sources to compile

[INFO] 

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example ---

[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!

[INFO] skip non existing resourceDirectory /Users/wil9dw/test-scala/dl4j-beta5/src/test/resources

[INFO] 

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ example ---

[INFO] No sources to compile

[INFO] 

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ example ---

[INFO] No tests to run.

[INFO] 

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ example ---

[WARNING] JAR will be empty - no content was marked for inclusion!

[INFO] Building jar: /Users/wil9dw/test-scala/dl4j-beta5/target/example-1.0-SNAPSHOT.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  02:15 min

[INFO] Finished at: 2019-09-12T16:39:46+10:00

[INFO] ------------------------------------------------------------------------

```



However, `sbt` (version 1.2.8) fails:



```scala

// build.sbt

ThisBuild / scalaVersion := "2.12.7"

ThisBuild / organization := "com.example"



lazy val test = (project in file("."))

  .settings(

    libraryDependencies += "org.deeplearning4j" %% "dl4j-spark" % "1.0.0-beta5",

    //dependencyOverrides += "org.datavec" %% "datavec-spark" % "1.0.0-beta5",

    name := "Test"

  )

```



```

$ sbt compile

[info] Loading settings for project global-plugins from idea.sbt ...

[info] Loading global plugins from /Users/wil9dw/.sbt/1.0/plugins

[info] Updating ProjectRef(uri("file:/Users/wil9dw/.sbt/1.0/plugins/"), "global-plugins")...

[info] Done updating.

[info] Loading project definition from /Users/wil9dw/test-scala/dl4j-beta5/project

[info] Updating ProjectRef(uri("file:/Users/wil9dw/test-scala/dl4j-beta5/project/"), "dl4j-beta5-build")...

[info] Done updating.

[info] Loading settings for project test from build.sbt ...

[info] Set current project to Test (in build file:/Users/wil9dw/test-scala/dl4j-beta5/)

[info] Executing in batch mode. For better performance use sbt's shell

[info] Updating ...

[warn] 	module not found: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT

[warn] ==== local: tried

[warn]   /Users/wil9dw/.ivy2/local/org.datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/ivys/ivy.xml

[warn] ==== public: tried

[warn]   https://repo1.maven.org/maven2/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-SNAPSHOT.pom

[warn] ==== local-preloaded-ivy: tried

[warn]   /Users/wil9dw/.sbt/preloaded/org.datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/ivys/ivy.xml

[warn] ==== local-preloaded: tried

[warn]   file:////Users/wil9dw/.sbt/preloaded/org/datavec/datavec-spark_2.12/1.0.0-SNAPSHOT/datavec-spark_2.12-1.0.0-SNAPSHOT.pom

[warn] 	::::::::::::::::::::::::::::::::::::::::::::::

[warn] 	::          UNRESOLVED DEPENDENCIES         ::

[warn] 	::::::::::::::::::::::::::::::::::::::::::::::

[warn] 	:: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found

[warn] 	::::::::::::::::::::::::::::::::::::::::::::::

[warn] 

[warn] 	Note: Unresolved dependencies path:

[warn] 		org.datavec:datavec-spark_2.12:1.0.0-SNAPSHOT

[warn] 		  +- org.deeplearning4j:dl4j-spark_2.12:1.0.0-beta5 (/Users/wil9dw/test-scala/dl4j-beta5/build.sbt#L6)

[warn] 		  +- com.example:test_2.12:0.1.0-SNAPSHOT

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found

[error] 	at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:332)

[error] 	at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:208)





...



[error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

[error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

[error] 	at java.lang.Thread.run(Thread.java:748)

[error] (update) sbt.librarymanagement.ResolveException: unresolved dependency: org.datavec#datavec-spark_2.12;1.0.0-SNAPSHOT: not found

[error] Total time: 5 s, completed 12/09/2019 4:46:12 PM

```



Uncommenting the dependency override section means resolution and compilation works:



```

$ sbt compile

[info] Loading settings for project global-plugins from idea.sbt ...

[info] Loading global plugins from /Users/wil9dw/.sbt/1.0/plugins

[info] Loading project definition from /Users/wil9dw/test-scala/dl4j-beta5/project

[info] Loading settings for project test from build.sbt ...

[info] Set current project to Test (in build file:/Users/wil9dw/test-scala/dl4j-beta5/)

[info] Executing in batch mode. For better performance use sbt's shell

[info] Updating ...

[info] downloading https://repo1.maven.org/maven2/org/datavec/datavec-spark_2.12/1.0.0-beta5/datavec-spark_2.12-1.0.0-beta5.jar ...

[info] downloading https://repo1.maven.org/maven2/org/datavec/datavec-hadoop/1.0.0-beta5/datavec-hadoop-1.0.0-beta5.jar ...

[info] 	[SUCCESSFUL ] org.datavec#datavec-spark_2.12;1.0.0-beta5!datavec-spark_2.12.jar (1022ms)

[info] 	[SUCCESSFUL ] org.datavec#datavec-hadoop;1.0.0-beta5!datavec-hadoop.jar (1317ms)

[info] Done updating.

[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.

[success] Total time: 12 s, completed 12/09/2019 4:34:07 PM

```



#### Version Information



Please indicate relevant versions, including, if relevant:



* Deeplearning4j version: 1.0.0-beta5

* Platform information (OS, etc): macOS



Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]