← All tasks
javaapache/hudi #19680Not a task: not reproduced

hudi-spark: libthrift 0.12.0 from spark-hive shadows Hive 2.3.10's 0.14.1, hive-jdbc connects die with NoClassDefFoundError

envgap__apache__hudi-19680

01 / FAILURE SIGNATURE

As reported upstream

java.lang.NoClassDefFoundError: org/apache/thrift/TConfiguration
Not a benchmark task.
  • In a clean container the reported failure did not reproduce, or the known fix did not make the project run.

02 / ENVIRONMENT RECIPE

Base commit
5a11a4fce20620afac143358bc771dfe2d14d3c8
Manifest
pom.xml
Reproduce
Awaiting issue-specific recipe
Run under trace
Awaiting a meaningful runtime command

03 / ORIGINAL ISSUE TEXT

apache/hudi #19680 · read the original issue
**Describe the problem**

Hive 2.3.10 bumped its libthrift to 0.14.1 (CVE-2020-13949), so the Hive 2.3.10 client jars hudi builds against are compiled against thrift 0.14.1 APIs: `hive-common`'s `HiveAuthUtils.getSocketTransport` does `new TSocket(new TConfiguration(), host, port, timeout)`. In hudi-spark, maven dependency mediation resolves libthrift 0.12.0 instead (nearest win via `spark-hive_2.12:3.5.5 -> libthrift:0.12.0`; the hive jars' 0.14.1 is "omitted for conflict with 0.12.0"), and 0.12.0 has no `TConfiguration`.

As a result, any `DriverManager.getConnection("jdbc:hive2://...")` on this classpath dies in transport setup instead of connecting or failing with an `SQLException`:

```
java.lang.NoClassDefFoundError: org/apache/thrift/TConfiguration
	at org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(HiveAuthUtils.java:47)
	at org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:519)
	at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:539)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:309)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:196)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
	at ReproHive2.main(ReproHive2.java:6)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TConfiguration
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 9 more
```

(Repro: `DriverManager.getConnection("jdbc:hive2://unused", "", "x")` on hudi-spark's resolved test classpath from `mvn dependency:build-classpath -Dmdep.includeScope=test`.)

Concretely, `sync_validate`'s `complete` / `latestPartitions` record-count modes are unusable on this dependency set, and in `complete` mode the error is further masked into an NPE by #19635. It is also why the fail-fast pins in `TestValidateHoodieSyncProcedure` fail deterministically wherever libthrift 0.12.0 wins resolution, see #19679.

**Suggested fix**

Align libthrift with the Hive 2.3.10 client jars, e.g. a dependencyManagement pin or direct dependency on libthrift 0.14.1 in hudi-spark. 0.14.1 keeps the `TSocket(String, int, int)` constructor spark-hive 3.5's 0.12.0-era callers use, but the spark-hive / thriftserver and HMS-based test suites need a verification pass before claiming the bump safe.
Continue on GitHub ↗

04 / LABELS

Labels from the report text only; not yet run

No supported category has been assigned.

Label rules and the text that matched
[]