hudi-spark: libthrift 0.12.0 from spark-hive shadows Hive 2.3.10's 0.14.1, hive-jdbc connects die with NoClassDefFoundError
envgap__apache__hudi-19680
01 / FAILURE SIGNATURE
As reported upstream
java.lang.NoClassDefFoundError: org/apache/thrift/TConfiguration
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
5a11a4fce20620afac143358bc771dfe2d14d3c8- Manifest
pom.xml- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
apache/hudi #19680 · read the original issue
**Describe the problem**
Hive 2.3.10 bumped its libthrift to 0.14.1 (CVE-2020-13949), so the Hive 2.3.10 client jars hudi builds against are compiled against thrift 0.14.1 APIs: `hive-common`'s `HiveAuthUtils.getSocketTransport` does `new TSocket(new TConfiguration(), host, port, timeout)`. In hudi-spark, maven dependency mediation resolves libthrift 0.12.0 instead (nearest win via `spark-hive_2.12:3.5.5 -> libthrift:0.12.0`; the hive jars' 0.14.1 is "omitted for conflict with 0.12.0"), and 0.12.0 has no `TConfiguration`.
As a result, any `DriverManager.getConnection("jdbc:hive2://...")` on this classpath dies in transport setup instead of connecting or failing with an `SQLException`:
```
java.lang.NoClassDefFoundError: org/apache/thrift/TConfiguration
at org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(HiveAuthUtils.java:47)
at org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:519)
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:539)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:309)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:196)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at ReproHive2.main(ReproHive2.java:6)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TConfiguration
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 9 more
```
(Repro: `DriverManager.getConnection("jdbc:hive2://unused", "", "x")` on hudi-spark's resolved test classpath from `mvn dependency:build-classpath -Dmdep.includeScope=test`.)
Concretely, `sync_validate`'s `complete` / `latestPartitions` record-count modes are unusable on this dependency set, and in `complete` mode the error is further masked into an NPE by #19635. It is also why the fail-fast pins in `TestValidateHoodieSyncProcedure` fail deterministically wherever libthrift 0.12.0 wins resolution, see #19679.
**Suggested fix**
Align libthrift with the Hive 2.3.10 client jars, e.g. a dependencyManagement pin or direct dependency on libthrift 0.14.1 in hudi-spark. 0.14.1 keeps the `TSocket(String, int, int)` constructor spark-hive 3.5's 0.12.0-era callers use, but the spark-hive / thriftserver and HMS-based test suites need a verification pass before claiming the bump safe.
04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]