[Bug] [flinkcdc yaml] flinkcdc yml not support transform
envgap__DataLinkDC__dinky-4372
01 / FAILURE SIGNATURE
As reported upstream
java.lang.NoSuchFieldError: names
Not a benchmark task.
- In a clean container the reported failure did not reproduce, or the known fix did not make the project run.
02 / ENVIRONMENT RECIPE
- Base commit
7996c9eda698f05007abe541d720a7c9d7f3f9e0- Manifest
pom.xml- Reproduce
Awaiting issue-specific recipe- Run under trace
Awaiting a meaningful runtime command
03 / ORIGINAL ISSUE TEXT
DataLinkDC/dinky #4372 · read the original issue
### Search before asking
- [x] I had searched in the [issues](https://github.com/DataLinkDC/dinky/issues?q=is%3Aissue) and found no similar issues.
### What happened
I successfully started the flink task with console commands, but an exception occurred when using dinky。
i add transform,start job error:
`
java.lang.NoSuchFieldError: names
at org.apache.flink.cdc.runtime.parser.TransformParser.generateProjectionColumns(TransformParser.java:348)
at org.apache.flink.cdc.runtime.operators.transform.PostTransformOperator.lambda$cacheSchema$7(PostTransformOperator.java:252)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.apache.flink.cdc.runtime.operators.transform.PostTransformOperator.cacheSchema(PostTransformOperator.java:260)
at org.apache.flink.cdc.runtime.operators.transform.PostTransformOperator.processElement(PostTransformOperator.java:233)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Thread.java:834)
`
flink-version:1.81
flinkcdc-version:3.2.0
dinky:dinky-release-1.18-1.2.1
### What you expected to happen
The yml task is working properly
### How to reproduce
`SET 'execution.checkpointing.interval' = '60s';
EXECUTE PIPELINE WITHYAML (
source:
type: mysql
name: xka-order-business-db
hostname: 10.21.203.13
port: 3306
username: xxxx
password: xxx
tables: order.order_detail,order.order,marketing.goods_sku
scan.newly-added-table.enabled: true
server-id: 5430-5434
server-time-zone: Asia/Shanghai
-- transform:
-- - source-table: order.order_detail
-- projection: \*, concat('20',substr(REGEXP_REPLACE(order_no,'[a-zA-Z]',''),0,6)) as dt
-- primary-keys: dt, id
-- partition-keys: dt
-- - source-table: order.order
-- projection: \*, concat('20',substr(REGEXP_REPLACE(order_no,'[a-zA-Z]',''),0,6)) as dt
-- primary-keys: dt, order_no
-- partition-keys: dt
-- - source-table: marketing.goods_sku
-- projection: \*
-- partition-keys: dt
sink:
type: starrocks
name: StarRocks Sink
jdbc-url: jdbc:mysql://xxx:9030
load-url: xx:8080
username: xx
password: ''
table.create.properties.replication_num: 1
pipeline:
name: Sync MySQL Database to StarRocks
parallelism: 1
route:
- source-table: order.order_detail
sink-table: ods.ods_order_detail
- source-table: order.order
sink-table: ods.ods_order
- source-table: marketing.goods_sku
sink-table: ods.ods_goods_sku
)
`
### Anything else
_No response_
### Version
1.2.1
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
04 / LABELS
Labels from the report text only; not yet run
No supported category has been assigned.
Label rules and the text that matched
[]