https://delta.io logo
j

Jalil Alchy

04/07/2023, 8:45 PM
Maybe a simple question, but when using the Delta Connector for Flink, if I set the RowKind to Update or Delete, will this be written or does the connector strictly support inserts?
s

Scott Sandre (Delta Lake)

04/08/2023, 3:03 AM
Do you mean a row kind other than RowData?
Anyways, currently the connector only supports inserts
And we are working hard on adding Table API and SQL support!
If you’d like to see higher level DML support (delete, update, merge) please add a ticket on the delta-io/connectors repo!
k

Krzysztof Chmielewski

04/08/2023, 12:07 PM
Do you mean a row kind other than RowData?
@Scott Sandre (Delta Lake) I think what @Jalil Alchy was refering to by RowKind is Flink's RowKind. The Flink's RowData::getRowKind() return a reference to RowKind. Currently Delta Source And Sink do not use this field so it will have no effect on the processing. As small side note, somewhat relevant to this is that for SQL/Table API that Im working on, both Delta Table Sink and Table Source will not work with CDC connectors that produces RowKind other than Insert. Flink table planner for every query verifies if ChangelogMode for source and Sink in the pipeline is compatible. I think that you could use CDC connectors with Delta connector in DataStreamAPI, where you could easily filter out RowData that has different RowKind than Insert. Flink SQL does not provide such a feature.
36 Views