Hi, we are getting with CDC the following error: "java.lang.IllegalArgumentException: CDC range from start 4 to end 1 was invalid. End cannot be before start." . We are reading from append only table by using the following code. We are not providing endingVersion:

spark.readStream.format("delta")

.option("readChangeFeed", "true")

.option("mergeSchema", "true")

.option("startingVersion", starting_version)

What could be the problem? Thanks for the help, Regards, Gapy

what was the value for

starting_version

in your case? 4 or 1?

it was 4, but the goal table had the version 1

so version 4 did not exist in the source?

it did, the source had version 4, but the goal table had version 1 ...already one version (from before) was merged inside

not sure but I think its related to the version of the CDC vs. the version of the table

Hi, okay we need to check this up. Yes we should have it in place - first thing as the table is created is ALTER TABLE... SET TBLPROPERTIES (delta.enableChangeDataFeed = true) , but we need to check it out. Thanks

not sure if you can force a stream to start (again) directly with version 4 if version 1 was the last one already loaded as this would technically break the stream as you are skipping transactions