Hi All, i have a readStream job to delta, but the job failed due to some anomaly in one partition, the partition contains many files (around 5000 files). I run the compaction for that partition, when I check with

describe detail

the total of files has been reduced. but, when I run the readStream job, the job still read the old files (scan more than 5000 files), then I try to run the

vacuum(0)

, but the readStream still read the file that has been deleted by vacuum command. But, when I try to run the

select

command, there is no error. Is there any way to invalidate the delta table? why the readStream still read the file that has been deleted? note: I use delta version 1.0.0 Thanks

compaction/optimize has no impact on you stream. It creates a new version in your delta-log but as no data was actually changed by compaction/optimize but just the file layout, they are not considered for streaming. If they were, you would stream the very same data twice

hi @Gerhard Brueckl thanks for your reply. I see. But, after I run vacuum(0) & re-run the streaming (the checkpoint is still empty), why is the streaming still trying to read a file deleted by vacuum? i assume, after vacuum(0) they delta history must be clear

yes, but your stream is broken as it is missing some data. lets say your stream has read until the 2nd of January and you get new data every day. If you run

vacuum

on the 5th and delete all changes before, and restart the stream, it will complain that the files between 2nd and 5th are missing. you could use ``ignoreMissingFiles`` but obviously the changes between 2nd and 5th will be missing from your stream then https://learn.microsoft.com/en-us/azure/databricks/kb/delta/file-transaction-log-not-found

Stream reads progress through the log starting at the first version and move forward. Any optimization/compaction sets dataChange=false in the version they commit, and the stream reads the data from the earliest add action it encounters, and it skips future add actions that have dataChange=false that were created by optimizations. Now that you have broken the version history by aggressively vacuuming, you need to re-start the stream with both a new checkpoint and starting version after you vacuumed.

Does z-order or optimize re-arrange data across files?

Yes, unless you restrict it by partition predicate.