Hello I am wondering how you guys have handled delta streami

Question

Hello I am wondering how you guys have handled delta streaming + eventual schema updates changing column data types Has your solution been to simply delete checkpoints create a new checkpoint mirroring the new schema or have you found some other nice workaround It s in the classic bronze silver gold Azure platform with delta streaming happening between all the zones Thank you

Karolina A · Answer

This has been a thorn in my side actually There s no good way that I ve found but the hacky way is as follows stop any running tasks involving the delta source table and checkpoint directory assert that the checkpoints have fully consumed everything up to now from the delta source table if not can t migrate because the next step will lead to data loss

Karolina A · Answer

do the schema migration this creates new version of delta table write some commit offset file pairs in the checkpoint directory to act as if the version after the schema update has also been consumed This is hacky I have written some code to write those text files didn t find something in pyspark or delta to do it after this is done restart any tasks The checkpoints will skip over the schema migration and process microbatches only when new data is added to the table

Karolina A · Answer

We also have the bronze silver gold architecture so generally we stop the entire pipeline migrate all tables write the new checkpoint file pairs and restart the pipeline

Patrik Ekman · Answer

Thank you for answering What s the motivation for messing with the checkpoint files rather than removing them and start a new checkpoint I suppose it s something about keeping the offset so that you don t have to read everything from start once restarting slightly smiling face

Kashyap Bhatt · Answer

gt I have written some code to write those text files < Karolina A> could you share this code

Karolina A · Answer

gt gt Thank you for answering What s the motivation for messing with the checkpoint files rather than removing them and start a new checkpoint I suppose it s something about keeping the offset so that you don t have to read everything from start once restarting Yeah we do incremental processing so we don t want to reprocess months worth of data again whenever we change the schema slightly

Kashyap Bhatt · Answer

any chance you can share that code < Karolina A>

Karolina A · Answer

No I can t share as it s not open But if you go to checkpoint directories you will see that the offsets are plain json files which refer to a version that they ve processed This can be changed

Patrik Ekman · Answer

Cool thank you so much Karolina Are there any considerations on e g the reservoirVersion field in the offset file

Karolina A · Answer

The reservoir Version is the version of the source delta table up to which the data has been marked as processed by the checkpoints roughly I haven t found the spec

Karolina A · Answer

I think only if a corresponding commit file exists then you can know that the data has been processed and committed up to he version