sabari dass

02/02/2023, 8:06 AM
Can anyone please help me on this?

Yerachmiel Feltzman

02/03/2023, 8:55 AM
A risky solution is to write the checkpoints yourself. I've in a super specific case edited it myself even though I'm not supposed to. If you understand how to write it, you can do it and only write the last one, since the only the last one (or the two depending where the streaming stopped) is relevant for restart
Other solution is to restart the streaming from a specific source table version or timestamp. Delta supports it if I remember correctly. In this case, you will need to be able to deal with duplicates. Ideally if your target operations are idempotent you are good to go. You will only have to wait for the streaming pipeline close the gap between the configured starting point and "now" (eventually consistent, as they say)
👍 1