Question about Delta table versioning and time travel: If a Delta table has two versions v1 and v2, does v2 contain everything that v1 had plus the difference between d1 and d2 OR does v2 only contain the difference between v1 and v2 ?

it’s a rolling set of changes.. so cumulative

It’s a little more complicated than that (for example, the answer is different depending on whether V2 was created from an Append transaction or a Delete transaction). What are you trying to understand? How much storage each version should take up? Or something else?

I have a use case where I’d like to read data from Delta table via Delta connector for a streaming application. To add job resilience and restarting from failures, I want to use the startingTimestamp to continue job processing only from the point where the job had failed and process NEW data from that point onwards. Any ideas on how I can do that? P.S My delta table is a collection of log files in a S3 bucket, there won’t be any updates or deletion to logs files once uploaded, but there will be new log files periodically.

ya as long as your source delta table is append only, you can stream using the timestamp... i'd prefer to use startingVersion though b/c it's more explicit

Do I have to specify in some configuration that the source delta table is append?

Do I have to specify in some configuration that the source delta table is append?

no, the stream will throw an exception if you do perform an update/delete https://docs.delta.io/2.2.0/delta-streaming.html#ignore-updates-and-deletes

I think I understand what ignore deletes and updates is doing but the naming is a little confusing. Intuitively, I think that ignore updates should basically result in no change being propagated to the downstream in case of an update. Here it rewrites the file containing that change and propagates that entire file as a change to downstream.

Do I have to specify in some configuration that the source delta table is append?

Yes, for streaming you don’t need to modify anything on the streaming table. But to protect the table from any accidental updates or deletes you can use table property

delta.appendOnly