Does anyone here have experience running high-write throughput Delta Lake tables? One thing I've noticed is the size of the checkpoint files can quickly outgrow the dataset (like 10x). Curious to hear any learnings about these types of tables.

We run what I would consider some high throughput append only tables, what does high throughput mean to you? We have had memory challenges with checkpoint creation, but I expect some recent changes in delta-rs will have improved that but we haven't tested yet.

For a table, commits every 30s or sooner depending on buffer size.

hm, I'm not sure how frequently we're seeing commits from our #kafka-delta-ingest processes 🤔 Is there a strong motivation to have checkpoints created so frequently? You can certainly disable it, originally the checkpoint lambda we have floating around in incubator was created so we could create checkpoints out of band from the appends to a table

No strong reason to have them checkpoint so frequently, will likely increase it or change it to a regular interval.

what's the read load look like on the table?

Writes are append only, partitioned by

date=YYYY-MM-DD

and reads would typically be either most recent data (last N minutes) or historical ranges

that's very similar to my append only tables. An out of band checkpoint creation every few minutes could do quite well IMHO

Yeah, thats what I am considering. Basically out of band compaction and checkpointing. I've got other work required before I get there but thats the goal

We have end of ds partition

optimize

happening in Spark for these tables too, that helps mucho

Yeah, without an external locking source, multi-writer support is not possible on Cloudflare R2 (yet) https://github.com/delta-io/delta-rs/issues/1356 So right now I am doing checkpoints in the single writer flow but I imagine it will be combination of regular optimize and checkpointing once multi-writer is possible

why not make an external locking thingie?

I would prefer to not have the additional component if the object store supported it out-of-the-box. I have some ideas around an external locker using etcd or https://developers.cloudflare.com/workers/learning/using-durable-objects/

I have not, though I don't have a good reason for that other than 🤷