Hello, I'm implementing a Delta Lake reader for my Master thesis. Is there any way for a writer to control the checkpoint interval? I want to deviate from the default 10 and measure how the performance of my reader behaves if I use checkpoints less or more frequently (e.g. 0, 10, 30, 50, ...). I thought there might be a table property that controls the checkpoint interval, but can't find the right one. Thanks 🙂

https://github.com/delta-io/delta/blob/dc061f7b5b47ffb4cf980b445d85100690bedf05/sp[…]est/scala/org/apache/spark/sql/delta/DeltaAlterTableTests.scala delta.checkpointInterval

Thank you 🙂!

What language are you using out of curiosity?

Not sure if you are referring to the writer or my reader 😄 I'm using pyspark for writing my tables and C++ for my reader implementation

i believe clickhouse have a delta lake reader in C++ . maybe that can help you

Thank you! I'll have a look at it 🙂

@Jan Schweizer a few of us from #delta-rs have been exploring what a lower-level and more portable layer might look like, which we hope to include FFI bindings for languages like C and C++. I don't want to interrupt your studies in any way, but I am interested to learn more about you work. I am in PST but happy to join a video call to chat at any mutually convenient time

That sounds interesting 😄

@Jan Schweizer - yea, @rtyler is right, we’re building out the Delta Kernel layer in Rust & Java right now. @Yi Wang is working on building out a Delta Kernel-like layer in Golang. Definitely send us the repo with your C++ implementation. Depending on the delta-kernel-rs C FFI would make it way easier to build out the C implementation of course.

There are still some number of unanswered questions for a kernel in Rust which I also think are interesting academic topics 😉

Thank you for the info. I had a very interesting chat with Tyler 🙂