Does delta rs currently have support for the following 1 get

Question

Does delta rs currently have support for the following 1 get the last checkpoint I do see a function that s public at the crate level but not outside the crate 2 get the total size of the table at the current loaded version 3 get the size of the parquet file created from the json writer s flush and commit

Will Jones · Answer

gt 1 get the last checkpoint Do you want to just know what s the most recent version with a checkpoint Or you want to know the content of the file

Will Jones · Answer

gt 2 get the total size of the table at the current loaded version You can compute this by summing the `size bytes` column in `add actions table ` <https docs rs deltalake 0 14 0 deltalake table state struct DeltaTableState html method add actions table>

Will Jones · Answer

gt 3 get the size of the parquet file created from the json writer s flush and commit I don t think that s part of the API but seems like a reasonable request if you want to make a Github issue

Ryan Aston · Answer

On 1 I d just need the version Makes sense on 2 and I ll put in a request for 3

Ryan Aston · Answer

On 3 I did see that the writer has a `buffer len` function to return the current size of the parquet buffers in bytes but the value seems to always be returned as zero I am successfully writing data so I m not sure why it never shows any bytes I m printing the value of buffer len after a ` write ` but before the ` flush and commit ` ```let mut wrtr deltalake writer JsonWriter = deltalake writer json JsonWriter for table amp table unwrap wrtr write events json clone await unwrap println size of buffers wrtr buffer len always prints 0 wrtr flush and commit amp mut table await unwrap ```

Will Jones · Answer

IIRC buffer len is the current size of serialized Parquet data that hasn t been written to disk If you haven t gotten enough rows to fill up a row group it will be zero

Will Jones · Answer

I think `buffered record batch count` should be non zero for you is it

Ryan Aston · Answer

Yes events json is a Vec lt Value gt with 487 tuples After the write above `buffer len ` returns 0 and `buffered record batch count ` returns 1 The description for `write ` is Writes the given values to internal parquet buffers for each represented partition and the description for `buffer len ` is Returns the current byte length of the in memory buffer This may be used by the caller to decide when to finalize the file write which implies it would be the size of the data waiting to be flushed to disk but that doesn t seem to be the case For now I m working around it by calling the helper function I created for question 2 just before I flush and just after and finding the difference For 1 I m working around it by basically creating my own helper function using inspiration from these two functions in the delta rs source

Ryan Aston · Answer

Hi Will I had one more follow up here The add actions approach is working well but is there a way to get only the add actions that occurred as part of a particular version commit Also should I be scoping it to active adds only would not doing so not take into account deletes

Will Jones · Answer

gt but is there a way to get only the add actions that occurred as part of a particular version commit No I don t think there is

Will Jones · Answer

gt Also should I be scoping it to active adds only would not doing so not take into account deletes The table only includes add actions that are active not any that are removed

Ryan Aston · Answer

Even if there s not a native way are the building blocks there to build my own function to get the adds for a particular version

Will Jones · Answer

I think what you want is a function to get the logs for a particular log file we don t expose anything that low level as far as I can find