Found an issue where the column statistics fail to parse during query time for

timestamp

fields. https://github.com/delta-io/delta-rs/issues/1372

any documentation on how exactly to configure the delta-rs crate to work with the individual storage backends it supports?

@Cole MacKenzie just wanted to give you a 🌮 👏 , I know creating repro cases and good bug reports takes time and I appreciate it

That issue we saw in CI with

target/

exploding and taking up heaps of disk space, I think I'm seeing that with #kafka-delta-ingest a bit too. There's something wacky either with

rustc

or with a dependency of ours that creeps up every now and again 🤔

Does anyone know if there is some threshold before

PruningStatistics

for

Datafusion

are used?

Few community updates: • I gave a presentation to the Dask community showcasing deltadask. Here’s if you’re interested. The Dask community is very excited about Delta Lake. • I will chat with a core member of the cuDF team on Monday. They have a bunch of users that are also asking for Delta Lake support. • I will be presenting about why Delta Lake is a great file format for pandas at the Data + AI Summit next month. I will be showing how file skipping allows for certain queries to run much faster. Thanks to this PR from @shingo, we will be able to showcase a more beautiful syntax. It would be ideal if someone could implement this Z Order feature, so we could highlight this at the talk as well. Even a naive Z Order implementation will have huge data skipping opportunities for the pandas community and should make them quite excited.

1 question, maybe not just specific to delta-rs but delta in general. what is the main reason for the tight coupling with parquet format? I was thinking that feature-wise delta is great and has tons of external connectors but one thing that it imposes is to migrate to parquet because the files store metadata/ stats. but then this can be done with other formats as well like AVRO/ ORC. shouldn't delta be a thin wrapper of the current features around "an existing data lake" rather than "a data lake made of parquet only"? file format implementation can be decoupled from the delta protocol exposed via an interface that connectors implement to support a file format. This will also be a more future facing - what if tomorrow there's a new shiny amazing file format that beats parquet. delta will require lots of refactoring to possibly support it. but via the interfaces I mentioned above it should be relatively easy. just some thoughts here, whats your take on this ?

With the object store abstractions we have a pretty reasonable way to plug new backends into delta-rs. I once upon a time hacked on supporting #delta-sharing natively, and was discouraged for a number of reasons. I think we now have enough of an interesting abstraction layer to support sharing at the rust layer, I'm curious for feedback on supporting delta-sharing like it's Just Another Storage Backend (read-only)

arrow-rs v40 released 🤔

Is there any particular reason for these being defined as type aliases? https://github.com/delta-io/delta-rs/blob/main/rust/src/schema.rs#L14-L20

For delta writes , how SSE and KMS KEY can be mentioned in write_deltalake while writing in S3 , any leads ?

I wonder how hard it is for us to add new data types. In particular I need u128, u256, u512 as well as decimal512 & decimal1024 types.

Are there any guarantees or tests around the "correctness" of a query using DataFusion on a Delta Lake table (using

delta-rs

)? Basically, how do we know we can trust the result? I was looking at sqlite and see they have a tool (https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki) to validate the result against other DB engines.

datafusion 25 landed as well based on arrow 39

Hey guys: loving using delta-rs in python to break away from spark for the more simple things I want to do. One question, how does one set the delta option (for write) of "dataChange = false"? I have not found a way to do this. Thanks!

Hi folks, Delta v2 checkpoint design is up for review -- comments and suggestions very welcome! https://github.com/delta-io/delta/issues/1793 tl;dr: It tightens up the failure atomicity story for multi-file checkpoints, which allows to remove the constraints on data layout (e.g. can partition, sort, etc however we want after this)

what I have learned thus far is that getting @Andy Grove’s work on the frontpage of HN is a great way to get him to spend more time on it 🤔

Also any limitations in the "vacuum" dry run support? Running this on one of our large tables shows 44K files to remove, while the Spark based version shows 24.4M files to remove.

Would there be any heartache if I worked on a way to do locking with a Postgres instance vs dynamodb on s3? Use case is we're using MinIO (S3 Compliant) cloud storage on premises and can't use Dynamo for locking

Heads up, I just spoke more with @John Darrington about the need for an alternative lock client. He is going to explore making dynamodb_lock into "distributed_lock" and put dynamodb behind a feature so that postgres can also sit behind a feature.. That way our use of

LockClient

in the S3 storage layer can remain generic and we can start to support alternative locking implementations

dask-deltatable was recently donated to the dask-contrib GitHub organization. We’re going to incorporate delta-incubator/dat (Delta Acceptance Testing) and collaborate with the Dask community to get the reader methods working for the reference tables. Will be cool to dogfood delta-rs with dask-deltatable to make sure we provide good interfaces for connectors 🚀

I guess Python binding also supports checkpoint creation, am I missing something? https://github.com/delta-io/delta-rs/blob/930d16e583fbd784f5f3d4bb8bcd753e2d6488a6/python/deltalake/table.py#L579

Hello Delta rs community! I am planning to use delta-rs to write to Delta tables stored in S3 or on Databricks. Is this possible? If yes, how? I've read the documentation and couldn't find a topic making this clear.

@Robert I saw your comments on @John Darrington’s pull request. Would you and @Will Jones be up for me moving dynamodb_lock into a repo in delta-incubator? I think that crate needs to come out especially if we're to use a LockingObjectStore implementation

Are there any unit tests available to verify the correct functionality of the load_version method, particularly when there are existing checkpoints? I believe I may have encountered a bug while implementing a benchmark code.

@Will Jones with those great writer changes of yours, shall I cut us a new release?

that plus @Robert’s performance improvements 🙂

I think once https://github.com/apache/arrow-rs/pull/4305 is released, I'm good to switch to using multipart upload when writing data files. There any other blockers for that switch that anyone knows of?

@rtyler @Will Jones, as we are moving the lock client, I was wondering how we should proceed the the proofs (tla / stateright) in the delta-rs repo?