Ah ok. We use microseconds..hence we need to fix the docs

That would be μs / us, not ms

@Ian Joiner https://github.com/delta-io/delta-rs/issues/1286 is related I believe. Basically Delta/Spark does something annoying by default and 10 years out of date 😆

Hello all. I'm planning to use delta-rs with datafusion for a streaming sql processing use case. Because my query nodes are going to frequently re-process queries over a time window, a local filesystem cache is a must. I would be interested in implementing https://github.com/delta-io/delta-rs/issues/769 but could use guidance on the design. The current questions that I have are: • Does the scala implementation implement a local cache feature? • Would it be safe to design a system that memoizes calls to get and get_range in the ObjectStore trait, from a delegate ObjectStore instance, to a local cache directory? • Should the cache have a configurable expiry? If so is it size based, or time based, or both?

One more change before I release Python

deltalake

0.9.0 https://github.com/delta-io/delta-rs/pull/1337

@Will Jones @Florian Valeye @Matthew Powers while watching ☝️ wait for GitHub Actions to finish, I am feeling more and more like it is time for us to put the python bindings in its own repository so that it can be developed more independently from the Rust bindings. Curious what y'all think

Looking for a gut check here, I'm working with S3 Bucket Notifications, they come in with a bucket name and the object created (e.g. the full prefix and file name of a put object). I am wondering if it would be a safe assumption to make that the first non-partition segment of object key should be where

_delta_log/

goes. E.g. •

some/path/to/a/prefix/alpha.parquet

, the

_delta_log/

should be in

some/path/to/a/prefix/

•

/some/path/ds=2023-05-05/site=<http://delta.io/beta.parquet|delta.io/beta.parquet>

, the

_delta_log/

should be in

some/path/

Are there instance where this assumption might not hold true?

Currently updating the Delta Lake docker to the latest

deltalake version

. When running

pip install deltalake==0.9.0

- it appears that Rust needs to be installed first but this was not a requirement for

0.8.1

. Was this on purpose for 0.9.0?

https://github.com/delta-io/delta-rs/blob/main/rust/src/builder.rs#LL349C1-L355C2 I think we should add

wasbs

which provides TLS-encrypted access to Azure Blob Storage

Hi everyone! I wasn't aware of the existence of this channel!

Hey guys, have any of you experience with setting up the dynamodb lock with delta-rs in rust? I mean I have read the documentation and I am sure that as far as I could follow everything is set correctly. But the

RecordBatchWriter

panicked at

flush_and_commit

with an

ObjectStore { source: Generic { store: "DeltaS3ObjectStore", source: LockClientRequired } }

. Do any of you have any ideas or experience with this? In thread I will post some code snippets to better understand how my setup looks like.

May I ask when we plan to do our next Rust release (0.10.1?)

Does delta-rs support mergeSchema now?

Hello team , while writing to delta table via delta-rs python binding , I am getting deltalake.PyDeltaTableError: Schema error: Invalid data type for Delta Lake: Null for None cols , how best this can be handled , want to keep null in delta table.

is:pr is:open draft:false

is a very useful filter predicate for our pull requests simple smile

datafusion 24 was released today which allows us to bump to arrow 38, of course arrow 39 was also released today 😆 Upgrading datafusion and arrow are dependent on my work completing with this pull request since some deprecated symbols are now fully removed as of 38.

I was extra caffeinated 🤓 this afternoon, so, as promised, here is a document detailing how PyArrow datasets work with deltalake: https://docs.google.com/document/d/1XGg1pf9Nep9GHlSdvO65Ao1kyQ_Z_g55uyHuTYVyeT0/edit?usp=sharing cc@Matthew Powers

Any of you have a Windows machine available to try to test my pull request, CI keeps getting

fatal error LNK1180: insufficient disk space to complete link

which I cannot reproduce on *nix.

Any plans to support change data feed in delta-rs? https://docs.databricks.com/delta/delta-change-data-feed.html

Is there any concern regarding enabling the

hdfs

feature in the python bindings?

This document summarizes the pandas interfaces I think we should expose to end users: https://docs.google.com/document/d/1QnvRPX_Oy5XCnow5wxBy_FNOcvaWwK_PcXxPXbIG6AE/edit?usp=sharing It shows how we can give pandas users the ability to restore to a previous version, evolve the schema, update columns, delete rows, create tables with generated columns, etc. The doc is intentionally high-level. I think it’d be good to have a high-level brainstorming exercise to discuss the “ideal end state for pandas users” without diving into the weeds with implementation details.

FYI this pull request is a pre-requisite for the next datafusion and arrow upgrades since the old

Decoder

is removed. I have a follow up change to make both those upgrades possible

quick question, I noticed that we have a bunch of commits on main that do not comply with conventional commits. i think this is due to us sometimes forgetting to squash commits in a pr and using another merge method. should we configure the repo to enforce this? i guess otherwise setting up generating changelogs from our commits may not work as we are hoping to 🙂

@Will Jones @Robert @QP Hou @Denny Lee @Christian Williams @misha @Florian Valeye @Ian Joiner I would like to cut

main

for a rust binding release of v0.11.0 with datafusion 24, arrow 38, and a few other improvements and fixes today. Can I get a 👍 or 👎

<!here> I am very pleased to share that the

deltalake

crate v0.11 has been published! 🎊 You can read the changelog graciously prepared by @Ian Joiner for more details here

any ideas on how to merge-schema with the latest deltalake ? I'm trying to do the following over an existing table,

write_deltalake('/tmp/write-delta-test', df, mode='append', overwrite_schema=True)

 line 180, in write_deltalake
    raise ValueError(
ValueError: Schema of data does not match table schema

write_deltalake('/tmp/write-delta-test', df, mode='overwrite', overwrite_schema=True)

This works but it creates a version only with the new columns. I am trying to replicate the functionality with the

mergeSchema

option that's available on spark connector

I think with the Rust/Python library you can only overwrite the schema (no schema evolution ).

Found a minor bug here (the error statement shows the swapped schemas). I also missed a statement for checking against

.to_pyarrow(True)

shall I raise a PR for this?

FYI, I've also raised a PR on polars' side to write dfs as delta tables https://github.com/pola-rs/polars/pull/7616 please check it out if this interests you. @Matthew Powers if and when this gets merged, I was thinking of publishing a blog on this, would love to get your pointers 🙂