Any thoughts on using GitHub milestones for delta-rs? https://github.com/delta-io/delta-rs/milestones It’d be awesome to have some insight into what’s coming in the new releases. We could use that as our roadmap. BTW - everyone is doing an amazing job with the release notes!

Hello people, I have a question regarding authenticating to Azure using the Python library. In particular, I'm trying to use the managed identity authentication. While it works just fine using pure Python code (the fsspec library) and the Azure Function runtime sets all the necessary environment variables, deltalake fails to obtain the access token. I know this logic is technically a part of the arrow / object_store library in Rust, but maybe someone here would be able to help me out here. Any pointers are much appreciated! Thanks

I am trying to work around this issue since it is taking a while to fix: https://github.com/delta-io/delta-rs/issues/1411. What I am thinking is to read the json after the delta write and remove the infinity value from the stats. I can do this using boto3 but wanted to see if there is an easier hack using the Deltalake library. Ideas?

Hi, everyone. I was going to submit a new issue in GitHub, but decided to confirm something before I do. I would like to use delta-rs to read a delta table in Azure blob storage for a client. Usually the URL, then, is something like

https://$<http://ACCOUNT_NAME.blob.core.windows.net|ACCOUNT_NAME.blob.core.windows.net>

. It seems delta-rs makes this assumption, too, and I can't seem to find a way to change it. I'm not exactly proficient with Rust, but my amateurish review of the code seems to confirm it's not possible. The problem is, there are technically multiple "registered clouds" with Azure. For instance:

❯ az cloud list | jq .[].name
"AzureCloud"
"AzureChinaCloud"
"AzureUSGovernment"
"AzureGermanCloud"

In these other "clouds", the URL for, say, Azure blob storage changes. Is there any way to indicate to delta-rs that the endpoint suffix should not be

<http://blob.core.windows.net|blob.core.windows.net>

, but instead something else? I've also been able to work around this in the past by just passing a connection string that includes the

EndpointSuffix=$SUFFIX

, but that doesn't really seem to work, either (or I'm missing something). Like I said, I was planning to submit a GH issue, but I figured it was time to ask around first before I go that far. EDIT: I'm trying to use delta-rs in Python. I should have said that to begin with.

Is this value correct? https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_arrow.rs#L170C21-L170C25 I seem to get a panic from parquet writer, switching this to

false

"fixes" it. That would be inline with https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_arrow.rs#L79 and https://github.com/apache/arrow-rs/issues/1697

https://github.com/delta-io/delta-rs/pull/1622 will fix the CI job so we can merge https://github.com/delta-io/delta-rs/pull/1549

One last PR to review, and we'll be releasing Python 0.10.2: https://github.com/delta-io/delta-rs/pull/1613

@Will Jones - anything I can do to help move this PR to add Python API references to mkdocs forward? I’d really love to get these docs setup such that I can start contributing!

Hey maintainers, could someone take a peak and/or provide feedback on this PR that was put together by a colleague of mine? https://github.com/delta-io/delta-rs/pull/1621

Using Python

deltalake

with Pyarrow and S3 and trying to minimize query latency, I noticed that Pyarrow makes HEAD requests for every file it reads. I believe this is to determine the file size so the parquet footer can be read. However, I think the delta log already contains the file size, so in principle this could be communicated to Arrow to avoid the HEAD requests (see e.g. https://github.com/apache/arrow/pull/7547 ). Has anyone looked into something like this?

is DataFusion still the best method for querying DeltaLake tables from Rust?

chrono semver lolz this was clearly an urgent change worth introducing a deprecation in a patch release 😆

QQ - do you think it makes sense to link the Delta Docker (https://go.delta.io/docker and https://go.delta.io/DockerHub) under the "Getting Started" section of the readme? I'm guessing "no" actually but thought I'd ask, eh?!

Anyone know if liquid clustering will be supported (or is supported) in delta-rs?

I’ve been working on a branch that enables Arrow schema’s using Dictionary columns to convert into Delta schema, using the dictionary value type, also changed how the partition stringification works so it works when partition column is encoded as Dictionary in Arrow. This cleans up some ergonomics around these columns we’re dealing with and prevents failures doing schema conversions. One thing we’d like to also leverage is to preserve dictionary encoding when reading out from Parquet. Parquet stores the arrow schema metadata when a batch is written out. However all the delta-rs write paths will convert to the Arrow schema for writing using the Delta schema which has no concept of Dictionary afaik. So i’m coming up with ways ensure the Arrow schema has Dictionary columns where we want when its getting written out so Parquet will read them into dictionary columns when querying. So far I have: • when writing out, add option to use the write execution plan’s schema vs reading teh schema from delta and converting to arrow, some validation would still occur ◦ don’t like this as it hits more code paths and is a lower level detail that’s hard to communicate upwards to users using the API • when defining a delta schema, add metadata to fields to suggest they should be encoded as dictionary. Then on Delta -> Arrow schema conversion that metadata is checked for, if present, wrap arrow type in Dictionary ◦ like this better, as its encoded into the delta schema, inline with how the parquet arrow schema hint works ◦ is this too specific to the rust/arrow impl though? is there a precedent for options/metadata that only apply to one delta impl? Would love feedback and thoughts, thanks!

Does delta-rs suport automatic schema evolution? (addition of columns/renaming columns/missing columns in newer versions of data) on append?

@Will Jones can you assign the issue regarding adding the merge statement in the python API to me? I've tagged you but perhaps you missed it :)

This is the issue: https://github.com/delta-io/delta-rs/issues/1357

I have gone ahead and done so @Ion

@rtyler 🙏

@Matthew Powers are you open to being added as a maintainer to delta-rs to help curate docs, as mentioned here: https://github.com/delta-io/delta-rs/pull/1646#pullrequestreview-1634511810

@Will Jones I've done some tidying up of https://github.com/delta-io/delta-rs/pull/1602, up for taking it out of draft and merging?

Has anyone managed to cross compile the python lib on apple silicon to a linux (x86_64) target? I tried zig, docker and podman but failed miserably with all

@Robert - message from the Dagster team:

Hey Matt - resurrecting this thread!

We now have the CI infrastructure set up to allow community members to run the full test suite. Additionally, we’ve taken the Delta Lake PR and rebased it for the author here: https://github.com/dagster-io/dagster/pull/16463

We’re just waiting to hear back from the author as there are a few more things that need to be fixed. If he doesn’t get to it we plan on having our engineers take it over the finish line at the end of October. However, if the author ends up finishing it we will ship it earlier.

Let me know if you still want to do comms on this because we definitely do!

This is really exciting!!

Does

list_with_offset

with S3 currently push the offset down to the S3 API, or does it fall back to the ObjectStore default implementation that uses a simple

list

with post-filtering? Given that

list_with_offset

is not defined here: https://github.com/delta-io/delta-rs/blob/a74589be7c39315360925049c716d1d70b906970/rust/src/storage/s3.rs#L470

@Robert I am planning to cut rust-v0.16 this weekend with some of the things in

main

and one or two fixes in flight. I'm planning to pick up your modularization work after that has been released.

Hey I think there is a bug with the max_concurrent_tasks setting in z_order command. I tried to set it to 1, and it still uses all the cores on my computer leading to OOM 😞 I am on latest Python version

Has anyone tried to use that flag with success?

@Will Jones it should pass the checks now for my PR.. I usually use Pyright instead of mypy, so was checking those instead of mypy. I find it actually strange mypy wants me to explicitly add an else to return a None, since it's being skipped anyway.