Anyone know if liquid clustering will be supported (or is supported) in delta-rs?

I’ve been working on a branch that enables Arrow schema’s using Dictionary columns to convert into Delta schema, using the dictionary value type, also changed how the partition stringification works so it works when partition column is encoded as Dictionary in Arrow. This cleans up some ergonomics around these columns we’re dealing with and prevents failures doing schema conversions. One thing we’d like to also leverage is to preserve dictionary encoding when reading out from Parquet. Parquet stores the arrow schema metadata when a batch is written out. However all the delta-rs write paths will convert to the Arrow schema for writing using the Delta schema which has no concept of Dictionary afaik. So i’m coming up with ways ensure the Arrow schema has Dictionary columns where we want when its getting written out so Parquet will read them into dictionary columns when querying. So far I have: • when writing out, add option to use the write execution plan’s schema vs reading teh schema from delta and converting to arrow, some validation would still occur ◦ don’t like this as it hits more code paths and is a lower level detail that’s hard to communicate upwards to users using the API • when defining a delta schema, add metadata to fields to suggest they should be encoded as dictionary. Then on Delta -> Arrow schema conversion that metadata is checked for, if present, wrap arrow type in Dictionary ◦ like this better, as its encoded into the delta schema, inline with how the parquet arrow schema hint works ◦ is this too specific to the rust/arrow impl though? is there a precedent for options/metadata that only apply to one delta impl? Would love feedback and thoughts, thanks!

Does delta-rs suport automatic schema evolution? (addition of columns/renaming columns/missing columns in newer versions of data) on append?

@Will Jones can you assign the issue regarding adding the merge statement in the python API to me? I've tagged you but perhaps you missed it :)

This is the issue: https://github.com/delta-io/delta-rs/issues/1357

I have gone ahead and done so @Ion

@rtyler 🙏

@Matthew Powers are you open to being added as a maintainer to delta-rs to help curate docs, as mentioned here: https://github.com/delta-io/delta-rs/pull/1646#pullrequestreview-1634511810

@Will Jones I've done some tidying up of https://github.com/delta-io/delta-rs/pull/1602, up for taking it out of draft and merging?

Has anyone managed to cross compile the python lib on apple silicon to a linux (x86_64) target? I tried zig, docker and podman but failed miserably with all

@Robert - message from the Dagster team:

Hey Matt - resurrecting this thread!

We now have the CI infrastructure set up to allow community members to run the full test suite. Additionally, we’ve taken the Delta Lake PR and rebased it for the author here: https://github.com/dagster-io/dagster/pull/16463

We’re just waiting to hear back from the author as there are a few more things that need to be fixed. If he doesn’t get to it we plan on having our engineers take it over the finish line at the end of October. However, if the author ends up finishing it we will ship it earlier.

Let me know if you still want to do comms on this because we definitely do!

This is really exciting!!

Does

list_with_offset

with S3 currently push the offset down to the S3 API, or does it fall back to the ObjectStore default implementation that uses a simple

list

with post-filtering? Given that

list_with_offset

is not defined here: https://github.com/delta-io/delta-rs/blob/a74589be7c39315360925049c716d1d70b906970/rust/src/storage/s3.rs#L470

@Robert I am planning to cut rust-v0.16 this weekend with some of the things in

main

and one or two fixes in flight. I'm planning to pick up your modularization work after that has been released.

Hey I think there is a bug with the max_concurrent_tasks setting in z_order command. I tried to set it to 1, and it still uses all the cores on my computer leading to OOM 😞 I am on latest Python version

Has anyone tried to use that flag with success?

@Will Jones it should pass the checks now for my PR.. I usually use Pyright instead of mypy, so was checking those instead of mypy. I find it actually strange mypy wants me to explicitly add an else to return a None, since it's being skipped anyway.

Why are the docs builds failing in the ci? @Will Jones Do we need to change something there

Are there plans for implementing the ability to update the schema of an existing DeltaTable?

Quick question, I started working on exposing merge to python bindings. I was thinking on mimicking the mergebuilders in python, where each class method updates the class attributes of python MergeBuilder and only with .execute you dispatch it to rust. Does this make sense?

https://github.com/ion-elgreco/delta-rs/commit/f62abab996936e7e362b5ae62f861ab8308aaf41 this is what I got to now, let me know if I am way off ( :

Looks like the InfluxDB 3.0 release will swap out the backend from Go => Rust. Here’s is the InfluxDB Rust codebase. Anyone use InfluxDB / interested in building a Delta Lake connector?

Am I missing something or is there no way to read, set table properties from the python bindings? Hmm apparently only during write_deltalake can you add a table property

I see it's only possible during a write. I'll create an issue about it, makes sense for user to also just set table properties manually without writing

This message was deleted.

I'm hoping I can release the 0.16 of the rust crate today, only one or two more things to close out 🤞

Hi! I'm trying to configure dynamodb to write using

write_deltalake()

from delta-rs using Python. I have set the following env vars:

AWS_S3_LOCKING_PROVIDER=dynamodb
DYNAMO_LOCK_TABLE_NAME=delta_log
AWS_REGION=eu-west-1
AWS_DEFAULT_REGION=eu-west-1

Then I try to write a table

write_deltalake("<s3://my-bucket/tmp/some_delta_table>", my_table, mode="overwrite", schema=data_schema)

And I'm getting the following error from dynamodb:

Generic DeltaS3ObjectStore error: Dynamo Error: Get item error: The provided key element does not match the schema (GetItemError(Validation("The provided key element does not match the schema"))).
ERROR MicroBatchExecution: Query [id = f3781630-6c8b-45c3-b494-06af9c2f893a, runId = 35b19a83-a98c-453b-8242-482ad9dfe406] terminated with error
py4j.Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/home/pysparkarrow.py", line 67, in write_parquet
    write_deltalake("<s3://my-bucket/tmp/some_delta_table>", my_table, mode="overwrite", schema=data_schema)
  File "/home/venv/lib/python3.9/site-packages/deltalake/writer.py", line 324, in write_deltalake
    _write_new_deltalake(
OSError: Generic DeltaS3ObjectStore error: Dynamo Error: Get item error: The provided key element does not match the schema (GetItemError(Validation("The provided key element does not match the schema"))).

@Will Jones are you planning to update the roadmap? 🙂 I think quite some stuff changed or has been implemented. Personally I am interested to see whether ADBC drivers is something that still will be added as a feature?

Would it be possible to do a small release after this branch is merged? 😆 https://github.com/delta-io/delta-rs/pull/1668 Then I could immediately start working on a fix in polars

Did something change on the way hive partitions are written it seems the encoding is not working anymore