Hi everyone! Is there a way to calculate the length of the Delta table using delta-rs effectively?

I believe we should be strongly encouraging downstream consumers to use our re-exported

arrow

and

parquet

modules in our documentation. I am finding downstream users importing these themselves and not being able to take on new versions of delta-rs without some manual upgrade work. This is especially problematic for downstream consumers who are 2 orders removed from us. Do y'all have opinions on why somebody should be adding both

arrow

and

deltalake

to their

Cargo.toml

?

Hi everyone! Is there any way to write a field's metadata with Python delta-rs? Looking at the API reference for

write_deltalake()

, it seems like we can only specify schemas with pyarrow, which unfortunately doesn't preserve the metadata dictionary in the conversion to a

deltalake.schema

.

Hello. Is there any known issue if you append concurrently from let’s say

dask

to a delta table with

delta-rs

? Will it be able to handle the log properly or will it fail silently?

🙀 my

target/

directory for building/testing delta-rs is 3.4GB of stuff 🙀

Before I create an issue I wanted to check here. Why does

JsonWriter::buffer_len()

always return 0?

Hi everyone! I noticed that delta_datafusion load partitioned files into a filegroups declared as HashMap<Vec<ScalarValue>, Vec<PartitionedFile>>. Does that mean it will load data lost sequence which write in order. For example, the partition key is datetime.

I have a proposal for some of the committers: it's not automated yet, but what if we start pushing more regular patch versions? We're really lethargic about pushing releases of the Rust crate and I that makes working downstream a bit more of a PITA. Here's an approach I would like y'all to consider: • Contributions which make API changes, change the version in

Cargo.toml

, e.g. the next API changes should result in

0.10.0

. • Every two weeks we take

main

and assuming we're stable and there is desire (e.g. "I want to consume this feature that landed") we cut a patch release, e.g. 0.9.1, 0.9.2, 0.9.3.

I had a similar idea for Python: every two weeks, we cut a pre-release version and upload to PyPI (but only if there hasn’t been a formal release in the past week). This wouldn’t increment the version, but would just have a

.devXXXX

suffix. This is similar to how DuckDB operates, and it nice because folks can try out new features and fixes without having to build the packages themselves. https://pypi.org/project/duckdb/#history This should be completely automated and require no manual intervention (unless there is a CI failure)

Is it possible to just optimize files between commit 0 and some commit version N? During my testing, when loading table version N and then running optimize, it looks like it just overwrites commit N+1 instead of trying to commit N+1 and then advancing to the next available version. Sounds like a bug? Or is this not possible to do?

With Datafusion, is it possible to determine which file a particular record came from? Something similar to

input_file_name

from Spark would be beneficial since it would help with determining the origin a record which can be used to support delete, update, and merge operations.

Hi Folks, I noticed delta-rs rust package version 0.9.0 was released with HDFS support. We'd love to start evaluating delta-rs with the python bindings. Is a released for python 0.9.0 planned? Or should we look into building it ourselves?

thankfully datafusion 23 is released which relies on arrow 37, so I'm going to prepare an upgrade in my

arrow-37

topic branch 🤞

I am running one query from the h2o benchmarks that were recently updated by duckdb, see here. I ran the first groupby benchmark locally (M1 Mac, 64 GB of RAM) with different query engines. The 100 million row CSV file (1e8) is 5GB on disk and the 1 billion row file (1e9) is 50GB on disk. I am counting data loading time + query runtime. It’s a 9 column datasets and here’s the query:

SELECT id1, sum(v1) AS v1 from the_table GROUP BY id1

. Here are my results: • DuckDB: query errors out, even on the 1e8 dataset (notebook) • DataFusion: 9.8 seconds 1e8 / 99 seconds 1e9 (notebook) • Polars: 7.5 seconds 1e8 / 229 seconds 1e9 (notebook) • Pandas 2.0 with usecols: 19.3 seconds 1e8 / 204 seconds 1e9 (notebook) I tried a bunch of tweaks to get DuckDB working, but it only worked on the tiny 1e7 dataset, so not sure what I’m doing wrong. I was really impressed with DataFusion’s performance on the 1e9 dataset. The Polars performance surprised me on the 1e9 dataset. And Pandas 2.0 just seems way better than what it used to be. I am going to use this dataset and structure some queries that show how Delta Lake can allow for faster query runtimes.

I made good progress on arrow 37 last night, I am working through the datafusion changes now, and I'm hoping that we can cut a 0.10.0 release this weekend 🤞

@Will Jones the commit rules totally get screwy with links "✖️ body's lines must not be longer than 100 characters [body-max-line-length]" Can you disable this check?

🌮 🌮 to @Florian Valeye for this very interesting and promising pull request: https://github.com/delta-io/delta-rs/pull/1315

I created a benchmark that shows how to run a pandas query 20 times faster with a Z ORDERed Delta Lake compared with a CSV file (for a 100 million row / 5GB dataset). I think this is going to be one of the most compelling reasons we can give the pandas community to start using Delta Lake.

I am updating a few branches we have open in pull requests right now to pick up the changes for arrow/datafusion, hoping to land a lot of this soon 😛

Committers! I would like to take

HEAD

of

main

and release 0.10.0 of the Rust crate today/tomorrow. Yay 👍 or Nay 👎 @Florian Valeye @misha @Robert @Will Jones @Christian Williams @Thomas Vollmer (Microsoft) @QP Hou @Denny Lee

@Matthew Powers have you or @Denny Lee discussed North Bay Python at all? I think it'd be cool for somebody other than me to submit to talk about doing big cool data work with delta io and pandas/polars https://2023.northbaypython.org/speak 😆

Oh - yes, we have @Nick Karpov submitting and I'm talking to three others to also submit. 🙂

tests/test_fs.py::test_roundtrip_azure_env

keeps timing out for unrelated changes, I'm inclined to just disable the dang thing. It feels like more trouble than it's worth, but then again I'm not using deltalake for Azure 😛

@Robert omg https://github.com/delta-io/delta-rs/pull/1320 🧵

With delta.rs on 0.10.0 - should we update delta.rs python bindings?

Here’s a POC Streamlit app that’s built with DuckDB and Delta Lake: https://nemtracker.streamlit.app/ The code is so simple. Here’s the post from the author in case you’d like to upvote it.

Does that mean all our timestamps are milliseconds?