I created a library called mack that exposes some helper functions delta-spark helper functions that users commonly need: https://github.com/MrPowers/mack. The library has gotten a lot of traction in a short period of time. We might want to build some of these features directly into delta-rs eventually. Deduplicating a Delta table is a great example. Lots of users want to perform this function and we could provide them with a performant out-of-the-box implementation. Thoughts?

Shall we tell dependabot to ignore arrow, datafusion and parquet updates since we usually update them manually anyway?

I was going to start working on the PyO3 blog post today and then found that there is already another blog post on the same topic that’s currently on the front page of HackerNews: https://news.ycombinator.com/item?id=34968186 😐 I still think a “How delta-rs exposes Python bindings to a Rust codebase with PyO3” would be a compelling post. Thoughts? My main takeaway here is that @rtyler has good blog post ideas and I need to execute faster next time 🚀

See my comment on the Pandas 2.0 and the Arrow revolution thread. Lots of users are really excited about using Polars / DuckDB to filter datasets and then convert them to pandas datasets to avoid memory errors. I think we have a great opportunity to educate the pandas community that Delta Lake allows for predicate pushdown filtering and can help solve their memory problems. I have an idea for benchmarks we can prepare for the pandas community demonstrating how we can solve their out of memory exceptions / make their analyses run faster. Would anyone like to collab with me on these benchmarks? It’s probably like 2 hours of work cause I already have the dataset and problem loaded in my head. I’d be able to generate some really compelling content for the pandas community with these benchmarks.

Has anyone tried s3 feature with deltalake rust 0.7.0 ? It might still have some version conflicting issues with

rusoto_dynamodb

https://github.com/delta-io/delta-rs/issues/1191

pypi package downloads of deltalake have fallen pretty significantly. From 1.1 million monthly downloads (if I recall correctly) to 450k. I think we should investigate…

We just released the New features in the deltalake 0.7.0 release blog. Here is the LinkedIn post if you’d like to upvote / reshare.

I’d like to start reaching out to different communities and giving them a high level overview of the next steps for Python deltalake (new word for delta-rs) as it pertains their roadmaps. I’m pretty tight with the Polars and Dask communities and can start there. Here’s the messaging I’m thinking of at a high level: “We’ve built a connector that makes it easy to read Delta tables into Dask DataFrames. This allows users to query data faster than regular Parquet data lakes because of transaction log level metadata skipping and avoiding inefficient file listing operations. We’re currently integrating the Arrow Database Connectivity (ADBC) library into delta-rs, which will allow us to build Dask writers and support for delete, merge, and update transactions more easily. We will let you know when the ADBC work is done so we can start working on the Dask writers/DML support.” I’ll plan on sending Ritchie from Polars a similar message. Does this sound like a good plan?

This is handy: https://cheats.rs/

I just created a project called levi that leverages the new

get_add_actions

API to give users new insights into the sizes of files in their Delta table. Here’s example usage:

import levi
from deltalake import DeltaTable

dt = DeltaTable("some_folder/some_table")
levi.delta_file_sizes(dt)

# return value
{
    'num_files_<1mb': 345, 
    'num_files_1mb-500mb': 588,
    'num_files_500mb-1gb': 960,
    'num_files_1gb-2gb': 0, 
    'num_files_>2gb': 5
}

This makes it clear that the Delta table contains lots of small files and 5 huge files. This Delta table will almost certainly cause slow queries, so the user should optimize. This is the first API that makes it easy to access this data. Think this is a big step in the right direction to making Delta tables more usable!

I made this PySpark OSS Contribution Opportunity post on the Apache Spark Subreddit, which encouraged a bunch of new people to commit. We have 24 issues with the “good first issue” tag (wonderful job making these) and think it’d be good to make a post in the Rust subreddit to let folks know we’re happily accepting contributions. My last post in the Rust subreddit was upvoted a lot. Thoughts?

@Will Jones - I was chatting with Marc Garcia, the creator of PDEP-9, the pandas discussion on how to “Allow third-party projects to register pandas connectors with a standard API”. Looks like they’re going to solve this problem with Arrow. Might be good to chat with them about ADBC.

Preparing for Rust 0.8.0 release now.

Sadly we are slowed by https://github.com/github-changelog-generator/github-changelog-generator/issues/1027. I will create the changelog manually then.

Just added a

levi.skipped_stats(delta_table, filters=[('a_float', '=', 4.5)])

function that returns this

{'num_files': 2, 'num_files_skipped': 1, 'num_bytes_skipped': 996}

. This allows users to see how many files / how much data gets skipped for different predicates. It’ll help them figure out when they should Z ORDER, etc. The new

get_add_actions

API is opening up all sorts of new possibilities to get insights on Delta tables. Here’s the code if you’re interested.

I manually updated the changelog for Rust 0.8.0. Pure Python issues/PRs, CI / development process / dependabot / issues without PRs etc have been ignored. Please check whether this is accurate. Really thanks!

Noah Gift just gave delta-rs a nice shoutout on LinkedIn. Feel free to like / comment.

Saw this collection of CI scripts last night: https://github.com/jonhoo/rust-ci-conf Explained at the beginning of this stream: The one that stood out to me for our project was the minimal version test (

-Zminimal-versions

or probably better with the upcoming

-Zdirect-minimal-versions

). That might allow us to be more flexible with our Arrow dependency so we could for example support a range like

arrow >= 30, <= 32

. Datafusion probably has more frequent breaking changes in it’s API, but maybe as it gets more stable we could support a wider range for that too.

FYI I released 0.8.0 for Python but shortly after discovered a few bugs in the main new feature (overwrite partition). So I have a PR up to fix those, and will cut a bugfix release once that is merged.

Hello everyone! 👋 We have just released the next version of the Python deltalake package (based on delta-rs): deltalake 0.8.1 This release fixes several bugs and adds the ability to overwrite partitions. Credits: • @ismoshkov • @rbushri - first contribution! • @houqp • @guyrt - first contribution! • @reoap • @marijncv

is there a plan to enable reading from a delta table's change data feed from

delta-rs

? i don't see it on the roadmap

Anybody happen to have example Rust code floating around which writes a String,String map? I am having a hell of a time battling arrow's wacky types

Hey folks, could anyone chime in on this discussion https://github.com/delta-io/delta-rs/discussions/1178 Wondering if there is a way to work with spark-written delta tables with array<T> columns via delta-rs. thanks!

I finally got around to writing a blog post about writing Python modules in Rust using Arrow. Big emphasis on how much easier it is to maintain than a Cython/C++ extension. I hope to eventually do a larger tutorial on using the arrow-rs pyarrow submodule, since I think that’s really under appreciated. Thanks to @Matthew Powers for continuously reminding me to write this 🙂 https://www.datawill.io/posts/pandas-arrow-rust/

Hey all - does anybody have experience with running delta-rs from a lambda (using Rust) in AWS? I'm struggling to work out how to open the Delta table with the IAM permissions I've given to the lambda. When testing locally I've used my temporary credentials (session token etc) alongside

open_table_with_storage_options

, but not sure how to do this from the lambda itself Thanks

Hi team, I ran an optimize + vacuum operation on a unpartitioned table in rust. The table size approximately doubled in size after the optimize/vacuum. Is it an expected behaviour? thanks!

The more time I spend with the arrow crate the more convinced that I am we should switch to arrow2 by default as soon is it has the necessary support we need. It's insanely complicated and has a fair bit of

unsafe

floating around I am running into more and more. 🤦 🙀

Here’s a draft blog post on How to create and append to Delta Lake tables with pandas. Feel free to respond in the thread or DM me with your email if you’d like permissions to comment on the doc. This post highlights the ability to overwrite specific partitions of Delta tables using pandas in the new release. I put this feature in a broader context and set the blog post title so that it’ll rank well in SEO searches and get more traffic over time. I plan to publish this one on April 5th!

@Will Jones I'm hacking on top of your recently merged changes to arrow-rs, there's some API changes in arrow upstream that are incompatible with delta-rs, have you already started working on that , or should I continue cleaning up the compat in my branch?