I was poking last night and I found that Cargo can output a bunch of details about compilation units in JSON and I was wondering if I should set up a delta io ingestion pipeline off our

main

branch to start building a data set there 🤔

Does

delta-rs

support replacing tables? I might just be missing it in the docs, or not sure how I would accomplish it with the current setup https://docs.delta.io/latest/best-practices.html#replace-the-content-or-schema-of-a-table

best practice for converting an array of strings, lets say from a csv reader, into an arrow-array

Int32Array

or something similar?

Z Ordering is cool, but I don’t think it’s always necessary. A lot of times sorting the data is all you need. Sorting the data is good for tables that are always filtered by a certain column for example. Hierarchical sorting is also good in a lot of instances. I am assuming that sorting would be easier to implement than Z Ordering (also easier to explain and use perhaps). What do folks think about a

delta_table.optimize.sort(["col1", "col2"])

method that hierarchically sorts the data? We can also add a

delta_table.optimize.z_order

method of course.

I'm going to cut rust-v0.12.0 now

If a crate is published without a changelog, does anybody really notice? https://crates.io/crates/deltalake/0.12.0

lol

@rtyler I can add one

How do you represent the decimal precision and scale in the

String

in this

SchemaDataType

?

Is there any minimal example of writing to Azure blob storage using an

wasb

or

wasbs

URL? In particular how do we authenticate?

Does anyone know if it is possible to track stats for datafusion delta-rs queries? Basically I want to see how many files where considered, pruned and then how many bytes fetched from the object store.

Anyone have the power to rerun builds or should I just push a new commit? https://github.com/delta-io/delta-rs/actions/runs/5136523475/jobs/9243367539

when we were chatting earlier how exciting it is I mentioned this to earlier to @Christian Williams how exciting it is to see delta-rs starting to really experience a snowball effect of users and contributors jumping in to improve the state of big data in Rust (and Python 🙂 )

Why is the python build lint suddenly angry about this now? https://github.com/delta-io/delta-rs/actions/runs/5149497047/jobs/9272568670?pr=1421

@Robert the support for unity catalog (https://github.com/delta-io/delta-rs/pull/1338) seems to be languishing a bit, that draft now has some conflicts unfortunately. Do you need some help bringing that support across the line? I can deploy some Databricks resources in AWS to validate and test if necessary

@rtyler do you have the power to edit this release and generate the release notes? https://github.com/delta-io/delta-rs/releases/tag/rust-v0.12.0 From the edit screen, above the text box you can click the "Generate release notes" button

Hi, I seem to be facing an issue while reading delta tables where the partition column has been renamed, is this a known issue?

I am giving a talk at the Data + AI Summit in a few weeks on why Delta Lake is great for pandas. Here’s the high level flow of the talk: • use the 1e9 h2o groupby dataset (it’s 50 GB when stored in an uncompressed CSV file) • Run a pandas query on the CSV file and show the runtime • Run the same query on a Parquet file & show how that helps • Run the same query on a Z Ordered Delta table and see how much faster that makes the query • Then discuss all the quality of life improvements Delta Lake offers pandas users (versioned data, time travel, etc.) Here are a few things that could make my presentation better: • translating this R Script to Rust (the R Script is so slow and hard to run) • adding the

delta_table.optimize.z_order(["col1", "col2"])

interface to delta-rs (I am currently Z Ordering with PySpark) • Making a release so I can access the to_pandas filters syntax that @shingo added in this PR, so the code snippet looks nicer • I will be highlighting time travel with Delta Lake. Any of the other interfaces outlined in this doc would also be a good way to show the quality of life improvements deltalake gives pandas users compared with Parquet data lakes.

Hello, I’m looking to trial delta-rs library using Python. Before I start a few questions: • Is this production ready or anyone using it in production? • Can we still write data (without transactions)? • Any other caveats I should be aware of that may not be documented?

@Chitral Verma - so cool running a Polars query and then seeing how a Z Ordered Delta table can make the runtime 30x faster. Thanks again for adding this feature.

Datafusion 26 is out and I've started the work to upgrade to that and arrow 40 + object_store 0.6.0. Unfortunately there are some API changes (

CastOptions

) which may cause us to need to have some API changes as well (lifetimes were introduced which is tripping me up). 🙍

I tried looking to https://github.com/delta-io/delta-rs/issues/1445 a bit yesterday but couldn't really track it down. I think it might be some issue with datafusion and/or arrow?

I see Python library has ability to get fragments from DeltaTable by taking filters. Is there anything similar in rust version? It would help if we can get fragments of filenames in rust DeltaTable also.

This might be a dumb question - but for DataFusion to run a query against a parquet/csv file, does it have to load entire file into memory? And how does it query against a full delta table spread across multiple files?

Python

deltalake

v0.10.0 has just been published 🥳 This release includes Z-order optimize as well as various performance improvements to both optimize and vacuum commands. https://github.com/delta-io/delta-rs/releases/tag/python-v0.10.0

Not sure if you can help with this one - looking at this in DataFusion https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html# It says it uses ObjectStore to read the files but I'm not seeing any methods for passing in an ObjectStore. Does it infer and build it based on the file uri? And if so, does that mean I'd need to include auth information in that URI somehow?

documentation says but when you dive into the function for creating a new

ListingOptions

it only takes the path

The Dask community is making some exciting progress on their connector: https://github.com/dask-contrib/dask-deltatable My high level thoughts on the existing code: • They should just use the delta-rs history() command and not reimplement it • They should use the delta-rs vacuum() command • I’m not sure about the catalog logic, but my guess is that there should not be catalog logic in the Dask connector itself See the thread for the POC reader I created.

If anyone feels so inclined, feel free to open an issue / comment on the issues I created, so we can help the Dask community build this connector.

Does anyone here have experience running high-write throughput Delta Lake tables? One thing I've noticed is the size of the checkpoint files can quickly outgrow the dataset (like 10x). Curious to hear any learnings about these types of tables.