Just tried and this tool works quite well with Python + Rust: py-spy. It’s quite convenient!

Up to python-0.6.2, we had AWS access keys in the conftest.py, wondering why we had those keys within the tests? thanks!

Do you have to install rust and cargo before running

pip install deltalake

?

I know that you don’t have to install rust & cargo when using conda, but is it a requirement when using pip?

TIL about Column Invariants from @TD which are 100% not being looked at when the Rust writer does its writes 🤣 https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-invariants

🤗 https://masto.ai/@rfc1149/109671360502970461

Hey! I've been playing with delta-rs and really enjoying it. I had a couple questions though: 1. In the deltalake.table.DeltaTable init method, what is the purpose of the

without_files

parameter? 2. What's the best way to get a reference to the filesystem object being used by a DeltaTable if I want to pass it to another method but with the same configuration / credentials?

Would anyone like to collaborate with @Florian Valeye and I on a blog post on how to read Delta tables in AWS Lambda functions using aws-sdk-pandas? We’ll need to build the AWS Lambda “layer” (prebuild layers don’t include the optional dependencies unfortunately) and then read the data. If someone can help with the instructions on how to build the layer / the code snippets, I can do the writing. Here’s an example of a release with the prebuilt layers. We might want to provide the community with prebuilt layers cause they’re actually annoying to build.

Probably obvious to most folks here, but I just stumbled across

cargo clippy --fix

, and I like it 😄.

Hi All. In this documentation the basic functionality of delta table is supported for writer's version >= 2. I checked the rust library and the max supported writer version = 1. Is there any plan to support to writer version 2?

Hi all, I'm trying to use pyarrow to parse Delta's AddFile.stats field (a JSON string), but there doesn't seem to be any way to parse a

StringArray

full of json into an

Array

or

Table

of nested data. This surprises me, becuase pyarrow.json.read_json does exactly the right thing... but only for line-delimited json files. At least, I didn't see anything e.g. in pyarrow.compute and a google search came up empty. Am I missing something obvious here?

This talk was pretty interesting: Probably not at a point where we focus on optimization yet, but when we get there this causal profiler will likely be useful. https://github.com/plasma-umass/coz

It has been exactly a month since rust 0.6.0 was released so it is time to do 0.7.0. I plan to file a PR to add the 0.7.0 CHANGELOG tonight or tomorrow morning.

@Nick Karpov and I are working on a blog post for how to use Delta Lake and aws-sdk-pandas. Thanks for the awesome contribution from @Florian Valeye, delta-rs is now an optional dependency for that project. Since it’s an optional dependency, it’s not going to be included in the layers that are attached to the releases. Hopefully we can just add a layer with delta-rs + the official layer and that’ll work (multiple layers can be added to an AWS Lambda function). In any case, we need to spell this out clearly for the community, so they know exactly how to start using Delta Lake in AWS Lambda functions.

There are two types of content I’d like to collaborate on this year: • content to get more Python users: the Reading Delta Tables into Polars DataFrames and How to version your data with pandas and Delta Lake are examples. Lots of Python users are completely unfamiliar with Delta Lake and how it can help them. We want to spread the value that Delta Lake will add to their workflows (e.g. tell pandas users that they can skip data with Delta which will help with their memory errors). • content to get more Rust contributors: We also want to create content that will get Rust programmers excited about delta-rs and hopefully encourage them to contribute. @rtyler suggested a post on PyO3 which is a great idea. I’ll create a draft for that post soon and circulate it for feedback. If you ever have any content suggestions or would like to collaborate on a post, let me know. I am all ears!

Is there a way to read a delta table in azure adls gen2 from outside spark (say Python installed in my local machine) using Service Principal and secret?

Hello all! The below issue of being able to write delta tables to s3 , has seemingly been resolved in https://github.com/delta-io/delta-rs/issues/911 , as fixed in https://github.com/delta-io/delta-rs/pull/893

{PyDeltaTableError('Failed to read delta log object: Generic DeltaS3ObjectStore error: Atomic rename requires a LockClient for S3 backends. Either configure the LockClient, 
        or set AWS_S3_ALLOW_UNSAFE_RENAME=true to opt out of support for concurrent writers.')}

However, the fix seems to just be allowing to pass

AWS_S3_ALLOW_UNSAFE_RENAME

as an option. This can cause issues with concurrent writes, as a lock is not acquired before renaming. I was hoping that the solution would involve being able to set a lock on the bucket, and releasing the lock after a write was performed. Does the above solution mean that concurrent writes are not supported/advised with delta lake ? Keen to hear how others are approaching this issue 🙏

Hello again! I'm checking to see if AWS Glue can be registered as the data catalog when creating delta tables, but from the documentation, it looks like Glue is only referred to when reading delta tables https://delta-io.github.io/delta-rs/python/usage.html?highlight=glue

Will the ADBC work allow us to add Golang bindings easier? Curious on this issue.

Hi all! Is it writing to GCS currently supported from python bindings? I’m faced with this error:

deltalake.PyDeltaTableError: Failed to read delta log object: Generic GCS error: Error performing copy request test_delta/_delta_log/_commit_0025723a-ff2f-4bb6-9f7f-7c80f95fd5b9.json.tmp: response error
......
<p>POST requests require a <code>Content-length</code> header.  <ins>That's all we know.</ins>

to reproduce it:

import os

import pyarrow as pa
import deltalake

storage_options = {"SERVICE_ACCOUNT": os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")}
pylist = [{'n_legs': 2, 'animals': 'Flamingo'}, {'year': 2021, 'animals': 'Centipede'}]
table = pa.Table.from_pylist(pylist)
deltalake.write_deltalake("<gs://some_bucket/test_delta>", table, storage_options=storage_options)

Hello everyone 👋, We have just released the next version of the Python binding for delta-rs: deltalake 0.7.0. Credits: • @fvaleye • @hayesgb - first contribution! • @roeap • @damiondoesthings - first contribution! • @gyscos - first contribution! • @andrei-ionescu • @Fazzani - first contribution! • @ismoshkov - first contribution!

We got rid of the Ruby bindings per your suggestion @rtyler: https://github.com/delta-io/delta-rs/pull/1102. Thanks for getting this done Robert!

Can someone with repo admin permissions update the project description? cc: @Denny Lee

Mehdi Ouazza, a popular data LinkedIn person, just joined MotherDuck and pinged me about if we’d like to collab on the Delta Lake / DuckDB integration. Is this something we’d like to invest in now, or should we wait till the ADBC work is a bit farther along?

We just nuked that Ruby bindings skeleton and then along comes https://bundler.io/blog/2023/01/31/rust-gem-skeleton.html It looks like Rust and Ruby are friends again 😆

We were finally able to get delta-spark published to conda, see here: https://anaconda.org/conda-forge/delta-spark Looks like we created a PR to get deltalake added a few months ago and it hasn’t been responded to yet: https://github.com/conda-forge/staged-recipes/issues/21102 @Nick Karpov - any suggestions on how to move the deltalake request forward? Perhaps just pinging on the PR?

I took put our 2023 H1 roadmap into a GH issue here: https://github.com/delta-io/delta-rs/issues/1128 The issue is now pinned, so it will be very visible to new contributors :)

@Will Jones - the New features in the deltalake 0.7.0 release blog post is ready for your review. If anyone else would like to comment, just DM me your email and I’ll share the doc with you. Thanks for adding these awesome features!!

It seems like most of our users are using the Python bindings. What do we want to call this project when reaching out to the Python community? “Delta Rust” seems off putting for most Python programmers. They’ll probably think “well, that’s not the project for me”. Should we just call this project “deltalake” when chatting with the Python community? That’s the pypi package name and seems like a good option - thoughts?

Just noticed an interesting delta-rs / delta-spark difference. Delta Spark doesn’t let you instantiate a Delta Table with a specific table version, but delta-rs does. • delta-rs:

DeltaTable("../rust/tests/data/simple_table", version=2)

• delta-spark:

DeltaTable.forPath(spark, "/path/to/table")

- no version argument available Are there any implications of this difference we should think about?