https://delta.io logo
j

John Darrington

05/17/2023, 1:04 AM
any documentation on how exactly to configure the delta-rs crate to work with the individual storage backends it supports?
w

Will Jones

05/17/2023, 1:07 AM
That's pretty broad. In general, we don't have extensive docs for the Rust crate yet.
Is there a particular backend you want to know more about?
More of them are just backed by the object-store crate, so the same configuration options apply. S3 is a little more complicated.
j

John Darrington

05/17/2023, 1:08 AM
Azure Blob Storage particularly, though S3 would be a close second
ah didn't look deep enough to see it was using object-store
j

John Darrington

05/17/2023, 1:10 AM
thanks 🙂
what makes s3 more complicated to configure?
w

Will Jones

05/17/2023, 1:11 AM
Unlike GCS or Azure blob store, S3 doesn't have any way to do atomic commits out of the box. So you must either opt out of concurrent writer support or set a dynamodb lock client.
j

John Darrington

05/17/2023, 1:12 AM
ah
well I guess it's a good thing I'm on azure infra atm then lol
though I was hoping that I could get things working with Minio
y

Yousry Mohamed

05/17/2023, 1:13 AM
This post has some details on configuring things for AWS https://www.cuusoo.com.au/guide/navigating-the-data-lake-using-rust-part-two
👀 1
👍 1
j

John Darrington

05/17/2023, 1:15 AM
awesome
d

Denny Lee

05/17/2023, 3:45 AM
Really awesome post @Yousry Mohamed 🙂
y

Yousry Mohamed

05/17/2023, 3:52 AM
Thank you @Denny Lee, glad to help spread the awesomeness of Rust flavour of Delta Lake 🙂
🎉 1
😀 1
j

John Darrington

05/17/2023, 1:51 PM
Yeah we're looking at using it over having to stand up spark
🎉 1
and using the rust code interacting with delta tables in python/node 😄
d

Denny Lee

05/17/2023, 3:01 PM
I like it 🙂
j

John Darrington

05/17/2023, 3:01 PM
We wrote an open-source data lake that we're going to back with delta tables so we can reach the petabyte range of data stored, it's a big project
d

Denny Lee

05/17/2023, 3:22 PM
That sounds cool - definitely reach out to us if we can help, eh?! Any details of your project that you can share - sounds intriguing 🙂