Hello Delta rs community! I am planning to use delta-rs to write to Delta tables stored in S3 or on Databricks. Is this possible? If yes, how? I've read the documentation and couldn't find a topic making this clear.

Yes, we support S3, as well as Azure blob store and GCS. https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table

Awesome ! One more question, does delta rs support multiple writers to the same table stored in s3 ?

No, not without special configuration.

@Luan Carvalho Databricks operates what's commonly known as the S3 Commit Service inside a given workspace. This allows for concurrent writers by DBR-based applications. With delta-rs we have some DynamoDB-based locking code which can be used, but that will not guarantee safe concurrent writes from an unmodified Rust and unmodified Spark-based writer. The pattern that I take is that tables which Rust writers will write to are _only _ written to by Rust writers. Except some for special scenarios where I make my Spark code use the same DynamoDB lock semantics

When you mentioned dbr based applications you are talking about Spark application inside databricks environment or delta rs inside databricks ?

Spark via DBR, there is no version of delta-rs code which can utilize the Databricks proprietary commit service

Awesome guys

readers need not concern themselves with the lock at all, but if you have multiple threads/processes attempting to write transactions independently, those should use the dynamodb lock, but that's built-in to the writer code already in delta-rs so all you need to do is configure the dynamodb table appropriately