https://delta.io logo
r

Rodrigo Witzel

06/16/2023, 10:04 AM
Hi! Does https://github.com/delta-io/delta-rs (Python version) guarantee the delta-concurrency ?
y

Yi Wang

06/16/2023, 10:59 AM
as far as I understand, the concurrency control is achieved by the log store, which depends on the what the storage service is used. For example, Azure Blob, GCS has built-in support that only one writer can succeed. But for s3, in the case of the multi-cluster(drivers), it is a bit complex to achieve, you have to use the DynamoDB together within your configuration.
🙏 1
s

Sadiq Kavungal

06/22/2023, 11:38 AM
Is HDFS can achieve the concurrency control
y

Yi Wang

06/23/2023, 3:14 PM
i do not think so
r

Rodrigo Witzel

06/25/2023, 11:27 AM
Thank you, Yi, this was very helpful.