Has anyone considered using redis as a lock state db instead of dynamodb? I am working on a big personal project where I am building everything in native k8s. I like the dynamodb solution but I thought maybe I could achieve the same thing just via redis? And maybe somebody already done it. Maybe you already thought about that @User.

@S Thelin while I haven't considered redis explicitly, @John Darrington has a need for a PostgreSQL lock. Just yesterday I was returning to our work in https://github.com/delta-incubator/locking-object-store to make a quality abstraction here for locking.

Ah interesting, would redis not be a good solution here? I was naively thinking it is pretty light weight, and I was playing around writing the state myself from my parallel tasks. So it does not have a direct connection with delta-rs framework. What you have there looks better than what I am trying to do at the moment.

redis is a good cheap locking mechanism IMHO if you've got it available. The objective for this repository is to kind of hide/abstract the details of locking with the

ObjectStore

trait so that a user can just create the

ObjectStore

and let it handle the locking transparently without needing to deal with a bunch of custom code in delta-rs

Sounds great, I will follow this one closely, will fiddle around a bit with redis and see if I can share anything from that.

That would be great. I’ve had the same thought as I’ve worked on the Postgres one when I’ve had the chance

@rtyler @John Darrington @Matthew Powers I have made a very stupid python example now with

from redis import StrictRedis, lock

With No Redis Lock input

df1 = DataFrame({'id': [1]})
df2 = DataFrame({'id': [2]})
df3 = DataFrame({'id': [3]})
df4 = DataFrame({'id': [4]})

Fed via

joblib

to ensure to try to mimic a multi process situation. Here you can see it writes the physical, files, but only 3 commits in the delta log.