Would there be any heartache if I worked on a way to do locking with a Postgres instance vs dynamodb on s3? Use case is we're using MinIO (S3 Compliant) cloud storage on premises and can't use Dynamo for locking

there would be no heartache

cool - shouldn't be too difficult honestly

was said before, but no heartache at all 😄. It may be worthwhile to consioder though which way we want to go with this.. i,e, there have been some discussions around offering alternative locking mechanisms, to be able to use databricks and delta.rs for the same table. While postgres would not offer that, there may be soem inspiration for a simpler approach rather then using locks… https://github.com/delta-io/delta-rs/issues

using the Databricks S3 commit service would require them to post an actual API, which I've given them grief about since before the delta-rs project was founded

there is a design document linked in the issue as well.. all in all that is going to be a larger effort, and we will be supporting the lock based going forward, so implementing a lock client shopuld also be fine

I think you meant to link a specific issue @Robert but not everything

I found it 🙂

Honestly it seems like the quicker, stopgap for my situation would be to quickly implement a new locking client for now - then work on getting rid of the need in the future. I guess I could also run it in unsafe mode, I didn't see that one before

Databricks has the S3 Commit service which serializes writes to Delta tables in a given workspace. It's not the same as the "dynamodb one" you might be referring to, which is the DynamoDB Log Store implementation which is a very different beast compared to just locking

I can't do databricks, they have no on premises solution

The big problem I have with the dynamodb log store is that it's an all or nothing proposition, a table has to move its log store to DynamoDB and it just lives there, rather than as JSON in object storage. That means all clients have to understand how to interact with the log store in dynamo

wait, is that how it is currently, or was that the proposed route?

hmm if i read this document right, it does put a line for each commit into dynamo, but the actuial files still reside in the object store, it just about not allwoing duplicate entries on a table, so the write to the db table will fail, if the commit has been made before. https://docs.google.com/document/d/1Gs4ZsTH19lMxth4BSdwlWjUNR-XhKHicDvBjd2RqNd8/edit#heading=h.xgcsc4h146hh

yeah for just one person and needing to get something out the door soon - this is the way

@Robert this is what I was thinking of with its warning, which honestly is not much different than our limitation, so it is perhaps just around write operations that the dynamodb log store comes into play. I cannot make heads or tails of the actual java behind it

My hope is anyhow, that S3 will eventually join the party and offer atomic writes.. all others do, and IIRC R2 will offer that on top of S3 🫰

Considering that this postgres lock would be it's own crate - any issues with me using some helper packages like thiserror?

we already have it 😉

ya the java land dynamodb store intercepts write and list calls, & therefore indeed heavy in that sense (but as robert linked, it does not move the log entirely to dynamo) (cc @shingo you may find this thread helpful)

Thanks, @Nick Karpov!

Does someone have some time to do a quick huddle that's familiar with the current dynamodb locking paradigm? I wanted to make sure I understood it correctly before attempting to implement a postgres lock

@John Darrington what timezone are you in? I have the day off but am floating around some today

I’m in the mountain timezone @rtyler , USA

aight, I'm PST and in the office for a couple hours here

I'm free now

Late to the party here, but:

The big problem I have with the dynamodb log store is that it's an all or nothing proposition, a table has to move its log store to DynamoDB and it just lives there, rather than as JSON in object storage. That means all clients have to understand how to interact with the log store in dynamo

AFAIK, the DDB client writes back the .json file to its usual place in S3 after the winning commit has been confirmed, and the reader doesn't talk to DDB at all. I don't know that code tho, so I'd have to double check to be 100% certain.

exactly correct @Ryan Johnson. everything is synced back to s3 and the json are in their usual location.