@Robert I saw your comments on @John Darrington’s pull request. Would you and @Will Jones be up for me moving dynamodb_lock into a repo in delta-incubator? I think that crate needs to come out especially if we're to use a LockingObjectStore implementation

Is the idea: • dynamodb_lock ->

delta-incubator/dynamodb-lock

•

pub trait LockingObjectStore

lives in delta-rs • dynamodb_lock is used to implement

LockingObjectStore

for S3 in delta-rs ?

There's a bunch of ways that I think we can approach this actually. I think my preference would be to create a repo locking_object_store in incubator and have that depend on dynamodb_lock releases. The only reason dynamodb_lock lives inside the delta-rs repo is because we had no where else to sensibly put it at the time

That seems reasonable too. I feel like there is a lot of refactoring to do here and I haven't wrapped my head around it all yet.

I dropped a quick sub-tree cut out of the crate into an incubator repo https://github.com/delta-incubator/dynamodb-lock-rs

So in the end we would end up with a repo that contains ``LockingObjectStore``, which exposes the

LocakCLient

trait? Then the dynamodb-lock-rs repo would implement that for dynamo and there would be a new repo for the ongoing posgres client?

@Robert that was my intention, I was going to drop a couple other lambdas I have written in there when I found the time 🙂

makes sense to me 👍. Was just wondering, since we created the dynamodb-lock repo. WHat would be contained in that, or is that just temporary?

Since dynamodb-lock-rs is basically some standalone crate which implements a simple DynamoDB lock that others may be dependent on, I wanted to drop that elsewhere so that maybe others would contribute to it, or at the very least we can set it off to the side and forget about it 😛

makes sense 😆

So what should be my course of action at this point regarding where the code lives and where I should put the Postgres stuff? @rtyler @Robert

My suggestion would be to do it in the delta-rs repo for now, as there will be changes to the core codebase and its probably easier for development until thigs are sorted out. Depending on you like to work, we could also build it in a separate repo, and integrate with delta-rs once we have a somewhat clean state. Publishing whatever new crates exist then would then be a pre-requisite though. Also, I cannot create repos in delta-incubator, so @rtyler would have to help out there. In short, I'm fine either way...

Ok I think it makes the most sense to pull out into the incubator or as a separate crate as well.

absolutely. IIUC, we landed on having the whole

LockedObjectStore

in that crate, so a name along those lines would be great 🙂 . The first implementation would then be the postgres backed one... As this would also pull in object_store, which by now exposes AWS credentials, I hope to get that rusoto-free some time ...

Oh so you don’t want me to pull the existing dynamodb stuff over wholesale?

well - if i understood @rtyler correctly, he wants to publish dynamo db lock as a standlalone crate. So the traits etc defined over there would live in the new crate, but the actual implementation of the dynamo lock i guess not...

Ah

Personally I believe for object store, we can implement the dynamo db lock without any sdks, which would be great in terms of depoendencies, and also compatibility with how the S3 object store itself is configured.

Ok. My goal is to get the Postgres backed lock up this week so I can start experimenting with delta on our data warehouse. I’m just glad works paying for my time on this :D

We are happy to help wherever we can as well - I think many users may appreciate an alternative to DynamoDB, as we see many using non AWS S3 stores...

Yeah we’re using Minio here

Nice! Do you have a link, so I can browse a bit once I find some time? 🙂

https://www.linkedin.com/in/johnwdarrington/ https://github.com/idaholab/Deep-Lynx/

Thanks!

Yep. I’m very excited for delta tables. No one wants to manage a petabyte sized Postgres cluster on prem lol

indeed not. 😄

Good to know.

^^ same reason that got me started 😄

And this works with data that’s been ingested by data bricks?

as usual, it depends ... databricks already supports more features. e.g. deletion vectors. SO if a table is written with unsupported features we cannot (reliably) read it. That said, there is some stuff coming up, that will make following new feature releases along much easier!

Cool

concurrent writes from databricks and delta-rs are also not possible rigth now, with S3, as a shared locking machanism between both environments is required.

That’s fine. I’m looking at leveraging delta live tables for ingestion but on prem storage for long term keeping

if you are interested in delta sharing, @shingo has a working rust implementation...

Awesome!

@Robert Thank you for the call-out to my project here! @John Darrington I would be happy to collaborate with you!

Cool. I’m a ways off of sharing but I’ll let you know.

maybe I'm just blind - but where is the incubator repo so I can fork it?

@Robert I'll bring you into the incubator now

ok so just to clarify we're pulling out dynamodb-lock into it's own repo on incubator then basically not touching it. Then I'll be writing

locked-object-store

(or should we name it

distributed-lock

?) as it's own thing with Postgres being the first backer which will also live in the incubator

@John Darrington what's your github ID? I can just create a locked-object-store repo right now while I'm here

DnOberon

added to https://github.com/delta-incubator/locked-object-store, @Robert you and @Will Jones also have invites in your inbox to join the incubator org, your absence was an oversight

I'm in, thanks