Hey Joydeep, will be great for the mack contributors to learn about your awesome Change Data Feed contributions!

Sure @Matthew Powers, do we port some of these functionalities to mack

Databeans just open source a tool called lighthouse to provide users with insights on when data maintenance tasks should be performed on their Delta tables. Check it out @Joydeep Banik Roy @Brayan Jules and @Nick Karpov: https://github.com/Databeans/lighthouse

@Abidi Gassen - welcome 👋

Really excited about your new project!

Thank you @Matthew Powers, indeed we’re super excited about the first release of #lighthouse project and looking forward to the community’s review and feedback. Looking forward to answer any questions about the project and discuss it’s future given the huge uncovered axis of cleaning up your lakehouse from a data layout perspective.

For those interested in a more profound dive into the whys and hows of the project, I strongly recommand reading thses two blog posts: https://databeans-blogs.medium.com/z-ordre-take-the-guesswork-out-bad0133d7895

@Abidi Gassen great project, I just checked it out. Have a few ideas. let's connect

Hi delta io guys Please check out my new blog on Delta Merge Optimization:- https://www.linkedin.com/posts/joydeep-banik-roy_delta-mergeoptimisation-strategies-a[…]867588489216-WDby?utm_source=share&utm_medium=member_desktop

Just made a fairly large quinn refactoring to make the Ruff lint checks happy: https://github.com/MrPowers/quinn/pull/104

Hey @Sem Sinchenko, glad to have you in this chat!

@Joydeep Banik Roy - this PR to add users more insights into the sizes of files in their Delta table is cool. I wanted to make you aware of another project called levi that I am working on that is providing similar functionality. Levi uses delta-rs, which exposes easier APIs for figuring out the list of files / file sizes in a Delta Table. Let me know what you think.

@shingo - Hey there. This channel is a smaller group of folks that are collaborating on different projects in the Delta open source ecosystem. Happy to have you in this channel 👋

👋

👋

Thanks for the invitation!

@Michael Shtelma - welcome! Everyone else - Micheal just donated deltatorch to the delta-incubator organization. This is a great way to read Delta tables to perform PyTorch analyses. We will add some issues with the “good first issue” label soon if you’d like to contribute.

Thanks a lot, @Matthew Powers

I just published a blog post on how to deploy a self-hosted Delta Sharing server on AWS. Please check it out, and any feedback would be welcome! https://medium.com/@ognis1205/how-to-deploy-self-hosted-delta-sharing-server-on-aws-7aa2a7f6f46

I have started writing a blog post series that explains the delta-rs Python bindings: https://medium.com/@ognis1205/delta-rs-as-a-python-rust-ffi-example-part-1-b11f53b40f9c

This is my second part of the blog post series that explains the delta-rs Python bindings: https://medium.com/@ognis1205/delta-rs-as-a-python-rust-ffi-example-part-2-f0cb386a04bd I hope you enjoy the read 🙂

This post may not be a very good blog entry. The writing may not be intuitive and easy to understand😅

I will edit the blog posts so that the articles are more concise and easy to read later.

Thanks for writing these blogs @shingo - they’re really useful.

Hey everyone! Does anyone know how to get the DynamoDB Log Store going within Databricks for writes to Delta? I tried doing something simple like

spark.conf.set("spark.delta.logStore.s3.impl", "io.delta.storage.S3DynamoDBLogStore")
spark.conf.set("spark.delta.DynamoDBLogStore.ddb.region", "us-west-2")
spark.conf.set("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", "ddb_test")

df = spark.createDataFrame(
    [
        (1, "foo"),
        (2, "bar"),
    ],
    ["id", "label"]
)

df.write.mode("append").format("delta").save(<S3 path>)

and didn't see a DynamoDB table created. Would appreciate any pointers, thanks!