Update: I’ve merged the PR to add in the new initial tables. Now I’m working on testing the release mechanism and establishing the tests in delta-rs repo. Once that work is done, we should be clear to have other connectors start using the test cases and work on adding more test cases. https://github.com/delta-io/delta-rs/pull/909

@Will Jones - how do you recommend I integrate the DAT reference tables into deltadask? I just manually copied them over in this directory for now. I think it’s good for the DAT reference tables to get checked into deltadask source control so Dask developers don’t need to think about DAT. You think I should just run the download / unpack instructions here and check in the tables to source control? cc: @Jim Hibbard as this will probably help you with the Ray connector tests 🤓

The first DAT meeting of the year is starting in a half hour. Let me know if you’re not on the invite and would like to join. We’ll talk about integrating DAT into delta-rs / deltadask, writer tests, and overall next steps for the project. Looking forward to chatting with everyone soon!

It feels like DAT has grown up and is now ready to graduate from delta-incubator/dat to delta-io/dat. What does the community think about this suggestion?

Looking forward to chatting with everyone at the DAT meeting later today. @Will Jones - if you’re up for it, would you mind driving the meeting, giving a walk through of the code, and helping to guide us on how to add the reference table for the writer integration test? Perhaps just walking us through the recommended contributor experience for other developers to add reference tables would help. Looking forward to chatting with everyone!

Good chat today, summary of next steps: • I will add a PR to generate a writer reference table • We will dogfood DAT in deltaray, added an issue here • I will take a look at all the issues and make sure they’re all still relevant. Will add new ones for next steps. • Once the issue list is clean, we’ll try to recruit some new contributors.

Meeting agenda for today: • deltalake 0.7.0 blog post is almost published. Facing a weird bug, but will hopefully be resolved soon. • PyO3 blog post next steps • Update on integrating DAT in deltaray • We’re working on a Delta Lake + Ray blog post • Update on Delta Lake + AWS Lambda. Next steps for Layer and blog post • Update on Will’s progress on integrating ADBC

Also just noticed Will’s issue on how to document how to write from DataFusion. Do we already have read / write support? If so, that seems like really low hanging fruit from a content perspective.

Just cloned DAT on a new machine and it was smooth sailing with three commdands: •

poetry install

•

make write-generated-tables

•

poetry run pytest tests

@Will Jones - I am dogfooding DAT in the newly created levi repo as well: https://github.com/MrPowers/levi It’s a different type of dogfood cause it’s not actually for building a connector. It’s just to easily access Delta tables without building them again.

The presence of these tables confused another contributor, so we might want to consider renaming the tables / giving clearer instructions on how they should be included in other projects.

I've noticed that the structure of the

dat

download has changed from

v0.0.1

to

v0.0.2

, was this intentional? The

table_content.parquet

is missing under the

all_primitive_types

test tables for example.

@Jim Hibbard - you can just read the

table_content

directory now. It’s not a single file anymore.

I see, so the README.md is for

v1.0.0

of dat tests and the documentation is just lagging a bit. Thanks Matt!

@Jim Hibbard - feel free to submit a PR with a fix if any documentation is lagging!

I submitted one PR and am already working on another 😛

So don't worry, you won't wait long for those PR's 🙌

I added unit tests for the DAT reader reference tables to deltadask: https://github.com/MrPowers/deltadask/blob/main/tests/test_public_interface.py The partitioned tables were causing me some problems.

Looks like some strange partition values are getting written. I was going to put this in an issue but looks like GitHub is broken right now.

Here’s my proposal: • update basic_partitioned to not include any null partition values • update multi_partitioned to not include any null partition values • Add basic_partitioned_with_null (which will be the existing basic_partitioned just renamed) • Add multi_partitioned_with_null (just the existing multi_partitioned renamed)

@Will Jones - can you take a look at this PR when you have a sec? This build stopped working. Looks like other poetry users are facing this issue as well and are upgrading the poetry version as a work-around: https://stackoverflow.com/questions/76175361/firebase-authentication-httpresponse-object-has-no-attribute-strict-status

This is becoming a fun little yak shaving expedition: https://about.codecov.io/blog/message-regarding-the-pypi-package/

@Will Jones - had a fun DAT experience when upgrading deltadask to properly incorporate file skipping. The “no_stats” acceptance test started failing. So handy to 1. know that this bug exists, and 2. get this acceptance test for free without having to figure out the bug in a production setting the hard way. I’m getting a lot of value from DAT while writing deltadask.

I unsuccessfully tried to suppress the CRC files in this PR: https://github.com/delta-incubator/dat/pull/39 I actually think this is important. I tried a bunch of different things. You know how to do this @Nick Karpov?

this should work

val conf = spark.sparkContext.hadoopConfiguration
val fs = org.apache.hadoop.fs.FileSystem.get(conf)
fs.setWriteChecksum(false)