@Andy Grove The datafusion 21.0 release has been prepared, but it doesn't appear on crates.io, was there a failure on the release or is the release still in process?

@Will Jones I cannot thank you enough for figuring out arrow's

take()

with

MapArray

that unblocked yesterday's hacking to where I now have a very good Lambda streaming data from SQS into Delta tables in production 🥳 (production builds off

master

branches yolo 🙂 )

I ended up writing and testing an optimize lambda with some tables in S3 tonight: https://github.com/buoyant-data/lambda-delta-optimize

With https://github.com/delta-io/delta-rs/pull/1249 we're going to have to finally address the deprecation of arrow:jsonreader:Decoder upstream. I have been working with that code a bit more lately so I might try to tackle it, but I pushed this change to a topic branch so others can collaborate on it if desired. I think the migration to

RawDecoder

will help some of the checkpoint memory pressure that @Christian Williams and I have identified in production #kafka-delta-ingest usage

Our next scheduled release (tracked by @Ian Joiner) would be April 6th (a week from Thursday). I am not sure if the upstream Arrow changes which fix #1239 or #1226 will have been released by then. I think the Datafusion upgrade in #1249 will be ready however. I would like to plan to delay the release of 0.9 to incorporate those upstream arrow changes if possible @misha @Christian Williams @QP Hou @Florian Valeye @Will Jones @Robert

@Robert I am working with the

OptimizeBuilder

tonight, and I'm curious why

with_filters

takes

PartitionFilter<&'a, &'a str>

where T =

&'a str

rather than an allocated `String? Lifetimes are not my strength but I'm not sure how it would be possible to create

PartitionFilter

with anything other than static strings for the partition values. I have tried a couple different options with no success, and I think the

T

used for

PartitionFIlter

is just too restrictive 🤔

Hello, Team I think there is a bug in delta-rs while getting schema, we use the below json input from user and convert it to spark compatible schema with a custom function scenario:

schema": {
    "date": {
      "data_type": "string",
      "partition_column": true,
      "nullable": false
    },
    "shift": {
      "data_type": "string",
      "partition_column": false,
      "nullable": false
   }
}

*the above is the schema of the table that we have created but when using deltaTable.sc*hema() it gives the below schema

schema": {
    "date": {
      "data_type": "string",
      "partition_column": true,
      "nullable": true
    },
    "shift": {
      "data_type": "string",
      "partition_column": false,
      "nullable": true
   }
}

nullable is returned true when we have set it to false we are using spark to create the delta table

This message was deleted.

Would someone be able to review my pull request to fix a documentation error? Really excited to potentially make my first contribution to delta-rs! Still need to brush up on my Rust for more, thanks!

@Will Jones @Robert @Florian Valeye will any of you be coming to San Francisco for Data and AI Summit this summer in June?

I wanted to share some updates on the datafusion upgrade: https://github.com/delta-io/delta-rs/pull/1249 Basically datafusion 21 incorporates a newer (not newest) version of arrow which has some bugs preventing our use of it. As such we need datafusion 22 to come out which upgrades to arrow 36 😦

I tried out the mdBook tool that’s written in Rust and was really impressed. It’s easy to use and generates SEO friendly URLs. mdBook is used to create the Polars User Guide which is beautiful. I think the amazing Polars User Guide is part of the reason that project is so successful. Here are our current docs: https://delta-io.github.io/delta-rs/python/index.html The URLs aren’t SEO friendly. Here’s an example: https://delta-io.github.io/delta-rs/python/usage.html#querying-delta-tables. Google ignores anchor tags for SEO. This would be a better url: https://delta-io.github.io/delta-rs/python/usage/querying-delta-tables. I’m also not sure how to update our docs currently. Would folks be open to migrating to mdBook? I am happy to do all the work. Migrating to a Rust solution also seems aligned with the delta-rs philosophy 🤓

In case you just saw a bunch of notifications in #delta-rs-notifications, I closed some dependabot pull requests which are basically addressed by https://github.com/delta-io/delta-rs/pull/1249

The How to create and append to Delta Lake tables with pandas blog post was released today. This highlights the Python deltalake 0.8.1 feature that lets pandas users easily overwrite a partition. Yet another advantage of Delta Lake over Parquet for the pandas community 🚀

Just spotted https://eurorust.eu/ which could be a fun event for you folks east of the atlantic to share some cool Delta rusty things 🙂

I am planning the next batch of blog posts. Let me know if you have any special requests. Happy to run with your ideas or coauthor posts for the delta.io/blog 🤓

Cross-posted on @rtyler request: Is there any way to write a field's metadata with Python delta-rs? Looking at the API reference for

write_deltalake()

, it seems like we can only specify schemas with pyarrow, which unfortunately doesn't preserve the metadata dictionary in the conversion to a

deltalake.schema

.

Would one of you committers be willing to be added to the #kafka-delta-ingest repository so I can request code reviews? I have been moving a few more changes through, and there's just not enough reviewers available 😞

@Yousry Mohamed I read through your blog post about doing writes with the deltalake crate. I'm curious how you feel about the ergonomics of using the

CreateBuilder

compared to some patterns we have floating around which use JSON for schema definition such as in kafka-delta-ingest

Hi! Sorry if this has already been asked by I couldn't find anything when I search the channel history and the documentation is not clear. We are planning to use this library - the python bindings - to write a meltano destination for Databricks. The problem is that some of our sources evolve the schema by adding new fields every now and then, and we don't know if this library supports mergeSchema.

@Will Jones and @Robert as far as I can tell you're the only two committers that have touched the pyspark integration tests. They're failing in

main

right now because we have some dependency ranges that are unspecified and were upgraded upstream. I tried to fix the object_store deprecation warning but I do not really understand why the pyspark tests would suddenly start failing or what dependency might have changed to cause that 😕

Working through the datafusion-upgrade branch, I'm still wrapping my head around this change and how we might change

files_matching_predicate

https://github.com/apache/arrow-datafusion/issues/4695

Can we cut a Rust release soon? Have upstream issues been taken care of? I’ll do the CHANGELOG as usual. I would love to have https://github.com/delta-io/delta-rs/pull/1273 merged first though.

For you arrow hackers, what's the deal with 37 being on crates.io but not tagged 😕 ❓

Looks like I'll have to figure out the datafusion upgrade in order to get the arrow upgrade in 😆 😿

Datafusion 22 has been pushed, so that's one less variable to wait for 🎉

Hi all, did you see this?

From today through April 16, we invite you to review the current draft of the updated Rust Trademark Policy and share any feedback you might have using this form. 

Your responses will help us ensure the Rust Trademark Policy is useful for all! 🦀  

<https://t.co/XMFKluoQVs|https://t.co/XMFKluoQVs>

Just got off the phone with @Michael Shtelma who is working on the torchdelta connector. I think a PyTorch connector is a wonderful idea! Suggested next steps (feel free to chime in with thoughts): • possibly rename the project to deltatorch consistent with deltaray & deltadask • switch to poetry consistent with the other Delta Lake Python ecosystem projects • Publish to PyPi • I can then make a Jupyter notebook & write a blog post. We can work on the story for why Delta Lake is great for PyTorch analyses. I am guessing that the file skipping and versioned data would be huge benefits for the PyTorch community. Anyone else interested in getting involved with this project?

@Matthew Powers or @Jim Hibbard I need to do some optimization testing, I know that @Denny Lee has had some bigger delta tables floating around for demo purposes of public data sets, do y'all have something I can copy into my own S3 or minio bucket that can be optimized?