https://delta.io logo
r

rtyler

04/28/2023, 3:42 PM
šŸŒ® šŸŒ® to @Florian Valeye for this very interesting and promising pull request: https://github.com/delta-io/delta-rs/pull/1315
šŸš€ 2
šŸŽ‰ 1
@Florian Valeye how close to completion do you anticipate this work? I think it would be really cool to release the python bindings with this after rust 0.10.0 comes in with arrow 37 and datafusion 23
f

Florian Valeye

04/28/2023, 4:11 PM
Hey @rtyler šŸ‘‹, It's already working on my laptop (good sentence when there is an incident in production), I built and loaded the DeltaTableAsync asynchronously using Python. So it will depend on the reviews (and I will also add documentation on how to use the
DeltaTableAsync
in an async context). I limited the read operations before attacking the writer (except
update_incremental
), which will be created in a future PR. We could also improve the builder and other writing operations to refactor the codebase and avoid duplication. One last point, I was forced to create its own struct because of
pyo3-asyncio
implementation, but I was wondering if a Rust expert could help me to do a kind of reflection wrapper to call the
RawDeltaTable
by default if the method is not found in
RawDeltaTableAsync
(but it's a tradeoff between code readability and Rust accessibility. My goal is to provide full asynchronous capabilities in the Python binding to be aligned with the Tokyo async features in Rust. Also, I'm thinking of a lot of use cases where the async method would be nice (for instance a web application using FastAPI for example). Besides, we could also decide to split the build and the load phase when creating the synchronous
DeltaTable
as well. FYI @Denny Lee and @Matthew Powers it could be a nice blog post "Async Python binding available"
m

Matthew Powers

04/28/2023, 5:01 PM
Yea, will definitely be happy to blog about this!!
šŸ™ 3
d

Denny Lee

04/28/2023, 9:10 PM
YAYYYYY!!!!
j

Jim Hibbard

04/28/2023, 10:31 PM
This is awesome! Great work šŸ˜„
šŸ™ 1
f

Florian Valeye

05/05/2023, 5:29 PM
Here is an improved version by making the integration of asynchronous more user-friendly by suffixing the methods with
_async
šŸ˜„ I kept the capability of lazily loading the DeltaTable (
lazy val
of Scala, I miss this feature) It's ready for reviews šŸš€ cc @Will Jones @rtyler
šŸ‘ 2
w

Will Jones

05/05/2023, 5:32 PM
Iā€™d like to release 0.10.0 today, but happy to release another version soon once this is ready.
šŸ‘ 2
f

Florian Valeye

05/05/2023, 5:32 PM
Sure šŸ‘