https://delta.io logo
s

shingo

07/25/2023, 5:37 AM
Copy code
# delta-rs: Rust meets Data Engineering

# Elevator Pitch
There are many fantastic open-source projects that support the "democratization of data and AI."
Depending on the scale of the data, appropriate tools are chosen, ranging from the ecosystem
centered around Spark running on clusters to pandas, which operates on laptops.
Now, what kind of ecosystem would be suitable for data of intermediate size, say several terabytes?
I would like to share with you the possibilities of a new ecosystem that utilizes Rust in data
engineering communities, drawing on my experience participating in the delta-rs community.

# Description

I donated the Rust implementation of the data sharing protocol called Delta Sharing Protocol to
the Delta Lake community. Since then, I have been actively involved with developers worldwide
in implementing the Delta Lake Transaction Protocol in Rust. Through my participation in the
delta-rs development community, I have gained insights and experiences on how Rust's memory
management at the compiler level and its characteristics as a native compiled language can be
leveraged in certain types of data engineering problems. During my sharing session, I will mainly
discuss the following topics:

 - Why Rust is gaining attention in the field of data engineering.
 - Introducing some awesome Rust projects for data engineering.
 - Rust's role as a core language for inter-language communication.

By delving into these areas, I hope to shed light on how Rust's unique features can benefit data
engineering and contribute to the development of innovative solutions in the data community.
2 Views