Could you please suggest a solution? We have a 20GB customer persona in the delta table, and our i*n-house application needs to retrieve customer persona data through an API call*. We require the results to be obtained <*1 second*. Is it possible to create an API on top of the Delta Table for this purpose?

Wouldn't delta sharing be the protocol you want? https://www.databricks.com/product/delta-sharing

thanks f_or the reply. Can we use Delta sharing in_ On Premise

Though this is a reference implementation, you can deploy the server on-premise: https://github.com/delta-io/delta-sharing

isn't there someone working on a Rust delta sharing server too?

haha, that's me 🙂 https://github.com/kotosiro/sharing

😄

❤️🙂

@shingo Could we set up a Rust delta sharing server in an on-premises environment and read on-premises HDFS paths? Thanks for your answer! 😍

@陳冠穎 Thank you for your mention and suggestion! Unfortunately, the current implementation does not support on-premise object stores since the Delta Sharing protocol relies on the secure pre-signed short-lived URLs for the data sharing, which is not available in HDFS (according to my knowledge). You can implement your own pre-signed short-lived URL generator module for on-premise (HDFS-based) data sharing but I believe there are some additional considerations and it will be another crate apart from the delta-sharing-rs.