Hi all. Any thoughts, or recommendations on how to stream data from

delta-rs

using datafusion? This discussion thread is asking exactly this. See https://github.com/apache/arrow-datafusion/discussions/6760 and would be appreciated if anyone here who has got some pointers to help chime in

I'm not too familiar with the details, but one of the driving use cases for creating delta-rs was streaming data from kafka into delta tables. Perhaps it would be instructive to check out that repo? https://github.com/delta-io/kafka-delta-ingest/blob/main/src/writer.rs

I haven't used it but Polars supports streaming and can also read delta but not sure if both are compatible.