Hi everyone I have a question regarding the use of spark to

Question

Hi everyone I have a question regarding the use of spark to write data in deltalake Are there any alternatives apart from delta rs

Roberto · Answer

My use case is very simple I only need to read data from a Kafka topic and write the output directly to deltalake maybe some little transformations For that using Spark Structured streaming looks like an overkill

Matthew Powers · Answer

Checkout kafka delta ingest <https github com delta io kafka delta ingest>

Roberto · Answer

Hi Matthew looks great but I m not sure if this project is ready to use in production is it Anyway I would like to perform some transformations over the data

Matthew Powers · Answer

It s used in production <https www youtube com watch v=do4jsxeKfd4 amp ab channel=Databricks>

Roberto · Answer

Perfect I ll take a look at it Thank you

Roberto · Answer

Apart from that There is no alternative right