i'd love to use this for azure and move our bronze pipeline out of DLT. are anyone working on azure compability?

Hey guys. How can I read kafka topic with dynamic schema in value? I use this in batch process to read these cases: df_read = spark.read.json(df.rdd.map(lambda x :x.value),multiLine=True), but this doesn't work for streaming.

I've been looking at upgrading kafka-delta-ingest to the latest delta-rs and the refactoring to use the object-store crate is quite... a challenge

FYI I have added a temporary restriction onto the

main

branch to only allow merges by members of the

delta-rs-committers

team, so I could safely add @KyJah Keys as a collaborator on the repository so we can bring the

azure-writes

branch across the line and get that merged

I am trying out the kafka-delta-ingest and was suprised to find that the table has to exist before the utility will write to it. Is there any reason for this? Is there a quick fix I can make to the code to have it initialize the log if it is empty?

The Azure support from @KyJah Keys and @Thomas Vollmer (Microsoft) is about to be landed in

main

, after which I have some other refactors that need to happen: • Sourcing

arrow

and

parquet

from

deltalake

to make keeping up to date easier • Un-ignoring some tests • Upgrading dependencies as far as I can take them without refactoring After that I think we'll be in a good position to start talking about another "release" for whatever that may be worth

@Christian Williams @misha @Thomas Vollmer (Microsoft) comically enough, because I created this shared pull request of other people's commits, I cannot merge it after reviewing it. If any of y'all have a change to ✅ I would appreciate it. Otherwise I'll override that setting on the repo in an hour https://github.com/delta-io/kafka-delta-ingest/pull/136

https://github.com/apache/arrow-rs/pull/3979 has me very excited, I think we can remove some code once this lands 👯‍♂️

Question: does my "source" kafka topic always have to have the same schema? Or can I also configure kafka-delta-ingest to just take whatever JSON is in the topic and make that available in a structured way?

Hey guys. I have workflow every hour a day ingesting from kafka to Delta table bronze and another notebook to process to Silver. In the first job I use Trigger Once and the load lasts an average of 8 minutes, but for Silver I use Available Now and foreachBatch in this job for 45 minutes. The company needs the data as close to real time as possible. My question is how can I improve this workflow to reduce latency? I cannot use continuous because my sink is a delta table. Does anyone have any tips or advice for this case? I thought of DLT for Silver, does that make sense?

Hello. Has anyone used something else rather than dynamodb for state locking for this project? Regarding: https://github.com/delta-io/kafka-delta-ingest#writing-to-s3

Hi, does

kafka-delta-ingest

require messages to be in json format?

@KyJah Keys I have released #delta-rs v0.11.0 and merged that into our topic branch: https://github.com/delta-io/kafka-delta-ingest/pull/139 The tests are passing, but I see that some things were disabled in your last commit, is there anything more you have planned here or should I move forward?

@KyJah Keys do you think you'll have a chance to review ☝️ so we can merge today?

I'm doing a pretty major test refactor, this kind of spiraled out of control. 🤦

Part of what I'm learning as a result of this is that kafka-delta-ingest has an over-reliance on integration tests which makes development a much more onerous task than it probably should be. Anything that touches #delta-rs is linking a lot of crates at compile time. Each integration test is its own binary which must be linked, so if there are 400 crates that are needed to be linked, that's 400 crates linked for 10ish different integration tests.

👋 Hey y’all, doesn’t look like this channel is mostly quiet, but will try here first to get some feelers on some questions? 1. Has there been earnest intentions set for supporting

avro

and

protobuf

with

kafka-delta-ingest

? 2. What about a (possible) helm chart for deploying this on K8s?

https://github.com/delta-io/kafka-delta-ingest This project is currently highly experimental. Is this still relevant. When can we expect a stable release?

Hi, here's a question about the Avro support. How is it managing the fact that there are two avro schema per topic: one for the key and another for the kafka message value?