Anyone has experience of persisting data from SQS to Delta i

Question

Anyone has experience of persisting data from SQS to Delta in S3 I am considering SQS gt Kinesis gt Spark Streaming or use SQS gt Lambda + Python delta rs Which one do you think is more appropriate We just need append to the table and I am not sure if delta rs can handle concurrent write to a table out of the box

rtyler · Answer

SQS gt Lambda works quite well so long as you don t have strict ordering requirements that is fine

Li Sun · Answer

Thanks it simplified lots of steps I will do a POC on that

Matthew Powers · Answer

You might find this blog from < Nick Karpov> helpful

Li Sun · Answer

Thanks I am following this article and found that the deltalake python package jumped from 56MB to 100+MB from version 0 8 to 0 9 so I guess there is no way to use 0 9 if I want to use the lambda layer solution

Kees Duvekot · Answer

I found streaming from Kinesis not really handy Because it can only run continuously last time I tried

Kees Duvekot · Answer

We had AWS DMS as a source But that can now use Kafka as a target So we are now working on that part

Nick Karpov · Answer

< Li Sun> oof that s crappy I ll take a look I think it may be under size limit if you do the container approach for lambda though if you want to try that for now

Li Sun · Answer

Yeah thanks I will try the container approach

rtyler · Answer

yeah that ballooning in size is interesting to me as well FWIW writing Rust lambdas is pretty simple too I actually have code which I have open sourced which is exactly what you re looking for laughing

rtyler · Answer

The only missing component which I have not made a decision on how I want to handle is whether to have the Lambda do schema inference from the table and validate messages coming from SQS based off of that

Li Sun · Answer

I never used Rust before but I am interested to see your solution glitch crab Please share the repo once you finish