Hello Delta community, what is best and recommended way to ingest data from multiple sources like many s3 into Delta using spark structured streaming. Spark.readStream......load()

"Best" is hard to pin down without knowing more about your needs / requirements. But assuming you mean the standard way to ingest data handled by multiple separate clusters into a single Delta Table sink in S3... then you might want to take a look at this blog. You can have each Spark cluster write to the same Delta Table in S3 as long as you have a service that handles maintaining atomicity of writes. There are standard DynamoDB implementations, delta docs here! Hope that helps 😁

So multiple dataFrames for each data stream load() And then some unionDF =df1.union(df2) And then unionDF.writeStream.format("delta").options().start() Is this looks good choice or just create seperate spark clusters per s3 and then write from each stream.