https://delta.io logo
e

Edmondo Porcu

05/12/2023, 3:26 PM
Hello, how do you use Spark Streaming with Delta Lake, the pattern “forEachBatch”, when the table does not exists yet?
c

Christopher Grant

05/12/2023, 5:37 PM
Depends on how you're writing to the table, some of the time it's created for you. But if you're using
MERGE
, for example, the table is not created automatically so you'd have to do it yourself. For things like this, I typically a DeltaTable.createIfNotExists() which will no-op when the table already exists. If you go this way, I suggest using addColumns; with it you can easily take the schema of an existing DataFrame and inject it into the builder.
a

Amit Panwar

05/12/2023, 7:54 PM
using delta lake connector will right idea, u can pull data directly in delta format
64 Views