Jiri Vinarek06/12/2023, 2:01 PM
In another streaming query, you can continuously read deduplicated data from this Delta table. This is possible because an insert-only merge only appends new data to the Delta table.The target table will not be a streaming table, as I understand. I can get around this by enabling change data feed and then consuming INSERT operations from the feed. Is this the proposed way? Or is there another way how to get deduplicated streaming table?
Gerhard Brueckl06/13/2023, 6:56 AM
Jiri Vinarek06/13/2023, 7:08 AM
to have a "streaming" delta table. I think I got this part wrong. As I understand, just appending to a delta table with any operation will work and I can then use the table as a source for structured streaming.
Gerhard Brueckl06/13/2023, 7:09 AM