Hello team, I have 1 question. I am using Apache Spark Streaming to run Merge Query to Delta Lake table. My table turn on optimize performance. Meaning after run Merge Query complete, it is automatic run Optimize query. I want to turn of that to improve Streaming batch. How can I turn off this feature ?

just set the table property accordingly https://docs.databricks.com/delta/tune-file-size.html#auto-compact

I don't know why each merge statement completely, optimize operation automatically run. How to turn off that? I did not configure any special

yes, as I said, you can configure this by setting the table properties mentioned in the link above

base on your recommendation, I think delta.autoCompact configure will suitable, right ? I just want to turn of optimize statement

yes, setting

detal.autoCompact

to

False

in your table properties should do the trick

This is interesting. We don't use

MERGE

, we do

streaming_df.writeStream()

. Target delta table has both autoCompact and optimizeWrite enabled. We see that it runs OPTIMIZE only once in a while, which makes sense as it's waiting (I think) for enough small files to be there before optimizing.

yes for every complex business logic. Just resolve by sql spark. So I use merge statement

no, autoCompact does not support ZOrdering

that is our setting

can you try

delta.autoCompact

insterad of

delta.autoOptimize.autoCompact

I can not setup configure

i think you need to write the properties with quotes

IMG_8444.jpg

it should be `ALTER TABLE delta.

mytable_location

SET TBLPROPERTIES ('my property' = 'abc')`

Tks so much, I have change configure success. And monitor tmr