Sukumar Nataraj01/18/2023, 6:50 AM
but still no luck. are we missing something, any idea on this?. Appreciating your help.
ALTER TABLE dbx.tab1 SET TBLPROPERTIES ('delta.targetFileSize' = '104857600');
Omkar01/18/2023, 10:03 AM
is available in Databricks Delta and not in Delta Lake OSS right now. For full list of table properties which are honoured in Delta Lake OSS, refer to this list: https://docs.delta.io/latest/table-properties.html Databricks Delta seems to have some more supported table properties, that list is here: https://docs.databricks.com/delta/table-properties.html
Nick Karpov01/19/2023, 8:17 PM
Unfortunately, the table propertythis is not correct, this configuration is there and respected, you can see further down in the code path execution that the bin packing code uses the targerfilesize to determine the sizes to bin smaller files into https://github.com/delta-io/delta/blob/master/core/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala#L248is available in Databricks Delta and not in Delta Lake OSS right now.
Omkar01/23/2023, 11:31 AM
) that you've mentioned which is passed from here and initialized in the Delta SQL Config here (not
). Please let me know if I'm missing something here, thanks. @Sukumar Nataraj you can try to validate if the properties are set properly on your table using the
sql on your respective Delta table. Additionally, there's this Github issue similar to the issue you're facing which may also help: https://github.com/delta-io/delta/issues/1139
Nick Karpov01/23/2023, 10:02 PM