Hi, Any one here has success with using open source delta lake and partition pruning. I am using version

delta-core_2.12

version

2.1.0

and the partition pruning doesnt happen. Here is my merge statement.

<http://deltaTable.as|deltaTable.as>("oldData")
        .merge(
            finalDF.where("OP != 'd'")
                    .as("newData"),
            condition = "oldData.ID = newData.ID and oldData.created_date in ('null','2022-12','2023-01')")
        .whenMatched
        .updateAll()
        .whenNotMatched
        .insertAll()
        .execute()

In the physical plan I see no indication of partition filter being used on partition field

created_date

instead it ends up in the SortMergeJoin

+- SortMergeJoin [ID#43253], [ID#43121], FullOuter, created_date#43136 IN (null,2022-12,2023-01)

curious to know the answer

I've found myself in the same situation while using the MERGE api.