https://delta.io logo
s

Sukumar Nataraj

05/12/2023, 5:04 AM
Team, In production, when am doing the merge for a one delta table, kind of below log keeps printing and merge never ends. Can some one help me with this issue?. Delta 2.0.2 and Spark 3.2.1 , EMR 6.7.0, Can someone help me to debug this one. this something burning in production. Appreciate your help.
Copy code
23/05/12 05:01:16 INFO DataSourceStrategy: Pruning directories with: isnotnull(site_id#36183L),(site_id#36183L = 1007443),dynamicpruning#142232 [site_id#36183L]
23/05/12 05:01:16 INFO FileSourceStrategy: Pushed Filters: IsNull(deleted),IsNotNull(id)
23/05/12 05:01:16 INFO FileSourceStrategy: Post-Scan Filters: isnull(deleted#36235),isnotnull(id#36182L)
23/05/12 05:01:16 INFO FileSourceStrategy: Output Data Schema: struct<id: bigint, code: string, deleted: string ... 1 more fields>
23/05/12 05:01:16 INFO DataSourceStrategy: Pruning directories with: isnotnull(site_id#36173L),dynamicpruning#142231 [site_id#36173L]
23/05/12 05:01:16 INFO FileSourceStrategy: Pushed Filters: IsNotNull(id)
23/05/12 05:01:16 INFO FileSourceStrategy: Post-Scan Filters: isnotnull(id#36172L)
23/05/12 05:01:16 INFO FileSourceStrategy: Output Data Schema: struct<id: bigint>
2 Views