https://delta.io logo
r

Robin Kraft

06/02/2023, 9:37 AM
hey guys, we are facing challenges with delta tables and upserts for delta tables with many columns (approx. 4000) in spark@k8s. essentially, after the executors spin up we have period of 15-20mins where no job/task gets scheduled/executed. debug log statements say following DEBUG BaseSessionStateBuilder$$anon$1: Resolving 'col1 to col1#14457. so it seems like delta maps internally the source and target table columns. I described the problem and settings in more details also here https://github.com/delta-io/delta/issues/1806. We use Spark 3.3.0 and delta 2.3.0 (Seems like it is related to the fix from delta 1.2.0 https://github.com/delta-io/delta/commit/9a86ca79). any ideas or suggestions what to do?