Robin Kraft

06/02/2023, 9:37 AM
hey guys, we are facing challenges with delta tables and upserts for delta tables with many columns (approx. 4000) in spark@k8s. essentially, after the executors spin up we have period of 15-20mins where no job/task gets scheduled/executed. debug log statements say following DEBUG BaseSessionStateBuilder$$anon$1: Resolving 'col1 to col1#14457. so it seems like delta maps internally the source and target table columns. I described the problem and settings in more details also here We use Spark 3.3.0 and delta 2.3.0 (Seems like it is related to the fix from delta 1.2.0 any ideas or suggestions what to do?