jacktby jacktby

08/10/2023, 4:01 PM
for writeAllChanges, if there are only match clauses, you choose "rightOuter" , I think you should use "inner join",but why not?
Copy code
val joinType = if (shouldOptimizeMatchedOnlyMerge(spark)) {
    } else {
👀 1

Nick Karpov

08/10/2023, 6:14 PM
merge is operating at the file level... if you do an inner join here, you'll effectively delete all the rows that didn't match, even though they should persist aka. we rewrite the rows that evaluate to a no-op