Samuel LaFell
08/31/2023, 12:19 AM<http://www.baseball-reference.com/teams/ARI/2023-schedule-scores.shtml>
23/08/28 00:01:12 WARN MergeIntoCommand: Merge source has SQLMetric(id: 47838, name: Some(number of source rows), value: 163) rows in initial scan but SQLMetric(id: 47839, name: Some(number of source rows (during repeated scan)), value: 0) rows in second scan
<http://www.baseball-reference.com/teams/ATL/2023-schedule-scores.shtml>
23/08/28 00:01:18 WARN MergeIntoCommand: Merge source has SQLMetric(id: 48322, name: Some(number of source rows), value: 163) rows in initial scan but SQLMetric(id: 48323, name: Some(number of source rows (during repeated scan)), value: 0) rows in second scan
<http://www.baseball-reference.com/teams/BAL/2023-schedule-scores.shtml>
23/08/28 00:01:24 WARN MergeIntoCommand: Merge source has SQLMetric(id: 48806, name: Some(number of source rows), value: 163) rows in initial scan but SQLMetric(id: 48807, name: Some(number of source rows (during repeated scan)), value: 0) rows in second scan
Hey all, pulling in data to update a delta lake table.
The code that updates:
# Join on game date and team
deltaExisting.alias("existingTable").merge(
deltaNew.alias("newTable"),
"newTable.Date = existingTable.Date AND <http://newTable.Tm|newTable.Tm> = <http://existingTable.Tm|existingTable.Tm>",
).whenNotMatchedInsertAll().execute()
Trying to understand what these warnings are coming from, can’t quite figure out from docs/github how to fix this or silence it?