ritwik singh05/31/2023, 3:11 AM
Divyansh Jain05/31/2023, 5:40 AM
Suraj Malthumkar05/31/2023, 8:29 AM
Robin Moffatt05/31/2023, 12:03 PM
Alessandro Biagi05/31/2023, 4:04 PM
operation. What does it happen if during a merge operation the underlying data is changing? Can this lead to data inconsistencies? This would explain some issues that we're facing..
Anton Abilov05/31/2023, 4:24 PM
dynamically? I would like for my Spark streaming job to process large batches (i.e. 50GB) when there is a lot of data to backfill, however when it’s caught up it should process smaller batches.
Alessandro Biagi05/31/2023, 4:39 PM
Iván Ramírez05/31/2023, 6:53 PM
Kiran Kondamadugula05/31/2023, 7:35 PM
Kenny Ma06/01/2023, 12:17 AM
sharath06/01/2023, 8:20 AM
Iris Shainsky06/01/2023, 9:13 AM
Dominic Copeland06/01/2023, 9:53 AM
Gal Stainfeld06/01/2023, 11:40 AM
. The above method worked for me locally after i z-ordered a table, but i was trying to use it on a table i know for sure was z-ordered in the past (half a year ago) and it returned empty results. Is it possible for a table to just lose it’s z-ordered when the only operations made on it are add and update ones ? Thanks
Bhupendra Singh Chauhan(TECH-BLR)06/01/2023, 12:23 PM
Bhupendra Singh Chauhan(TECH-BLR)06/01/2023, 3:01 PM
Sujith Kumar.S06/02/2023, 6:50 AM
Robin Kraft06/02/2023, 9:37 AM
Divyansh Jain06/02/2023, 1:21 PM
Bhupendra Singh Chauhan(TECH-BLR)06/02/2023, 2:32 PM
Alber Tadrous06/05/2023, 3:34 PM
Mohanraj M06/05/2023, 4:13 PM
Cam Gallucci06/05/2023, 5:51 PM
function. 2. An update to record x occurs in version 6 3. Another update to record x occurs in version 7 4. I load the data from version 5 to 7 to get the most recent updates with
Will step 4 give me one combined update for record x (with the
delta_sharing.load_table_changes_as_spark(table_url, starting_version=5, ending_version=7)
from version 5), or will it have both updates (two
, one from version 5 and one from version 6)? And then say in step 3 record x was deleted instead of updated, would only the delete come? Appreciate any clarification on this, thanks!
Christina06/05/2023, 11:54 PM
creates external tables. Is there a way to create a managed delta table with an existing dataset?
CREATE TABLE USING DELTA LOCATION …
Ajex06/06/2023, 4:13 AM
Please let me know if you have any suggestions or solutions to resolve this issue (execute 2 job at the same time without conflict).
"File does not exist: /delta_path/part-00068-23256f70-a297-416f-8c5d-3650343929b3.c000.zstd.parquet"
Rahul Sharma06/06/2023, 6:42 AM
GapyNi06/06/2023, 12:24 PM
? Main goal would be to merge schemas while streaming (adding new columns, ...). When running this code in Notebook (12.2.x-scala2.12) it works as opposed to Job (10.4.x-scala2.12). Was it some change done? Thanks and regards, GapyNi
set spark.sql.streaming.stateStore.stateSchemaCheck to false
Dinesh N06/06/2023, 1:05 PM
Alber Tadrous06/06/2023, 8:08 PM
John Darrington06/06/2023, 8:09 PM