https://delta.io logo
s

Sujit Pattnaik

03/20/2023, 6:23 AM
can we copy delta table using distcp to a different cluster ? or there is any other better way of copying the complete delta table ?
j

JosephK (exDatabricks)

03/20/2023, 11:24 AM
Delta Sharing is a great way. Are you on a physical hadoop cluster?
s

Sujit Pattnaik

03/20/2023, 11:25 AM
Yeah JosephK, ours is a physical cluster on bare metal. We are able to copy the whole delta table using distcp , but that's relatively slower due to multi-directory structure of delta lake I am planning for small file compaction and do the copy again. (Delta Lake Small File Compaction with OPTIMIZE) Do you suggest any other way ?
j

JosephK (exDatabricks)

03/20/2023, 11:26 AM
Delta sharing should work best then
s

Sujit Pattnaik

03/20/2023, 11:29 AM
any documents on delta sharing ? Let me go through that.
j

JosephK (exDatabricks)

03/20/2023, 11:30 AM
s

Sujit Pattnaik

03/20/2023, 11:30 AM
đź‘Ť
is it only for cloud storages ? ours is plain hdfs files system!!
j

JosephK (exDatabricks)

03/20/2023, 11:39 AM
I don’t know. You can ask on the #delta-sharing channel for more details. I thankfully moved away from on prem stuff 8 years ago and haven’t looked back.
s

Sujit Pattnaik

03/20/2023, 11:41 AM
thanks !! I know the pain of on-prem , lot of blockers for creativity
5 Views