can we copy delta table using distcp to a different cluster ? or there is any other better way of copying the complete delta table ?

Delta Sharing is a great way. Are you on a physical hadoop cluster?

Yeah JosephK, ours is a physical cluster on bare metal. We are able to copy the whole delta table using distcp , but that's relatively slower due to multi-directory structure of delta lake I am planning for small file compaction and do the copy again. (Delta Lake Small File Compaction with OPTIMIZE) Do you suggest any other way ?

Delta sharing should work best then

any documents on delta sharing ? Let me go through that.

👍

I don’t know. You can ask on the #delta-sharing channel for more details. I thankfully moved away from on prem stuff 8 years ago and haven’t looked back.

thanks !! I know the pain of on-prem , lot of blockers for creativity