https://delta.io logo
m

Martin

01/26/2023, 1:55 PM
Changing physical LOCATION of a Delta Table (without breaking metastore ref) Hi everyone, Is there a simple way to move the physical location of a Delta Table that is registered in Hive metastore? I know that the Delta Table itself is "portable": I can move the Delta Lake table folder using operating system capabilities. However, this breaks the Hive metastore reference. There is a
ALTER TABLE x SET LOCATION
statement (https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-alter-table.html#set-location-and-set-file-format). But if I run this after I changed the physical location, it fails since at this point the table in Hive metastore is already corrupt since there is no Delta Table to be found any more are the stored location. I can work around this issue by creating a dummy delta table (without registering it in Hive metastore) at the original location after I moved the Delta Lake table folder to its destination, run the the
ALTER TABLE x SET LOCATION
statement and then simply delete the dummy delta table at the original location. Another way for moving unmanaged Tables would be to move Delta Lake table folder using OS, run
DROP TABLE x
and then
CREATE TABLE x USING DELTA LOCATION
(https://docs.delta.io/latest/delta-batch.html#control-data-location). Downside of this approach is that we lose table properties and metadata. Both approaches feel very clumsy to my. In my point of view, this should be a build-in feature. Am I missing something?
g

Gerhard Brueckl

01/26/2023, 2:00 PM
table properties etc. are stored as part of the delta table so they are not lost when you drop/create the table in your metastore
👍 2
6 Views