Hi all, I want to move a delta live table(a dlt pipeline loads this currently) from one schema to a different schema without doing a full reload. Also need to repoint the dlt pipeline to load incremental to table in new schema. Anyone had this situation before? Any thoughts on how to go about this?
08/09/2023, 4:52 PM
To move a table you should be able to just change the target schema of your pipeline and rerun.
You cannot however change a stream to read from a different source.
For this you need the new create flow API (going into private preview). It will let you create a new stream that writes to an existing table.
08/09/2023, 8:03 PM
#1 would need full refresh. I need to avoid that also want the new table to hold complete history and incremental going forward
08/09/2023, 8:08 PM
You do not need to full refresh to change the target of a pipeline.
In both HMS and UC the entries in the metastore are just pointers to the actual tables.