Hi All, recently I've been asked if we can use any database cd tool when deploying our Delta Tables (and also views) to the Databricks workspace. So far wherever I worked with Spark/Hive we didn't defined DDL for our Tables - it was always derived from the set of transformations and finally saved with .saveAsTable(). Now we've got set of Delta Tables, many views that stays on top of them (with relations between the views) and we wanted to automate our deployment. But due to the fact that part of the tables is defined with pyspark transformations and views that depend on these tables are just SQL, I don't know how to connect together these areas and use let's say Flyway. Any ideas? 🙂

DBT ?

@raphaelauv thanks for the tip, any specific service/tool from DBT you have in mind? I've given it a quick look, I don't see anything that automates database migration there :(