Hi All Is it a good practice to have side effects in the del

Question

Hi All Is it a good practice to have side effects in the delta live table pipeline I would like to write data into Postgres DB as a last step

JosephK (exDatabricks) · Answer

Could you put the DLT pipeline into a workflow and then have a notebook in the workflow that does the write

vijay · Answer

This would be continuously running DLT Will I be able to integrate that with the regular workflow

JosephK (exDatabricks) · Answer

Hmm I think continuous won t work because it never finishes and wouldn t trigger the second part of the workflow I think in the python API you can have a write to jdbc as a sink

JosephK (exDatabricks) · Answer

I think the first bullet addresses ```save ``` <https docs databricks com workflows delta live tables delta live tables python ref html limitations>

vijay · Answer

Thank you

vijay · Answer

Here it says However you can include these functions outside of `table` or `view` function definitions because this code is run once during the graph initialization phase

vijay · Answer

It says it will run only once during the initiliazation phase I want to run this every time I process the stream data

vijay · Answer

Can I include the sink inside table definition Something like this

vijay · Answer

``` dlt table def table df = read stream data df save to db return df```

JosephK (exDatabricks) · Answer

I think it has to be outside but I m not 100 sure DLT is too new