Hello, I am using batch processing to load previous day data to write to Delta table which is partitioned by Folder Date and using Overwrite mode. Now my requirement is to run this Job every hour, what would be the best way to do it ? Note: I should have re-run option for a particular date

You can try Airflow to schedule your job

Scheduling is not a problem I can do the same thing with Databricks workflow jobs but it’s about writing to the delta table with overwrite mode