I have an ETL process which adds files to a s3 bucket of parquet files.
I would like to create a delta table which is up to date with these files.
I do not want to use convert every time to the whole path (might take too long).
Ideally, I would like to update the delta table as new files are arriving.
However, the documentation states that:
After the table is converted, make sure all writes go through Delta Lake.
So, my question is, how do I add to delta new add parquet files which already are in the delta location?
01/10/2023, 7:14 AM
are you using Databricks? Autoloader might be your friend…