Hi. I have a delta table which automatically generates symlink format manifests via

delta.compatibility.symlinkFormatManifest.enabled=true.

Occasionally I am finding that I receive errors if the write that triggers the manifest generation results in a checkpoint file being created and pre-checkpoint transaction log files being deleted. If I run the job again it will work just fine. Is the best way to avoid this to disable the automatic generation and instead do it manually after the data write with

DeltaTable.forPath(...).generate()

? Or am I overlooking something?

org.apache.spark.sql.delta.DeltaRuntimeException: Committing to the Delta table version 210 succeeded but error while executing post-commit hook Generate Symlink Format Manifest: Generate Symlink Format Manifest

Encountered error while reading file <s3://my-bucket/my-table/_delta_log/00000000000000000201.json>

Caused by: java.io.FileNotFoundException: File not present on S3