Is there a way to set

maxBytesPerTrigger

dynamically? I would like for my Spark streaming job to process large batches (i.e. 50GB) when there is a lot of data to backfill, however when it’s caught up it should process smaller batches.

Not a delta lake question. Doesn't available now address this? https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers

My bad, I guess it’s more related to Spark — but was hoping delta streaming might have a solution for this. From the docstring of availableNow:

availableNow : bool, optional
    if set to True, set a trigger that processes all available data in multiple batches then terminates the query. Only one trigger can be set.

This is a continuous query with new data arriving — we don’t want it to terminate

For streaming, the only thing I've seen is the delta live tables autoscaler or roll your own.