Vignesh Chandramohan

05/01/2023, 4:34 PM
Hi, Opened this issue for a spec question. #837 added multi-part checkpoint. Specification says that the checkpoint file can be split into multiple files. Are there any restrictions on how the actions must be split into different files? As an example, say we know that the only actions since last checkpoint have been add file actions. Then the additional datapoint in the new checkpoint is just the newly added files since last checkpoint. With multi-part checkpoint, can we copy the previous checkpoint as part 0 and add newly added files alone as part 1 of the checkpoint? If (1) is valid, are there ways to optimize this case, either now or in the roadmap? For example, a way to refer to the previous checkpoint file.