in the README table there's a line for "Write transactions" and "High-level file writer", what's the difference? from the API docs it seems like the Python bindings support writing to Delta Lake tables but not sure if there's something i'm missing
w
Will Jones
06/15/2023, 4:24 PM
high-level file writer handles writing data files, making the action list, and then writing the transaction.
Write transactions just means committing the new actions to the log. It's for when you've already written the data files in some other process.
j
Jeremy Jordan
06/15/2023, 4:26 PM
thanks for the clarification! do you have an example of a typical scenario where another process would have already written the data files?
w
Will Jones
06/15/2023, 6:54 PM
If you are building your own high-level file writer basically. Maybe the writes happen in a distributed system and you collect the actions to a single node and call the write transaction API. You can just think of it as a low-level API for building new connectors.