Hi team,
Just I am trying to understand at which level Delta file format is operating...
Does Delta file format act like a server (a process keeps running and responds to requests) ?
Analogous to Hadoop file system, where the base file system is Unix File System and on top of it, HDFS operates where name node manages the HDFS files.
But if this is to be true, I don't see any server process described in Delta file format.
So, what is responsible for its features (ACID, upsert, delete, data governance etc..) ?
To add on, it is told that many other tools can interact with Delta file format.
For example: DBT (a SQL based transformation tool) can read/write data.
If this is the case, which process is responsible to provide the aforementioned features?
Also it is mentioned that Delta format supports only tables. If yes, is it a RDBMS product?
For HDFS, it very clear that it operates on top of host OS file system and different processes (name node, data'node etc) are available to interact with it.
Similarly I don't get any clarity about the Delta format.
Any help will be much appreciated.
Thanks