Hi everyone, I’m developing new open-source delta sharing server named deltastore. My main goals are near real-time query against the up-to-date regardless of the size of deltalogs, and access controls for data governance. Belows are key features.
• Caching DeltaLogs
◦ We support deltalogs caching using a key-value storage like RocksDB. We create a cache database per each deltatable, and update newly appended deltalogs to the cache database incrementally and periodically to keep the up-to-date.
◦ We support partition filterings using range queries if you are using RocksDB for deltalogs caching. A key in a cache database is a AddFile’s path which contains partition columns and values as a prefix. So, we can scan deltalogs very fast using a prefix which is generated from predicate hints.
• Configuration and Authorization
◦ We support simple REST protocols to help you develop your own configuration and authorization servers.
Please let me know if you need any further information! Enjoy!
Thank to all contributors for delta, delta-sharing, and delta-standalone!