Yousry Mohamed

01/17/2023, 10:34 AM
According to this announcement, idempotent writes have been improved in version 2.0.2. I guess those improvements do not exist in 2.2.0 for some reason (I could be wrong). I wonder what is the rational here. Relating to having some visibility of deltalake version, this Scala expression retrieves deltalake version
For an OSS local environment with 2.2.0, it returns 2.2.0 as expected. On DBR 12.0, same expression returns 1.1.0. It’s just a bit confusing specially the the DBR release notes page of 12.0 for example shows 2.2.0. I know that deltalake on Databricks does not necessarily have to match a certain OSS version (at lease before open sourcing delta). I wonder if there is a robust/consistent way to get the version of deltalake installation whether on OSS or Databricks.


01/17/2023, 11:38 PM
1. yeah we backported a few usability features in this 2.0.2 release since that is the last release on Spark 3.2 and users are blocked from using those until they upgrade to Spark 3.3. I agree that its "temporarily" weird that one of those features is not released yet 2.1/2.2. But its backported to all those branches (include master) and next release of 2.3.0/2.2.1/2.1.2 will have the feature as well. 2. Regarding DBR's reported version... thats probably DBR bug in how it is integrating with Delta OSS. Please report that to the DB support.

Yousry Mohamed

01/17/2023, 11:55 PM
Thank you @TD, makes complete sense now.