https://delta.io logo
r

Randy Sims

04/11/2023, 3:12 PM
Hey everyone! Just joined and I have a few questions: • What version of delta lake is used in Synapse? • Is there a query I can run to answer my own question?
j

JosephK (exDatabricks)

04/11/2023, 3:13 PM
m

Martin

04/11/2023, 4:14 PM
I'd also appreciate if there was a programmatic way to look up the version in a python session
c

chris fish

04/11/2023, 5:23 PM
is there not? databricks has that in its notebooks
v

Venki Balakrishnan

04/12/2023, 5:42 AM
The version details are also displayed/selectable (select the Spark version) when you create a Synapse spark pool.
m

Martin

04/12/2023, 6:47 AM
@chris fish in Python packages usually have a
<package_name>.___version___
attribute. Delta does not have this currently. There were some efforts but it was abandoned: https://github.com/delta-io/delta/pull/1356
Concering Azure Synapse and https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime: From my experience: Microsoft is very bad at keeping doc up-to-date. Also: unfortunately they maintain a Microsoft-proprietary fork of delta. They add another digit at the end of the official delta version, see here: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime#:~:text=delta%2Dcore_2.12-,2.2.0.1,-kusto%2Dingest In a Synapse Notebook you can do this:
Copy code
>>> delta.__path__
['/usr/hdp/current/spark3-client/jars/delta-core_2.12-2.2.0.2.jar/delta']
delta-core_2.12-2.2.0.2.jar
indicates that they actually run version 2.2.0.2 (= second revision of MS flavored Delta 2.2.0)
3 Views