Attribute Lineage for PySpark DataFrame My question is not Delta specific, sorry for that. I was wondering if it would be possible to extract attribute lineage from the execution plan of a PySpark DataFrame. Pseudo-Example:

df = spark.table("myTable").withColumn("C", col("A") + col("B")).withColumnRenamed("A", "Z")

magicLineageAnalyzer(df)
>> df.Z <-- myTable.A
>> df.B <-- myTable.B
>> df.C <-- myTable.A, myTable.B

There is a tool for that called Spline. I’ve never used it, but I am considering it recently. https://github.com/AbsaOSS/spline-spark-agent