Jens Waage09/06/2023, 7:04 AM
) and pyspark (v.
) packages to attempt to unit test our Databricks notebooks locally. I ran into a problem where the
method throws a
when using a 3-layer-namespace for the
parameter. Below is an example error:
The method works correctly when naming a table using a 2-layer-namespace, i.e.
pyspark.errors.exceptions.captured.ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near '.'.(line 1, pos 21) == SQL == catalog.database.test_table ---------------------^^^
. Given that 3-layer-namespace support was added to Spark 3.4 and delta-spark 2.4.x should be compatible with Spark 3.4, I expected the
method to support this. This thread on the Databricks Community forums also discusses this issue, and the poster mentions that there may be something missing in the delta-core libraries. Is this indeed a bug/missing feature or have I misunderstood something here?