Anita A

03/10/2023, 4:49 PM
Hi all, has anybody tried to run delta lake unit tests in Docker? I am using pytest, and was following this guide but the delta package won't load. On
I get this error:
object is not callable
. Thanks!

Kashyap Bhatt

03/10/2023, 5:37 PM
Post more code and full stack trace.

Anita A

03/10/2023, 6:24 PM
Here's the code
Copy code
from delta import configure_spark_with_delta_pip
        spark_jars_packages = [""]
        builder = SparkSession.builder.appName("MyApp") \
        .config("spark.sql.extensions", "") \
        .config("spark.sql.catalog.spark_catalog", "")
        spark = configure_spark_with_delta_pip(builder, extra_packages=spark_jars_packages).getOrCreate()
Copy code
def test_table():
        assert DeltaTable.isDeltaTable(spark, "/some/path") == 1
And i have tried many different variation of injecting the delta core package. This is the error that i get:
Copy code
return, identifier)
TypeError: 'JavaPackage' object is not callable
If i invoke that same code as a regular phyton code within Docker i sill get the error unless i invoke it like this
spark-submit --packages
So basecally i need a way to load this delta core package as part of the pytest invocation.
Any update?