Hello i run my AWS Glue Jobs locally in the Docker Container.
I want to convert a parquet file to delta file.
I tried using pandas api with “to.Delta()” function and inside the container the command “spark-submit myjob.py --deltalake delta” as argument. and i tried pip install delta-spark inside the container
It fails always with “Failed to find data source: delta.”