Hi all! I am trying to run a simple spark script in Docker to create a delta table in aws, wondering what Docker image people are using and would recommend ?
I have tried a few different ones but I run into version issues with hadoop-aws and delta-core !!
I am using Mac m1 , so the image would have to be compatible (arm64)
Edit - I have been using image apache/spark-py:v3.3.0 , but I need to install hadoop separately .. which is a pain