Grainne B

02/07/2023, 10:45 PM
Hi all! I am trying to run a simple spark script in Docker to create a delta table in aws, wondering what Docker image people are using and would recommend ? I have tried a few different ones but I run into version issues with hadoop-aws and delta-core !! I am using Mac m1 , so the image would have to be compatible (arm64) Edit - I have been using image apache/spark-py:v3.3.0 , but I need to install hadoop separately .. which is a pain