Hi fellows my question is not about delta lake but generally

Question

Hi fellows my question is not about delta lake but generally for Spark on EMR sorry about that So if it does not fit here please feel free to delete the question I have a Spark job which I am running on EMR This job has to process some 2 TB of data and I am getting error ``` java lang OutOfMemoryError Java heap space XX OnOutOfMemoryError= kill 9 p Executing bin sh c kill 9 27571 ``` I have tried using various instance types like 20 25 core instances of m5 16xlarge and r5 12xlarge Also tried playing around spark configurations like spark driver memory and spark executor memory from 30g to 300g but nothing helped The job has not any major computation but simply its spark coalesce write partitionBy parquet Also tried setting HADOOP HEAPSIZE in the configuration to 100g and 200g Please ask if more information is required Thanks Here is the screenshot of EMR executors

JosephK (exDatabricks) · Answer

Coalesce is trying to reduce all your data to fewer memory partitions and these are too big to fit into memory and giving you oom You should use repartition before a write operation You can also just use delta lake to do optimization compaction

Zohaa Qamar · Answer

i will try to increase the number of partitions in coalesce to maybe 100 as currently they are set to 20

Zohaa Qamar · Answer

< JosephK exDatabricks > I replaced coalesce with repartition and also increased the number of partitions from 20 to 1000 Below are my other configurations ```spark driver memory=100g spark executor memory=100g spark yarn executor memoryOverhead=2g spark yarn driver memoryOverhead=2g spark executor cores=5 spark driver cores=5 spark executor instances=60 spark network timeout=50000s spark executor heartbeatInterval=5000s``` But still it is failing in EMR I have the below errors now ```ERROR AsyncEventQueue Dropping event from queue eventLog This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler ``` and also this in stderr ```java lang NullPointerException null value for spark executor cores``` I have tried with several variations in the spark configurations and coalesce repartition etc But there is no way it is succeeding Any help please

Mike M · Answer

What is the source data format and how are you reading it If CSV are you defining the schema manually

Zohaa Qamar · Answer

source is json with gzip compression ```sparkSession read schema getInputSchema json sourceFilesPath ```

Mike M · Answer

How many files are you reading from

Zohaa Qamar · Answer

they are lots like really too many the size is around 2TB

Zohaa Qamar · Answer

I followed this which says this will be good for 10 TB but no luck <https aws amazon com blogs big data best practices for successfully managing memory for apache spark applications on amazon emr >

Mike M · Answer

Are they similarly sized

Zohaa Qamar · Answer

yeah more or less equi sized files

Mike M · Answer

How large compressed uncompressed JSON can be a pain in the rear to parse in large sizes and it wouldn t surprise me if that s a part of the issue Also do you know if it s pure JSON or more of a JSON lines style

Zohaa Qamar · Answer

pure JSON file size is 2 0 to 2 3 MB each previously I had around 1 5TB of data and that worked fine with the same code this is sudden increase in data for one day and it is failing even after several configuration tweaks

Mike M · Answer

Ah one of those deals Usually to work around them I end up splitting steps writing out intermediate steps to parquet bumping node size etc

Zohaa Qamar · Answer

i tried bumping node size and instance count from m5 12xlarge to m5 16xlarge and also as suggested in the blog above r5 12xlarge 20 25 instances but no luck at all

Mike M · Answer

What is your `spark sql shuffle partitions` value

Zohaa Qamar · Answer

this would be default as not setting this explicitly

Zohaa Qamar · Answer

any suggestions to what value do I set it

Mike M · Answer

I believe the default is 200 I ve almost never had it make an effect but you can try setting to 400 or 800 I m not sure if it would make a difference here

Zohaa Qamar · Answer

right any other suggestions seems so nothing is working here though

Mike M · Answer

Other than the partitions thing which in theory could fit I believe that setting is how many partitions it sets up by default to process into usually I don t hit it on read due to larger fewer files but it might make a difference in your case I d try looking at the SparkUI and see if anything seems particularly skewed Beyond that possibly try a parquet conversion on a subset of the files then merge the two after

Zohaa Qamar · Answer

Yeah partitions I have increased and also used repartition instead of coalesce

Zohaa Qamar · Answer

I am not much knowledgeable about Spark UI and identifying the issues using that if you would be happy to spare some time for a screen sharing to help me out tomorrow shall be really thankful

JosephK (exDatabricks) · Answer

Def a case of too many files What appears to be happening is that you have hundreds of thousands of files and it s causing scheduling problems As mentioned above breaking it up in to smaller jobs may help What is happening is that the drive is trying to get the list of files to read and it s so massive that it can t hold the list in memory I also hate to shill for my own company but many of these kinds of problems don t exist on databricks It may cost more but saves you money in the end

Zohaa Qamar · Answer

```breaking it up in to smaller jobs may help``` does this mean I process my job for chunks of data multiple times

JosephK (exDatabricks) · Answer

yeah read in like 20gb at a time

Zohaa Qamar · Answer

sorry but can you guide how this can be done read by restricting the data Thanks

JosephK (exDatabricks) · Answer

It depends on how your data is laid out If there are any partitions that will make it easier You can also do streaming of 1 file per trigger

Zohaa Qamar · Answer

its year month day hour and my process is hourly so its already reading from the last partition i e hour

JosephK (exDatabricks) · Answer

yeah you can do it in parts maybe day at a time

Zohaa Qamar · Answer

its already on hour