Vishal Kadam06/09/2023, 4:41 AM
Sukumar Nataraj06/09/2023, 7:34 AM
error usually come when disk space volume is low on EC2 instances. Increase the EBS volume when spinning up EMR. It is not related to delta issue.
no space left on the device
Vishal Kadam06/09/2023, 7:35 AM
Sukumar Nataraj06/09/2023, 7:41 AM
Vishal Kadam06/09/2023, 7:49 AM
Kashyap Bhatt06/09/2023, 1:32 PM
Nothing to do with spark or Delta as far as I know because this is due to infra.I do not agree this with at all. You could cross-join a billion rows with a billion rows and then try to throw as much hardware at it as you like and it won't work (e.g. will take 3 years to run the job). If you're running outta memory then first thing I'd check is what I'm doing wrong instead of increasing memory, unless I was being cheap to begin with and using small hardware.