Is there any way to do ‘dos2unix’ cmd in Databricks notebook. My usecase is I have a csv file in ADLS location but with some junk chars so I need to remove the junk chars while reading the file using pyspark in my databricks notebook. I can handle this using dos2unix cmd running manually but this suppose to be automate. Can anyone plz help on this?
02/15/2023, 9:26 PM
You can use %sh to call shell commands, though dos2unix is not installed on the dbr images. You could use sed or awk to do the same thing.