Is it adviceable to write databricks notebook's pyspark code in Object Oriented way (using python classes and methods) ? Embedding all pyspark code in python class & methods. Will it have performace issues same as python udfs ? Any input is appreciated. Thanks.

That’s a great question. If all the code inside a function is spark code, you won’t run into any trouble. so for example:

def function(input):
spark.read.path(input).filter().groupby().count().write()

won’t give you any problems. I always suggest that you write in any way you like as long as everyone in the organization follows the same style. You don’t want 4 different ways of writing code. Coming up with a style guide can be very useful.

Thanks for your reply. Completely agree. From where this question is coming is the idea of modularizing notebook's code - creating functions for reusable codes

Yes, it might be useful if you have 15 transformations for dates that you always apply. You can do a %run at the top of subsequent notebooks to include those resuable fucntions