Hi All, Can anyone help me resolve the below issue: I have a scenario where I'm converting dataframe to dictionary and iterating over every value to get the absolute value. But getting error. Refer SS-1 & SS-2. However, when I'm running the same code directly on hard coded dictionary same code is working fine. Refer SS-3 & SS-4. Please help. Thanks!

Print both dictionaries and see what they contain first before executing further logic. It is a bit weird the stack trace in SS2 seems like coming from Spark due to using wrong column name in a dataframe or something while you are using collect and if something fails it should not fail on such line. The exception shown usually means a UDF is called or a DataFrame expression is missing something like wrapping a constant with

lit

function.

@Yousry Mohamed I tried to print both the dictionaries i.e. Dict from Screenshot SS1 and even SS2. Both are same. Still facing same issues.

Try to print both dictionaries after line 3 in SS2

Did that too..

Can you share the code for the failing cell along with the full stacktrace

SS2 is the code which is failing

Please share the most recent stack trace

Hope this helps

phew, that's a know issue

Oh okay gotcha

you are welcome