Hi Brahmareddy! Thank you for your answer, it was really useful for testing what was going on with our pipeline. 

The main problem was that, we were importing a custom library and creating custom DataFrame objects using classes from it in our notebook, which hasn't the same spark version as the compute. Therefore this was causing an issue while calling spark built-in functions because of that spark session conflict .

In order to solve this we refactored our code and aligned the spark version.

Thank you very much for your help.