Hello @Jaclaglez13 

I am really interested in your ticket because we have the same kind of error.
We loose the SQL built-in function when we run the notebook a second time without restarting our "all purpose compute".
There is a point I would like to clarify regarding your last message.
What do you mean exactly by: "we were importing a custom library and creating custom DataFrame"?
Does the issue was that this "custom library" (uber jar?) was embedding some Spark libraries in conflict with the Spark version installed on the Databricks compute?
In my case, the only external dependency that we use is org.apache.common.commons-csv, otherwise, all other dependencies are compiled with the maven "scope" property set to "provided", so we use dependencies already installed on the Databricks Runtime.

Your help would be really appreciate!

Loïc