Hi,

I am dealing with a situation where i need to secure data at rest on storage (azure data lake), hence saving the data as encrypted text into the delta table. While serving, i want to create dynamic view which will be created from the delta table and decrypt it on the fly so that required user can see the decrypted data. I am storing the decryption key into azure KV which is not accessible via azure Databricks. Want to avoid the secret scope route as those user need read access and can see the key. Hence I have created a python whl which will read the decryption key and is obfuscated so that no one can see the code. Now the issue with that is performance as 1>SQL function context don't have SparkSession. I am forced to use normal python code instead of pandas udf as sparksession is not available within sql function. 2> Can't create global UDF so that i can use it from any notebook.

1> Can anyone help how to improve the performance of the udf?

2> Is there any way #databricks enable a> enable sparksession within sql function, b> enable user to create global udf so that it can be accessible from any notebook

Many thanks