topic Serverless Compute - How to determine if being used programatically in Data Engineering

Serverless Compute - How to determine if being used programatically

Dave1967 — Thu, 03 Oct 2024 06:39:14 GMT

Hi, We use a common notebook for all our "common" settings, this notebook is called in the first cell of each notebook we develop. This issue we are now having is that we need 2 common notebooks, one for a normal shared compute and one for serverless as serverless does not allow (need) the same spark config commands. What I'd like to do is determine in our common notebook whether it is running on serverless compute or not so I can have just one common notebook, any ideas if and how I can do this.

Re: Serverless Compute - How to determine if being used programatically

filipniziol — Thu, 03 Oct 2024 07:41:42 GMT

Hi @Dave1967 ,

If you know any spark config command that is not supported in serverless, then build your logic around this command using try, catch:

def is_config_supported(): try: spark.sparkContext.getConf() return True except: return False print(is_config_supported())

Re: Serverless Compute - How to determine if being used programatically

Dave1967 — Thu, 03 Oct 2024 11:12:48 GMT

Many thanks,

I did think about doing it this was, just wondered if there was something neater but actually this is fine.

Many thanks