I have API that triggers Spark calculations - with API hosted by Python 3.12 pod in AKS and connects to Databricks cluster using Databricks 18.1.1.

Initially I was using getOrCreate call on my API requests and all works.

But problem is - as Spark session is shared.. after a while when new API request comes in, it fails with "INVALID SESSION" - this is because Cluster expired the Spark session after waiting enough due to inactivity - looks like.

I also felt like sharing same Spark session is not my intention as I was to isolate each API call/request and create NEW Spark session per request.

So now I am using create NEW Spark session per API request and dont have any issues.

But do I need to do any clean up?

If I am creating NEW session , I feel tempted to call Stop session or do any clean up once I am done.

But documentation seems to suggest never explicitly call STOP Session.

I am not seeing any issues so far by not calling STOP but not sure if this causes any resource leaks and want to do what is right?

What is right way to clean up Spark Session in this  case when its created explicitly per API request? Do nothing like what I am doing OR call STOP - please suggest