I am getting NoneType error when running a query from API on cluster

[Nastia]

When I am running a query on Databricks itself from notebook, it is running fine and giving me results. But the same query when executed from FastAPI (Python, using databricks library) is giving me "TypeError: 'NoneType' object is not iterable".

I can see both queries (from notebook and from API) from SparkUI list of jobs and both of them are marked as SUCCESS.

Differences between two jobs: 
API one doesn't have any stdout logs (giving error "The requested Spark UI page does not exist."), whilst notebook one has logs.

API one has 3 stages in DAG visualization, whilst notebook one has 2 stages (additional stage is mapPartitionsInternal)

API is sending a query to "thriftsever-session-..." pool)

[NandiniN]

Hi @Nastia , can you please share the entire stacktrace and the query that you are running. 

There is currently not much detail with which I can help you understand this. But it is totally possible it is a bug that's causing this, because there should be a uniform behavior on both API and notebook.