I am getting NoneType error when running a query from API on cluster

Nastia — Wed, 17 Apr 2024 14:01:15 GMT

When I am running a query on Databricks itself from notebook, it is running fine and giving me results. But the same query when executed from FastAPI (Python, using databricks library) is giving me "TypeError: 'NoneType' object is not iterable".

I can see both queries (from notebook and from API) from SparkUI list of jobs and both of them are marked as SUCCESS.

Differences between two jobs:
API one doesn't have any stdout logs (giving error "The requested Spark UI page does not exist."), whilst notebook one has logs.

API one has 3 stages in DAG visualization, whilst notebook one has 2 stages (additional stage is mapPartitionsInternal)

API is sending a query to "thriftsever-session-..." pool)

Re: I am getting NoneType error when running a query from API on cluster

NandiniN — Thu, 09 Jan 2025 07:20:38 GMT

Hi @Nastia , can you please share the entire stacktrace and the query that you are running.

There is currently not much detail with which I can help you understand this. But it is totally possible it is a bug that's causing this, because there should be a uniform behavior on both API and notebook.

topic I am getting NoneType error when running a query from API on cluster in Data Engineering

I am getting NoneType error when running a query from API on cluster

Re: I am getting NoneType error when running a query from API on cluster