When I am running a query on Databricks itself from notebook, it is running fine and giving me results. But the same query when executed from FastAPI (Python, using databricks library) is giving me "TypeError: 'NoneType' object is not iterable".
I can see both queries (from notebook and from API) from SparkUI list of jobs and both of them are marked as SUCCESS.
Differences between two jobs:
API one doesn't have any stdout logs (giving error "The requested Spark UI page does not exist."), whilst notebook one has logs.
API one has 3 stages in DAG visualization, whilst notebook one has 2 stages (additional stage is mapPartitionsInternal)
API is sending a query to "thriftsever-session-..." pool)
