I am getting NoneType error when running a query from API on cluster

[Nastia]

When I am running a query on Databricks itself from notebook, it is running fine and giving me results. But the same query when executed from FastAPI (Python, using databricks library) is giving me "TypeError: 'NoneType' object is not iterable".

I can see both queries (from notebook and from API) from SparkUI list of jobs and both of them are marked as SUCCESS.

Differences between two jobs: 
API one doesn't have any stdout logs (giving error "The requested Spark UI page does not exist."), whilst notebook one has logs.

API one has 3 stages in DAG visualization, whilst notebook one has 2 stages (additional stage is mapPartitionsInternal)

API is sending a query to "thriftsever-session-..." pool)

[Kaniz_Fatma · Accepted solution]

Hi @Nastia, The “TypeError: ‘NoneType’ object is not iterable” error typically occurs when you try to iterate over a variable that has a value of None.

Let’s explore some possible solutions to address this issue:

1. Check for None before Iterating: Ensure that the variable you’re trying to iterate over is not None. You can add a check like this before the loop:

   if my_list is not None:
       for item in my_list:
           # Your code logic here
   

2. Verify the Query Execution: Since both queries are marked as SUCCESS in SparkUI, the issue might not be with the query execution itself. However, it’s essential to verify that the query execution is indeed successful when called from FastAPI.

3. Investigate the API Execution: Since the API query doesn’t have stdout logs, it’s challenging to diagnose the issue directly. Here are some steps you can take to investigate further:

   • Check if the API call is correctly passing the query to Databricks.
   • Verify that the API call is using the same configuration (e.g., cluster settings, authentication) as the notebook execution.
   • Inspect any relevant logs or error messages specific to the API execution.

4. Pool Configuration: The fact that the API query is sending to the “thriftsever-session-…” pool might be relevant. Ensure that the pool configuration (e.g., concurrency, resources) is consistent between the notebook and API executions.

5. Additional Stages in API Execution: The extra stage (mapPartitionsInternal) in the API execution could be a clue. Investigate what this stage does and whether it’s expected in your query.

Remember to thoroughly review your API code and compare it with the notebook execution to identify any discrepancies. If you encounter any specific error messages or logs related to the API execution, please share them for further analysis. 😊

For more information, you can refer to the Databricks community forum.¹ Additionally, you can explore general solutions for handling ‘NoneType’ errors in Python here² and here³.⁴

Feel free to provide more details or ask additional questions if needed! 🚀