ApplyInPandas failing at a particular grouped item

[mbejarano89]

Hello,

I have a code that performs a forecast for 21k items in parallel. It looks like this:

 def forward_forecast(data):
    model = ETSModel(window_data, error='add', trend='add', seasonal=None)
                fitted_model = model.fit(disp=0)
                # Forecast the missing value
                forecast = fitted_model.forecast(steps=1).values[0]
                # Replace the missing value with the forecasted value
                data.loc[start_index:end_index-1, 'y_hat'] = forecast
                return data
result = data.groupBy("item").applyInPandas(foreward_forecast,schema)

When I run this code with a couple of items it run fine, but when I try using the 21k items, it fails at one item and gives this error: "unsupported operand type(s) for -: 'NoneType' and 'int'"

I am trying to figure out how to troubleshoot it and find  out at which item my function is failing.

Thanks

[Kaniz_Fatma]

Hi @mbejarano89, The error message you’re encountering, “unsupported operand type(s) for -: ‘NoneType’ and ‘int’”, indicates that you’re trying to perform a subtraction operation between a NoneType and an integer.

Let’s break down the issue and explore how to troubleshoot it:

1. Understanding the Error:

   • The error occurs when you attempt to subtract something from a None value, which is not allowed in Python.
   • In your code snippet, there’s likely a line where you’re trying to subtract an integer from a value that turns out to be None.

2. Potential Causes:

   • The error could be related to various parts of your code, such as:
     • The forecast value being assigned to None.
     • The data.loc[start_index:end_index-1, 'y_hat'] assignment resulting in a None.
     • An issue with the applyInPandas function.

3. Troubleshooting Steps: To identify the specific item causing the error, follow these steps:

   a. Check the forecast Value:

   • Inspect the forecast value after the line forecast = fitted_model.forecast(steps=1).values[0].
   • Ensure that it’s not None. If it is, investigate why the forecast is not being computed correctly.

   b. Inspect the Dataframe Assignment:

   • Look at the line data.loc[start_index:end_index-1, 'y_hat'] = forecast.
   • Verify that the slicing (data.loc[start_index:end_index-1]) is working as expected.
   • Confirm that the forecast value is not None.

   c. Debugging with Print Statements:

   • Add print statements to your code to track the execution flow.
   • Print relevant variables (e.g., forecast, start_index, end_index) to see their values during execution.
   • This will help you pinpoint the problematic item.

   d. Iterate Over Items Individually:

   • Instead of processing all 21k items at once, try running your function on a smaller subset (e.g., a few hundred items).
   • Gradually increase the number of items until you encounter the error.
   • This will help you identify the specific item causing the issue.

4. Fixing the Issue:

   • Once you’ve identified the problematic item, focus on understanding why it’s failing.
   • Check for any edge cases or unexpected data that might lead to a None value.
   • Consider handling missing or unexpected data gracefully within your function.

Remember to use print statements and step through your code to isolate the issue.

Good luck troubleshooting, and feel free to provide additional details if you need further assistance! 🚀