Hello,

forecast_date = '2017-12-01'
spark.conf.set('spark.sql.shuffle.partitions', 500 ) 
 
# generate forecast for this data
forecasts = (
  history
  .where(history.date < forecast_date) # limit training data to prior to our forecast date
  .groupBy('store', 'item', lit(30).alias('days_to_forecast'))
    .applyInPandas(get_forecast, "store integer, item integer, date timestamp, sales float, sales_pred_mean float, sales_pred_lower float, sales_pred_upper float")
    .withColumn('forecast_date', lit(forecast_date).cast(TimestampType())) 
    ).cache()
 
forecast_evals = (
  forecasts      
    .select('forecast_date', 'store', 'item', 'sales', 'sales_pred_mean')
    .where(forecasts.date < forecasts.forecast_date)
    .groupBy('forecast_date', 'store', 'item')
    .applyInPandas(evaluate_forecast, "forecast_date timestamp, store integer, item integer, mse float, rmse float, mae float, mape float")
    )
 
forecast_evals_cv = (
  forecasts      
    .select('forecast_date', 'store', 'item', 'sales', 'sales_pred_mean')
    .where(forecasts.date < forecasts.forecast_date)
    .groupBy('forecast_date', 'store', 'item', lit(30).alias('days_to_forecast'))
    .applyInPandas(evaluate_forecast_cv, "forecast_date timestamp, store integer, item integer, horizon integer, mse float, rmse float, mae float, mape float, mdape float, coverage float")
    )
 
forecasts.createOrReplaceTempView('forecasts_tmp')
forecast_evals.createOrReplaceTempView('forecast_evals_tmp')
forecast_evals_cv.createOrReplaceTempView('forecast_evals_cv_tmp')

When I run the above code, it's throwing error

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

I'm using a shared cluster with 12.2 LTS Databricks Runtime and unity catalog is enabled.

Also getting similar error for faker package.