cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

Mr_K
New Contributor

Hello,

forecast_date = '2017-12-01'
spark.conf.set('spark.sql.shuffle.partitions', 500 ) 
 
# generate forecast for this data
forecasts = (
  history
  .where(history.date < forecast_date) # limit training data to prior to our forecast date
  .groupBy('store', 'item', lit(30).alias('days_to_forecast'))
    .applyInPandas(get_forecast, "store integer, item integer, date timestamp, sales float, sales_pred_mean float, sales_pred_lower float, sales_pred_upper float")
    .withColumn('forecast_date', lit(forecast_date).cast(TimestampType())) 
    ).cache()
 
forecast_evals = (
  forecasts      
    .select('forecast_date', 'store', 'item', 'sales', 'sales_pred_mean')
    .where(forecasts.date < forecasts.forecast_date)
    .groupBy('forecast_date', 'store', 'item')
    .applyInPandas(evaluate_forecast, "forecast_date timestamp, store integer, item integer, mse float, rmse float, mae float, mape float")
    )
 
forecast_evals_cv = (
  forecasts      
    .select('forecast_date', 'store', 'item', 'sales', 'sales_pred_mean')
    .where(forecasts.date < forecasts.forecast_date)
    .groupBy('forecast_date', 'store', 'item', lit(30).alias('days_to_forecast'))
    .applyInPandas(evaluate_forecast_cv, "forecast_date timestamp, store integer, item integer, horizon integer, mse float, rmse float, mae float, mape float, mdape float, coverage float")
    )
 
forecasts.createOrReplaceTempView('forecasts_tmp')
forecast_evals.createOrReplaceTempView('forecast_evals_tmp')
forecast_evals_cv.createOrReplaceTempView('forecast_evals_cv_tmp')

When I run the above code, it's throwing error

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

I'm using a shared cluster with 12.2 LTS Databricks Runtime and unity catalog is enabled.

Also getting similar error for faker package.

2 REPLIES 2

User16502773013
Contributor II

Hello @Mr_K ,

Running applyInPandas on UC enabled cluster is not currently supported.

As an alternative/interim solution we suggest to implement the forecast function as a Spark UDF 

For more information on currently supported Python UDF's please check release notes here

Regards

Tharun-Kumar
Honored Contributor II

@Mr_K 

ApplyInPandas is a higher order function in Python. As of now, we do not support higher order functions in Unity Catalog. We do support direct calls made to python UDFs. 

Here is an example of how to reference UDFs in UC - https://docs.databricks.com/en/udf/python.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group