Hi,
In my code I am creating a Spark session, that can be to used query the Unity Catalog delta tables.I have an MLFlow Pyfunc model that uses these table to retrieve information. Things work well from my cluster but getting below error from model serving:
"Error retrieving information for table catlogue_name.schema_name.table_name: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `catlogue_name`.`schema_name`."
Code Snippet:
schema="catlogue_name.schema_name"
table=table_name
full_table_name = f"{schema}.{table_name}"
try:
df = spark.read.table(full_table_name)
except Exception as e:
logger.error(f"Error retrieving information for table {full_table_name}: {e}")
What am I missing here? Can spark sessions be used from Model Serving? Do I need some authentication to retrieve information from delta tables? Is any specific model packaging step while model registration?