Hi,
I am using databricks runtime 17.3.x-scala2.13 ,
But I am unable to create python stored procedures, (functions are possible but they dont support a spark session like below) , any thoughts/help is much appreciated ?
[
INVALID_STATEMENT_OR_CLAUSE] The statement or clause: CREATE PROCEDURE ... LANGUAGE PYTHON is not valid. SQLSTATE: 42601 == SQL (line 2, position 1) == LANGUAGE PYTHON ^^^^^^^^^^^^^^^
This is the code I am trying
CREATE OR REPLACE PROCEDURE test_cat.rg_test_landing.get_max_table_version(IN table_name STRING, OUT p_max_version BIGINT)
LANGUAGE PYTHON
SQL SECURITY INVOKER
AS $$
df = spark.sql(f"DESCRIBE HISTORY {tbl_name}")
target_operations = ["WRITE", "MERGE", "UPDATE", "DELETE"]
filtered_df = df.filter(df.operation.isin(target_operations))
if filtered_df.count() > 0:
max_version = filtered_df.select("version").orderBy("version", ascending=False).first()[0]
--print(max_version)
p_max_version = max_version
else:
p_max_version = None
$$;
Thanks!
