Hi team,
I believe you cannot create or access a SparkSession or run Spark operations like spark.sql() directly inside a Python UDF. input_table is a table argument, not a string with a table name. You receive it as a pandas DataFrame when using RETURNS TABLE

You need to define your logic outside SQL in a notebook and use regular Spark APIs:

Then call process_table("my_table") in your notebook or job. Hope this helps!