Using UDF in an insert command

Dp15 — Tue, 09 Apr 2024 17:05:55 GMT

Hi,
I am trying to use a UDF to get the last day of the month and use the boolean result of the function in an insert command. Please find herewith the function and the my query.

function:

import calendar

from datetime import datetime, date, timedelta

def getMonthEndDate(curr_dt: int😞

curr_dt = datetime.strptime(str(curr_dt),'%Y%m%d').date()

print("Provided date is: ",curr_dt)

me_date=getLastBusinessDateInMonth(curr_dt.year,curr_dt.month)

print("Last Business Day of the Month is: ",me_date)

if me_date == curr_dt:

return True

else:

return False

def getLastBusinessDateInMonth(year: int, month: int) -> int:

return date(year, month, max(calendar.monthcalendar(year, month)[-1][:5]))

spark.udf.register("isMonthEnd1", getMonthEndDate)

%sql

INSERT INTO test.metadata(id, reporting_date, run_id, run_type, batch_type, is_latest, create_date, create_time,is_month_end)

VALUES(11, 20240409, 00, 0, 'Test', 1, NULL, NULL,isMonthEnd1(20240409));

when I execute this notebook I am getting this following error
Could someone please help?

AnalysisException: [INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE] Invalid inline table. Cannot evaluate the expression "isMonthEnd1(20240409)" in inline table definition.

Re: Using UDF in an insert command

Dp15 — Tue, 16 Apr 2024 16:32:19 GMT

Thank you @Retired_mod for your detailed explanation

topic Re: Using UDF in an insert command in Data Engineering

Using UDF in an insert command

Re: Using UDF in an insert command