Lakeflow SDP partition error

IM_01
Valued Contributor

Hi,
I was trying to log an exception in Lakeflow SDP , firstly I am creating an empty streaming dataframe in case of exception and writing log into audit table as shown below

try:
	raise Exception("testexception")
	return df
except Exception as e:
	df=spark. createDataFrame([{f"error _msg": str (e) }], schema="error_msg
	string")
	df.write.insertInto("cat.sch.tbl_stg_tst_audit")

	df=spark.readstream.format("rate").load()
	df=df.select(*[lit (None).cast(coltype).alias(colname) for colname, coltype in tb_schema l)
	df=df.where("1==0")
	return df

but was facing below error in case of write to original table

 

Category: Error
Message: The number of partitions
(0) used in previous microbatch is different from the current number of partitions (8). There could be two possible reasons:
1. Option "numpartitions" of the rate source gets changed during query restart.
2. The size of the cluster might change during query restart.
Explicitly set option
"numpartitions" of the rate source to 0 to fix this issue.
Error class:
STREAMING_RATE_SOURCE_V2_PARTITION_NUM_CHANGE_UNSUPPORTED

Could anyone please help with this issue