Lakeflow SDP partition error
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-09-2026 10:38 AM - edited 05-09-2026 10:40 AM
Hi,
I was trying to log an exception in Lakeflow SDP , firstly I am creating an empty streaming dataframe in case of exception and writing log into audit table as shown below
try:
raise Exception("testexception")
return df
except Exception as e:
df=spark. createDataFrame([{f"error _msg": str (e) }], schema="error_msg
string")
df.write.insertInto("cat.sch.tbl_stg_tst_audit")
df=spark.readstream.format("rate").load()
df=df.select(*[lit (None).cast(coltype).alias(colname) for colname, coltype in tb_schema l)
df=df.where("1==0")
return dfbut was facing below error in case of write to original table
Category: Error
Message: The number of partitions
(0) used in previous microbatch is different from the current number of partitions (8). There could be two possible reasons:
1. Option "numpartitions" of the rate source gets changed during query restart.
2. The size of the cluster might change during query restart.
Explicitly set option
"numpartitions" of the rate source to 0 to fix this issue.
Error class:
STREAMING_RATE_SOURCE_V2_PARTITION_NUM_CHANGE_UNSUPPORTEDCould anyone please help with this issue