Abond
New Contributor II

‎03-04-2025 04:09 PM

Hi, 

Any news regarding that issue? I have the same one on job cluster with 15.4 LTS when using asset bundles with foreachBatch  in .py file and call it from notebook. When the same code is located in notebook - it works file.

(prep_silver_df(bronze_table_fqn_df)
    .writeStream
    .trigger(availableNow=True)
    .foreachBatch(lambda df, batchId: upsertToDelta(df, batchId, silver_table_fqn))
    .option("checkpointLocation", silver_checkpoint_path)
    .outputMode("update")
    .start()
    .awaitTermination()
    )
0 Kudos