Python project || write write_micro_batch in structure streaming

manish1987c — Fri, 14 Jun 2024 05:10:24 GMT

Hi Team,

We are in progress of developing a framework in python using datatbricks-connect in VSO,

However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: cannot pickle 'ClientThreadLocals' object"

Although we are able to run writestream without for each batch.. but since we have to perform transformations / DQ checks and metadata entries created in different table.. we are trying to encapsulate all these in same micro batch.

I tried removing all these eventhough just a simple code with foreachbatch function is not working in VSO using datatbricks-connect.

Please provide your inputs and help.. thanks

#databricks-connect

#python-project

topic Python project || write write_micro_batch in structure streaming in Data Engineering

Python project || write write_micro_batch in structure streaming