Python project || write write_micro_batch in structure streaming

[manish1987c]

Hi Team,

We are in progress of developing a framework in python using datatbricks-connect in VSO,

However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: cannot pickle 'ClientThreadLocals' object"

Although we are able to run writestream without for each batch.. but since we have to perform transformations / DQ checks and metadata entries created in different table.. we are trying to encapsulate all these in same micro batch.

I tried removing all these eventhough just a simple code with foreachbatch function is not working in VSO using datatbricks-connect.

Please provide your inputs and help.. thanks

#databricks-connect

#python-project

[Kaniz]

Hi @manish1987c, 

• Ensure that the Python version you’re using locally matches the one on your Databricks cluster. Minor version differences are usually acceptable (e.g., 3.10.11 versus 3.10.10).