Hi Team,

We are in progress of developing a framework in python using datatbricks-connect in VSO,

However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: cannot pickle 'ClientThreadLocals' object"

Although we are able to run writestream without for each batch.. but since we have to perform transformations / DQ checks and metadata entries created in different table.. we are trying to encapsulate all these in same micro batch.

I tried removing all these eventhough just a simple code with foreachbatch function is not working in VSO using datatbricks-connect.

Please provide your inputs and help.. thanks

#databricks-connect

#python-project