Hi Team,
We are in progress of developing a framework in python using datatbricks-connect in VSO,
However when we are trying to run micro batches in the foreachbatch function it is giving us error message saying that "few objects are not serializable: cannot pickle 'ClientThreadLocals' object"
Although we are able to run writestream without for each batch.. but since we have to perform transformations / DQ checks and metadata entries created in different table.. we are trying to encapsulate all these in same micro batch.
I tried removing all these eventhough just a simple code with foreachbatch function is not working in VSO using datatbricks-connect.
Please provide your inputs and help.. thanks
#databricks-connect
#python-project
