I dont know the answer to the first 2 questions, but its a shared compute, and the library is build with python 3.12 (compatible with 3.11 and 3.10). The compute is on databricks 15.4.

The problem is related to this question: https://community.databricks.com/t5/data-engineering/replay-stream-to-migrate-to-liquid-cluster/m-p/...

By limiting the number of records we transform from bronze to silver I was able to stream 100 records to silver with liquid clustering. So perhaps access control is not an issue. However the full transformation which for one record in bronze perhaps produce 200 000 records in silver - the job either times out (with G1GC garbage collector) or the driver restarts with the parallel garbage collector. This job ran fine when silver table was partitioned and unity catalog was not enabled.

Please reach out!

Kind regards, Endre