Databricks Community

Thanks for the suggestions. I investigated all of the above, but they didn't provide a solution. What did work was using another logging package within my custom package: Loguru. Not sure why this helped?

Hi, would it work if you try to remove "landing" at the end? Seems the error is complaining that the external location should be a directory.   

I ran another test this week where I changed my source table to not have deletion vectors and no longer believe that step is limiting factor. Without it the compute times seemed to be as follows:reading in data + wscg + exchange = 7.9 hours, ~1/3 wal...

Hi @mourakshit , I tried all the three methods you mentioned . None of them workedmethod_1 returne pipeline_name or pipelin_id as printed value : {{dlt_pipeline.name}} {{dlt_pipeline.id}} not the actual valuesmethond_2 returned not conf like spark.da...

Hi, I'm still not able to recreate this issue with Standard_DS3_v2.  I'm not sure if this is relevant, but do you also have this issue on an old High Concurrency cluster with custom access mode for the Standard_DS3_v2 cluster too? 

Hello @Faizan_khan8171, Thanks for your question. In Unity Catalog, external location names must be unique within the metastore. This restriction prevents naming conflicts and ensures that every external location is easily identifiable and manageable...

By the way I can acess all chunks using the endpoint /api/2.0/sql/statements/{statement_id}/result/chunks/{chunk_index}