Databricks Community

melbourne · 02-06-2024

Hi,I've a DLT pipeline running in Unity Catalog, and one of the task is to write content into a file within volume.I was able to write to file within volume using just PySpark, however when I do the same in DLT, I get an error:OSError: [Errno 30] Rea...

melbourne · 02-07-2024

I checked all the privileges and it looks alright.However, I figured out that DLT uses a shared cluster and using that files cannot be written into a Volume.

melbourne · 02-02-2024

Well it’s maybe for future and will be beneficial for serverless clusters

melbourne · 02-02-2024

You can switch to 'Triggered' pipeline in this case.Next, create a job in workflow and attach a trigger of type 'file arrival' to it. Next add the notebook and cluster to the job. If you're not using DLT, then set cluster timeout minutes to 0, so tha...

melbourne · 02-02-2024

Ideally one would expect clusters used for DLT pipeline to terminate after the pipeline execution has finished. However, while running in `development` environment, you'll notice it doesn't terminate on its own, whereas in `production` it terminates ...

melbourne · 12-18-2023

Hi @afisl Use following SQL command to add tags programmatically:alter table <table_name> set tags ('key1' = 'value1', 'key2' = 'value2')

Databricks Community

User Stats

User Activity

Unable to write to Volume from DLT pipeline

Re: Unable to write to Volume from DLT pipeline

Re: DLT Job Clusters: Continuous vs Triggered Cluster Start Times

Re: stop autoloader with continuous trigger programatically

Re: DLT Job Clusters: Continuous vs Triggered Cluster Start Times

Re: Apply unitycatalog tags programmatically