Databricks Community

ashraf1395

Hi there,I have a usecase where I want to set the dlt pipeline id in the configuration parameters of that dlt pipeline.The way we can use workspace ids or task id in notebook task task_id = {{task.id}}/ {{task.name}} and can save them as parameters a...

ashraf1395

We have complete sql based dlt pipelines. Where bronze tables are read from UC volumes. There can be situations when no new data comes so of the endpoints in the UC volume. in that case the SQL code blocks gets failed which results in failing the ent...

ashraf1395

I have 7 - 8 different dlt pipelines which have to be run at the same time according to their batch type i.e. hourly and daily. Right now they are triggered effectively according to their batch type. I want to move to a next stage where I want to clu...

ashraf1395

So, We are migrating a hive metastore to UC catalog. We have some fivetran connections.We are creating all tables as external locations and we have specified the external locations at the schema level.So when we specify the destination in the fivetra...

ashraf1395

Working on an ingestion workflow in databricks which extracts data from on-prem sources in databricks following all standard practices of incremental load, indempotency, upsert, schema evolution etc and storing data properly.Now we want to optimize t...

ashraf1395

Any ideas on this @VZLA, @Alberto_Umana

ashraf1395

Hi @VZLA , I got the idea. There will be a small change in the way, we will use it. Since we don't schedule the workflow in databricks we trigger it using the API. So I will pass a job parameter along with the trigger according to the timestamp wheth...

ashraf1395

Hi there, Thanks I understood the approach. But how to implement is what I am not able to figure out.If can give a small demo example. It will be really helpful

ashraf1395

Databricks asset bundles can be a great solution for this. Clear and straightforward. https://docs.databricks.com/en/dev-tools/bundles/index.html

ashraf1395 · 01-05-2025

@Walter_C If I cannot move my databricks partner workspace. Then I will lose my access to the free credits I have there. Any way I can move it as well.

Databricks Community

User Stats

User Activity

How to capture dlt pipeline id / name using dynamic value reference

Adding if statements or try/catch block in sql based dlt pipelines

Databricks Workflow design

Connecting Fivetran with databricks

Solution Design for an ingestion workflow with 1000s of tables for each source

Re: How to capture dlt pipeline id / name using dynamic value reference

Re: Databricks Workflow design

Re: Databricks Workflow design

Re: what is the best way to deploy workflows with different notebooks to execute in different worksp

Re: Can we change our cloud service connected with our Databricks account