- 357 Views
- 3 replies
- 0 kudos
Handling single table in multiple dlt pipeines
So this is my workflow which i am trying to buildThere are 2 databricks job Lets say A and B.So Job A has 3 tasks and the3rd task Check if target table is present or not if present it updates its schema if any changes or normally refreshes the pipeli...
- 357 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @ashraf1395 Yes, your understanding is partially correct. Let me clarify:Only one pipeline can own and manage a target table, including operations like schema evolution, maintenance, and refreshes, etcWhen other pipelines are mentioned as "produci...
- 0 kudos
- 269 Views
- 3 replies
- 0 kudos
Getting error while using Live.target_table in dlt pipeline
I have created a target table in the same dlt pipeline. But when I read that table in different block of notebook with Live.table_path. It is not able to read it Here is my code block 1 Creating a streaming table # Define metadata tables catalog = sp...
- 269 Views
- 3 replies
- 0 kudos
- 0 kudos
Cant we use Live.table_name on a target dlt table with @Dlt.append_flow decorator.If yes can you share the code bcz when I tried I am getting error.
- 0 kudos
- 420 Views
- 1 replies
- 0 kudos
Can Databricks write query results to s3 in another account via the API
I work for a company where we are trying to create a Databrick's integration in node using the @DataBricks/sql package to query customers clusters or warehouses. I see documentation of being able to load data via a query from s3 using STS tokens whe...
- 420 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you been able to get a response on this topic, based on the information I can see it might not be supported to write on an S3 outside your account
- 0 kudos
- 406 Views
- 1 replies
- 0 kudos
I keep on getting Parse_syntax_error on autoloader run foreachbatch
Hey guys, I keep on getting this error message when trying to call a function with soda DQ's: [PARSE_SYNTAX_ERROR] Syntax error at or near '{'. SQLSTATE: 42601 File <command-81221799516900>, line 4 1 dfBronze.writeStream \ 2 .foreachB...
- 406 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sakuraDev , this looks like a Soda syntax issue. Try fixing the "fail" and "warn" fields in your Soda checks. For example, instead of writing: - missing_count(site) = 0: name: Ensure no null values fail: 1 warn: 0 Use Soda's thres...
- 0 kudos
- 265 Views
- 2 replies
- 1 kudos
HOW TO SKIP A STEP FROM AN EXISTING WORKFLOW
Hi, can any one guide me in below scenario:Suppose we have created a workflow with 10 steps and our batches are running properly but due to some business requirement/testing purpose we have to skip step 4 and run the rest of job from step 5 onwords t...
- 265 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @17abhishek ,You can try if/else conditions between the tasks with a Job parameter of "IsActiveTask" with a True or False.But it would be great if databricks team just add ability to simply disable some task from UI.
- 1 kudos
- 11660 Views
- 6 replies
- 6 kudos
Resolved! How can I trigger the execution of a specific step within a Databricks Workflow job?
I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?
- 11660 Views
- 6 replies
- 6 kudos
- 6 kudos
Really interested in this feature. I'd love to be able to programmatically orchestrate tasks with more granularity.
- 6 kudos
- 517 Views
- 3 replies
- 0 kudos
Run continuous job for a period of time
Hi there,I have a job where the Trigger type is configured as Continuous. I want to only run the Continuous job for a period of time per day, e.g. 8AM - 5PM. I understand that we can achieve it by manually starting and cancelling the job on the UI, o...
- 517 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @MuthuLakshmi , thank you for your answer. However, your answer doesn't help with my question. Let me rephrase my question.In short, my question is how to configure a Continuous job to run for a period of time, e.g. from 8AM to 5PM every day, and ...
- 0 kudos
- 341 Views
- 4 replies
- 0 kudos
Resolved! DABs job name must start with a letter or underscore
Hi,In UI I used the pipeline name 00101_source_bronze. I wanted to do the same in the Databricks Asset Bundles.but when the configuration is refreshed against Databricks Workspace I see this error:I found that this issue can be connect to Terraform v...
- 341 Views
- 4 replies
- 0 kudos
- 0 kudos
As mentioned above, this is a limitation directly with Terraform due to this our engineering team is limited on the actions that can be done, you can find more information about this limitation on the Terraform documentation: https://developer.hashic...
- 0 kudos
- 336 Views
- 1 replies
- 1 kudos
Resolved! workflow set maximum queued items
Hi all,I have a question regarding Workflows and queuing of job runs. I'm running into a case where jobs are running longer than expected and result in job runs being queued, which is expected and desired. However, in this particular case we only nee...
- 336 Views
- 1 replies
- 1 kudos
- 1 kudos
Unfortunately there is no way to control the number of jobs that will be moved to queue status when queuing is enabled.
- 1 kudos
- 294 Views
- 3 replies
- 0 kudos
Guidance Needed for Developing CI/CD Process in Databricks Using Azure DevOps
Hi everyone,I am working on setting up a complete end-to-end CI/CD process for my Databricks environment using Azure DevOps. So far, I have developed a build pipeline to create a Databricks artifact (DAB). Now, I need to create a release pipeline to ...
- 294 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @alcatraz96 ,One question, why don't you use Databricks Assets Bundles? Then the whole process would be much simplerHere you have a good end to end example:CI/CD Integration with Databricks Workflows - Databricks Community - 81821
- 0 kudos
- 849 Views
- 10 replies
- 0 kudos
Limitations for Unity Catalog on single user access mode clusters
Hello! According to Databricks documentation on azure :"On Databricks Runtime 15.3 and below, fine-grained access control on single user compute is not supported. Specifically:You cannot access a table that has a row filter or column mask.You cannot ...
- 849 Views
- 10 replies
- 0 kudos
- 0 kudos
@Nes_Hdr Single user compute uses fine-grained access control to access the tables with RLS/CLM enabled.There is no specific details about OPTIMIZE being supported in Single user mode. Under this doc limitations of FGAC mentions that "No support for...
- 0 kudos
- 1862 Views
- 3 replies
- 0 kudos
Databricks Asset Bundle to deploy only one workflow
Hello Community -I am trying to deploy only one workflow from my CICD. But whenever I am trying to deploy one workflow using "databricks bundle deploy - prod", it is deleting all the existing workflow in the target environment. Is there any option av...
- 1862 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Team, the deployment via DAB(Databricks Asset Bundle) reads all yml files present and based on that workflows are generated. In the previous versions of Databricks CLI prior to 0.236(or latest one), it use to delete all the workflow by making dele...
- 0 kudos
- 10627 Views
- 5 replies
- 2 kudos
Experiences using managed tables
We are looking into the use of managed tables on databricks. As this decision won’t be easy to reverse I am reaching out to all of you fine folks to learn more about your experience with using this.If I understand correctly we dont have to deal with ...
- 10627 Views
- 5 replies
- 2 kudos
- 2 kudos
Databricks recommends to ALWAYS use Managed Tables always UNLESS:Your tables are not DeltaYou explicitly need to have the table files in a specific location Managed Tables are just better... Databricks manages:the upgrades (Deletion Vectors? Column M...
- 2 kudos
- 178 Views
- 1 replies
- 0 kudos
Deriving a relation between spark job and underlying code
For one of our requirement, we need to derive a relation between spark job, stage ,task id with the underlying code executed after a workflow job is getting triggered using a job cluster. So far we are able to develop a relation between the Workflow ...
- 178 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Subhrajyoti thanks for your question! I'm not sure if you have tried this already, but by combining listener logs with structured tabular data, you can create a clear mapping between Spark job executions and the corresponding notebook code. You c...
- 0 kudos
- 455 Views
- 1 replies
- 0 kudos
Delta live tables - foreign keys
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
- 455 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, thank you for your question. Delta Live Tables (DLT) does not currently have built-in support for creating or enforcing foreign key constraints within a pipeline. This is because DLT is designed to focus on declarative data transformations and...
- 0 kudos