by
Skr7
• New Contributor II
- 12 Views
- 1 replies
- 0 kudos
Hi, I'm implementing Databricks Asset bundles, my scripts are in GitHub and my /resource has all the .yml of my Databricks workflow which are pointing to the main branch git_source:
git_url: https://github.com/xxxx
git_provider: ...
- 12 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Skr7 , Let’s break down your requirements:
Dynamically Changing Git Branch for Databricks Asset Bundles (DABs): When deploying and running your DAB, you want the Databricks workflows to point to your feature branch instead of the main branch....
- 87 Views
- 5 replies
- 0 kudos
Hi,I have configured 20 different workflows in Databricks. All of them configured with job cluster with different name. All 20 workfldows scheduled to run at same time. But even configuring different job cluster in all of them they run sequentially w...
- 87 Views
- 5 replies
- 0 kudos
Latest Reply
HI @jainshasha i tried to replicate your problem but in my case i was able to run jobs in parallel(the only difference is that i am running notebook from workspace, not from repo)As you can see jobs did not started exactly same time but it run in par...
4 More Replies
- 56 Views
- 4 replies
- 0 kudos
Hello, I am trying to connect the power bi semantic model output (basically the data that has already been pre processed) to databricks. Does anybody know how to do this? I would like it to be an automated process so I would like to know any way to p...
- 56 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @madhumitha, Connecting Power BI semantic model output to Databricks can be done in a few steps.
Here are a couple of options:
Databricks Power Query Connector:
The new Databricks connector is natively integrated into Power BI. You can configu...
3 More Replies
by
dbdude
• New Contributor II
- 4518 Views
- 7 replies
- 0 kudos
Why can I use boto3 to go to secrets manager to retrieve a secret with a personal cluster but I get an error with a shared cluster?NoCredentialsError: Unable to locate credentials
- 4518 Views
- 7 replies
- 0 kudos
Latest Reply
Hi @dbdude and @drii_cavalcanti , The NoCredentialsError you’re encountering when using Boto3 to retrieve a secret from AWS Secrets Manager typically indicates that the AWS SDK is unable to find valid credentials for your API request.
Let’s explor...
6 More Replies
by
Skr7
• New Contributor II
- 1143 Views
- 2 replies
- 1 kudos
Hi ,I have a Databricks job that results in a dashboard post run , I'm able to download the dashboard as HTML from the view job runs page , but I want to automate the process , so I tried using the Databricks API , but it says {"error_code":"INVALID_...
- 1143 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Skr7, You cannot automate exporting the dashboard as HTML using the Databricks API. The Databricks API only supports exporting results for notebook task runs, not for job run dashboards.
Here's the relevant excerpt from the provided sources:
Exp...
1 More Replies
by
Anske
• New Contributor II
- 93 Views
- 1 replies
- 0 kudos
Hi,I have a DLT pipeline that applies changes from a source table (cdctest_cdc_enriched) to a target table (cdctest), by the following code:dlt.apply_changes( target = "cdctest", source = "cdctest_cdc_enriched", keys = ["ID"], sequence_by...
- 93 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Anske, It seems you’re encountering an issue with your Delta Live Tables (DLT) pipeline where updates from the source table are not being correctly applied to the target table.
Let’s troubleshoot this together!
Pipeline Update Process: Whe...
- 80 Views
- 1 replies
- 0 kudos
Hello Community Folks -Did anyone implemented migration of notebooks that is in workspace to production databricks workspace using Databricks Asset Bundle? If so can you please help me with any documentation which I can refer? Thanks!!RegardsNiruban ...
- 80 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @niruban, Migrating notebooks from one Databricks workspace to another using Databricks Asset Bundles is a useful approach.
Let me guide you through the process and provide relevant documentation.
Databricks Asset Bundles Overview:
Databricks ...
- 104 Views
- 1 replies
- 0 kudos
I have a DLT pipeline, where all tables are non-streaming (materialized views), except for the last one, which needs to be append-only, and is therefore defined as a streaming table.The pipeline runs successfully on the first run. However on the seco...
- 104 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Oliver_Angelil, It appears that you’re encountering an issue with your DLT (Databricks Delta Live Tables) pipeline, specifically related to having an append-only table at the end of the pipeline.
Let’s explore some potential solutions:
Stream...
- 74 Views
- 1 replies
- 0 kudos
I use AWS Databricks which has an SSO&Scim integration with AAD. I generated an SPN in AAD, synced it to Databricks, and want to use this SPN with using AAD client secrets to use Databricks SDK. But it doesnt work. I dont want to generate another tok...
- 74 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @BerkerKozan, It sounds like you’re trying to set up provisioning to Databricks using Microsoft Entra ID (formerly known as Azure Active Directory) and encountering some issues.
Let’s break down the steps and address your concerns:
Provisionin...
by
sasi2
• New Contributor II
- 247 Views
- 1 replies
- 0 kudos
Hi, Is there any connectivity pipeline established already to access MuleSoft or AnyPoint exchange data using Databricks. I have seen many options to access databricks data in mulesoft but can we read the data from Mulesoft into databricks. Please gi...
- 247 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @sasi2, Connecting MuleSoft or AnyPoint to exchange data with Databricks is possible, and there are several options you can explore.
Let’s dive into some solutions:
Using JDBC Driver for Databricks in Mule Applications:
The CData JDBC Driver...
- 2477 Views
- 7 replies
- 4 kudos
Hello, we have Databricks Python workbooks accessing Delta tables. These workbooks are scheduled/invoked by Azure Data Factory. How can I enable Photon on the linked services that are used to call Databricks?If I specify new job cluster, there does n...
- 2477 Views
- 7 replies
- 4 kudos
Latest Reply
When you create a cluster on Databricks, you can enable Photon by selecting the "Photon" option in the cluster configuration settings. This is typically done when creating a new cluster, and you would find the option in the advanced cluster configura...
6 More Replies
by
subha2
• New Contributor II
- 340 Views
- 1 replies
- 0 kudos
There are some tables under schema/database under Unity Catalog.The Notebook need to read the table parallel using loop and thread and execute the query configuredBut the sql statement is not getting executed via spark.sql() or spark.read.table().It ...
- 340 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @subha2, It seems you’re encountering an issue related to executing SQL statements in Spark.
Let’s troubleshoot this step by step:
Check the Unity Catalog Configuration:
Verify that the Unity Catalog configuration is correctly set up. Ensure t...
- 79 Views
- 1 replies
- 0 kudos
I have Data Engineering Pipeline workload that run on Databricks.Job cluster has following configuration :- Worker i3.4xlarge with 122 GB memory and 16 coresDriver i3.4xlarge with 122 GB memory and 16 cores ,Min Worker -4 and Max Worker 8 We noticed...
- 79 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @DBX-2024,
Let’s break down your questions:
High CPU Utilization Spikes: Are They Problematic?
High CPU utilization spikes can be problematic depending on the context. Here are some considerations:
Normal Behavior: It’s common for CPU utilizat...
- 82 Views
- 1 replies
- 0 kudos
As of this morning we started receiving the following error message on a Databricks job with a single Pyspark Notebook task. The job has not had any code changes in 2 months. The cluster configuration has also not changed. The last successful run of ...
- 82 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @smukhi, The error message you’re encountering, specifically the “Py4JJavaError” with the “Missing Credential Scope” issue, can be quite puzzling.
Let’s explore some potential solutions and ideas to troubleshoot this problem:
Check Cluster Con...
- 336 Views
- 1 replies
- 0 kudos
The chunk of code in questionsys.path.append(
spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/")
)
from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_ud...
- 336 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @gabe123 , It seems like you’re encountering a ModuleNotFoundError when trying to import the broker_utils module in your Python code.
Let’s troubleshoot this issue step by step:
Check Module Location:
First, ensure that the broker_utils.py fil...