Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
16134 Posts
Activity in Discussions
- 0 Views
- 0 replies
- 0 kudos
Request to Expedite Exam Reschedule — Databricks Certified Data Engineer Professional (#00891706)
Hi @Cert-Team @Cert-TeamOPS,Thank you for reaching out regarding my suspended exam (Ticket #00891706). I have already replied to the support team with my preferred rescheduling dates and times:15th April 2026, 8:00 PM IST16th April 2026, 8:00 PM ISTH...
- 0 Views
- 0 replies
- 0 kudos
- 47 Views
- 1 replies
- 0 kudos
Delta Jira data import to Databricks
We need to import large amount of Jira data into Databricks, and should import only the delta changes. What's the best approach to do so? Using the Fivetran Jira connector or develop our own Python scripts/pipeline code? Thanks.
- 47 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @greengil, Have you considered Lakeflow Connect? Databricks now has a native Jira connector in Lakeflow Connect that can achieve what you are looking for. It's in beta, but something you may want to consider. It ingests Jira into Delta with incr...
- 0 kudos
- 70 Views
- 2 replies
- 1 kudos
Registering Delta tables from external storage GCS , S3 , Azure Blob in Databricks Unity Catalog
Hi everyone,I am currently working on a migration project from Azure Databricks to GCP Databricks, and I need some guidance from the community on best practices around registering external Delta tables into Unity Catalog.Currenlty I am doing this but...
- 70 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @muaaz, On GCP Databricks, the SQL pattern you are using is fine, but the recommended best practice is to back it with a Unity Catalog external location instead of pointing tables directly at arbitrary gs:// paths. In practice, that means first cr...
- 1 kudos
- 43 Views
- 2 replies
- 0 kudos
Accessing secrets(secret scope) in pipeline yml file
How can I access secrets in pipeline yaml or directly in python script file?
- 43 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @prakharsachan ,In Declarative Automation Bundles YAML (formerly known as Databricks Assets Bundles) you can only define secret scopes:If you want to read secrets from secret scope you can use dbutils in python script:password = dbutils.secrets.ge...
- 0 kudos
- 303 Views
- 1 replies
- 1 kudos
Data System & Architecture - PySpark Assignment
Title: Spark Structured Streaming – Airport Counts by CountryThis notebook demonstrates how to set up a Spark Structured Streaming job in Databricks Community Edition.It reads new CSV files from a Unity Catalog volume, processes them to count airport...
- 303 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
That's cool ! why not git it ?
- 1 kudos
- 199 Views
- 7 replies
- 0 kudos
Unable to connect to any cluster from a notebook
I'm experiencing an unusual issue following my return from annual leave. I'm unable to connect to any compute from a notebook (both Classic Compute and Serverless) this is despite having Can Manage permissions on the clusters.The error shown is: "Unk...
- 199 Views
- 7 replies
- 0 kudos
- 45 Views
- 0 replies
- 0 kudos
[Urgent] Exam Suspended Without Prior Warning: #00891706
Hi @Cert-Team, @SujithaI was attempting my Databricks Certified Data Engineer Professional exam on 12th April 2026 at 5:45 PM IST. The exam was going smoothly, and I had only 17 questions remaining when it was abruptly suspended. There was no warning...
- 45 Views
- 0 replies
- 0 kudos
- 56 Views
- 1 replies
- 1 kudos
pipeline config DAB
I am deploying DLT pipeline in dev environment using DABs. source code is in a python script file. In the pipeline's yml file the configuration key is set to true(with all correct indentations), yet the pipeline isnt deploying in the continuous mode....
- 56 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @prakharsachan ,Continuous must be set inside the pipeline resource definition, not under configuration.The configuration block in a SDP (former DLT) pipeline definition is for Spark/pipeline settings (key-value string pairs passed to the runtime)...
- 1 kudos
- 54 Views
- 1 replies
- 0 kudos
Driver memory utilization grows continuously during job
I have a batch job that runs thousands of Deep Clone commands, it uses a ForEach task to run multiple Deep Clones in parallel. It was taking a very long time and I realized that the Driver was the main culprit since it was using up all of its memory ...
- 54 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @tsam ,I think your problem might be caused by the fact that each call "CREATE OR REPLACE TABLE ... DEEP CLONE" accumulates state on the driver even though you're not collecting data.The main culprits are:1. Spark Plan / Query Plan Caching Every S...
- 0 kudos
- 75 Views
- 4 replies
- 0 kudos
- 75 Views
- 4 replies
- 0 kudos
- 301 Views
- 6 replies
- 2 kudos
Resolved! Get task_run_id that is nested in a job_run task
Hi, I'm wondering if there is an easier way to accomplish this.I can use Dynamic Value reference to pull the run_id of Parent 1 into Parent 2, however, what I'm looking for is for Child 1's task run_id to be referenced within Parent 2.Currently I am ...
- 301 Views
- 6 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @ChristianRRL you're absolutely right, and I apologize for the earlier suggestion. I've verified that task values from child jobs are not propagated back through run_job tasks. Your instinct about the REST API was correct. Here's the fix: Solutio...
- 2 kudos
- 144 Views
- 2 replies
- 2 kudos
Resolved! Get task_run_id (or job_run_id) of a *launched* job_run task
Hi there, I'm finding this a bit trickier than originally expected and am hoping someone can help me understand if I'm missing something.I have 3 jobs:One orchestrator job (tasks are type run_job)Two "Parent" jobs (tasks are type notebook)parent1 run...
- 144 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi, I ran into the same confusion and did some testing on this. Here's what I found: Task values don't cross the run_job boundary. So even if child1 sets a task value with dbutils.jobs.taskValues.set(), the orchestrator can't read it. But {{tasks.par...
- 2 kudos
- 14578 Views
- 18 replies
- 3 kudos
Is it possible to view Databricks cluster metrics using REST API
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
- 14578 Views
- 18 replies
- 3 kudos
Latest Reply
- 3 kudos
@Walter_C Could you please share if there is any latest update on this ?
- 3 kudos
- 55 Views
- 1 replies
- 0 kudos
Exam Rescheduling
Hi Team,Could you please help with this ticket #00891919Thanks !
- 55 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Kunal55 ,Thank you for filing a ticket with our support team, Support team will respond shortly. Please note that we cannot provide support or handle exam suspensions via community.Thanks & Regards,@cert-ops
- 0 kudos
- 156 Views
- 4 replies
- 0 kudos
Databricks file based trigger to sharepoint
Hi,Can we create a file based trigger from sharepoint location for excel files from databricks. So my need is to copy the excel files from sharepoint to external volumes in databricks so can it be done using a trigger that whenever the file drops in ...
- 156 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
File-based triggers in Databricks are designed to work with data that already resides in cloud storage (such as ADLS, S3, or GCS). In this case, since the source system is SharePoint, expecting a native file-based trigger from Databricks is not feasi...
- 0 kudos
| User | Count |
|---|---|
| 1901 | |
| 992 | |
| 930 | |
| 479 | |
| 351 |
![Community [Industry BrickTalk #4] Transforming Commercial Real Estate Portfolio Management with AI](/t5/image/serverpage/image-id/25529iFB6AFAFAE433451D/image-size/large?v=v2&px=999)
