cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

TamD
by Contributor
  • 2895 Views
  • 0 replies
  • 0 kudos

ModuleNotFoundError Importing fuction modules to DLT pipelines

Following best practice, we want to avoid reusing code by putting commonly used transformations into function libraries and then importing and calling those functions where required.We also want to follow Databricks recommendations to use serverless ...

  • 2895 Views
  • 0 replies
  • 0 kudos
ozaaditya
by Contributor
  • 766 Views
  • 1 replies
  • 0 kudos

How to Configure a Policy in Job Cluster Using Databricks Linked Service in ADF?

I am trying to configure a policy in a job cluster using Databricks Linked Service in Azure Data Factory (ADF). The goal is to restrict access to a specific group or individuals who can use this cluster.I want to know:How to define and enforce a clus...

  • 766 Views
  • 1 replies
  • 0 kudos
Latest Reply
cgrant
Databricks Employee
  • 0 kudos

In the Linked Services panel in the Azure Data Factory UI, find your Databricks Linked Service and click the squiggly braces {} to edit the service. From here, you'll see a definition in JSON. Create a new key: properties.typeProperties.policyId and ...

  • 0 kudos
Dave_Nithio
by Contributor II
  • 2798 Views
  • 0 replies
  • 0 kudos

Preset Partner Connect Schema Changes

When using partner connect to connect Serverless Databricks to my BI tool Preset, you must manually define the schema that Preset has access to. In my case, I individually selected all databases currently in my hive_metastore:The problem is, once cre...

Dave_Nithio_0-1742329042090.png
  • 2798 Views
  • 0 replies
  • 0 kudos
dprutean
by New Contributor III
  • 14709 Views
  • 5 replies
  • 5 kudos

Resolved! JDBC Driver support for OpenJDK 17

Connecting to Databricks using OpenJDK 17 I got the exception below. Are there any plans to fix the driver for OpenJDK17?java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500540) Error caught in BackgroundFetcher. Foreground thread ID: 44. Ba...

  • 14709 Views
  • 5 replies
  • 5 kudos
Latest Reply
ameyabapat
New Contributor II
  • 5 kudos

I still see the above error with databricks jdbc driver 2.6.33. Anyone aware of fix available either in  driver or java? 

  • 5 kudos
4 More Replies
kulasangar
by New Contributor II
  • 2805 Views
  • 0 replies
  • 0 kudos

Permission Denied while trying to update a yaml file within a python project in Databricks

I have a python project and within that I do have a yaml file. Currently i'm building the project using poetry and creating an asset bundle to deploy it in Databricks as a workflow job.So when the workflow runs, I do have an __init__.py within my ent...

kulasangar_0-1742321907764.png
  • 2805 Views
  • 0 replies
  • 0 kudos
saraM
by New Contributor III
  • 593 Views
  • 1 replies
  • 0 kudos

Resolved! My databricks Exam still suspended request: 00635015

My exam has been suspended, and I received an email to schedule it. But I see that there is a delay of 10 to 15 days. Is it possible to schedule it for 19 March  at 11 a.m.?because I have a deadline that I must not exceed to obtain this certificate.o...

  • 593 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @saraM! Thank you for reaching out to us again and confirming.As you requested, the support team has rescheduled your exam for March 19, 2025, at 11:00 AM in Europe/Paris.Please log into your account and confirm. The rescheduled exam will be re...

  • 0 kudos
kulasangar
by New Contributor II
  • 987 Views
  • 2 replies
  • 1 kudos

ERROR: Could not find a version that satisfies the requirement

I'm using Databricks workflows along with asset bundle to run my pipeline and recently we moved our pipeline from development to acceptance. We are also using JFrog as our package artifactory.Even though I do see the release candidate version under t...

kulasangar_0-1741255645400.png
  • 987 Views
  • 2 replies
  • 1 kudos
Latest Reply
kulasangar
New Contributor II
  • 1 kudos

Thank you for the response. Apparently one of our internal library failed to install, since an init script was missing on the cluster. After adding that, this got resolved

  • 1 kudos
1 More Replies
nikhil_2212
by New Contributor II
  • 1072 Views
  • 1 replies
  • 0 kudos

Lakehouse monitoring metrices tables not created automatically

Hello,I have an external table created in databricks unity catalog workspace and trying to "Create a monitor" for the same from quality tab.While creating the same the dashboard is getting created however the two metrices tables "profile" & "drift" a...

  • 1072 Views
  • 1 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 0 kudos

Hi Nikhil,How are you doing today?, As per my understanding, It looks like the monitoring setup isn't fully completing, likely due to permissions, external table limitations, or system settings. First, check if your Unity Catalog schema has the right...

  • 0 kudos
lottiejanedev
by New Contributor
  • 441 Views
  • 1 replies
  • 0 kudos

Liquid Clustering with Derived / Generated Fields

I'm having issues with liquid clustering not working when creating derived/generated fields in an asset bundle. Are there any known limitations here? 

  • 441 Views
  • 1 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 0 kudos

Hi Lottiejanedev,How are you doing today?, As per my understanding, Yes, Liquid Clustering works best with directly stored columns, and it might not behave as expected when clustering on derived or generated fields. One possible issue is that generat...

  • 0 kudos
labromb
by Contributor
  • 2085 Views
  • 2 replies
  • 2 kudos

Capturing notebook return codes in databricks jobs

Hi, I currently am running a number of notebook jobs from Azure Data Factory. A new requirement has come up where I need to capture a return code in ADF that has been generated from the note. I tried using  dbutils.notebook.exit(json.dumps({"return_v...

  • 2085 Views
  • 2 replies
  • 2 kudos
Latest Reply
JaviRuiz
New Contributor III
  • 2 kudos

I tried dbutils.notebook.exit(<value>) where value is 0 and 1. I also tried sys.exit(<value>). None of this produces a difference in the output of the databricks CLI commanddatabricks jobs run-now --json '{"job_id": <JOB_ID>, "job_parameters": {"will...

  • 2 kudos
1 More Replies
jim12321
by New Contributor II
  • 5341 Views
  • 4 replies
  • 1 kudos

Databricks CLI how to start a job and pass the parameters?

I try to start a job ID 85218616788189 and pass one parameters 'demo' in Windows Shell.This works:databricks jobs run-now 85218616788189  If I try this one,databricks jobs run-now --json '{"job_id":85218616788189,"notebook_params": {"demo":"parameter...

jim12321_0-1710267172994.png
  • 5341 Views
  • 4 replies
  • 1 kudos
Latest Reply
JaviRuiz
New Contributor III
  • 1 kudos

If the job itself has parameters you have to use 'job_parameters' instead. No need for escaping as of version 0.243.0databricks jobs run-now --json '{"job_id": <JOB_ID>, "job_parameters": {"will_exit": "yes"}}' 

  • 1 kudos
3 More Replies
saraM
by New Contributor III
  • 373 Views
  • 1 replies
  • 0 kudos

My databricks exam got suspened on 18th march and it is still in the suspened state.

My databricks exam got suspened on 18th march and it is still in the suspened state. I have raised a support request using the below link.https://help.databricks.com/s/contact-us?ReqType=trainingand my request id : #00635015 I would really like to ge...

  • 373 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @saraM! It looks like this post duplicates the one you recently posted. The Original post has already been answered. I recommend continuing the discussion there to keep the conversation focused and organised.

  • 0 kudos
ewe
by New Contributor III
  • 2697 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks apps (streamlit) not able to install python libs

So, I have a databricks streamlit app that is not able to install any python lib defined on the requirements.txt.Issue is not specific to one lib, tried other ones but no python lib can be installed. Anyone with similar issue to help ? [2025-02-19 10...

  • 2697 Views
  • 3 replies
  • 2 kudos
Latest Reply
ewe
New Contributor III
  • 2 kudos

Hi, returning to inform how the issue was solved. A bit of context, looks like the databricks apps runs on the databricks/microsoft backend, and not on our clusters. The control of the network ingress rules is done via SEG (https://www.databricks.com...

  • 2 kudos
2 More Replies
Upendra_Dwivedi
by Contributor
  • 1196 Views
  • 3 replies
  • 0 kudos

ON-Prem SQL Server Direct Connection with Azure Databricks

Hi All,I have SSMS installed and have some data there which i want to export to Databricks and process it there. Databricks is hosted on Azure Cloud and I am wondering if it is possible. I have tested it using JDBC connection but i am getting error: ...

  • 1196 Views
  • 3 replies
  • 0 kudos
Latest Reply
MariuszK
Valued Contributor III
  • 0 kudos

If you want to play with SQL Server the easy way is to use Azure SQL that will be visuable for Databricks. Alternativly you can use ADF with slef-hosted run time to extract data to Azure from SQL Server.Network configuration requires many steps and i...

  • 0 kudos
2 More Replies
Mumrel
by Contributor
  • 7699 Views
  • 5 replies
  • 1 kudos

Resolved! Usage of forum: How to find the threads where I commented on and my bookmarks

Hi quick questions, 1) the other day I commented on a thread and cannot find it. Is there a feature to find all my posts? I cannot find it under my profile.2) I find interesting threads and want to watch (in the sense: email me when updates occure) t...

  • 7699 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Jan St.​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your ...

  • 1 kudos
4 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels