cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

8966 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

374 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

88 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

810 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

496 Posts

Activity in Databricks Platform Discussions

Jaku6
by > • New Contributor
  • 234 Views
  • 1 replies
  • 1 kudos

Run now with different parameters doesn't pass parameter to pipeline tasks

I have a job with some tasks. Some of the tasks are pipeline_task's some are notebook_task's.When I run the job with "Run now with different parameters" and enter a new key-value, I see that the key-value is available in the notebook_task's with dbut...

Data Engineering
Reply
  • 234 Views
  • 1 replies
  • 1 kudos
Reply
Latest Reply
Walter_C
Honored Contributor
  • 1 kudos

As per docs it seems that pipeline task type is currently not supported to pass parameters: https://docs.databricks.com/en/jobs/create-run-jobs.html#pass-parameters-to-a-databricks-job-taskYou could create a notebook task that runs before your pipeli...

  • 1 kudos
data-digger
by > • New Contributor
  • 124 Views
  • 0 replies
  • 0 kudos

Hide function defination

Hi - I would like to hide function definition from user, what is the best way to do it in unity catalog

Data Governance
Reply
  • 124 Views
  • 0 replies
  • 0 kudos
Reply
eric-c
by > • New Contributor
  • 201 Views
  • 1 replies
  • 0 kudos

Alert ACL's not available

The list of available ACL's for alerts suggests there's a way to set "can run" and "can manage" permissions but neither the REST API nor the Databricks CLI show options for setting permissions. Is there a way to set acl's on alerts? If so, how?Alert ...

Administration & Architecture
Reply
  • 201 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

The API call needed to set the ACLs for an alert and other object types as (alerts | dashboards | data_sources | queries) is:https://docs.databricks.com/api/workspace/dbsqlpermissions/set

  • 0 kudos
cadull
by > • New Contributor II
  • 89 Views
  • 1 replies
  • 1 kudos

Permission Issue with IDENTIFIER clause

Hi all,we are parameterizing environment specific catalog names (like `mycatalog_dev` vs. `mycatalog_prd`) in Lakeview dashboard queries like this:SELECT * FROM IDENTIFIER(:catalog_name || '.myschema.mytable')Which works fine in most cases. We have o...

Data Engineering
Reply
  • 89 Views
  • 1 replies
  • 1 kudos
Reply
Latest Reply
madams
New Contributor
  • 1 kudos

I've had quite a bit of fun with UC and view permissions.  I don't think this is specific to using the IDENTIFIER() function, but I suspect it's related to UC permissions.  What you'll need to ensure:The user or group who owns the view on catalog_b h...

  • 1 kudos
tdk
by > • New Contributor
  • 81 Views
  • 1 replies
  • 0 kudos

Cannot install jar to cluster: invalid authority.

Hi allI want to access on-prem Oracle Database data from the python notebooks. However, the install of the jar (ojdbc8.jar) results in an error, which occurs while the cluster is starting up.The error message:"Library installation attempted on the dr...

Data Engineering
Reply
  • 81 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

The error message suggests that the jar file located at abfss:/jars/ojdbc8.jar has an invalid authority. This could be due to a number of reasons such as incorrect file path, insufficient permissions, or network restrictions. Here are a few steps you...

  • 0 kudos
semsim
by > • Contributor
  • 45 Views
  • 0 replies
  • 0 kudos

Can't query Legacy Serving Endpoint

Hi,I was able to deploy an endpoint using legacy serving (It's the only option we have to deploy endpoints in DB). Now I am having trouble querying the endpoint itself. When I try to query it I get the following error:    Here is the code I am using ...

semsim_0-1726245119742.png
Machine Learning
Reply
  • 45 Views
  • 0 replies
  • 0 kudos
Reply
FedericoRaimond
by > • New Contributor
  • 323 Views
  • 7 replies
  • 2 kudos

Azure Databricks Workflows with Git Integration

Hello,I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.As per documentation: "For a Git repository, the path relative to the repository root."Then, I use directly the name of the notebook file...

Data Engineering
Reply
  • 323 Views
  • 7 replies
  • 2 kudos
Reply
Latest Reply
madams
New Contributor
  • 2 kudos

Ah, I have had that same error before when cloning from git.  I'm guessing you got the Repo URL by hitting the "clone" button in ADO and copy/pasting it in to Databricks.  One thing I've done in the past is in the Repository URL, remove the "org@" pa...

  • 2 kudos
6 More Replies
Visakh_Vijayan
by > • New Contributor
  • 114 Views
  • 3 replies
  • 0 kudos

Uninstall whl file from databricks cluster via CLI

Hello, we have a need to uninstall older versions of whl files from personal cluster via databricks CLI - could you please provide the exact command to be used here. We tried with many found on the documentations but none of them worked to do the act...

Machine Learning
Reply
  • 114 Views
  • 3 replies
  • 0 kudos
Reply
Latest Reply
szymon_dybczak
Contributor
  • 0 kudos

Hi @Visakh_Vijayan ,Did you try to use databricks libraries uninstall? It's exactly crafted for this purposedatabricks libraries uninstall --json YOUR_JSON_WITH_REQUEST_BODYAlso, when you uninstall a library from a cluster, the library is removed onl...

  • 0 kudos
2 More Replies
hyedesign
by > • New Contributor II
  • 1619 Views
  • 6 replies
  • 0 kudos

Getting SparkConnectGrpcException: (java.io.EOFException) error when using foreachBatch

Hello, I am trying to write a simple upsert statement following the steps in the tutorials. here is what my code looks like:from pyspark.sql import functions as Fdef upsert_source_one(self df_source = spark.readStream.format("delta").table(self.so...

Data Engineering
pyspark
Reply
  • 1619 Views
  • 6 replies
  • 0 kudos
Reply
Latest Reply
seans
New Contributor III
  • 0 kudos

I figured out the error is hiding an underlying issue with the code, which you can get to if you deploy the bundle (if you are using asset bundles) and run from a notebook in a browser.So the issue is more about the debugger not being able to stop on...

  • 0 kudos
5 More Replies
juan_barreto
by > • New Contributor
  • 179 Views
  • 1 replies
  • 1 kudos

Problem with dropDuplicates in Databricks runtime 15.4LTS

Hi,I'm testing the latest version of the databricks runtime but I'm getting errors doing a simple dropDuplicates.Using the following codedata = spark.read.table("some_table") data.dropDuplicates(subset=['SOME_COLUMN']).count() I'm getting this error....

juan_barreto_0-1726153266526.png
Data Engineering
Reply
  • 179 Views
  • 1 replies
  • 1 kudos
Reply
Latest Reply
kellys
New Contributor
  • 1 kudos

Wanted to add to this thread. Seeing the same issue. This appears to be recent problem.

  • 1 kudos
Elderion
by > • New Contributor II
  • 104 Views
  • 3 replies
  • 1 kudos

Resolved! Delta Live Tables + Databricks Assets Bundles

Hi,I'm trying to setup CICD pipeline for Delta Live Table jobs using Databricks Bundles. I have a problem with path to notebook in pipeline. According to this example:https://docs.databricks.com/en/delta-live-tables/tutorial-bundles.htmlYAML file sho...

Elderion_0-1726152706169.png
Data Engineering
Reply
  • 104 Views
  • 3 replies
  • 1 kudos
Reply
Latest Reply
ThierryBa
New Contributor
  • 1 kudos

I had this error once.you need to specify the extension of your file. If you set the notebook to be python, then it must be .py at the end, likewise .sql if you used SQL     libraries:        - notebook:            path:  ${workspace.file_path}/datab...

  • 1 kudos
2 More Replies
Kurtis_R
by > • New Contributor II
  • 93 Views
  • 2 replies
  • 0 kudos

Excel Formula results

Hi all,Just wanted to raise a question regarding Databricks workbooks and viewing the results in the cells. For the example provided in the screenshot I want to view the results of an excel formula that has been applied to a cell in our workbooks. Fo...

Kurtis_R_0-1725568966348.png Kurtis_R_1-1725569630650.png
Data Engineering
Reply
  • 93 Views
  • 2 replies
  • 0 kudos
Reply
Latest Reply
User16756723392
New Contributor III
  • 0 kudos

@Kurtis_R do you want to display the value of 45 or formula of how 45 is achieved.?

  • 0 kudos
1 More Replies
Mario_D
by > • New Contributor
  • 40 Views
  • 1 replies
  • 0 kudos

Foreign key constraint in a dlt pipeline

As primary/foreign key constraints are now supported/available in Databricks, how are foreign key constraints handled in a dlt pipeline, i.e if a foreign key constraint is violated, is the record logged as a data quality issue and still added to the ...

Data Engineering
Reply
  • 40 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
szymon_dybczak
Contributor
  • 0 kudos

Hi @Mario_D ,I don't think primary/foreign constraint are supported by DLT. At least I can't find anything in documentation. But you obtain same result using DLT expectations:Manage data quality with Delta Live Tables - Azure Databricks | Microsoft L...

  • 0 kudos
Databrixer
by > • New Contributor
  • 193 Views
  • 2 replies
  • 4 kudos

Changes in Databricks rest api

Hi all, Just wanted to ask if there is an announcement regarding changes on Databricks REST API?  Because I had weird experiences in using the REST API. Specifically for Query History API Last Aug 29, 2024. I created a script to pull get request in t...

Administration & Architecture
Reply
  • 193 Views
  • 2 replies
  • 4 kudos
Reply
Latest Reply
szymon_dybczak
Contributor
  • 4 kudos

Yeah, totally agree with you. It should be documented/mentioned somewhere. Or maybe the API should be versioned, so if they introduce a new version, it won't break existing workflows.Anyway, thanks for sharing!

  • 4 kudos
1 More Replies
jacobholland
by > • New Contributor
  • 73 Views
  • 2 replies
  • 0 kudos

Removing Admin Users

Hello, I am trying to deal with something I thought would be straightforward, but am hitting some walls. Basically the original user associated with my Databricks account has left my organisation and, when trying to remove / disable this user, I am m...

Administration & Architecture
Reply
  • 73 Views
  • 2 replies
  • 0 kudos
Reply
Latest Reply
jacobholland
New Contributor
  • 0 kudos

Just to follow-up; I can administer & disable OTHER account admins and do the needful in every other case, but this one user account is particularly privileged.

  • 0 kudos
1 More Replies
Load more

Featured Posts

View all