- 1664 Views
- 4 replies
- 0 kudos
Is there any way to connect polaris catalog from unity catalog?
Hi databricks community, I'd like to access data managed by polaris catalog through unity catalog to manage all data one place. But is there any way to connect? (I could access the data with all-purpose cluster without unity catalog.)
- 1664 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @chandu402240 , we're looking at a very similar problem. Wondering if you were able to access the Polaris catalog from a Databricks cluster? Was the External Location documentation linked by @Alberto_Umana relevant?
- 0 kudos
- 277 Views
- 1 replies
- 1 kudos
Resolved! Injecting Databricks secrets into Databricks Asset Bundles.
Hey,I want to inject Databricks secrets into my Databricks Asset Bundles in order to avoid exposing secrets.I tried it as shown in the code block below but it gives the error below the code block.When I hardcode my instance_profile_arn it does work.H...
- 277 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @daan_dw ,Possible reason for your problem:Databricks Asset Bundles use Terraform under the hood, and Terraform cannot resolve Databricks secret references (like ${secrets.aws_secrets.cluster_profile_arn})at deployment time. Secrets are only acce...
- 1 kudos
- 561 Views
- 4 replies
- 0 kudos
Databricks unable to read ADLS external location
Hey Databricks forum, We are seeing a bit of issue in our azure databricks environment, from this sunday, that we are unable to list the files inside the containers. We have our unity catalogues and all configured in our external location, while we m...
- 561 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @databricks1111,thanks for the extra details The behavior you’re seeing (works fine on personal compute but fails on shared compute) usually comes down to which identity Databricks uses to access Azure Storage.When you use personal compute, opera...
- 0 kudos
- 2179 Views
- 1 replies
- 1 kudos
Resolved! How can I execute a Spark SQL query inside a Unity Catalog Python UDF so I can run downstream ML?
I want to build an LLM-driven chatbot using Agentic AI framework within Databricks. The idea is for the LLM to generate a SQL text string which then passed to a Unity Catalog-registered Python UDF tool. Within this tool, I need the SQL to be execute...
- 2179 Views
- 1 replies
- 1 kudos
- 1 kudos
There is currently no supported method for SQL-defined Python UDFs in Unity Catalog to invoke Spark SQL or access a SparkSession directly from within the SafeSpark sandbox. This limitation is by design: the SafeSpark/Restricted Python Execution Envir...
- 1 kudos
- 229 Views
- 1 replies
- 1 kudos
Resolved! Altair charts don't work in offline mode
Hello,We are running a secure Databricks environment (no internet access) within an Azure Virtual Desktop and are currently unable to get any charts to display. They work if we access the environment from outwith the AVD, but not when we access it fr...
- 229 Views
- 1 replies
- 1 kudos
- 1 kudos
Why It Works Outside AVD When working outside your AVD setup (such as on a local machine or a cloud environment with internet access), the widget JavaScript loads successfully from the CDN, enabling chart display. Solutions for Secure, Offline Envir...
- 1 kudos
- 272 Views
- 4 replies
- 2 kudos
Resolved! call job parameter in notebook
Notebook1 as a list output (d_list) and stored in a taskValueI have provided this as input to a loop for notebook 2These are the parameters for notebook2 How do I get the value of the parameters file_name and file_format inside notebook2?when I try ...
- 272 Views
- 4 replies
- 2 kudos
- 2 kudos
Let me know if you managed to do it @Hritik_Moon in a way I described above. In case of any issues I can prepare for you an end to end example
- 2 kudos
- 432 Views
- 7 replies
- 4 kudos
Dynamic Jobs community Edition
Hello I tried this,Notebook 1 :dbutils.jobs.taskValues.set(key = "my_key", value = "hi From Notebook1")Notebook2:X = dbutils.jobs.taskValues.get(taskKey="01", key="my_key", debugValue = "Fail")print(X) Here I get "Fail" as output, its not fetching m...
- 432 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Ritesh-Dhumne ,Folllow my steps. I created 2 notebooks:- first one called Notebook1 with followign content- second one called Notebook2 with following content that will read value defined in Notebook1 Here's my definiton of workflow that is using...
- 4 kudos
- 191 Views
- 2 replies
- 3 kudos
Jobs and Pipeline input parameter
I wanted to extract all files in the volume I have uploaded , in notebook 1 and then in notebook 2 perform basic transformation on every files like missing values , nulls , also I want to store the null , dirty records seperately and a clean datafram...
- 191 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Ritesh-Dhumne ,I'm assuming that you mistakenly named Free Edition as Community since you're using volumes which are not available in community edition.I’m not sure if I’ve understood your approach correctly, but at first glance it seems incorre...
- 3 kudos
- 333 Views
- 2 replies
- 1 kudos
Resolved! Error filtering by datetime Lakehouse Federated SQL Server table
In unity catalog, I have a connection to a SQL Server database. When I try to filter by a datetime column using a datetime with fractional seconds, Databricks gives me this error:Job aborted due to stage failure: com.microsoft.sqlserver.jdbc.SQLServe...
- 333 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks, @Isi. Very helpful. It would be nice if Lakehouse federation would do this for us (the same way that it knows SQL Server uses ISNULL where Spark SQl uses NVL). Is there a way to bring it the dev's attention?
- 1 kudos
- 373 Views
- 2 replies
- 3 kudos
Resolved! Update Databricks App compute
Hello community!I have been using for a while the new Databricks feature 'Databricks Apps'. It has been incredible the amount of effort and time we have saved by using Databricks Apps to deploy our dashboards instead of deploying them directly in our...
- 373 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @juanjomendez96 ,You’ve explained this really well, and yes, what you’re experiencing is currently one of the main limitations of Databricks Apps.Right now, these apps run on fixed managed compute controlled by Databricks. That means we, as users...
- 3 kudos
- 273 Views
- 2 replies
- 2 kudos
Sharing Databricks Notebook Functionality Without Revealing Source Code
Hi All,I have a unique scenario in Databricks and would appreciate your insights.I’ve developed functionality in Databricks notebooks, and I’d like to share this with other developers within the same workspace. My goal is to allow colleagues to impor...
- 273 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey @Hari_P ,I believe this doesn’t exist today as a built-in feature. I reviewed the Databricks notebook permission model (docs link) and with the minimum level (“CAN READ”) users already have access to view the notebook’s source. The simplest and m...
- 2 kudos
- 1804 Views
- 3 replies
- 1 kudos
Resolved! How to Read and Wrire Data between 2 seperate instance of Databricks
How to Read and Wrire Data between 2 seperate instance of Databricks.I want to have bi-directional data read and write between Databricks A and Databricks B. Both are not in same instance.Please help
- 1804 Views
- 3 replies
- 1 kudos
- 1 kudos
Here are some patterns that you can utilize:1. If the workspaces are in different Databricks Accounts or different Azure regions. The recommended approach is Delta sharing.The simplest, most governed way to let A read B’s tables and B read A’s tables...
- 1 kudos
- 681 Views
- 6 replies
- 3 kudos
Dynamic value input to a job
How do I pass dynamic value to a databricks job?I created a notebook which will extract the names of files in the catalog, I want to pass these names as parameter to another notebook task in a job.What are the ways I can do this?
- 681 Views
- 6 replies
- 3 kudos
- 3 kudos
Hello, I think you have posted wrong picture for notebook1. Could you please verify once.I made some changes and its working now. Thanks a lot.
- 3 kudos
- 538 Views
- 3 replies
- 1 kudos
Resolved! Getting NO_TABLES_IN_PIPELINE error in Lakeflow Declarative Pipelines
Yesterday (10/1) starting around 12 PM EST we starting getting the following error in our Lakeflow Declarative Pipelines (LDP) process. We get this in environments where none of our code has changed. I found some info on the serverless compute abou...
- 538 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @cpollock Check the “Event log” and “Pipeline logs” in the Databricks UI for any clues.also can you please share screenshot as pasted in window, attachment is not really working and only scanning
- 1 kudos
- 1340 Views
- 21 replies
- 17 kudos
Resolved! Copy text from the integrated Web terminal
How do I copy text from the integrated web terminal? The selection goes away as soon as I lift my finger from the mouse.
- 1340 Views
- 21 replies
- 17 kudos
- 17 kudos
An update here. I raised a ticket and found out this is because tmux mode is enabled by default in the web terminal. You can disable the tmux mode by adding `export DISABLE_TMUX=true` to the ~/.bashrc of the compute.When I asked them about the defaul...
- 17 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 484 | |
| 349 | |
| 287 |
