- 219 Views
- 1 replies
- 0 kudos
split parse_url output for the information
Hi All,I have data in blobs which I am loading from blob store to Databricks delta tables. One of the blob types contains urls. From the Urls I want to extract knowledge from the path and query parts I can get those out easily with parse url. the pro...
- 219 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @turagittech ,Honestly, it all depends on how complex your URLs can get.UDFs will always be more flexible but less performant than native SQL functions.That said, if your team mainly works with SQL, trying to solve it natively in Databricks SQL...
- 0 kudos
- 204 Views
- 1 replies
- 0 kudos
Create wheels and install/configure automation
Can a notebook be created that pushes new versions of code w/o having to go thru the manual process of creating a whl and other configuration files? In other words, can I create a notebook that will setup/configure and install the wheel? So far all t...
- 204 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @adrianhernandez ,Technically yes, but it’s not recommended.You could technically build everything needed to compile the wheel directly from a Databricks notebook using a setup.py, and store it in a volume, CodeArtifact, or any supported cloud st...
- 0 kudos
- 469 Views
- 5 replies
- 1 kudos
Table ID not preserved using CREATE OR REPLACE TABLE
The When to replace a table documentation states that using CREATE OR REPLACE TABLE should preserve the table’s identity:Table contents are replaced, but the table identity is maintained.However, in my recent test the table ID changed after running t...
- 469 Views
- 5 replies
- 1 kudos
- 1 kudos
@Krishna_S thanks for your reply. In a non UC-managed table, is it valid to see a table ID change throughout the life time of the table?(Also, what value gives me to utilize UC to manage my tables?)
- 1 kudos
- 1621 Views
- 4 replies
- 0 kudos
Is there any way to connect polaris catalog from unity catalog?
Hi databricks community, I'd like to access data managed by polaris catalog through unity catalog to manage all data one place. But is there any way to connect? (I could access the data with all-purpose cluster without unity catalog.)
- 1621 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @chandu402240 , we're looking at a very similar problem. Wondering if you were able to access the Polaris catalog from a Databricks cluster? Was the External Location documentation linked by @Alberto_Umana relevant?
- 0 kudos
- 201 Views
- 1 replies
- 1 kudos
Resolved! Injecting Databricks secrets into Databricks Asset Bundles.
Hey,I want to inject Databricks secrets into my Databricks Asset Bundles in order to avoid exposing secrets.I tried it as shown in the code block below but it gives the error below the code block.When I hardcode my instance_profile_arn it does work.H...
- 201 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @daan_dw ,Possible reason for your problem:Databricks Asset Bundles use Terraform under the hood, and Terraform cannot resolve Databricks secret references (like ${secrets.aws_secrets.cluster_profile_arn})at deployment time. Secrets are only acce...
- 1 kudos
- 510 Views
- 4 replies
- 0 kudos
Databricks unable to read ADLS external location
Hey Databricks forum, We are seeing a bit of issue in our azure databricks environment, from this sunday, that we are unable to list the files inside the containers. We have our unity catalogues and all configured in our external location, while we m...
- 510 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @databricks1111,thanks for the extra details The behavior you’re seeing (works fine on personal compute but fails on shared compute) usually comes down to which identity Databricks uses to access Azure Storage.When you use personal compute, opera...
- 0 kudos
- 2164 Views
- 1 replies
- 1 kudos
Resolved! How can I execute a Spark SQL query inside a Unity Catalog Python UDF so I can run downstream ML?
I want to build an LLM-driven chatbot using Agentic AI framework within Databricks. The idea is for the LLM to generate a SQL text string which then passed to a Unity Catalog-registered Python UDF tool. Within this tool, I need the SQL to be execute...
- 2164 Views
- 1 replies
- 1 kudos
- 1 kudos
There is currently no supported method for SQL-defined Python UDFs in Unity Catalog to invoke Spark SQL or access a SparkSession directly from within the SafeSpark sandbox. This limitation is by design: the SafeSpark/Restricted Python Execution Envir...
- 1 kudos
- 187 Views
- 1 replies
- 1 kudos
Altair charts don't work in offline mode
Hello,We are running a secure Databricks environment (no internet access) within an Azure Virtual Desktop and are currently unable to get any charts to display. They work if we access the environment from outwith the AVD, but not when we access it fr...
- 187 Views
- 1 replies
- 1 kudos
- 1 kudos
Why It Works Outside AVD When working outside your AVD setup (such as on a local machine or a cloud environment with internet access), the widget JavaScript loads successfully from the CDN, enabling chart display. Solutions for Secure, Offline Envir...
- 1 kudos
- 232 Views
- 4 replies
- 2 kudos
Resolved! call job parameter in notebook
Notebook1 as a list output (d_list) and stored in a taskValueI have provided this as input to a loop for notebook 2These are the parameters for notebook2 How do I get the value of the parameters file_name and file_format inside notebook2?when I try ...
- 232 Views
- 4 replies
- 2 kudos
- 2 kudos
Let me know if you managed to do it @Hritik_Moon in a way I described above. In case of any issues I can prepare for you an end to end example
- 2 kudos
- 362 Views
- 7 replies
- 3 kudos
Dynamic Jobs community Edition
Hello I tried this,Notebook 1 :dbutils.jobs.taskValues.set(key = "my_key", value = "hi From Notebook1")Notebook2:X = dbutils.jobs.taskValues.get(taskKey="01", key="my_key", debugValue = "Fail")print(X) Here I get "Fail" as output, its not fetching m...
- 362 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi @Ritesh-Dhumne ,Folllow my steps. I created 2 notebooks:- first one called Notebook1 with followign content- second one called Notebook2 with following content that will read value defined in Notebook1 Here's my definiton of workflow that is using...
- 3 kudos
- 167 Views
- 2 replies
- 3 kudos
Jobs and Pipeline input parameter
I wanted to extract all files in the volume I have uploaded , in notebook 1 and then in notebook 2 perform basic transformation on every files like missing values , nulls , also I want to store the null , dirty records seperately and a clean datafram...
- 167 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Ritesh-Dhumne ,I'm assuming that you mistakenly named Free Edition as Community since you're using volumes which are not available in community edition.I’m not sure if I’ve understood your approach correctly, but at first glance it seems incorre...
- 3 kudos
- 306 Views
- 2 replies
- 1 kudos
Resolved! Error filtering by datetime Lakehouse Federated SQL Server table
In unity catalog, I have a connection to a SQL Server database. When I try to filter by a datetime column using a datetime with fractional seconds, Databricks gives me this error:Job aborted due to stage failure: com.microsoft.sqlserver.jdbc.SQLServe...
- 306 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks, @Isi. Very helpful. It would be nice if Lakehouse federation would do this for us (the same way that it knows SQL Server uses ISNULL where Spark SQl uses NVL). Is there a way to bring it the dev's attention?
- 1 kudos
- 320 Views
- 2 replies
- 3 kudos
Resolved! Update Databricks App compute
Hello community!I have been using for a while the new Databricks feature 'Databricks Apps'. It has been incredible the amount of effort and time we have saved by using Databricks Apps to deploy our dashboards instead of deploying them directly in our...
- 320 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @juanjomendez96 ,You’ve explained this really well, and yes, what you’re experiencing is currently one of the main limitations of Databricks Apps.Right now, these apps run on fixed managed compute controlled by Databricks. That means we, as users...
- 3 kudos
- 243 Views
- 2 replies
- 2 kudos
Sharing Databricks Notebook Functionality Without Revealing Source Code
Hi All,I have a unique scenario in Databricks and would appreciate your insights.I’ve developed functionality in Databricks notebooks, and I’d like to share this with other developers within the same workspace. My goal is to allow colleagues to impor...
- 243 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey @Hari_P ,I believe this doesn’t exist today as a built-in feature. I reviewed the Databricks notebook permission model (docs link) and with the minimum level (“CAN READ”) users already have access to view the notebook’s source. The simplest and m...
- 2 kudos
- 1786 Views
- 3 replies
- 1 kudos
Resolved! How to Read and Wrire Data between 2 seperate instance of Databricks
How to Read and Wrire Data between 2 seperate instance of Databricks.I want to have bi-directional data read and write between Databricks A and Databricks B. Both are not in same instance.Please help
- 1786 Views
- 3 replies
- 1 kudos
- 1 kudos
Here are some patterns that you can utilize:1. If the workspaces are in different Databricks Accounts or different Azure regions. The recommended approach is Delta sharing.The simplest, most governed way to let A read B’s tables and B read A’s tables...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 480 | |
| 349 | |
| 287 |
