- 449 Views
- 7 replies
- 4 kudos
Resolved! [CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)
Hello,after databricks update the Runtime from Release: 15.4.24 to Release: 15.4.25 we getting in all jobs the Error:[CANNOT_OPEN_SOCKET] Can not open socket: ["tried to connect to ('127.0.0.1', 45287)What we can do here?Greetings
- 449 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Hansjoerg,Apologies for the confusion earlier. You are right Bundles doesn't allow pinning to specific patch versions like 15.4.24.Your best option is to skip Bundles for now and use the regular Databricks Jobs setup (via UI or Jobs API) where yo...
- 4 kudos
- 2238 Views
- 1 replies
- 1 kudos
Resolved! Syncing lakebase table to delta table
I have been exploring Lakebase and I wanted to know if there is a way to sync CDC data from Lakebase tables to delta table in Lakehouse. I know the other way is possible and that's what was shown in the demo. Can you tell how I can I sync both the ta...
- 2238 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey AkhileshVB! Lakebase-to-Delta CDC sync is in Private Preview—GA/Preview dates are not firm yet. Doo you have a Databricks contact or account manager? They are the right place to go if you want early involvement. Workarounds & DIY Approaches: Fo...
- 1 kudos
- 315 Views
- 5 replies
- 2 kudos
Read files from adls in databricks
I have unity catalogue access connector but its not enabled as i have only admin access so i dont have access to the admin portal to enable this as its need global admin permissions.I am trying to read adls json data in databricks by using service pr...
- 315 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @SuMiT1 once networking issue is resolved , also make sure your service principal has at least Storage Blob Data Reader on the storage account/container.
- 2 kudos
- 401 Views
- 3 replies
- 2 kudos
Convert notebook to Python library
Looking for ways to convert a Databricks notebook to Python library. Some context :Don't want to give execute permissions to shared notebooks as we want to hide code from users.Proposed solution is to have our shared notebook converted into a Python ...
- 401 Views
- 3 replies
- 2 kudos
- 2 kudos
The best way to share code from a Databricks notebook as a reusable module while hiding implementation details from users—without using wheels or granting direct notebook execution permissions—is to convert your notebook into a Python module, store i...
- 2 kudos
- 293 Views
- 2 replies
- 2 kudos
Best Practice: Data Modeling for Customer 360 with Refined/Gold Source Data
Hi community,I'm looking for advice on the best data modeling approach for a Customer 360 (C360) project where our source data is already highly refined.I understand the standard Medallion architecture guidelines, which often recommend using Data Vau...
- 293 Views
- 2 replies
- 2 kudos
- 2 kudos
@BS_THE_ANALYST Thank you so much for your response.The goal is to keep it flexible as a platform rather than a data product mindset. Keeping this in mind, essentially the customer data platform should enable contribution from different teams prevent...
- 2 kudos
- 269 Views
- 3 replies
- 0 kudos
SQL Editor Auto complete
HiFrom time to time the SQL Editor Auto complete works and from time to time not.few times it knows the table columns and few time not - what is the rule for it?
- 269 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @pinikrisher ,To be honest I didn't notice this behaviour. Are you using SQL Editor v2 or legacy one?
- 0 kudos
- 516 Views
- 4 replies
- 4 kudos
How to Read Shared Drive Data in Databricks
Hi everyone,I am working on a project where the data is stored on a Shared Drive. How can I read an Excel file from the Shared Drive into a Databricks notebook?Thanks,
- 516 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Akshay_Petkar ,Could you provide more information. Share drive is pretty broad term. It could be Windows SMB / CIFS share , AWS FSx,, Google Shared Drive etc.
- 4 kudos
- 4387 Views
- 5 replies
- 3 kudos
Set time out or Auto termination for long running query
Hi ,We want to set auto termination for long running queries in data bricks adhoc cluster.I attempted below two approaches in my notebook. Despite my understanding that queries should automatically terminate after one hour, with both the approaches q...
- 4387 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @NehaR are the configurations(spark.sql.broadcastTimeout or spark.sql.execution.timeout) working when we set at cluster level. I am currently trying to do a similar configuration for compute clusters but it dosent work.
- 3 kudos
- 204 Views
- 1 replies
- 0 kudos
split parse_url output for the information
Hi All,I have data in blobs which I am loading from blob store to Databricks delta tables. One of the blob types contains urls. From the Urls I want to extract knowledge from the path and query parts I can get those out easily with parse url. the pro...
- 204 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @turagittech ,Honestly, it all depends on how complex your URLs can get.UDFs will always be more flexible but less performant than native SQL functions.That said, if your team mainly works with SQL, trying to solve it natively in Databricks SQL...
- 0 kudos
- 188 Views
- 1 replies
- 0 kudos
Create wheels and install/configure automation
Can a notebook be created that pushes new versions of code w/o having to go thru the manual process of creating a whl and other configuration files? In other words, can I create a notebook that will setup/configure and install the wheel? So far all t...
- 188 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @adrianhernandez ,Technically yes, but it’s not recommended.You could technically build everything needed to compile the wheel directly from a Databricks notebook using a setup.py, and store it in a volume, CodeArtifact, or any supported cloud st...
- 0 kudos
- 434 Views
- 5 replies
- 1 kudos
Table ID not preserved using CREATE OR REPLACE TABLE
The When to replace a table documentation states that using CREATE OR REPLACE TABLE should preserve the table’s identity:Table contents are replaced, but the table identity is maintained.However, in my recent test the table ID changed after running t...
- 434 Views
- 5 replies
- 1 kudos
- 1 kudos
@Krishna_S thanks for your reply. In a non UC-managed table, is it valid to see a table ID change throughout the life time of the table?(Also, what value gives me to utilize UC to manage my tables?)
- 1 kudos
- 1606 Views
- 4 replies
- 0 kudos
Is there any way to connect polaris catalog from unity catalog?
Hi databricks community, I'd like to access data managed by polaris catalog through unity catalog to manage all data one place. But is there any way to connect? (I could access the data with all-purpose cluster without unity catalog.)
- 1606 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @chandu402240 , we're looking at a very similar problem. Wondering if you were able to access the Polaris catalog from a Databricks cluster? Was the External Location documentation linked by @Alberto_Umana relevant?
- 0 kudos
- 180 Views
- 1 replies
- 1 kudos
Resolved! Injecting Databricks secrets into Databricks Asset Bundles.
Hey,I want to inject Databricks secrets into my Databricks Asset Bundles in order to avoid exposing secrets.I tried it as shown in the code block below but it gives the error below the code block.When I hardcode my instance_profile_arn it does work.H...
- 180 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @daan_dw ,Possible reason for your problem:Databricks Asset Bundles use Terraform under the hood, and Terraform cannot resolve Databricks secret references (like ${secrets.aws_secrets.cluster_profile_arn})at deployment time. Secrets are only acce...
- 1 kudos
- 446 Views
- 4 replies
- 0 kudos
Databricks unable to read ADLS external location
Hey Databricks forum, We are seeing a bit of issue in our azure databricks environment, from this sunday, that we are unable to list the files inside the containers. We have our unity catalogues and all configured in our external location, while we m...
- 446 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey @databricks1111,thanks for the extra details The behavior you’re seeing (works fine on personal compute but fails on shared compute) usually comes down to which identity Databricks uses to access Azure Storage.When you use personal compute, opera...
- 0 kudos
- 2155 Views
- 1 replies
- 1 kudos
Resolved! How can I execute a Spark SQL query inside a Unity Catalog Python UDF so I can run downstream ML?
I want to build an LLM-driven chatbot using Agentic AI framework within Databricks. The idea is for the LLM to generate a SQL text string which then passed to a Unity Catalog-registered Python UDF tool. Within this tool, I need the SQL to be execute...
- 2155 Views
- 1 replies
- 1 kudos
- 1 kudos
There is currently no supported method for SQL-defined Python UDFs in Unity Catalog to invoke Spark SQL or access a SparkSession directly from within the SafeSpark sandbox. This limitation is by design: the SafeSpark/Restricted Python Execution Envir...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 469 | |
| 349 | |
| 287 |
