cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

HariUmeshNaraya
by New Contributor II
  • 108 Views
  • 3 replies
  • 0 kudos

databricks-langchain PyPI installation stuck for 30+ minutes on multiple Azure Databricks clusters

Team,I am experiencing an issue installing the databricks-langchain Python package from PyPI on Azure Databricks.The package installation remains in the installing/resolving state for more than 30 minutes and does not complete.Testing performed:Origi...

  • 108 Views
  • 3 replies
  • 0 kudos
Latest Reply
ivanvyd
New Contributor II
  • 0 kudos

@HariUmeshNaraya you shouldn't need to install langchain or langchain-core beforehand. A normal installation of databricks-langchain resolves and installs its declared dependencies automatically.You can keep using the cluster's Libraries tab for note...

  • 0 kudos
2 More Replies
geramkumar
by New Contributor II
  • 81 Views
  • 2 replies
  • 1 kudos

Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation?

 Problem Statement:I have a Databricks Asset Bundles (DAB) project with the following requirement:I need to create an all-purpose cluster whose configuration references:An init script stored at /Volumes/<catalog>/<schema>/<volume>/scripts/init_script...

  • 81 Views
  • 2 replies
  • 1 kudos
Latest Reply
geramkumar
New Contributor II
  • 1 kudos

Thank you Sumesh. Let me explore the second option and check the latest DAB documentation aswell. Keep up the good work

  • 1 kudos
1 More Replies
rhlp
by New Contributor
  • 66 Views
  • 1 replies
  • 0 kudos

Allow Public Network Access - Disabled, I can still access the workspace from public internet

Hi everyone,We configured Private Link for an existing Azure Databricks workspace (Hybrid workspace):VNet injection and Secure Cluster Connectivity enableddatabricks_ui_api private endpoint createdpublicNetworkAccess = DisabledrequiredNsgRules = NoAz...

  • 66 Views
  • 1 replies
  • 0 kudos
Latest Reply
Gokul_Pillai1
Databricks Employee
  • 0 kudos

Hi @rhlp ,This doesnt sound like the expected behaviour if the request is genuinely reaching the workspace API from the public internet.A couple of things I would check first before coming into a conclusion Creating the databricks_ui_api private endp...

  • 0 kudos
jjjjjahs
by Visitor
  • 75 Views
  • 1 replies
  • 0 kudos

Databricks App suddenly unable to load valid Jupyter notebook

Hello Databricks Support,I am experiencing an issue with a Databricks App that was previously working correctly.The App has recently started failing when trying to load one of the Jupyter notebooks in its source directoryFailed 11 minutes ago: Error ...

  • 75 Views
  • 1 replies
  • 0 kudos
Latest Reply
SumeshKashyap
New Contributor III
  • 0 kudos

That specific error — "error downloading source code... may not be a valid notebook or may be an unsupported version" — comes from the Apps source-sync step's nbformat validation, which is stricter than the normal Workspace notebook viewer/downloader...

  • 0 kudos
abhi_dabhi
by Databricks Partner
  • 4078 Views
  • 5 replies
  • 0 kudos

Unable to Change Pipeline Ownership to Service Principal Despite Admin Permissions

Hello Databricks Community,I am facing an issue with changing the ownership of an ETL pipeline to a service principal. Despite being the Databricks administrator with workspace and metastore admin permissions, I receive the following error: "Only adm...

  • 4078 Views
  • 5 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Are you saying that you can't create a PAT token in the Databricks Workspace, or you can create a PAT token but don't have a secure way of using it with the API request? Please explain a bit more.  Thanks, Lou.

  • 0 kudos
4 More Replies
Marbricks
by New Contributor II
  • 244 Views
  • 4 replies
  • 3 kudos

Using Temporary Functions

Hi everyone,I'm currently doing my professional internship and working on a reporting project that integrates with Databricks through Unity Catalog functions.The reporting application is external to Databricks and uses Data Sources, Parameters, Resul...

  • 244 Views
  • 4 replies
  • 3 kudos
Latest Reply
data_pulse
New Contributor II
  • 3 kudos

@Marbricks Follow this approach: define the function’s interface first, then connect the real data. Use a temporary function for testing within the same session, and a persistent Unity Catalog function in a development schema for testing with the ext...

  • 3 kudos
3 More Replies
FastFoodBro
by New Contributor
  • 226 Views
  • 10 replies
  • 1 kudos

Can I run jobs continuously without interruption on a permanent free (Community Edition) account?

Chào mọi người,Tôi hiện đang sử dụng Databricks Community Edition (tài khoản miễn phí) và muốn hỏi về việc chạy các tác vụ theo lịch trình/liên tục trên đó.Cụ thể: 1. Liệu có thể chạy một tác vụ liên tục (ví dụ: tác vụ xử lý dữ liệu trực tuyến hoặc t...

  • 226 Views
  • 10 replies
  • 1 kudos
Latest Reply
balajij8
Esteemed Contributor II
  • 1 kudos

@FastFoodBro You can use Databricks Free account for personal use only (personal use, learning, experimentation - not for production or commercial use). You can upgrade to a paid plan to access full platform features as you will face interruptions in...

  • 1 kudos
9 More Replies
LokeshChikuru
by Databricks Partner
  • 361 Views
  • 4 replies
  • 0 kudos

Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via

We are using Azure Databricks Serverless Compute to connect to Azure SQL Managed Instance (SQL MI) through Network Connectivity Configuration (NCC) with Private Endpoint Rules. Connectivity works successfully when using the primary SQL MI FQDN. Howev...

  • 361 Views
  • 4 replies
  • 0 kudos
Latest Reply
LokeshChikuru
Databricks Partner
  • 0 kudos

@Samiko_db Thanks for your response. I had a similar thought and noticed the same recommendation in a few other blogs as well. However, I recently came across the new Private Network Gateway feature for Azure Databricks, which leverages a delegated s...

  • 0 kudos
3 More Replies
Darshanrawal971
by New Contributor
  • 697 Views
  • 3 replies
  • 0 kudos

How is Serverless Compute implemented under the hood?

Hi everyone,I've been reading about Serverless Compute and have a question about the underlying architecture.In the classic compute model, clusters are provisioned in the customer's cloud account. With Serverless, Databricks manages the compute, but ...

  • 697 Views
  • 3 replies
  • 0 kudos
Latest Reply
Samiko_db
Databricks Employee
  • 0 kudos

@Darshanrawal971 I think both replies have the answered questions, but the TLDR is: our engine manages clusters in warm start which means as soon as a query gets processed, a cluster gets attached to a shared network boundary that cluster would inher...

  • 0 kudos
2 More Replies
taglud_dbw
by New Contributor II
  • 110 Views
  • 2 replies
  • 0 kudos

Databrick Oauth federation

I try to get databrick token from my databricks workspace, I set up properly databricks federation policy, but i still get this error."invalid_client": "user '123456' is not a member of workspace '123456' "i have checked in databricks console workspa...

  • 110 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @taglud_dbw, The "invalid_client": "user '123456' is not a member of workspace '123456' "  error during token federation means that the OAuth token exchange endpoint could not resolve the user as a direct member of the target workspace at the mome...

  • 0 kudos
1 More Replies
vamsi_simbus
by Databricks Partner
  • 87 Views
  • 1 replies
  • 0 kudos

Using Delta Sharing to hold ISV entitlement data outside the customer's metastore — reasonable fit?

We're an ISV. Our product is installed into each customer's own Databricksworkspace as a dedicated deployment. We maintain the code and run alldeployments; the customer never installs anything themselves.We license per store, so the product needs to ...

  • 87 Views
  • 1 replies
  • 0 kudos
Latest Reply
Satyasai
New Contributor II
  • 0 kudos

See this bloghttps://www.databricks.com/blog/introducing-new-databricks-partner-program-and-well-architected-framework-isvs-and-data 

  • 0 kudos
Edyta
by New Contributor II
  • 35324 Views
  • 7 replies
  • 1 kudos

Resolved! Delete Databricks account

Hi everyone, as in the topic, I would like to delete unnnecesarily created account. I have found outdated solutions (e.g. https://community.databricks.com/t5/data-engineering/how-to-delete-databricks-account/m-p/6323#M2501), but they do not work anym...

  • 35324 Views
  • 7 replies
  • 1 kudos
Latest Reply
eddiejobs
New Contributor II
  • 1 kudos

What is the solution for this?I signed up with fooname@gmail.com by mistake instead of foo.name@gmail.com. Gmail ignores dots in the local part of an address and also ignores anything after a plus sign, so the mailbox still works. I do not want the D...

  • 1 kudos
6 More Replies
neerajdubey_86
by New Contributor
  • 200 Views
  • 2 replies
  • 1 kudos

Does enabling Photon improve performance while lowering compute costs?

Please refer screen shot and help in clarifying if enabling Photon lead to lower execution costs? 

neerajdubey_86_0-1788545169602.png
  • 200 Views
  • 2 replies
  • 1 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 1 kudos

Hi @neerajdubey_86,   Photon will almost certainly make your queries faster, but whether it also makes them cheaper depends on your workload.   Photon is Databricks native vectorized query engine. It replaces the JVM-based Spark execution layer with ...

  • 1 kudos
1 More Replies
mariajsm
by New Contributor
  • 396 Views
  • 1 replies
  • 1 kudos

Who pays cross-cloud egress for Delta Sharing from GCP to Azure?

Hi DBX community!! Are cross-cloud egress charges for a Databricks-to-Databricks #DeltaShare billed to the provider account, and do they apply when the shared objects are views backed by Cloudflare?I'm setting up a Delta Share using Databricks on GCP...

Administration & Architecture
cloudflare
cross cloud
delta share
egress
open share
  • 396 Views
  • 1 replies
  • 1 kudos
Latest Reply
emma_s
Databricks Employee
  • 1 kudos

Hi,   Interesting topic, I've pulled together some relevant info for you: TLDR If your underlying data is in Cloudflare R2, the cloud-provider egress charge for cross-cloud Delta Sharing is effectively zero because R2 does not charge for egress. Howe...

  • 1 kudos