cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

cyborg-de
by New Contributor
  • 4389 Views
  • 1 replies
  • 0 kudos

DNS resolution across vnet

Hi, I have created a new databricks workspace in Azure with backend private link. Settings are Required NSG rules - No Azure Databricks RuleNSG rules for AAD and azfrontdoor were added as per documentation. Private endpoint with subresource  databric...

  • 4389 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Based on your description, the error when creating a Databricks compute cluster in Azure with Private Link is likely due to DNS resolution issues between the workspace VNET and the separate VNET hosting your private DNS zone. Even with VNET peering a...

  • 0 kudos
Junda
by New Contributor III
  • 4069 Views
  • 1 replies
  • 0 kudos

How to install private repository as package dependency in Databricks Workflow

I am a member of the development team in our company and we use Databricks as sort of like ETL tool. We utilize git integration for our program and run Workflow daily basis. Recently, we created another company internal private git repository and wan...

  • 4069 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

You can install and use private repository packages in Databricks workflows in a scalable and secure way, but there are trade-offs and best practices to consider for robust, team-friendly automation. Here's a direct answer and a breakdown of solution...

  • 0 kudos
songhan89
by New Contributor
  • 4086 Views
  • 1 replies
  • 0 kudos

Why is writing direct to Unity Catalog Volume slower than to Azure Blob Storage (xarray -> zarr)

Hi,I have some workloads whereby i need to export an xarray object to a Zarr store.My UC volume is using ADLS.I tried to run a simple benchmark and found that UC Volume is considerably slower.a) Using a fsspec ADLS store pointing to the same containe...

songhan89_0-1738517230323.png
Administration & Architecture
Unity Catalog
Volume
  • 4086 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Writing directly to a Unity Catalog (UC) Volume in Databricks is often slower than writing to Azure Blob Storage (ADLS) using an fsspec-based store, especially for workloads exporting xarray objects to Zarr. This performance gap has been noted and di...

  • 0 kudos
gyorgyjelinek
by New Contributor II
  • 3811 Views
  • 1 replies
  • 0 kudos

How to calculate accurate usage cost for a longer contractual period?

Hi Experts!I work on providing and accurate total cost (in DBU and USD as well) calculation for my team for the whole ongoing contractual period. I'v checked the following four options:Account console: Manage account - Usage - Consumption (Legacy): t...

  • 3811 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Based on your description, the REST API for billable usage logs (Option 4) is likely the most comprehensive and reliable method for retrieving usage and cost data for the full contractual period, including potentially the missing first two months. Th...

  • 0 kudos
carlos_tasayco
by Contributor
  • 3462 Views
  • 1 replies
  • 1 kudos

Get managedResourceGroup from serverless

Hello,In my job I have a task where I should modify a notebook to get dynamically the environment, for example:This is how we get it:dic = {"D":"dev", "Q":"qa", "P":"prod"}managedResourceGroup = spark.conf.get("spark.databricks.xxxxx")xxxxx_Index = m...

  • 3462 Views
  • 1 replies
  • 1 kudos
Latest Reply
mark_ott
Databricks Employee
  • 1 kudos

To dynamically detect your Databricks environment (dev, qa, prod) in a serverless notebook, without relying on manual REST API calls, you typically need a reliable way to extract context directly inside the notebook. However, serverless notebooks oft...

  • 1 kudos
AiswaryaS
by New Contributor II
  • 3217 Views
  • 1 replies
  • 0 kudos

Query has been timed out due to inactivity while connecting from Tableau Prep

Hi,We are experiencing Query timed out error while running Tableau flows with connections to Databricks. The query history for Serverless SQL warehouse initially showing as finished in Databricks. But later the query status change to "Query has been ...

  • 3217 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

The "Query has been timed out due to inactivity" error with Tableau flows connected to Databricks Serverless SQL Warehouse is a known and intermittent issue impacting several users, even when the SQL warehouse does not auto-terminate during the proce...

  • 0 kudos
Saurabh_kanoje
by New Contributor
  • 229 Views
  • 2 replies
  • 2 kudos

Resolved! Learning Databricks

Hi All,I am new to databricks and trying to learn things around, i have experience in platform administration and Platform integration and managements roles.Can someone please guide a correct path learning path around platform administration and is t...

  • 229 Views
  • 2 replies
  • 2 kudos
Latest Reply
bianca_unifeye
Contributor
  • 2 kudos

Hi @Saurabh_kanoje , welcome to the Databricks community!In the Databricks Academy, there’s a free course called Databricks Platform Administration Fundamentals, which is a great starting point.I’d also recommend exploring the Azure, AWS AND GCP Data...

  • 2 kudos
1 More Replies
Vadimalk
by New Contributor II
  • 3822 Views
  • 1 replies
  • 0 kudos

Windows ODBC connection error

Hi all,I'm just started learning Databricks and have created a community-level workspace and loaded few tables.Now I'm trying to get access to the data from Excel ODBC connector following the guide here:https://docs.databricks.com/en/integrations/exc...

  • 3822 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

The “Status: 500 – Internal Server Error” when connecting Databricks to Excel via the ODBC connector usually means something on the Databricks end is not properly configured, or there is an issue with the authentication flow. Here are the main troubl...

  • 0 kudos
d_kailthya
by New Contributor
  • 4512 Views
  • 1 replies
  • 0 kudos

Implementing Databricks Persona in

Hi all,I am looking to implement the "persona" based access control across multiple workspaces for multiple user groups in Azure Databricks workspaces. Specifically,- I have a "DEV" workspace where the developer groups (Data Engineers and ML Engineer...

  • 4512 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

You can implement persona-based access control for Azure Databricks workspaces using Terraform and the Databricks provider, aligning with the setup you described for DEV and PROD environments. Terraform allows you to codify workspace configuration, u...

  • 0 kudos
camilo_s
by Contributor
  • 4123 Views
  • 1 replies
  • 0 kudos

Programmatically setting tags for securables

Unity Catalog securable objects can be tagged with key value pairs: https://learn.microsoft.com/en-us/azure/databricks/database-objects/tagsIs it possible tag objects via REST API calls?I initially thought any Unity Catalog resource in the Databricks...

  • 4123 Views
  • 1 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Hello @camilo_s ,  thanks for sharing the doc link and the details you observed in the UI network calls. Short answer There isn’t a documented, stable, public REST endpoint specifically for “tags on UC securables” today. You should use SQL DDL to man...

  • 0 kudos
daniel23
by New Contributor II
  • 3603 Views
  • 1 replies
  • 0 kudos

Delete Users that are Maintenance Readers

I am an Account Admin at Databricks (Azure), and trying to delete users that are being offboarded.I have managed to delete most users. However, for a couple, I get the following message (see screenshot):ABORTED: Account <account> is read-only during ...

abort-delete.PNG
  • 3603 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

When trying to delete users in Databricks (Azure) and encountering the message "ABORTED: Account <account> is read-only during maintenance and cannot be updated," this means that your Databricks account is currently in a maintenance mode where no cha...

  • 0 kudos
SolaireOfAstora
by New Contributor
  • 4172 Views
  • 1 replies
  • 0 kudos

Databricks report error: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)

Has anyone encountered this error and knows how to resolve it?"Unexpected end of stream, read 0 bytes from 4 (socket was closed by server)."This occurs in Databricks while generating reports.I've already adjusted the wait_timeout to 28,800, and both ...

  • 4172 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Yes, the "Unexpected end of stream, read 0 bytes from 4 (socket was closed by server)" error has been encountered by other Databricks users when generating reports with MySQL. You've already set the major MySQL timeout parameters to their maximums, w...

  • 0 kudos
axelboursin
by New Contributor II
  • 3660 Views
  • 1 replies
  • 0 kudos

Need to create an Identity Federation between my Databricks workspace/account and my AWS account

Hello,I need to set up an identification between my Databricks workspace/account and my AWS account, where Databricks is already deployed.The goal is to make an easy authentification without access and secret keys.So I thought that OIDC will be the s...

  • 3660 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

To set up identification between your Databricks workspace/account and your AWS account without using access or secret keys, you can leverage OIDC (OpenID Connect) federation. Instead of traditional SSO, what you’re looking for is a model where AWS t...

  • 0 kudos
Leo_310
by New Contributor II
  • 3874 Views
  • 2 replies
  • 0 kudos

OAuth Url and ClientId Validation

HiI am trying to setup an oauth connection with databricks, so I ask the user to enter their Workspace URL and ClientId.Once the user enters these values, I want to validate whether they are correct or not, so I ask them to login by redirecting them ...

  • 3874 Views
  • 2 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

If you’re using OAuth with Databricks and want to validate both the Workspace URL and ClientId before proceeding, you’re facing an issue seen by others: when the Workspace URL is correct but the ClientId is wrong, Databricks just displays a generic e...

  • 0 kudos
1 More Replies
evgenyvainerman
by New Contributor
  • 3781 Views
  • 1 replies
  • 0 kudos

Custom Runtime marketplace

Hi! Is there a possibility to share the solution accelerator on the custom runtime via the databricks marketplace?

  • 3781 Views
  • 1 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Greetings @evgenyvainerman , sorry for the delayed response. Your question is not entirely clear but I will take a swing at providing an answer.   Short answer: Yes, you can share a Solution Accelerator through Databricks Marketplace, but Marketplace...

  • 0 kudos