cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Start a conversation
SmileyVille
by • New Contributor II
  • 2319 Views
  • 2 replies
  • 0 kudos

Resolved! Leverage Azure PIM with DataBricks with Contributor role privilege

We are trying to leverage Azure PIM.  This works great for most things, however; we've run into a snag.  We want to limit the contributor role to a group and only at the resource group level, not subscription.  We wish to elevate via PIM.  This will ...

Administration & Architecture
Reply
  • 2319 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
SmileyVille
New Contributor II
  • 0 kudos

Thanks - think we were originally overthinking this.We determined we were doing this correctly, the user just needed to switch to 'groups' within PIM to request elevation of permissions.  The larger issue is actually the 40 min user provisioning cycl...

  • 0 kudos
1 More Replies
jrosend
by • New Contributor III
  • 8198 Views
  • 1 replies
  • 0 kudos

[Possible Bug] Repo Notebooks being modified without human interaction

Our production workspace has several Repos integrated with GitHub. These repos aways point to master and should never be modified manually by a human directly in the workspace as the pulls are triggered by a GitHub Actions workflow. This workflow cal...

MicrosoftTeams-image.png MicrosoftTeams-image (2).png
Administration & Architecture
Reply
  • 8198 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @jrosend , The issue you are facing is that there are minor changes occurring in the notebooks in your Databricks Repos, which are causing conflicts during the automatic update process triggered by the GitHub Actions workflow. These changes are no...

  • 0 kudos
abhaigh
by • New Contributor III
  • 2432 Views
  • 1 replies
  • 0 kudos

Error: cannot create permissions: invalid character '<' looking for beginning of value

I'm trying to use terraform to assign a cluster policy to an account-level group (sync'd from AAD via SCIM)My provider is configured like thisprovider "databricks" {alias = "azure_account"host = "accounts.azuredatabricks.net"account_id = "%DATABRICKS...

Administration & Architecture
Reply
  • 2432 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @abhaigh ,  • The issue is related to applying a cluster policy to an account-level group using Terraform.• The error message indicates that the returned value from the API endpoint is not as expected. • To resolve the issue, follow these steps:...

  • 0 kudos
_YSF
by • New Contributor II
  • 2385 Views
  • 1 replies
  • 0 kudos

Struggling with UC Volume Paths

I am trying to setup my volumes and give them paths in the data lake but I keep getting this message:Input path url 'abfss://my-container@my-storage-account.dfs.core.windows.net/' overlaps with managed storage within 'CreateVolume' callThere WAS some...

Administration & Architecture
Reply
  • 2385 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @_YSF, The error message "Input path URL'  abyss://my-container@my-storage-account.dfs.core.windows.net/' overlaps with managed storage within' CreateVolume' call" suggests that there is an issue with setting up volumes and giving them paths in th...

  • 0 kudos
diego_poggioli
by • Contributor
  • 5435 Views
  • 5 replies
  • 2 kudos

Resolved! Unable to list service principal in Job details RUN AS

I added the service principal in Admin Settings > Service Principal and then enabled all the Configurations "allow cluster creation", "databricks SQL access" and "workspace access". In the Permission settings I have enabled "Service principal: Manage...

Administration & Architecture
Reply
  • 5435 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
BilalAslamDbrx
Honored Contributor III
  • 2 kudos

For future readers - don't forget to add your email (e.g. me@foo.com) in the Service Principals permissions tab. This way, you will be able to see the newly-created service principal in the dropdown menu.

  • 2 kudos
4 More Replies
DevOps
by • New Contributor
  • 1958 Views
  • 1 replies
  • 0 kudos

Workspace creation via terraform provider fails on AWS

I'm trying to create a new workspace in a empty account. I have managed to create all the other resources without issues but when I try to create the workspace it fails with the following error:Error: cannot create mws workspaces: MALFORMED_REQUEST: ...

Administration & Architecture
Reply
  • 1958 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @DevOps, Your error is related to  credentials_id your workspace configuration. The credentials_id is a reference to the Databricks credentials configuration ID, which is generated when you create a new set of credentials. This ID represents your ...

  • 0 kudos
OU_Professor
by • New Contributor II
  • 11227 Views
  • 1 replies
  • 0 kudos

Resolved! Connect Community Edition to Power BI Desktop

I have submitted this question several times to Databricks over the past few weeks, and I have gotten no response at all, not even an acknowledgement that my request was received.Please help.How can I connect a certain dataset in Databricks Community...

Administration & Architecture
Reply
  • 11227 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @OU_Professor, To connect a particular data set in Databricks Community Edition to the Power BI desktop, you can follow these steps: 1. Install Databricks JDBC driver: You can download the JDBC driver from the Databricks JDBC driver download page....

  • 0 kudos
jaganadhg
by • New Contributor
  • 1272 Views
  • 1 replies
  • 0 kudos

Resolved! Clean up Databricks confidential computing resources

Hello All,I created a Databricks Premium Workspace for a Confidential Computing PoC. After creating a VM from Databricks UI, it came to notice that there is a new RG with managed identity, NAT Gateway, Public IP, security group, and a VNET (/16). I w...

Administration & Architecture
Confidential Compute
Reply
  • 1272 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @jaganadhg , To delete the resources once you complete the work involving confidential computing, you can use the DELETE API call provided in the given information. Here is an example of how to use it: bashcurl -X DELETE 'https://accounts.cloud.d...

  • 0 kudos
PetePP
by • New Contributor II
  • 1077 Views
  • 2 replies
  • 0 kudos

Extreme RocksDB memory usage

During migration to production workload, I switched some queries to use RocksDB. I am concerned with its memory usage though. Here is sample output from my streaming query:   "stateOperators" : [ { "operatorName" : "dedupeWithinWatermark", "...

Administration & Architecture
Reply
  • 1077 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
PetePP
New Contributor II
  • 0 kudos

Thank you for the input. Is there any particular reason why deduplication watermark makes it store everything and not just the key needed for deduplication? The 1st record has to be written to the table anyway, and its content is irrelevant as it jus...

  • 0 kudos
1 More Replies
Bagger
by • New Contributor II
  • 1782 Views
  • 1 replies
  • 1 kudos

Resolved! Monitoring job metrics

Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...

Administration & Architecture
jobs
metrics
prometheus
Reply
  • 1782 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Bagger, You can monitor Databricks jobs and get the required metrics using a combination of Databricks features and Prometheus. Here's a general idea of how you could approach each metric you mentioned. 1. Failed jobs: Databricks provides a REST ...

  • 1 kudos
Ajay3
by • New Contributor
  • 1650 Views
  • 1 replies
  • 1 kudos

How can I install maven coordinates using init script?

Hi,I need to install the below maven coordinates on the clusters using databricks init scripts.1. coordinate: com.microsoft.azure:synapseml_2.12:0.11.2 with repo https://mmlspark.azureedge.net/maven2. coordinate: com.microsoft.azure:spark-mssql-conne...

Administration & Architecture
Reply
  • 1650 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Ajay3, You can follow these steps to install the Maven coordinates via Databricks init scripts. 1. Create a shell script to install the Maven libraries on the cluster. Here is an example of how you can do it: bash#!/bin/bash# Install synapseml_2...

  • 1 kudos
nihar_ghude
by • New Contributor II
  • 4111 Views
  • 3 replies
  • 1 kudos

Resolved! How to change Workspace Owner?

Our Databricks workspace was created by a personal account. Now the person has left the organization. We would like to change the owner to a Service account(preferably, else to an Admin account).Questions:Is it possible to change the owner of the wor...

Administration & Architecture
admin
change owner
workspace owner
Reply
  • 4111 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @nihar_ghude ,  1. Yes, changing the owner of the Databricks workspace is possible. As per the provided information, the account's owner can be a user or service account. 2. However, the process to change the owner is not directly mentioned in the...

  • 1 kudos
2 More Replies
NadithK
by • Contributor
  • 5161 Views
  • 5 replies
  • 1 kudos

Resolved! Using a custom Hostname in Databricks CLI instead of per-workspace URL

Hi,At our organization, we have added front end privatelink connection to a Databricks workspace in Azure, and public access to the workspace is disabled. I am able to access the workspace UI with the private IP (in the browser), and able to call the...

Administration & Architecture
Reply
  • 5161 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
NadithK
Contributor
  • 1 kudos

Hi @Kaniz_Fatma ,Thank you for the support.Really appreciate it.Thanks

  • 1 kudos
4 More Replies
jaznarro
by • New Contributor II
  • 1536 Views
  • 3 replies
  • 2 kudos

Resolved! Modularization of Databricks Workflows

Given the size of a Workflow may become too big to manage in a single Terraform project, what would be your recommendation as a best practice to manage and deploy the workflows via code to maintainer a predictable result between environments?Would it...

Administration & Architecture
Reply
  • 1536 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @jaznarro, Given the size of a Workflow may become too big to manage in a single Terraform project, breaking the Workflow into multiple ones with references between each other would be a recommended best practice. This approach can help maintain ...

  • 2 kudos
2 More Replies
re
by • New Contributor II
  • 711 Views
  • 0 replies
  • 0 kudos

terraform/databricks setting default_catalog_name

While configuring databricks, we've set the "default_catalog_name", which sets the default schema when users connect via an ODBC connection. While the naming isn't consistent, this does have one desired effect, that is, when users connect, it default...

Administration & Architecture
Reply
  • 711 Views
  • 0 replies
  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels