cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

paritoshsh
by New Contributor II
  • 1228 Views
  • 1 replies
  • 1 kudos

Resolved! Terraform Repos Git URL Allow List

Hi,I am provisioning databricks workspaces using terraform and want to add specific github repo url that can be used. In UI there is an option for that but when it comes to terraform there is nothing specific. I came across custom_config option here ...

Screenshot 2023-08-08 174830.jpg
  • 1228 Views
  • 1 replies
  • 1 kudos
Latest Reply
Amine
New Contributor III
  • 1 kudos

Hello,This can normally be achieved using this terraform resource:resource "databricks_workspace_conf" "this" { custom_config = { "enableProjectsAllowList": true, "projectsAllowList": "url1,url2,url3", } }Cheers

  • 1 kudos
ArjenSmedes
by New Contributor
  • 5625 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks workspace in our own VNET

We have setup a Databricks workspace in our own Azure VNET, including a private endpoint. Connecting to the WS works fine (through the private ip address). However, when creating my first cluster, I run into this problem:"ADD_NODES_FAILED...Failed to...

  • 5625 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @ArjenSmedes, Yes, it is true. The Databricks data plane runs within your VNet. When you set up your private DNS zone, it will pick up the correct IP address, and the provisioning of the cluster should work. The error message you're seeing, "Faile...

  • 0 kudos
Palkers
by New Contributor II
  • 455 Views
  • 0 replies
  • 0 kudos

Data Marketplace private exchange

I want to use Data Markerplace but only as private / local mode, so don't want to publish any products outside my organization.I know I can create private listing , but it can be done only from provider console.I'm added to marketplace role but not s...

  • 455 Views
  • 0 replies
  • 0 kudos
NadithK
by Contributor
  • 1837 Views
  • 2 replies
  • 2 kudos

Resolved! Assigning Databricks Account Admin role to User group

At my current organization, we have a few users with the Databricks Account admin role assigned. But as per our company policy, individual users should not be given such elevated privileges. They should be given to user groups so that users in those ...

  • 1837 Views
  • 2 replies
  • 2 kudos
Latest Reply
NadithK
Contributor
  • 2 kudos

Hi Kaniz,Thank you for the feedback and I was able to find below solution from the article you have mentioned.https://docs.databricks.com/en/administration-guide/users-groups/groups.html#assign-account-admin-roles-to-a-groupSeems we can use Databrick...

  • 2 kudos
1 More Replies
jrosend
by New Contributor III
  • 870 Views
  • 2 replies
  • 1 kudos

Global search programatically

Hi!At the workspace header, there is a search box that allow us to look for a text in all notebooks in the workspace. Is there a way via CLI or API to call the global search https://<workspace-domain>/graphql/SearchGql so the result can be analysed a...

  • 870 Views
  • 2 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

if you checked in the notebooks into a git repo, the search in the git repo (or API) might save you.

  • 1 kudos
1 More Replies
Xyguo
by New Contributor
  • 1045 Views
  • 1 replies
  • 1 kudos

Resolved! REST API workspace list content doesn't work with Queries

Hi, I'm trying to export the SQL Queries in certain folders in workspace, but the list content API callGET  /api/2.0/workspace/list  doesnt work with queries? how should I export only queries in a certain folder in workspace? Thank you very much  

Xyguo_0-1691335659836.png
  • 1045 Views
  • 1 replies
  • 1 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 1 kudos

@Xyguo - Currently, exporting a SQL query file is not supported.  Kindly create an idea feature request by following the article listed - https://docs.databricks.com/en/resources/ideas.html#create-an-idea-in-the-ideas-portal to raise a feature reques...

  • 1 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 1196 Views
  • 1 replies
  • 2 kudos

Databricks Enhances Job Monitoring with Duration Thresholds for Workflow

Databricks has introduced Duration Thresholds for workflows. This new addition allows users to set time limits for workflow execution, significantly improving monitoring of job performance. When a job exceeds the preset duration, the system triggers ...

ezgif-4-8ab9474b98.gif
  • 1196 Views
  • 1 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi  @Hubert-Dudek, Good to see your posts. Welcome back!

  • 2 kudos
milicevica
by New Contributor
  • 4491 Views
  • 2 replies
  • 1 kudos

Resolved! Multiple orphan vms in managed resource group after starting and terminating my personal cluster

Hi today, i had problem with starting the cluster and therefore i did multiple times star and terminating. The problem is that this actions always started one new VM in the managed resource group but never turned it off. Therefore i ended up with mul...

  • 4491 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

It always takes a few minutes, even 10, before the machine is terminated in Azure. It is better to set a pool of machines in data bricks and use them from there so we will not be keeping Zombie machines.

  • 1 kudos
1 More Replies
vasu2
by New Contributor
  • 502 Views
  • 0 replies
  • 0 kudos

Multi bronze layers : multi client data

data coming in from multiple sources belonging to multiple customers.Should I create Single storage account name it bronze and then separate containers for each client's data to be put in.And then merge the data in the silver layer. What's the best p...

  • 502 Views
  • 0 replies
  • 0 kudos
niklas
by Contributor
  • 3480 Views
  • 2 replies
  • 0 kudos

Resolved! Can't set account admin using Terraform

I want to set the account admin for a service principal in order to create the Unity Catalog metastore. The Terraform code looks like this: data "databricks_service_principal" "application" { count = var.environment == "dev" ? 1 : 0 application_...

Administration & Architecture
access
account_admin
azure
role
Terraform
  • 3480 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @niklas, The error message "cannot read service principal role" is likely indicating that the user attempting to read the service principal is not assigned the Service Principal Manager role. This role allows users to manage functions on a service...

  • 0 kudos
1 More Replies
shawnbarrick
by New Contributor III
  • 2114 Views
  • 2 replies
  • 0 kudos

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error with S3 paths

We're trying to read from an S3 bucket using unity catalog and are selectively getting "INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error"  errors within the same bucket.  This works: "dbutils.fs.ls("s3://BUCKETNAME/dev/he...

  • 2114 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi,Could you please elaborate on the issue here? Running the list command on a managed directory is not supported in Unity Catalog. Catalog/schema storage locations are reserved for managed storage.Please tag @Debayan  with your next comment which wi...

  • 0 kudos
1 More Replies
DianGermishuize
by New Contributor II
  • 1205 Views
  • 1 replies
  • 0 kudos

Bitbucket Repo Add Assistance

I am trying to add a Bitbucket cloud repo to my workspace, but keep getting a credentials invalid error. I made sure to set a username and generate an app password with repo read write permissions. What am i doing wrong? 

  • 1205 Views
  • 1 replies
  • 0 kudos
Latest Reply
Harrison_S
New Contributor III
  • 0 kudos

Hello,Did you also set the password in User Settings > Git integration ? If so can you paste the full error?

  • 0 kudos
hattier
by New Contributor II
  • 2479 Views
  • 1 replies
  • 1 kudos

Resolved! OIDC account-level SSO and SAML workspace-level SSO

Hey folks! I am seeking clarity for this passage in documentation: "This means that SSO for your account and SSO for each workspace must be managed separately. You should configure SSO to the the same identity provider at the account and on your work...

  • 2479 Views
  • 1 replies
  • 1 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

If both are pointing to the same IDP then it should not be a problem.

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels