cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Start a conversation
Mukee
by • New Contributor II
  • 3060 Views
  • 1 replies
  • 0 kudos

How to get Workspace name with workspaceId?

I have an AWS Managed Databricks instance. I am trying to get a workspace name with workspace ID. Thank you very much for your time and assistance.

Administration & Architecture
Reply
  • 3060 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
I have an AWS Managed Databricks instance. I am trying to get a workspace name with workspace ID. Thank you very much for your time and assistance.

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
MarcoRezende
by • New Contributor II
  • 1567 Views
  • 0 replies
  • 1 kudos

It's possible to sync account group/user to workspace without the need to do it manually?

I am using Databricks SCIM for my Databricks Account, so when i add a user or group in the SCIM connector, the user or group its created in Databricks Account. After this, i need to manually assign the user/group to the workspaces. My boss wants to o...

Administration & Architecture
Reply
  • 1567 Views
  • 0 replies
  • 1 kudos
gwilson
by • New Contributor II
  • 2620 Views
  • 2 replies
  • 0 kudos

Setup unity catalog external location to minio

We have a minio server running in Azure that we have connected to the spark clusters directly. As we move to unity catalog, we would like to make the data stored in our minio servers accessible as an external location in Azure Databricks account via ...

Administration & Architecture
Reply
  • 2620 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
174817
New Contributor III
  • 0 kudos

Hi @Retired_mod ,I have a server on Azure that supports the S3 protocol, and I am trying to follow these instructions in order to use Unity on Azure DataBricks with it.  I am not sure about this part of your reply:Set the Spark configuration values i...

  • 0 kudos
1 More Replies
breaka
by • New Contributor III
  • 4805 Views
  • 3 replies
  • 2 kudos

Operations on Unity Catalog take too long

Hi!We are currently PoC-ing Databricks with Unity Catalog on AWS but it seems there are some issues.Creating a database in an existing (unity) catalog takes over 10 minutes. Creating an external table on top of an existing delta table (CREATE TABLE m...

Administration & Architecture
Reply
  • 4805 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
breaka
New Contributor III
  • 2 kudos

PS: Apparently I'm not allowed to use the world H E A L T H (without spaces) in my reply (The message body contains H e a l t h, which is not permitted in this community. Please remove this content before sending your post.)

  • 2 kudos
2 More Replies
gabriel_lazo
by • New Contributor II
  • 2417 Views
  • 1 replies
  • 0 kudos

How to configure an AWS so that workspace databricks can only access the s3 acces point using VPC

My team requires a configuration so that a databricks workspace can connect to aws s3 access point through VPC and that other databricks workspaces cannot access it if they are not within the route table.I have searched online, but I have only found ...

Administration & Architecture
Reply
  • 2417 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
My team requires a configuration so that a databricks workspace can connect to aws s3 access point through VPC and that other databricks workspaces cannot access it if they are not within the route table.I have searched online, but I have only found ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
zsucic1
by • New Contributor III
  • 8470 Views
  • 2 replies
  • 1 kudos

Current Azure Managed Identity capabilities 2024?

Hello everyone, I have a few questions about MI capabilites: Is it possible to define a managed identity for Azure Databricks Service resource and use it for e.g.: Writing to Azure SQL Server database Authenticating to Azure Devops in order to downlo...

Administration & Architecture
Reply
  • 8470 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
zsucic1
New Contributor III
  • 1 kudos

Kaniz, thank you very much, you are the best! I will get to work implementing your advice

  • 1 kudos
1 More Replies
Priyam1
by • New Contributor III
  • 3519 Views
  • 1 replies
  • 0 kudos

Access Logs

How can I check the timing when a particular AAD group was given access to a particular schema in a unity catalogue?Is there any API I can call to get this logs?

Administration & Architecture
Reply
  • 3519 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
How can I check the timing when a particular AAD group was given access to a particular schema in a unity catalogue?Is there any API I can call to get this logs?

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
migq2
by • New Contributor III
  • 3403 Views
  • 4 replies
  • 0 kudos

Use Unity External Location with full paths in delta_log

I have an external delta table in unity catalog (let's call it mycatalog.myschema.mytable) that only consists of a `_delta_log` directory that I create semi-manually, with the corresponding JSON files that define it. The JSON files point to parquet f...

Administration & Architecture
Reply
  • 3403 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I suggest you look at something else than UC for such cases.  I also wonder if delta lake is the right format.

  • 0 kudos
3 More Replies
avrm91
by • Contributor
  • 2842 Views
  • 1 replies
  • 1 kudos

GCP - (DWH) Cluster Start-up Delayed - Failing to start

I face the issue that my fresh new Databricks workspace is not capable to start any cluster."Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you."After 1830 seconds (30,5 minutes) the w...

Administration & Architecture
Reply
  • 2842 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
I face the issue that my fresh new Databricks workspace is not capable to start any cluster."Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you."After 1830 seconds (30,5 minutes) the w...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
rmubeenhsal
by • New Contributor II
  • 5840 Views
  • 2 replies
  • 0 kudos

authorizationfailure on ls fs on mount point files

One of our users has as of last week started seeing an authorization failure when he tries to list the files in the Azure storage account using Databricks Cli or Databricks API(using Python). He can list files on the Databricks portal or through the ...

Administration & Architecture
Reply
  • 5840 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Have you checked the list of allowed ip addresses that are set for the Storage account in Azure? Is user using VPN or internal network, we might need to confirm if the network where the user is trying to list is set as allowed. 

  • 0 kudos
1 More Replies
curiousoctopus
by • New Contributor III
  • 1942 Views
  • 1 replies
  • 0 kudos

User not authorised to copy files to dbfs

Hi,I'm trying to use a service principal to copy files to dbfs using the command line "databricks fs cp <source> <target>" but get back "User not authorised". I configured the authentication with PAT token and it is successful as I can deploy and lau...

Administration & Architecture
Reply
  • 1942 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

In Databricks, data access permissions are often managed separately from workspace permissions. For DBFS, access control is typically managed through the underlying cloud storage (Azure Blob Storage, S3, etc.). The service principal needs to have the...

  • 0 kudos
Learnit
by • New Contributor II
  • 1776 Views
  • 0 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

Administration & Architecture
Reply
  • 1776 Views
  • 0 replies
  • 0 kudos
m997al
by • Contributor III
  • 1659 Views
  • 0 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
Administration & Architecture
Reply
  • 1659 Views
  • 0 replies
  • 0 kudos
ducng
by • New Contributor II
  • 1584 Views
  • 1 replies
  • 0 kudos

Unity catalog metastore and databricks workspace provisioning

 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

Administration & Architecture
metastore
Terraform
Reply
  • 1584 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels