cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

gabriel_lazo
by New Contributor II
  • 1764 Views
  • 3 replies
  • 0 kudos

How to configure an AWS so that workspace databricks can only access the s3 acces point using VPC

My team requires a configuration so that a databricks workspace can connect to aws s3 access point through VPC and that other databricks workspaces cannot access it if they are not within the route table.I have searched online, but I have only found ...

  • 1764 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
2 More Replies
zsucic1
by New Contributor III
  • 5861 Views
  • 6 replies
  • 4 kudos

Resolved! Current Azure Managed Identity capabilities 2024?

Hello everyone, I have a few questions about MI capabilites: Is it possible to define a managed identity for Azure Databricks Service resource and use it for e.g.: Writing to Azure SQL Server database Authenticating to Azure Devops in order to downlo...

  • 5861 Views
  • 6 replies
  • 4 kudos
Latest Reply
zsucic1
New Contributor III
  • 4 kudos

Kaniz, thank you very much, you are the best! I will get to work implementing your advice

  • 4 kudos
5 More Replies
Priyam1
by New Contributor III
  • 2991 Views
  • 2 replies
  • 0 kudos

Access Logs

How can I check the timing when a particular AAD group was given access to a particular schema in a unity catalogue?Is there any API I can call to get this logs?

  • 2991 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Priyam1, To track when a specific Azure Active Directory (AAD) group was granted access to a particular schema in a Unity Catalog, you have a few options: Unity Catalog Privileges and Access Control: Unity Catalog allows you to control access...

  • 0 kudos
1 More Replies
migq2
by New Contributor III
  • 2332 Views
  • 5 replies
  • 0 kudos

Use Unity External Location with full paths in delta_log

I have an external delta table in unity catalog (let's call it mycatalog.myschema.mytable) that only consists of a `_delta_log` directory that I create semi-manually, with the corresponding JSON files that define it. The JSON files point to parquet f...

  • 2332 Views
  • 5 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I suggest you look at something else than UC for such cases.  I also wonder if delta lake is the right format.

  • 0 kudos
4 More Replies
avrm91
by Contributor
  • 2126 Views
  • 2 replies
  • 1 kudos

GCP - (DWH) Cluster Start-up Delayed - Failing to start

I face the issue that my fresh new Databricks workspace is not capable to start any cluster."Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you."After 1830 seconds (30,5 minutes) the w...

  • 2126 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @avrm91,  Verify that your project has sufficient CPU quota in the Google Cloud Platform (GCP) project associated with your Databricks workspace. If the quota is exceeded, it can prevent cluster nodes from launching.You can check your GCP quotas i...

  • 1 kudos
1 More Replies
rmubeenhsal
by New Contributor II
  • 1707 Views
  • 2 replies
  • 0 kudos

authorizationfailure on ls fs on mount point files

One of our users has as of last week started seeing an authorization failure when he tries to list the files in the Azure storage account using Databricks Cli or Databricks API(using Python). He can list files on the Databricks portal or through the ...

  • 1707 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

Have you checked the list of allowed ip addresses that are set for the Storage account in Azure? Is user using VPN or internal network, we might need to confirm if the network where the user is trying to list is set as allowed. 

  • 0 kudos
1 More Replies
curiousoctopus
by New Contributor II
  • 1556 Views
  • 1 replies
  • 0 kudos

User not authorised to copy files to dbfs

Hi,I'm trying to use a service principal to copy files to dbfs using the command line "databricks fs cp <source> <target>" but get back "User not authorised". I configured the authentication with PAT token and it is successful as I can deploy and lau...

  • 1556 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

In Databricks, data access permissions are often managed separately from workspace permissions. For DBFS, access control is typically managed through the underlying cloud storage (Azure Blob Storage, S3, etc.). The service principal needs to have the...

  • 0 kudos
mchirouze
by New Contributor
  • 1842 Views
  • 1 replies
  • 0 kudos

Send formatted html email from email distribution address

Hi, I have created an email distribution list "#MyList@mycompany.com". In the RShiny world I was able to send emails by a) getting the IP of the server I was sending the emails from and b) whitelisting that IP address within my company's SMTP Relay r...

  • 1842 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @mchirouze, To set up email services in Databricks, you have a few options depending on your requirements. Let’s explore them: Workspace Email Settings: As a workspace admin user, you can configure when users receive emails for certain events ...

  • 0 kudos
Learnit
by New Contributor II
  • 1181 Views
  • 1 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 1181 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Learnit, To resolve this issue, consider running the OPTIMIZE command on the shared Delta table to reduce the number of active files. After optimization, you may encounter the RemoveFiles limit if the OPTIMIZE command removed more than 100K files

  • 0 kudos
m997al
by Contributor II
  • 1290 Views
  • 1 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
  • 1290 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @m997al, Your understanding is correct: The Azure Databricks authentication workspace is unique per Azure region, and its uniqueness extends to each Azure-region-per DNS zone. If you have further questions or need additional clarification, feel fr...

  • 0 kudos
ducng
by New Contributor II
  • 985 Views
  • 2 replies
  • 0 kudos

Unity catalog metastore and databricks workspace provisioning

 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

Administration & Architecture
metastore
Terraform
  • 985 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @ducng, When working with Databricks metastores in Terraform, you can create a metastore using the databricks_metastore resource. This metastore acts as a top-level container for objects in Unity Catalog, including data assets like tables and view...

  • 0 kudos
1 More Replies
VJ3
by New Contributor III
  • 2998 Views
  • 1 replies
  • 1 kudos

Resolved! Workflow (Job) Cluster Permission Management

Hello Team,I understand that as the Job Owner, they can grant additional permissions to other users to manage/run/view the job.If "Can Manage" permission is given to the other users, that user can Edit the job including the Run-As parameter to themse...

  • 2998 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @VJ3, Let’s delve into the permissions and behaviour of jobs in Databricks when it comes to managing and running them. “Can Manage” Permission: When a secondary user is granted the “Can Manage” permission on a job owned by the primary user, th...

  • 1 kudos
Chaitanya07
by New Contributor
  • 2034 Views
  • 1 replies
  • 0 kudos

Databricks Rest APIs CORS Issue

Hello Team,We are currently integrating Databricks Rest APIs into our in-house application for managing access permissions. While testing with curl and Postman, we've successfully accessed certain APIs like listing cluster permission. However, we're ...

  • 2034 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Chaitanya07, Dealing with CORS (Cross-Origin Resource Sharing) issues can be a bit tricky, but I’ll provide some guidance to help you resolve this issue when integrating Databricks REST APIs into your in-house application. Understanding CORS:...

  • 0 kudos
Maxi1693
by New Contributor II
  • 1384 Views
  • 1 replies
  • 0 kudos

Error running 80 task at same time in Job, how limit this?

Hi! I have a Job running to process multiple streaming tables. In the beginning, it was working fine, but now I have 80 tables running in this job, the problem is that all the runs are trying to run at the same time throwing an error. Is there a way ...

  • 1384 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Maxi1693, It appears that you’re encountering issues with parallel execution of tasks in your Databricks job. Let’s address this by considering a few strategies: Concurrency Limit for Tasks: Databricks allows a maximum of 1000 concurrent tas...

  • 0 kudos
SirCrayon
by New Contributor
  • 1318 Views
  • 1 replies
  • 0 kudos

Do shared clusters have multiple drivers?

Hi,I know that with single clusters, theres a single driver node and one driver per cluster. With shared clusters, multiple jobs can run concurrently. Does this still run on a single driver container or multiple driver containers run per application?...

  • 1318 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @SirCrayon,  In the context of shared clusters, where multiple jobs can run concurrently, the behavior regarding driver containers differs from that of single clusters. Single Clusters: In a single cluster, there is indeed a single driver node...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels