cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

rocky5
by New Contributor III
  • 3465 Views
  • 5 replies
  • 0 kudos

System.billing.usage table - cannot match job_id from databricks api/UI

Hello, I have multiple continuous jobs that are running for many days (Kafka stream), however querying System.billing.usage table by job_id from UI or databricks job api not return any results for those jobs.1. What is the reason behind that?2. If I ...

  • 3465 Views
  • 5 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Hello, you are right, apologize for my misunderstanding as you have mentioned the job id persist, is the job run id the one that will be changed. So your issue is that as the job is continuing running the information of that run is not showing in the...

  • 0 kudos
4 More Replies
ossinova
by Contributor II
  • 1976 Views
  • 2 replies
  • 1 kudos

Error: Node.js SQL Driver auth using M2M

I am trying to follow the documentation in order to establish a M2M authentication through the Node.js SQL Driver. I am however having issues as it results in the following error message. What is it that I am not seeing here?Error msg:{"level":"info"...

  • 1976 Views
  • 2 replies
  • 1 kudos
Latest Reply
ta2
New Contributor II
  • 1 kudos

Not sure if you're still suck on this but I had the same issue and managed to resolve it by adding azureTenantId: <my-tenant-id> and useDatabricksOAuthInAzure: true in the client connection options. Hope this helps!

  • 1 kudos
1 More Replies
gwilson
by New Contributor II
  • 3134 Views
  • 2 replies
  • 0 kudos

Setup unity catalog external location to minio

We have a minio server running in Azure that we have connected to the spark clusters directly. As we move to unity catalog, we would like to make the data stored in our minio servers accessible as an external location in Azure Databricks account via ...

  • 3134 Views
  • 2 replies
  • 0 kudos
Latest Reply
174817
New Contributor III
  • 0 kudos

Hi @Retired_mod ,I have a server on Azure that supports the S3 protocol, and I am trying to follow these instructions in order to use Unity on Azure DataBricks with it.  I am not sure about this part of your reply:Set the Spark configuration values i...

  • 0 kudos
1 More Replies
breaka
by New Contributor III
  • 5517 Views
  • 3 replies
  • 2 kudos

Operations on Unity Catalog take too long

Hi!We are currently PoC-ing Databricks with Unity Catalog on AWS but it seems there are some issues.Creating a database in an existing (unity) catalog takes over 10 minutes. Creating an external table on top of an existing delta table (CREATE TABLE m...

  • 5517 Views
  • 3 replies
  • 2 kudos
Latest Reply
breaka
New Contributor III
  • 2 kudos

PS: Apparently I'm not allowed to use the world H E A L T H (without spaces) in my reply (The message body contains H e a l t h, which is not permitted in this community. Please remove this content before sending your post.)

  • 2 kudos
2 More Replies
gabriel_lazo
by New Contributor II
  • 2562 Views
  • 1 replies
  • 0 kudos

How to configure an AWS so that workspace databricks can only access the s3 acces point using VPC

My team requires a configuration so that a databricks workspace can connect to aws s3 access point through VPC and that other databricks workspaces cannot access it if they are not within the route table.I have searched online, but I have only found ...

  • 2562 Views
  • 1 replies
  • 0 kudos
Priyam1
by New Contributor III
  • 3605 Views
  • 1 replies
  • 0 kudos

Access Logs

How can I check the timing when a particular AAD group was given access to a particular schema in a unity catalogue?Is there any API I can call to get this logs?

  • 3605 Views
  • 1 replies
  • 0 kudos
migq2
by New Contributor III
  • 3787 Views
  • 4 replies
  • 0 kudos

Use Unity External Location with full paths in delta_log

I have an external delta table in unity catalog (let's call it mycatalog.myschema.mytable) that only consists of a `_delta_log` directory that I create semi-manually, with the corresponding JSON files that define it. The JSON files point to parquet f...

  • 3787 Views
  • 4 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I suggest you look at something else than UC for such cases.  I also wonder if delta lake is the right format.

  • 0 kudos
3 More Replies
rmubeenhsal
by New Contributor II
  • 6677 Views
  • 2 replies
  • 0 kudos

authorizationfailure on ls fs on mount point files

One of our users has as of last week started seeing an authorization failure when he tries to list the files in the Azure storage account using Databricks Cli or Databricks API(using Python). He can list files on the Databricks portal or through the ...

  • 6677 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Have you checked the list of allowed ip addresses that are set for the Storage account in Azure? Is user using VPN or internal network, we might need to confirm if the network where the user is trying to list is set as allowed. 

  • 0 kudos
1 More Replies
curiousoctopus
by New Contributor III
  • 2031 Views
  • 1 replies
  • 0 kudos

User not authorised to copy files to dbfs

Hi,I'm trying to use a service principal to copy files to dbfs using the command line "databricks fs cp <source> <target>" but get back "User not authorised". I configured the authentication with PAT token and it is successful as I can deploy and lau...

  • 2031 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

In Databricks, data access permissions are often managed separately from workspace permissions. For DBFS, access control is typically managed through the underlying cloud storage (Azure Blob Storage, S3, etc.). The service principal needs to have the...

  • 0 kudos
Learnit
by New Contributor II
  • 2046 Views
  • 0 replies
  • 0 kudos

Delta Sharing resulting in Bad_Request

Hi All,Recipeint encountering an issue while trying to access my organizational data (providers data) in a Delta Sharing scenario(databricks to databricks), and I'm hoping to get some guidance on how to resolve it. Here is the error message recipient...

  • 2046 Views
  • 0 replies
  • 0 kudos
m997al
by Contributor III
  • 1779 Views
  • 0 replies
  • 0 kudos

Azure Databricks with standard private link (only one Databricks authentication workspace)?

We have successfully set up Azure Databricks with standard private link (front-end and back-end).The front-end uses the authentication workspace as prescribed in the documentation.Suppose we use a "custom DNS" for the configuration below.  Can we set...

m997al_0-1710778785048.png
  • 1779 Views
  • 0 replies
  • 0 kudos
ducng
by New Contributor II
  • 1881 Views
  • 1 replies
  • 0 kudos

Unity catalog metastore and databricks workspace provisioning

 Hi,I am using terraform to create a databricks metastore using this.Now the issue is that I want to use this metastore reference in another terraform project where I create the databricks workspaces but there is no data source for metastore. Can any...

Administration & Architecture
metastore
Terraform
  • 1881 Views
  • 1 replies
  • 0 kudos
Chaitanya07
by New Contributor
  • 3889 Views
  • 0 replies
  • 0 kudos

Databricks Rest APIs CORS Issue

Hello Team,We are currently integrating Databricks Rest APIs into our in-house application for managing access permissions. While testing with curl and Postman, we've successfully accessed certain APIs like listing cluster permission. However, we're ...

  • 3889 Views
  • 0 replies
  • 0 kudos