cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

naveenanto
by New Contributor III
  • 1703 Views
  • 4 replies
  • 2 kudos

Resolved! Unable to get external groups from list group details API

Hi Team, I see that automatic group sync is enabled in our azure databricks by default but I am not able to get the groups using list groups endpoint. What is the right way to get it. I've an automated process to bring certain account groups to works...

  • 1703 Views
  • 4 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

I checked it and I can confirm that it worked. I enabled automatic identity management, then I've added EntraId group to the workspace and using above endpoint I was able to get information about that group.

  • 2 kudos
3 More Replies
noorbasha534
by Valued Contributor
  • 1505 Views
  • 2 replies
  • 0 kudos

Is there a way to see Job/Workflow in the lineage of a table/view

Hi allOne frequent question I get from the data users is - how often a view is refreshed? ((users are given access to views not tables))I was thinking to guide them to leverage lineage for the same. However, I noticed the lineage tab in the catalog e...

  • 1505 Views
  • 2 replies
  • 0 kudos
Latest Reply
noorbasha534
Valued Contributor
  • 0 kudos

@Vidhi_Khaitan the user wants to know the frequency of the schedule, not whether the view/table is fresh...

  • 0 kudos
1 More Replies
hajojko
by New Contributor
  • 1060 Views
  • 1 replies
  • 0 kudos

Job Runs Permissions for Runs Submit API via ADF

I wanted to assign group permissions to Job runs in Azure Databricks which are triggered from ADF via Runs Submit API.How can we assign Job Runs Permissions in Azure Databricks for ADF triggers by Runs Submit API?

  • 1060 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vidhi_Khaitan
Databricks Employee
  • 0 kudos

Hi @hajojko ,Good Day!I believe this is what you can try for your use case - https://docs.databricks.com/api/workspace/jobs_21/submit#access_control_list  

  • 0 kudos
sg-vtc
by New Contributor III
  • 1226 Views
  • 1 replies
  • 0 kudos

Databricks compute failed to start due to EC2 security group id change in AWS default VPC

I have been using Databricks on AWS for almost 2 years.  Recently, I cannot start up Databricks compute.        "databricks_error_message": "The VM launch request to AWS failed, please check your configuration. [details] InvalidGroup.NotFound: The se...

  • 1226 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
Honored Contributor II
  • 0 kudos

hey @sg-vtc You need to register a new network configuration in Databricks that points to the updated VPC and security group. Here’s how:- Go to the Databricks Account Console (not the workspace UI). (Click in workspaces dropbox and Manage Account)- ...

  • 0 kudos
alsetr
by New Contributor III
  • 376 Views
  • 1 replies
  • 0 kudos

Costs incurred by systems table usage

Doc about system tables says "The billable usage and pricing tables are free to use. Tables in Public Preview are also free to use during the preview but could incur a charge in the future."https://learn.microsoft.com/en-us/azure/databricks/admin/sys...

  • 376 Views
  • 1 replies
  • 0 kudos
Latest Reply
SP_6721
Contributor III
  • 0 kudos

Hi @alsetr ,The billable usage and pricing system tables are the clearly marked as free to use.Tables in Public Preview are also free for now.For the rest, there’s no mention of a direct cost, but you’ll still be billed for the compute (DBUs) used wh...

  • 0 kudos
vishwas7
by New Contributor II
  • 1398 Views
  • 3 replies
  • 1 kudos

Using logged in user's identity in Databricks Apps

Hi Databricks Community, I recently started using Datbricks apps where I list some schemas and tables in the UI.What I explicitly want to do is only show the schemas and tables on which user actually have permission. Currently the databricks apps wou...

  • 1398 Views
  • 3 replies
  • 1 kudos
Latest Reply
samlauncher
New Contributor II
  • 1 kudos

https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#retrieve-user-authorization-credentials : they launched this recently to support user identity in databricks apps

  • 1 kudos
2 More Replies
gt-ferdyh
by New Contributor
  • 1090 Views
  • 1 replies
  • 0 kudos

Custom hostname for Azure Databricks workspace

For a client it's required that all applications use a subdomain of an internal hostname. Does anyone have a suggestion on how to achieve this? Options could be in the line of CNAME or Reverse Proxy, but couldn't get that to work yet.Does Azure Datab...

  • 1090 Views
  • 1 replies
  • 0 kudos
Latest Reply
Renu_
Valued Contributor II
  • 0 kudos

Hi @gt-ferdyh, Azure Databricks doesn’t natively support custom subdomains. However, to make it accessible via an internal subdomain, you can:Enable Azure Private Link for your Databricks workspace.In your internal DNS, create an A record or CNAME th...

  • 0 kudos
xx123
by New Contributor III
  • 794 Views
  • 2 replies
  • 0 kudos

Resolved! "Aws Invalid Kms Key State" when trying to start new cluster on AWS

Hey,We just established new environment based on AWS. Our first step was to create Cluster but while doing so, we have encountered an error. We tried different policies, configurations and instance types. All resulted in:Aws Invalid Kms Key State: Th...

  • 794 Views
  • 2 replies
  • 0 kudos
Latest Reply
xx123
New Contributor III
  • 0 kudos

Yes, I followed this document and that fixed it. Thanks.

  • 0 kudos
1 More Replies
APJESK
by New Contributor III
  • 1017 Views
  • 2 replies
  • 2 kudos

Databricks Logs

I’m trying to understand the different types of logs available in Databricks and how to access and interpret them. Could anyone please guide me on:What types of logs are available in Databricks?Where can I find these logs?How can I use these logs to ...

  • 1017 Views
  • 2 replies
  • 2 kudos
Latest Reply
MadhuB
Valued Contributor
  • 2 kudos

Hi @APJESK,In addition, job logs are very useful to monitor and troubleshoot any job failures. They can be found under Workflows. Workspace admins role is required to have full access on all jobs unless explicitly granted to the user by Job Owner/adm...

  • 2 kudos
1 More Replies
funsjanssen
by New Contributor III
  • 4895 Views
  • 1 replies
  • 8 kudos

Azure Databricks Multi Tenant Solution

Hello Everyone,For the past few months, we’ve been extensively exploring the use of Databricks as the core of our data warehousing product. We provide analytics dashboards to other organizations and are particularly interested in the Column-Level Sec...

  • 4895 Views
  • 1 replies
  • 8 kudos
Latest Reply
Namanak47
New Contributor II
  • 8 kudos

Could you please let me know or provide a link how is it done in Databricks AWS using OIDC or SAML directly as SSO providers.I have exactly same kind of use case

  • 8 kudos
Yuki
by Contributor
  • 1947 Views
  • 3 replies
  • 2 kudos

Resolved! Clarification on Databricks Access Modes: Standard vs. Dedicated with Unity Catalog

Hello,I’d like to ask for clarification regarding the access modes in Databricks, specifically the intent and future direction of the “Standard” and “Dedicated” modes.According to the documentation below:https://docs.databricks.com/aws/ja/compute/acc...

  • 1947 Views
  • 3 replies
  • 2 kudos
Latest Reply
Vidhi_Khaitan
Databricks Employee
  • 2 kudos

Hi @Yuki Thank you, I am glad I could help!

  • 2 kudos
2 More Replies
andreapeterson
by Contributor
  • 1224 Views
  • 2 replies
  • 0 kudos

Have authenticator app set up but databricks still resorts to email mfa

Good monring,I set up an authenticator app through settings, profile, mfa. However, when I logout and log back into the workspace, I am still only getting the email verification to get in. Is there a way I can turn off email verification or have it d...

  • 1224 Views
  • 2 replies
  • 0 kudos
Latest Reply
Takuya-Omi
Valued Contributor III
  • 0 kudos

Hi, @andreapeterson Do you have Account Admin privileges?If you do, you can enforce MFA from the account console, just as shown in the attached image.If you’re using the Databricks Free Edition rather than a paid account, you can’t enforce MFA. This ...

  • 0 kudos
1 More Replies
prakash4
by New Contributor
  • 1161 Views
  • 1 replies
  • 0 kudos
  • 1161 Views
  • 1 replies
  • 0 kudos
Latest Reply
TheOC
Contributor
  • 0 kudos

hey @prakash4 I understand you're having trouble connecting to a remote server / getting your Databricks workspace to start. Would you be able to share a little more detail to help track down your issue?Specifically:- What happens when you try to sta...

  • 0 kudos
r_w_
by New Contributor II
  • 3035 Views
  • 6 replies
  • 1 kudos

Resolved! Best Practices for Mapping Between Databricks and AWS Accounts

Hi everyone, this is my first post here. I'm doing my best to write in English, so I apologize if anything is unclear.I'm looking to understand the best practices for how many environments to set up when using Databricks on AWS. I'm considering the f...

  • 3035 Views
  • 6 replies
  • 1 kudos
Latest Reply
Isi
Honored Contributor II
  • 1 kudos

Hey @r_w_ If you think my answer was correct, it would be great if you could mark it as a solution to help future users Thanks,Isi

  • 1 kudos
5 More Replies
Malthe
by Contributor
  • 490 Views
  • 1 replies
  • 3 kudos

Resolved! Cost of SQL Warehouse scaled to 0 clusters

A SQL Warehouse can be scaled to a minimum of 0.Presumably, there is still a cost to keeping the resource active, because we also have Auto-Stop which can completely stop the warehouse after a configurable amount of time.This cost is not documented. ...

  • 490 Views
  • 1 replies
  • 3 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 3 kudos

1. Just to clarify, minimum is 1, not 0. See https://docs.databricks.com/api/workspace/warehouses/create#min_num_clusters 2. How it works is based on concurrency. Assuming your max clusters is 2, and based on the concurrency, we don't need the 2nd cl...

  • 3 kudos