cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MadMax_71_2
by New Contributor II
  • 494 Views
  • 2 replies
  • 0 kudos

Resolved! Lineage & Query history table

Hi,I’m trying to perform a join between the Lineage and Query History system tables. Initially, I assumed statement_id could be used as the join key, but that didn’t seem correct. I then noticed that joining statement_id with entity_run_id also appea...

  • 494 Views
  • 2 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @MadMax_71_2 ,Check ERD diagram. Here you should find what's the correct key:docs.databricks.com/aws/en/assets/images/system-tables-erd-690b80220d077015a023f90d59ff7560.svg

  • 0 kudos
1 More Replies
Seunghyun
by Contributor
  • 757 Views
  • 5 replies
  • 3 kudos

Resolved! Questions regarding catalogs starting with __databricks_internal_catalog

Hello,I am a Data Engineer currently building a data platform in a Databricks-AWS environment.While querying system.information_schema.catalog_privileges, I noticed several catalogs starting with __databricks_internal_catalog. Specifically, there are...

  • 757 Views
  • 5 replies
  • 3 kudos
Latest Reply
Seunghyun
Contributor
  • 3 kudos

Thanks for everybody!!@stbjelcevic @Sumit_7 

  • 3 kudos
4 More Replies
Drew_Prof
by New Contributor II
  • 900 Views
  • 3 replies
  • 2 kudos

Resolved! College Course Use - Sharing Data With Students

Hello Everyone,I am creating a new college course on Database design and SQL analytics and have decided to use Databricks as our platform in the course.  We are going to be using the Free Edition so students do not need to pay for access.  I'm wonder...

  • 900 Views
  • 3 replies
  • 2 kudos
Latest Reply
Drew_Prof
New Contributor II
  • 2 kudos

Excellent information, thank you!  The data generator was something I was not aware of so I will check that out.

  • 2 kudos
2 More Replies
flynnpham920
by New Contributor II
  • 845 Views
  • 2 replies
  • 3 kudos

Resolved! Communication between two workspaces

Hi everyone, our team is implementing an application utilizing the Databricks App (in AzureDatabricks) to deploy a website in workspace A (sea). This web calls an API to the Databricks Dashboards in workspace B (jpe) to get a token to embed the dashb...

  • 845 Views
  • 2 replies
  • 3 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 3 kudos

The specific error message is a known symptom of Databricks network-policy enforcement for cross-workspace calls, not (only) an OAuth/service-principal problem.403 Cert validation failed. Both workspace comparison and snp system trusted checks did no...

  • 3 kudos
1 More Replies
imbrigliaf
by New Contributor II
  • 281 Views
  • 1 replies
  • 2 kudos

Resolved! Company card / billing in Databricks express setup

For Databricks express setup only, is there a way to add company's VAT number, name and address during (or after) the "add a payment method" form, in order to receive a valid company's billing document from Databricks?

  • 281 Views
  • 1 replies
  • 2 kudos
Latest Reply
pradeep_singh
Contributor III
  • 2 kudos

I dont thing this can be done. You can check with Tech support as well . https://help.databricks.com/

  • 2 kudos
Teo12333
by New Contributor II
  • 4452 Views
  • 2 replies
  • 0 kudos

How do I get rid of the GKE cluster?

hi!In our organisation we use databricks but I do not understand why this GKE cluster keeps getting created. We deploy workspaces and compute clusters through terraform and use the GCE tag"x-databricks-nextgen-cluster" = "true"From my understanding, ...

  • 4452 Views
  • 2 replies
  • 0 kudos
Latest Reply
Korakot
Databricks Partner
  • 0 kudos

Hello @Louis_Frolio I would like to confirm my understanding regarding cost attribution for workloads running on Next-Gen / Serverless / Ephemeral Compute that are tagged withx-databricks-nextgen-cluster = "true".Is it correct that, in this case, the...

  • 0 kudos
1 More Replies
fgeriksen
by New Contributor II
  • 613 Views
  • 3 replies
  • 1 kudos

Resolved! Enabling External Lineage on a free or trial account?

Hi,as part of a small OSS project I am doing, dbt-unity-lineage, I need to enable Bring your own data lineage (Public Preview as of December 2025). But it seems you can't enable that Preview in either free edition or Trial?I'd rather not use my emplo...

Administration & Architecture
dbt
dbt GA PublicPreview
  • 613 Views
  • 3 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

@fgeriksen , if you are satisfied with the response please "Accept as Solution" so that others will be informed as well.  Cheers, Louis.

  • 1 kudos
2 More Replies
4Twannie
by New Contributor III
  • 1678 Views
  • 5 replies
  • 3 kudos

Delta Sharing from Databricks to SAP BDC fails with invalid_client error

ContextWe are in the process of extracting data between SAP BDC Datasphere and Databricks (Brownfield Implementation).SAP Datasphere is hosted in AWS (eu10)Databricks is hosted in Azure (West Europe)The BDC Connect System is located in the same regio...

  • 1678 Views
  • 5 replies
  • 3 kudos
Latest Reply
anshu_roy
Databricks Employee
  • 3 kudos

The error DELTA_SHARING_INVALID_RECIPIENT_AUTH refers to an invalid authorization specification when accessing Delta Sharing resources. This maps to SQLSTATE code 28000 ("invalid authorization specification") and typically occurs when the recipient's...

  • 3 kudos
4 More Replies
Daniela_Boamba
by New Contributor III
  • 522 Views
  • 1 replies
  • 1 kudos

Databricks certificate expired

Hello,I have a databricks workspace with sso authentication. the IDP is on azure.The client certificate expired and now, I can't log on to databricks to add the new one.How can I do? Any idea is welcomed.Thank you!!Best regards,daniela 

  • 522 Views
  • 1 replies
  • 1 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 1 kudos

This is an AWS Databricks workspace and your SSO is with EntraID? You'll need to create a Support Ticket and then Engineering can disable-SSO temporarily allowing you to login with user+OTP. The long term solution here is that you should: Set up Acco...

  • 1 kudos
jAAmes_bentley
by Databricks Partner
  • 2812 Views
  • 2 replies
  • 2 kudos

Resolved! Databricks Apps - X-Forwarded-Access-Token not available

Hi all,I've recently been getting started with Databricks Apps, and have been trying to use on-behalf-of access. However, it seems that the X-Forwarded-Access-Token is not coming through in my app headers. I have lots of other X-Forwarded ones such a...

jAAmes_bentley_0-1753219027973.png jAAmes_bentley_1-1753219116820.png
  • 2812 Views
  • 2 replies
  • 2 kudos
Latest Reply
jlieow
Databricks Employee
  • 2 kudos

Just to add from my experience, in case James’s solution doesn’t work for you: I tried fully stopping and restarting the app, but the user authentication page which asks for permission didn’t appear until I used incognito mode

  • 2 kudos
1 More Replies
dbrixr
by New Contributor II
  • 2835 Views
  • 3 replies
  • 1 kudos

Reverse colors in dark mode

It seems that Databricks implements its dark mode by applying the invert filter so that all colors are reversed. This is problematic if one wants to create some sort of html widget or rich output since this filter is passed down to the result of disp...

  • 2835 Views
  • 3 replies
  • 1 kudos
Latest Reply
valentin9
New Contributor II
  • 1 kudos

Also struggled with that issue. What I found is that there is a style applied onto the iframe: "filter: var(--invert-filter);" which applies this CSS: filter: invert(1) saturate(0.5).I couldn't find any elements within the iframe that I can use to de...

  • 1 kudos
2 More Replies
Harish_Kumar_M
by New Contributor
  • 934 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks Genie - Space Creation Restriction

Hi,As Genie space can be created by anyone who has access to create by attaching a Delta Table and SQLWH. Is there a way to control or restrict space creation only by Admin Team group?We want no one apart Admins to create and manage spaces. Users sho...

  • 934 Views
  • 2 replies
  • 0 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 0 kudos

Yes—lock space creation to your Admins by controlling entitlements and warehouse permissions. The ability to create a Genie space isn’t a separate toggle today; it’s implied by (a) the Databricks SQL workspace entitlement and (b) CAN USE on at least ...

  • 0 kudos
1 More Replies
Ashash12
by New Contributor II
  • 720 Views
  • 2 replies
  • 2 kudos

Resolved! The Lakeflow connect Gateway setup, do we need to install the agent on-prem?

The Lakeflow connect Gateway setup to connect on-prem SQL serverPlease provide the steps the setup the gateway agent on the on-prem?  Where to download this agent? What are the firewall rules for outbound looks like. Kind regards,Asha 

  • 720 Views
  • 2 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @Ashash12 ,You need to have proper network connectivity to your on premise SQL Server. As they stated in the docs - connector supports SQL Server on-premises using Azure ExpressRoute and AWS Direct Connect networkinghttps://docs.databricks.com/aws...

  • 2 kudos
1 More Replies
Sai03
by Databricks Partner
  • 1182 Views
  • 2 replies
  • 1 kudos

Resolved! Queries Hanging Indefinitely

I spun up a databricks environment on AWS via the AWS marketplace.All the required infrastructure such as S3, VPC, Subnets are automatically created during the processOnce I get the Databricks environment up and running - I created a cluster. I attac...

  • 1182 Views
  • 2 replies
  • 1 kudos
Latest Reply
emma_s
Databricks Employee
  • 1 kudos

Hi, I believe this is happening as you haven't got the right ports open to connect between your classic compute and the UC Metatstore. When you try to select 1 it works as it doesn't need to talk to the metastore but when you do show catalogs it is t...

  • 1 kudos
1 More Replies
hietpas
by Databricks Partner
  • 1605 Views
  • 3 replies
  • 0 kudos

Resolved! PERMISSION_DENIED: Request for user delegation key is not authorized.

I am attempting to copy files from an Azure Storage container using an Azure Databricks Volume. When attempting to list files using dbutils.fs.ls('/Volumes/myCatalog/mySchema/myVolume' I get the following error:ExecutionError: (com.databricks.sql.man...

  • 1605 Views
  • 3 replies
  • 0 kudos
Latest Reply
hietpas
Databricks Partner
  • 0 kudos

@szymon_dybczak Can you provide a link to the documentation you noted? I confirmed this with my own testing, that the Storage Blob Delegator role must be at the ADLS account-level, and Storage Blob Data Reader can then be applied at the container-lev...

  • 0 kudos
2 More Replies