cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ErikApption
by New Contributor II
  • 852 Views
  • 1 replies
  • 1 kudos

Resolved! Timeout settings for Postgresql external catalog connection?

Is there any way to configure timeouts for external catalog connections? We are getting some timeouts with complex queries accessing a pgsql database through the catalog. We tried configuring the connection and we got this error  │ Error: cannot upda...

  • 852 Views
  • 1 replies
  • 1 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 1 kudos

Hello @ErikApption, there is no direct support for a connectTimeout option in the connection settings through Unity Catalog as of now. You might need to explore these alternative timeout configurations or consider adjusting your database handling to ...

  • 1 kudos
MaximeGendre
by New Contributor III
  • 2584 Views
  • 0 replies
  • 0 kudos

Dataiku connector limitation

Hello,I'm trying to read data from Unity Catalog and insert it into an Oracle Database using an "On Premise" Dataiku.It works well for a small dataset ~600Kb/~150 000 rows.[14:51:20] [INFO] [dku.datasets.sql] - Read 2000 records from DB [14:51:20] [I...

MaximeGendre_0-1739993758870.png
  • 2584 Views
  • 0 replies
  • 0 kudos
Christian_j
by New Contributor
  • 1459 Views
  • 3 replies
  • 0 kudos

Cannot create a workspace on GCP

Hi,I have been using Databricks for a couple of months and been spinning up workspaces with Terraform. The other day we decided to end our POC and move on to a MVP. This meant cleaning up all workspaces and GCP. after the cleanup was done I wanted to...

  • 1459 Views
  • 3 replies
  • 0 kudos
Latest Reply
MariuszK
Valued Contributor III
  • 0 kudos

Did you try from Marketplace? You may get there more detailed error.

  • 0 kudos
2 More Replies
CVveeLT
by New Contributor II
  • 736 Views
  • 2 replies
  • 0 kudos

Can we create an external location from a different tenant in Azure

We are looking to add an external location which points to a storage account in another Azure tenant. Is this possible? Could you point to any documentation around this.Currently, when we try to add a new credential providing a DBX access connector a...

  • 736 Views
  • 2 replies
  • 0 kudos
Latest Reply
CVveeLT
New Contributor II
  • 0 kudos

Thanks for the response @Alberto_Umana .Looks like the IDs are all provided correctly.  Here is the config -Tenant A                                                                Tenant BDatabricks is hosted here                                     ...

  • 0 kudos
1 More Replies
eondatatech
by New Contributor
  • 1721 Views
  • 1 replies
  • 0 kudos

creating Workspace in AWS with Quickstart is giving error

Hello, While  creating workspace in AWS using Quickstart, I get below error.  I used both admin Account and root account to create this but both gave the same issue. Any help is appreciated.  The resource CopyZipsFunction is in a CREATE_FAILED stateT...

  • 1721 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @eondatatech, Ensure that both the admin and the root account you are using to create the workspace have the necessary IAM permissions to create and manage Lambda functions. Specifically, check if the CreateFunction and PassRole permissions are...

  • 0 kudos
edouardtouze
by New Contributor II
  • 1277 Views
  • 3 replies
  • 1 kudos

Databricks on GCP with GKE | Cluster stuck in starting status | GKE allocation ressource failing

Hi Databricks Community,I’m currently facing several challenges with my Databricks clusters running on Google Kubernetes Engine (GKE). I hope someone here might have insights or suggestions to resolve the issues.Problem Overview:I am experiencing fre...

  • 1277 Views
  • 3 replies
  • 1 kudos
Latest Reply
chalkboardbrad
New Contributor II
  • 1 kudos

I am having similar issues. first time I am using the `databricks_cluster` resource, my terraform apply does not gracefully complete, and I see numerous errors about:1. Can’t scale up a node pool because of a failing scheduling predicateThe autoscale...

  • 1 kudos
2 More Replies
MDV
by New Contributor III
  • 1561 Views
  • 1 replies
  • 0 kudos

Resolved! ALTER TABLE ... ALTER COLUMN .... SYNC IDENTITY not working anymore ?

Hello,I recently noticed that the ALTER TABLE ALTER COLUMN SYNC IDENTITY command is no longer functioning as expected.I have an IDENTITY column on my table:D_Category_SID BIGINT GENERATED BY DEFAULT AS IDENTITY (START WITH 1 INCREMENT BY 1)Previously...

  • 1561 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @MDV, Thanks for your question. According to the recent updates, the SYNC IDENTITY command is now more restrictive and follows stronger invariants. Specifically, it no longer allows the high watermark to be reduced to ensure that there is no ri...

  • 0 kudos
ianchenmu
by New Contributor III
  • 1026 Views
  • 3 replies
  • 0 kudos

Is there a way to switch default cluster associated with a workflow job

Hi, I have a workflow job that is connected to a default cluster (see blow)I know I can swap the cluster. However, sometimes the cluster is not active but when I start the workflow job, I will wait for the cluster to beome activated. It will take som...

ianchenmu_0-1739506495579.png
  • 1026 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I suppose you can call the databricks api to run those workflows?Or is that a no go?

  • 0 kudos
2 More Replies
vidya_kothavale
by Contributor
  • 2951 Views
  • 0 replies
  • 0 kudos

Unity Catalog Not Enabled on Job Cluster When Creating DLT in GCP Databricks

I am trying to create a Delta Live Table (DLT) in my GCP Databricks workspace, but I am encountering an issue where Unity Catalog is not enabled on the job cluster.Steps I followed:Created a DLT pipeline using the Databricks UI.Selected the appropria...

  • 2951 Views
  • 0 replies
  • 0 kudos
Paladin
by New Contributor
  • 626 Views
  • 3 replies
  • 0 kudos

Does Databricks support configuring more than 1 Microsoft Entra ID in 1 Databricks account for SSO?

Can I configure more than 1 Microsoft Entra ID for a Databricks account for SSO? For example, I have 2 Microsoft Entra IDs: AD1 and AD2,  and I want to configure them into 1 Databricks account, so I can share the data or workspaces to the users in th...

  • 626 Views
  • 3 replies
  • 0 kudos
Latest Reply
Rjdudley
Honored Contributor
  • 0 kudos

No, an account is specific to the EntraID tenant and region, so you can only integrate SCIM with one tenant.  You'd have to make the users in AD2 guests in AD1 and then manage all the users in AD1.  We have a similar setup.  Clunky but works.

  • 0 kudos
2 More Replies
Prasad329
by New Contributor
  • 3172 Views
  • 1 replies
  • 0 kudos

Automating Version Control for Databricks Workflows

I am currently using Databricks Asset Bundles to manage and deploy workflows. While I have successfully automated the version control for notebooks, I am facing challenges with workflows. Specifically, I am looking to automate the process of fetching...

  • 3172 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

using the UI that is not possible I think.When using DAB and YML files it can be done.So I suggest you create the workflow using the UI (because it is easy to use) and then create a DAB out of that (using bundle generate).I admit, there is still some...

  • 0 kudos
erigaud
by Honored Contributor
  • 1026 Views
  • 1 replies
  • 2 kudos

Resolved! Running job within job fails

Hello,I have a job with a task of type "Run Job". Everything is deployed using asset bundles and the deployment works fine, however when running the job, the Job step fails with error "PERMISSION DENIED : User unknown does not have Manage Run or Owne...

  • 1026 Views
  • 1 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

the permissions of a main job are not copied to nested jobs, so the executing user needs the proper permissions for the main job and the nested job.This can be defined in the permissions seciont of the job (not the task).I for one am waiting for a ce...

  • 2 kudos
JamesDryden
by New Contributor II
  • 1960 Views
  • 3 replies
  • 5 kudos

How are you deploying graphs?

Hi all, I have a couple of use cases that may benefit from using graphs.  I'm interested in whether anyone has graph databases in Production and, if so, whether you're using GraphFrames, Neo4j or something else?  What is the architecture you have the...

  • 1960 Views
  • 3 replies
  • 5 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

Up to now the way to go is graphx or graphframes.There is also the possibility to use python libraries or others (single node that is), perhaps even Arrow-based.Another option is to load the data to a graph database and then move back to databricks a...

  • 5 kudos
2 More Replies
sri840
by New Contributor
  • 906 Views
  • 3 replies
  • 0 kudos

Databricks Asset bundles

Hi Team,In our company we are planning to migrate our workflows with Databricks Asset bundles, is it mandatory to install Databricks CLI tool for getting started with DAB ? Any one who integrated with Github with CICD pipeline please let me know the ...

  • 906 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I forgot the CI/CD part:that is not that hard.  Basically in DAB you define the type of environment you are using.If you use 'development', DAB assumes you are in actual development mode (feature branch).  so there you can connect git and put the fil...

  • 0 kudos
2 More Replies