- 4214 Views
- 1 replies
- 0 kudos
Resolved! Error: cannot create metastore data access
I'm in the progress of enabling Databricks Unity Catalog and encountered a problem with the databricks_metastore_data_access Terraform resource:resource "databricks_metastore_data_access" "this" { provider = databricks.account-level metastore_id...
- 4214 Views
- 1 replies
- 0 kudos
- 0 kudos
Found a solution. Please see my answer on Stack Overflow:https://stackoverflow.com/questions/77440091/databricks-unity-catalog-error-cannot-create-metastore-data-access/77506306#77506306
- 0 kudos
- 4095 Views
- 4 replies
- 2 kudos
Failed to start cluster: Large docker image
I have a large Docker image in our AWS ECR repo. The image is 27.4 GB locally and 11539.79 MB compressed in ECR.The error from the Event Log is:Failed to add 2 containers to the compute. Will attempt retry: true. Reason: Docker image pull failureJSON...
- 4095 Views
- 4 replies
- 2 kudos
- 2 kudos
I have a similar problem. a 10gb image pulls fine but a 31gb image doesnt. both workers and drivers have 64gb memory. i get the timeout error with "Cannot launch the cluster because pulling the docker image failed. Please double check connectivity fr...
- 2 kudos
- 5426 Views
- 0 replies
- 0 kudos
Is it possible to change the Azure storage account of Unity Catalog?
We have unity catalog metastore set up in storage account prod_1. Can we move this to prod_2 storage account and delete prod_1?Also, is it possible to rename the catalogs once they are created?
- 5426 Views
- 0 replies
- 0 kudos
- 5309 Views
- 3 replies
- 2 kudos
Connect to databricks from external non-spark cluster
Hi,I have an app/service on a non-spark kubernetes cluster. Is there a way to access/query a databricks service from my app/service? I see documentations on connectors, particularly on scala which is the code of my app/service. Can I use these connec...
- 5309 Views
- 3 replies
- 2 kudos
- 2692 Views
- 0 replies
- 0 kudos
Databricks deployment and automation tools comparison.
Hello All, As a newcomer to databricks, I am seeking guidance on automation within databricks environments. What are the best best practices for deployment, and how do Terraform, the REST API, and the databricks SDK compare in terms of advantages and...
- 2692 Views
- 0 replies
- 0 kudos
- 1676 Views
- 1 replies
- 0 kudos
Notebook Id level of uniqueness
Hi there,We know that notebook ids are unique. https://docs.databricks.com/en/workspace/workspace-details.html but I want to know in what level they're unique. For example, if Notebook Ids are unique within a workspace, or are they universally unique...
- 1676 Views
- 1 replies
- 0 kudos
- 3429 Views
- 2 replies
- 1 kudos
Have "viewer's credential" as default setting in Query access control, instead of "owner's credential"
Hi everybody,I am curious if there is a way to raise the security level by default. Specifically if it is possible to configure the default setting to limit the access of the query, rather than granting it open, as right now from my side the default ...
- 3429 Views
- 2 replies
- 1 kudos
- 1 kudos
I would like this too. Or perhaps the ability to change this programmatically in bulk. The CLI, SDK and Rest Endpoints don't appear to work and will not let me change this.
- 1 kudos
- 1864 Views
- 0 replies
- 0 kudos
Ubuntu 22 ODBC Connectivity Issue with PHP - SQL error: [unixODBC][Driver Manager]Can't open lib
Dear Friends,I'm having trouble connecting to Databricks ODBC from Ubuntu 22. I followed the steps documented here: https://docs.databricks.com/en/integrations/jdbc-odbc-bi.html#odbc-linuxHere is my odbc.ini file: [ODBC Data Sources] Databricks=Datab...
- 1864 Views
- 0 replies
- 0 kudos
- 1524 Views
- 0 replies
- 0 kudos
Support for setting R repository URLs in Databricks
The documentation for using R in Databricks states that the session can be configured by modifying the /usr/lib/R/etc/Rprofile.site file. This works for most things, however the repository URLs set by the `repos` option is overridden by another scrip...
- 1524 Views
- 0 replies
- 0 kudos
- 8751 Views
- 4 replies
- 1 kudos
Resolved! DBT job stuck when running on databricks
HiI'm trying to run a DBT job on a databricks instance. The query should be run on the same instance.When I run the job, I get to: Opening a new connection, currently in state initIt is stuck in that phase for a long time. I'm using IP access list wh...
- 8751 Views
- 4 replies
- 1 kudos
- 1 kudos
I recreated the databricks (there's no other way to solve that). If it was a production databricks workspace it was a disaster!I have created a VM with static public IP and added this IP to the IP access list. Hopefully it'll become the last resort i...
- 1 kudos
- 3932 Views
- 3 replies
- 0 kudos
Databricks Access Bundles - config data needed by notebook
I have this structure - Folder-1 - the root of databricks access directory. "databricks.yaml" file is in this directoryFolder-1 / Folder-2 has notebooks. One of the notebook, "test-notebook" is used for *job* configuration in databricks.yaml file.Fo...
- 3932 Views
- 3 replies
- 0 kudos
- 0 kudos
@GiggleByte @Yes based on demo test that I have done, it is working as you said. JSON converted yaml config for job setting need to be placed under resources, that yaml has job config setting, it looks similar to rest api json request converted in f...
- 0 kudos
- 9293 Views
- 7 replies
- 0 kudos
Can't access to manage account console
Hello!I have got problem with logging into manage account console.After clicking the Manage Account button, it takes me to the Workspace selection window and after selecting any of them, my workspace opens again.The first login was done by Acitve Dir...
- 9293 Views
- 7 replies
- 0 kudos
- 0 kudos
I'm facing the same problem, I´m admin on my workspace. Please someone from the Databricks team could respond what is happening when users can't access Accounts to create catalog ??????? It stay in infinite loops requesting the workspace name.
- 0 kudos
- 4572 Views
- 3 replies
- 0 kudos
Resolved! Small files and discrepancy in S3 vs catalog
Hello all,I'm in the process of optimizing my tables and I'm running into a confusing situation. I have a table named "trace_messages_fg_streaming_event". If I navigate to the Databricks catalog, it shows stats:Size: 6.7GB, Files: 464But when I look ...
- 4572 Views
- 3 replies
- 0 kudos
- 0 kudos
Nevermind, I'm dumb, figured it out, high frequency transactions leave many little files that need to be vacuumed.
- 0 kudos
- 5443 Views
- 5 replies
- 2 kudos
Cannot access admin group members
Hi,I'm using Azure databricks premium tier and lately, when I'm creating a new workspace and after that trying to browse to the admin members page, I get the following error:Uncaught Error: No QueryClient set, use QueryClientProvider to set oneReload...
- 5443 Views
- 5 replies
- 2 kudos
- 5142 Views
- 2 replies
- 1 kudos
Resolved! Azure databricks issue
Hi,I'm using Azure databricks service and having trouble blocking public access to my workspace.The security groups that are attached to the subnets that databrics is using are being ignored.From what I've learned in the last couple of days, the data...
- 5142 Views
- 2 replies
- 1 kudos
- 1 kudos
HiForgot to mention that I was using standard edition when writing this question but after struggling too long with that we've decided to upgrade to premium and use the IP access list feature and that worked!ThanksGil
- 1 kudos
