- 21996 Views
- 5 replies
- 1 kudos
Insufficient privileges:User does not have permission SELECT on any file
Hello,after switching to "shared cluster" usage a python job is failing with error message: Py4JJavaError: An error occurred while calling o877.load. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User...
- 21996 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @GeKo The checkpoint directory, is that set on cluster level or how do we set that ? Can you please help me with this ?
- 1 kudos
- 1506 Views
- 1 replies
- 0 kudos
Databricks UC Data Lineage Official Limitations
Hi all.I have a huge data migration project using medallion architecture, UC, notebooks and workflows . One of the relevant requirements we have is to capture all data dependencies (upstreams and downstreams) using data lineage. I've followed all re...
- 1506 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @RobsonNLPT , Yes SQL CTE are supported by the data lineage service. You can track table that were created using CTEs. Here is an example that demonstrate the feature. CREATE TABLE IF NOT EXISTS mpelletier.dbdemos.menu ( recipe_id INT, ...
- 0 kudos
- 2914 Views
- 3 replies
- 1 kudos
Resolved! Ingesting and Transforming NetCDF Data in Delta Table on Databricks Cluster
Hi,I need to ingest and transform historical climate data into a Delta table. The data is stored in .nc format (NetCDF). To work with this format, specific C libraries for Python are required, along with particular versions of Python libraries (e.g.,...
- 2914 Views
- 3 replies
- 1 kudos
- 1922 Views
- 5 replies
- 0 kudos
Service Principal Access to Users Directory in Databricks - Creating Git Folders
I am trying to automate the creation of git folders in user workspace directories triggered by GitHub feature branch creation. When developers create feature branches in GitHub, we want a service principal to automatically create corresponding git fo...
- 1922 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Brianhourigan, Can you please DIM your suggestions? I can add it to our internal AHA idea.
- 0 kudos
- 2831 Views
- 5 replies
- 0 kudos
Restore deleted databricks jobs and job runs
Hi All,Is there a way to restore deleted databricks jobs?Thank you.
- 2831 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @iptkrisna ,Currently, there is no option to recover deleted items. In architectures, it not necessary to control or manage the final code available in the system. Instead, the focus should be controlling and managing how code and jobs are deploye...
- 0 kudos
- 2431 Views
- 1 replies
- 1 kudos
Resolved! Unity Catalog : RDD Issue
In our existing notebooks, the scripts are reliant on RDDs. However, with the upgrade to Unity Catalog, RDDs will no longer be supported. We need to explore alternative approaches or tools to replace the use of RDDs. Could you suggest the best practi...
- 2431 Views
- 1 replies
- 1 kudos
- 1 kudos
To transition from using RDDs (Resilient Distributed Datasets) to alternative approaches supported by Unity Catalog, you can follow these best practices and migration strategies: Use DataFrame API: The DataFrame API is the recommended alternative to...
- 1 kudos
- 2285 Views
- 7 replies
- 0 kudos
Resolved! Tutorial docs for running a job using serverless?
I'm exploring whether serverless (https://docs.databricks.com/en/jobs/run-serverless-jobs.html#create-a-job-using-serverless-compute) could be useful for our use case. I'd like to see an example of using serverless via the API. The docs say "To learn...
- 2285 Views
- 7 replies
- 0 kudos
- 888 Views
- 1 replies
- 1 kudos
Resolved! Is there a cluster option for dashboards?
Hi everyone,I do not want to use 4 DBU/h XS warehouse since I have very tiny data on the new startup. I want to create a minimal cluster and run it as the underlying SQL engine for my dashboard.Thanks.
- 888 Views
- 1 replies
- 1 kudos
- 1 kudos
Unfortunately no, as dashboards are part of the SQL service on the platform they are designed to work with SQL warehouses only, you can create Notebook dashboards that will be able to work with regular clusters but functionalities will be limited in ...
- 1 kudos
- 1572 Views
- 5 replies
- 1 kudos
Resolved! Databricks workflow with sequenced tasks
I have a continuous workflow. It is continuous because I would like it to run every minute and if it has stuff to do the first task will take several minutes. As I understand, continuous workflows won't requeue while a job is currently running, where...
- 1572 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @h2p5cq8, No problem! and you can have the queue option disabled to stop it. Go to the Advanced settings in the Job details side panel and toggle off the Queue option to prevent jobs from being queued
- 1 kudos
- 1097 Views
- 1 replies
- 0 kudos
How Development Target works for multiple users?
Hi, I'm using the Databricks asset bundle to deploy my job to Azure Databricks.I want to configure the Databricks bundle so that when anyone runs the Azure pipeline, a job is created under their name in the format dev_username_job.Using a personal ac...
- 1097 Views
- 1 replies
- 0 kudos
- 0 kudos
Were you able to solve this?
- 0 kudos
- 6488 Views
- 5 replies
- 0 kudos
Azure Databricks Enterprise Application User Impersonation Token Group Claims Issue
Hi all, I am using the Azure Databricks Microsoft Managed Enterprise Application scope (2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/user_impersonation) to fetch an access token on behalf of a user. The authentication process is successful; however, the acce...
- 6488 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ahsan_aj, You can modify your token request by adding a claims parameter const claimsRequest = { "access_token": { "groups": null } https://learn.microsoft.com/en-us/security/zero-trust/develop/configure-tokens-gro...
- 0 kudos
- 1849 Views
- 1 replies
- 0 kudos
Best practices for tableau to connect to Databricks
Having problem in connecting to Databrikcs with service principal from tableau . Wanted to how how tableau extracts refreshing connecting to databricks , is it via individual Oauth or service principal
- 1849 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @cheerwthraj, To connect Tableau to Databricks and refresh extracts, you can use either OAuth or service principal authentication. For best practices, please refer to the below link, https://docs.databricks.com/en/partners/bi/tableau.html#best-pr...
- 0 kudos
- 1420 Views
- 1 replies
- 1 kudos
New Cluster 90% memory already consumed
Hi, seeing this on all new clusters (single or multi-node) I am creating. As soon as the metrics start showing up, the memory consumption shows 90% already consumed between Used and Cached (something like below). This is the case with higher or lower...
- 1420 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @AbhishekNegi I understand your concern. The reason for you to see memory consumption before initiating any task and regarding the comment taking time to execute. This is how Spark internally works. The memory consumption observed in a Spark clust...
- 1 kudos
- 6779 Views
- 15 replies
- 3 kudos
Delta Live Tables Permissions
Hi allI'm the owner of delta live tables pipelines but I don't see the option described on documentation to grant permissions for different users. The options available are "settings" and "delete"In the sidebar, click Delta Live Tables.Select the nam...
- 6779 Views
- 15 replies
- 3 kudos
- 3 kudos
Ok might be that the version of the workspaces could be different and the new patch will be implemented soon.
- 3 kudos
- 3547 Views
- 1 replies
- 0 kudos
How the Scale up process done in the databricks cluster?
For my AWS databricks cluster, i configured shared computer with 1min worker node and 3 max worker node, initailly only one worker node and driver node instance is created in the AWS console. Is there any rule set by databricks for scale up the next ...
- 3547 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks uses autoscaling to manage the number of worker nodes in a cluster based on the workload. When you configure a cluster with a minimum and maximum number of worker nodes, Databricks automatically adjusts the number of workers within this ra...
- 0 kudos
