- 7160 Views
- 5 replies
- 3 kudos
Drop table - permission management
Hello,I'm trying to wrap my head around the permission management for dropping tables in UC enabled schemas.According to docs: To drop a table you must have the MANAGE privilege on the table, be its owner, or the owner of the schema, catalog, or meta...
- 7160 Views
- 5 replies
- 3 kudos
- 3463 Views
- 2 replies
- 3 kudos
Resolved! Unity Catalog design in single workspace: dev/prod catalogs and schemas for projects — should we add
Hello everyone,We are currently designing our Unity Catalog structure and would like feedback on whether our approach makes sense and how it could be improved.Context:We use a single Databricks workspace shared by Data Engineering and Data Science/ML...
- 3463 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @JoaoPigozzo — great question. This one comes up all the time with the customers I train. I’ve been doing this for quite a while now and have had the chance to see a wide range of implementations and approaches out in the wild. While there’s no ...
- 3 kudos
- 1053 Views
- 1 replies
- 0 kudos
Resolved! Need help with changing RunAs owner
Dear Contributors,I have a scenario where I am required to trigger a couple of jobs (lets say 4) based on the success of a master job. The master job has a Service Principal attached to it, which is the owner. It is going to produce a list of items w...
- 1053 Views
- 1 replies
- 0 kudos
- 0 kudos
@ckunal_eng - One single Databricks Job run cannot dynamically change its "Run As" identity during execution. Rather you will need a pattern that separates the triggering identity from the executing identity.I would pre-configure 4 dependent jobs wit...
- 0 kudos
- 2258 Views
- 3 replies
- 3 kudos
Resolved! How to disable storage account key access of workspace storage accounts?
Each Azure Databricks workspace has an associated Azure storage account in a managed resource group known as the workspace storage account. The workspace storage account includes workspace system data (job output, system settings, and logs), DBFS roo...
- 2258 Views
- 3 replies
- 3 kudos
- 2596 Views
- 1 replies
- 1 kudos
Databricks AWS deployment with custom configurations (workspace root storage)
Hi everyone,I have a question about the IAM role for workspace root storage when deploying Databricks on AWS with custom configurations (customer-managed VPC, storage configurations, credential configurations, etc.).At an earlier stage of our deploym...
- 2596 Views
- 1 replies
- 1 kudos
- 1 kudos
Great question. [1] In the pre-UC world when you created a workspace you would designate a bucket/container that was used for what was most commonly known as DBFS. ie it's where the hive-metastore managed tables would be stored by default along with ...
- 1 kudos
- 2585 Views
- 3 replies
- 3 kudos
Resolved! system.lakeflow.job_task_run_timeline table missing task parameters on for each loop input
The system.lakeflow.job_task_run_timeline does not include the task level parameters on the input of the for each loop if dynamically setting the parameter in another notebook using dbutils.jobs.taskValues.set. This information is not included in the...
- 2585 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @hgintexas , You’re right, the system job timeline tables and the Runs API don’t currently surface the resolved per‑iteration inputs for a For-each task when those inputs are sourced via task values set in another notebook with dbutils.jobs.taskVa...
- 3 kudos
- 1076 Views
- 2 replies
- 0 kudos
Databricks import directory false positive import
Hello evryone,I'm using databricks CLI to moving several directories from Azure Repos to Databricks Workspace.The problem is that files are not updating properly, with no error to display.The self-hosted agent in pipeline i'm using has installed the ...
- 1076 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Giuseppe_C , Can you try running the command with the --debug flag to see if there is any additional information that you can gather? Full command: databricks import-dir <source> <dest> --overwrite --debugAlso, verify that the target path is not ...
- 0 kudos
- 4980 Views
- 6 replies
- 4 kudos
Resolved! DLT-Asset bundle : Pipelines do not support a setting a run_as user that is different from the owner
Hello !We're using Databricks asset bundles to deploy to several environments using a devops pipeline. The service principal running the CICD pipeline and creating the job (owner) is not the same as the SPN that will be running the jobs (run_as).This...
- 4980 Views
- 6 replies
- 4 kudos
- 4 kudos
Maybe I'm not catching this or missing something else but I've got the following job in one of my demo workspaces:Creator is my user and the job runs as a service principal account. Those are different identities. I got this by deploying the job with...
- 4 kudos
- 3806 Views
- 2 replies
- 1 kudos
Resolved! aws databricks with frontend private link
In aws databricks documentation, frontend PrivateLink assumes a separate transit VPC connected via Direct Connect/VPN. However, I'm implementing a different architecture using Tailscale for private network access.My setup: Tailscale subnet router de...
- 3806 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi everyone,I have a question about the IAM role for workspace root storage when deploying Databricks on AWS with custom configurations (customer-managed VPC, storage configurations, credential configurations, etc.).At an earlier stage of our deploym...
- 1 kudos
- 1710 Views
- 3 replies
- 2 kudos
Resolved! Removing access to Lakehouse and only allowing Databricks One?
Hello, I am trying to set up a user group for business users in our Azure Databricks that will only be able to query data. It looks like Databricks One is the solution to use. So I followed the documentation and granted the user group Consumer Access...
- 1710 Views
- 3 replies
- 2 kudos
- 2 kudos
Yeah, that was it. I had set up Databricks with Entra groups from the beginning and had done all my permission work there. I didn't even think of checking the default groups. Thank you!
- 2 kudos
- 1923 Views
- 1 replies
- 2 kudos
Resolved! how to complete connection to snowflake
Hi all,Simple test creating connection to my snowflake trial account was successful, and..Anyway:dbx > external data > connections > create connection > ... > [Test] successful but still see last 2 steps todo (5 access, 6 metadata) but don't see any ...
- 1923 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @emanueol ,From you description it seems that you were able to set up connection. Not every step in the wizard has "next button". From example in step 4 (catalog basic) if you create a catalog then you will be automatically transfered to the next ...
- 2 kudos
- 1418 Views
- 1 replies
- 0 kudos
Resolved! scroll bar disappears on widgets in dashboards
Databricks newbie.I've created a dashboard that has several widgets to allow users to select multiple values from a drop-down list. When I first open the widget to select the values, there is a scroll bar on the right side of the box which allows me...
- 1418 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @RichC! You’re not missing any setting here. This is expected behavior. The scrollbar auto-hides after a couple of seconds, but it’s still active. If you start scrolling again (mouse wheel or trackpad), the scrollbar will reappear.
- 0 kudos
- 2475 Views
- 3 replies
- 1 kudos
Tracing SQL costs
Hello, Databricks community!In our Account Usage Dashboard, the biggest portion of our costs are labeled simply "SQL".We want to drill deeper to see where the SQL costs are coming from.By querying the `system.usage.billing` table we see that it's mos...
- 2475 Views
- 3 replies
- 1 kudos
- 1 kudos
@simenheg - first of all, It’s not an error as Serverls SQL often produces null metadata fields.So you will need to follow below steps for the costUse SQL Warehouse Query Historyjoin billing data with SQL query history - system.billing.usage.usage_da...
- 1 kudos
- 4145 Views
- 4 replies
- 3 kudos
Resolved! AI/BI Dashboard embed issue in Databricks App
Hi everyone,I’ve created an AI/BI Dashboard in Azure Databricks and successfully published it, generated an embed link. My goal is to embed this dashboard inside a Databricks App (Streamlit) using an iframe.However, when I try to render the dashboard...
- 4145 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Louis_Frolio ,I have made changes my master menu with page navigation and used iframe inside submenu and it does work... Thanks for your insightful solution.
- 3 kudos
- 1091 Views
- 2 replies
- 0 kudos
Databricks Job : Unable to read Databricks job run parameter in scala code and sql query.
we are created data bricks job with Jar (scala code) and provided parameters/jar parameters and able to read those as arguments in main method. we are running job with parameters (run parameters / job parameters) , those parametes are not able to re...
- 1091 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Louis_Frolio Thank you for your suggestion. We are following the approach you recommended, but we encountered an issue while creating the job.We are creating a Databricks job using a JSON file through a pipeline. When we declare job-level parame...
- 0 kudos
