- 1394 Views
- 2 replies
- 2 kudos
Resolved! The Lakeflow connect Gateway setup, do we need to install the agent on-prem?
The Lakeflow connect Gateway setup to connect on-prem SQL serverPlease provide the steps the setup the gateway agent on the on-prem? Where to download this agent? What are the firewall rules for outbound looks like. Kind regards,Asha
- 1394 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Ashash12 ,You need to have proper network connectivity to your on premise SQL Server. As they stated in the docs - connector supports SQL Server on-premises using Azure ExpressRoute and AWS Direct Connect networkinghttps://docs.databricks.com/aws...
- 2 kudos
- 2296 Views
- 2 replies
- 1 kudos
Resolved! Queries Hanging Indefinitely
I spun up a databricks environment on AWS via the AWS marketplace.All the required infrastructure such as S3, VPC, Subnets are automatically created during the processOnce I get the Databricks environment up and running - I created a cluster. I attac...
- 2296 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, I believe this is happening as you haven't got the right ports open to connect between your classic compute and the UC Metatstore. When you try to select 1 it works as it doesn't need to talk to the metastore but when you do show catalogs it is t...
- 1 kudos
- 3030 Views
- 3 replies
- 0 kudos
Resolved! PERMISSION_DENIED: Request for user delegation key is not authorized.
I am attempting to copy files from an Azure Storage container using an Azure Databricks Volume. When attempting to list files using dbutils.fs.ls('/Volumes/myCatalog/mySchema/myVolume' I get the following error:ExecutionError: (com.databricks.sql.man...
- 3030 Views
- 3 replies
- 0 kudos
- 0 kudos
@szymon_dybczak Can you provide a link to the documentation you noted? I confirmed this with my own testing, that the Storage Blob Delegator role must be at the ADLS account-level, and Storage Blob Data Reader can then be applied at the container-lev...
- 0 kudos
- 1744 Views
- 2 replies
- 0 kudos
Resolved! Need the cost explorer
We would like each workspace admin to be able to easily check the cost of their own workspace for free.Currently, we have a Usage dashboard, but it shows account-level costs and requires a SQL Warehouse to view. This means that additional costs are i...
- 1744 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks currently does not provide a free, workspace‑level cost viewer—all usage/cost dashboards are account‑level and require a SQL Warehouse, which does create cost.So your request is valid and aligns with what many customers want.Here are few o...
- 0 kudos
- 3550 Views
- 1 replies
- 2 kudos
Resolved! How to connect to AWS Custom VPC endpoint
Hello,Could somebody help me with the connection issue to my VPC endpointI have created a customer-managed VPC in AWS and setup new workspace with that VPC. There is an RDS in other vpc that I want to connect from Databricks, and I have created a VP...
- 3550 Views
- 1 replies
- 2 kudos
- 2 kudos
“Connection refused” means the TCP handshake reached your endpoint ENI and the backend actively rejected the connection. That’s different from a timeout (routing/DNS), so your PrivateLink plumbing and DNS are mostly correct. Short fixes you can try:...
- 2 kudos
- 857 Views
- 1 replies
- 0 kudos
Resolved! force_destroy/force_update option for workspace APIs
The Unity catalog DELETE API supports the force option:https://docs.databricks.com/api/workspace/catalogs/deleteIs it possible to support the force option for the workspace APIs?https://docs.databricks.com/api/account/workspaces/deleteContext:At the ...
- 857 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @littlewat ,Currently this is not supported by API. You can raise feature request though
- 0 kudos
- 7005 Views
- 5 replies
- 3 kudos
Drop table - permission management
Hello,I'm trying to wrap my head around the permission management for dropping tables in UC enabled schemas.According to docs: To drop a table you must have the MANAGE privilege on the table, be its owner, or the owner of the schema, catalog, or meta...
- 7005 Views
- 5 replies
- 3 kudos
- 3210 Views
- 2 replies
- 3 kudos
Resolved! Unity Catalog design in single workspace: dev/prod catalogs and schemas for projects — should we add
Hello everyone,We are currently designing our Unity Catalog structure and would like feedback on whether our approach makes sense and how it could be improved.Context:We use a single Databricks workspace shared by Data Engineering and Data Science/ML...
- 3210 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @JoaoPigozzo — great question. This one comes up all the time with the customers I train. I’ve been doing this for quite a while now and have had the chance to see a wide range of implementations and approaches out in the wild. While there’s no ...
- 3 kudos
- 961 Views
- 1 replies
- 0 kudos
Resolved! Need help with changing RunAs owner
Dear Contributors,I have a scenario where I am required to trigger a couple of jobs (lets say 4) based on the success of a master job. The master job has a Service Principal attached to it, which is the owner. It is going to produce a list of items w...
- 961 Views
- 1 replies
- 0 kudos
- 0 kudos
@ckunal_eng - One single Databricks Job run cannot dynamically change its "Run As" identity during execution. Rather you will need a pattern that separates the triggering identity from the executing identity.I would pre-configure 4 dependent jobs wit...
- 0 kudos
- 2096 Views
- 3 replies
- 3 kudos
Resolved! How to disable storage account key access of workspace storage accounts?
Each Azure Databricks workspace has an associated Azure storage account in a managed resource group known as the workspace storage account. The workspace storage account includes workspace system data (job output, system settings, and logs), DBFS roo...
- 2096 Views
- 3 replies
- 3 kudos
- 2476 Views
- 1 replies
- 1 kudos
Databricks AWS deployment with custom configurations (workspace root storage)
Hi everyone,I have a question about the IAM role for workspace root storage when deploying Databricks on AWS with custom configurations (customer-managed VPC, storage configurations, credential configurations, etc.).At an earlier stage of our deploym...
- 2476 Views
- 1 replies
- 1 kudos
- 1 kudos
Great question. [1] In the pre-UC world when you created a workspace you would designate a bucket/container that was used for what was most commonly known as DBFS. ie it's where the hive-metastore managed tables would be stored by default along with ...
- 1 kudos
- 2456 Views
- 3 replies
- 3 kudos
Resolved! system.lakeflow.job_task_run_timeline table missing task parameters on for each loop input
The system.lakeflow.job_task_run_timeline does not include the task level parameters on the input of the for each loop if dynamically setting the parameter in another notebook using dbutils.jobs.taskValues.set. This information is not included in the...
- 2456 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @hgintexas , You’re right, the system job timeline tables and the Runs API don’t currently surface the resolved per‑iteration inputs for a For-each task when those inputs are sourced via task values set in another notebook with dbutils.jobs.taskVa...
- 3 kudos
- 1020 Views
- 2 replies
- 0 kudos
Databricks import directory false positive import
Hello evryone,I'm using databricks CLI to moving several directories from Azure Repos to Databricks Workspace.The problem is that files are not updating properly, with no error to display.The self-hosted agent in pipeline i'm using has installed the ...
- 1020 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Giuseppe_C , Can you try running the command with the --debug flag to see if there is any additional information that you can gather? Full command: databricks import-dir <source> <dest> --overwrite --debugAlso, verify that the target path is not ...
- 0 kudos
- 4856 Views
- 6 replies
- 4 kudos
Resolved! DLT-Asset bundle : Pipelines do not support a setting a run_as user that is different from the owner
Hello !We're using Databricks asset bundles to deploy to several environments using a devops pipeline. The service principal running the CICD pipeline and creating the job (owner) is not the same as the SPN that will be running the jobs (run_as).This...
- 4856 Views
- 6 replies
- 4 kudos
- 4 kudos
Maybe I'm not catching this or missing something else but I've got the following job in one of my demo workspaces:Creator is my user and the job runs as a service principal account. Those are different identities. I got this by deploying the job with...
- 4 kudos
- 3598 Views
- 2 replies
- 1 kudos
Resolved! aws databricks with frontend private link
In aws databricks documentation, frontend PrivateLink assumes a separate transit VPC connected via Direct Connect/VPN. However, I'm implementing a different architecture using Tailscale for private network access.My setup: Tailscale subnet router de...
- 3598 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi everyone,I have a question about the IAM role for workspace root storage when deploying Databricks on AWS with custom configurations (customer-managed VPC, storage configurations, credential configurations, etc.).At an earlier stage of our deploym...
- 1 kudos
