cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

dbrixr
by New Contributor II
  • 3010 Views
  • 3 replies
  • 1 kudos

Reverse colors in dark mode

It seems that Databricks implements its dark mode by applying the invert filter so that all colors are reversed. This is problematic if one wants to create some sort of html widget or rich output since this filter is passed down to the result of disp...

  • 3010 Views
  • 3 replies
  • 1 kudos
Latest Reply
valentin9
New Contributor II
  • 1 kudos

Also struggled with that issue. What I found is that there is a style applied onto the iframe: "filter: var(--invert-filter);" which applies this CSS: filter: invert(1) saturate(0.5).I couldn't find any elements within the iframe that I can use to de...

  • 1 kudos
2 More Replies
Harish_Kumar_M
by New Contributor
  • 1300 Views
  • 2 replies
  • 0 kudos

Resolved! Databricks Genie - Space Creation Restriction

Hi,As Genie space can be created by anyone who has access to create by attaching a Delta Table and SQLWH. Is there a way to control or restrict space creation only by Admin Team group?We want no one apart Admins to create and manage spaces. Users sho...

  • 1300 Views
  • 2 replies
  • 0 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 0 kudos

Yes—lock space creation to your Admins by controlling entitlements and warehouse permissions. The ability to create a Genie space isn’t a separate toggle today; it’s implied by (a) the Databricks SQL workspace entitlement and (b) CAN USE on at least ...

  • 0 kudos
1 More Replies
Ashash12
by New Contributor II
  • 992 Views
  • 2 replies
  • 2 kudos

Resolved! The Lakeflow connect Gateway setup, do we need to install the agent on-prem?

The Lakeflow connect Gateway setup to connect on-prem SQL serverPlease provide the steps the setup the gateway agent on the on-prem?  Where to download this agent? What are the firewall rules for outbound looks like. Kind regards,Asha 

  • 992 Views
  • 2 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 2 kudos

Hi @Ashash12 ,You need to have proper network connectivity to your on premise SQL Server. As they stated in the docs - connector supports SQL Server on-premises using Azure ExpressRoute and AWS Direct Connect networkinghttps://docs.databricks.com/aws...

  • 2 kudos
1 More Replies
Sai03
by Databricks Partner
  • 1577 Views
  • 2 replies
  • 1 kudos

Resolved! Queries Hanging Indefinitely

I spun up a databricks environment on AWS via the AWS marketplace.All the required infrastructure such as S3, VPC, Subnets are automatically created during the processOnce I get the Databricks environment up and running - I created a cluster. I attac...

  • 1577 Views
  • 2 replies
  • 1 kudos
Latest Reply
emma_s
Databricks Employee
  • 1 kudos

Hi, I believe this is happening as you haven't got the right ports open to connect between your classic compute and the UC Metatstore. When you try to select 1 it works as it doesn't need to talk to the metastore but when you do show catalogs it is t...

  • 1 kudos
1 More Replies
hietpas
by Databricks Partner
  • 2009 Views
  • 3 replies
  • 0 kudos

Resolved! PERMISSION_DENIED: Request for user delegation key is not authorized.

I am attempting to copy files from an Azure Storage container using an Azure Databricks Volume. When attempting to list files using dbutils.fs.ls('/Volumes/myCatalog/mySchema/myVolume' I get the following error:ExecutionError: (com.databricks.sql.man...

  • 2009 Views
  • 3 replies
  • 0 kudos
Latest Reply
hietpas
Databricks Partner
  • 0 kudos

@szymon_dybczak Can you provide a link to the documentation you noted? I confirmed this with my own testing, that the Storage Blob Delegator role must be at the ADLS account-level, and Storage Blob Data Reader can then be applied at the container-lev...

  • 0 kudos
2 More Replies
littlewat
by New Contributor II
  • 1324 Views
  • 2 replies
  • 0 kudos

Resolved! Need the cost explorer

We would like each workspace admin to be able to easily check the cost of their own workspace for free.Currently, we have a Usage dashboard, but it shows account-level costs and requires a SQL Warehouse to view. This means that additional costs are i...

  • 1324 Views
  • 2 replies
  • 0 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 0 kudos

Databricks currently does not provide a free, workspace‑level cost viewer—all usage/cost dashboards are account‑level and require a SQL Warehouse, which does create cost.So your request is valid and aligns with what many customers want.Here are few o...

  • 0 kudos
1 More Replies
alex_data
by New Contributor
  • 2967 Views
  • 1 replies
  • 2 kudos

Resolved! How to connect to AWS Custom VPC endpoint

Hello,Could somebody help me with the connection issue to my VPC endpointI have created a customer-managed VPC in AWS and setup new workspace with that VPC. There is an RDS  in other vpc that I want to connect from Databricks, and I have created a VP...

  • 2967 Views
  • 1 replies
  • 2 kudos
Latest Reply
nayan_wylde
Esteemed Contributor II
  • 2 kudos

“Connection refused” means the TCP handshake reached your endpoint ENI and the backend actively rejected the connection. That’s different from a timeout (routing/DNS), so your PrivateLink plumbing and DNS are mostly correct.  Short fixes you can try:...

  • 2 kudos
littlewat
by New Contributor II
  • 652 Views
  • 1 replies
  • 0 kudos

Resolved! force_destroy/force_update option for workspace APIs

The Unity catalog DELETE API supports the force option:https://docs.databricks.com/api/workspace/catalogs/deleteIs it possible to support the force option for the workspace APIs?https://docs.databricks.com/api/account/workspaces/deleteContext:At the ...

Administration & Architecture
force_destroy
workspace api
  • 652 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Esteemed Contributor III
  • 0 kudos

Hi @littlewat ,Currently this is not supported by API. You can raise feature request though

  • 0 kudos
PiotrM
by New Contributor III
  • 5893 Views
  • 5 replies
  • 3 kudos

Drop table - permission management

Hello,I'm trying to wrap my head around the permission management for dropping tables in UC enabled schemas.According to docs: To drop a table you must have the MANAGE privilege on the table, be its owner, or the owner of the schema, catalog, or meta...

  • 5893 Views
  • 5 replies
  • 3 kudos
Latest Reply
PiotrM
New Contributor III
  • 3 kudos

Hey,any news about this functionality being implemented? Br

  • 3 kudos
4 More Replies
JoaoPigozzo
by New Contributor III
  • 1890 Views
  • 2 replies
  • 3 kudos

Resolved! Unity Catalog design in single workspace: dev/prod catalogs and schemas for projects — should we add

Hello everyone,We are currently designing our Unity Catalog structure and would like feedback on whether our approach makes sense and how it could be improved.Context:We use a single Databricks workspace shared by Data Engineering and Data Science/ML...

  • 1890 Views
  • 2 replies
  • 3 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 3 kudos

Hey @JoaoPigozzo  — great question. This one comes up all the time with the customers I train. I’ve been doing this for quite a while now and have had the chance to see a wide range of implementations and approaches out in the wild. While there’s no ...

  • 3 kudos
1 More Replies
ckunal_eng
by New Contributor III
  • 645 Views
  • 1 replies
  • 0 kudos

Resolved! Need help with changing RunAs owner

Dear Contributors,I have a scenario where I am required to trigger a couple of jobs (lets say 4) based on the success of a master job. The master job has a Service Principal attached to it, which is the owner. It is going to produce a list of items w...

  • 645 Views
  • 1 replies
  • 0 kudos
Latest Reply
Raman_Unifeye
Honored Contributor III
  • 0 kudos

@ckunal_eng - One single Databricks Job run cannot dynamically change its "Run As" identity during execution. Rather you will need a pattern that separates the triggering identity from the executing identity.I would pre-configure 4 dependent jobs wit...

  • 0 kudos
Charuvil
by Contributor
  • 1553 Views
  • 3 replies
  • 3 kudos

Resolved! How to disable storage account key access of workspace storage accounts?

Each Azure Databricks workspace has an associated Azure storage account in a managed resource group known as the workspace storage account. The workspace storage account includes workspace system data (job output, system settings, and logs), DBFS roo...

  • 1553 Views
  • 3 replies
  • 3 kudos
Latest Reply
Charuvil
Contributor
  • 3 kudos

Thank you @Raman_Unifeye  and @nayan_wylde  it helped

  • 3 kudos
2 More Replies
margarita_shir
by New Contributor III
  • 1495 Views
  • 1 replies
  • 1 kudos

Databricks AWS deployment with custom configurations (workspace root storage)

Hi everyone,I have a question about the IAM role for workspace root storage when deploying Databricks on AWS with custom configurations (customer-managed VPC, storage configurations, credential configurations, etc.).At an earlier stage of our deploym...

  • 1495 Views
  • 1 replies
  • 1 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 1 kudos

Great question. [1] In the pre-UC world when you created a workspace you would designate a bucket/container that was used for what was most commonly known as DBFS. ie it's where the hive-metastore managed tables would be stored by default along with ...

  • 1 kudos
hgintexas
by New Contributor II
  • 1526 Views
  • 3 replies
  • 2 kudos

Resolved! system.lakeflow.job_task_run_timeline table missing task parameters on for each loop input

The system.lakeflow.job_task_run_timeline does not include the task level parameters on the input of the for each loop if dynamically setting the parameter in another notebook using dbutils.jobs.taskValues.set. This information is not included in the...

  • 1526 Views
  • 3 replies
  • 2 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 2 kudos

Hi @hgintexas , You’re right, the system job timeline tables and the Runs API don’t currently surface the resolved per‑iteration inputs for a For-each task when those inputs are sourced via task values set in another notebook with dbutils.jobs.taskVa...

  • 2 kudos
2 More Replies
Giuseppe_C
by New Contributor
  • 811 Views
  • 2 replies
  • 0 kudos

Databricks import directory false positive import

Hello evryone,I'm using databricks CLI to moving several directories from Azure Repos to Databricks Workspace.The problem is that files are not updating properly, with no error to display.The self-hosted agent in pipeline i'm using has installed the ...

  • 811 Views
  • 2 replies
  • 0 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 0 kudos

Hi @Giuseppe_C , Can you try running the command with the --debug flag to see if there is any additional information that you can gather? Full command: databricks import-dir <source> <dest> --overwrite --debugAlso, verify that the target path is not ...

  • 0 kudos
1 More Replies