Hello All,I created a Databricks Premium Workspace for a Confidential Computing PoC. After creating a VM from Databricks UI, it came to notice that there is a new RG with managed identity, NAT Gateway, Public IP, security group, and a VNET (/16). I w...
During migration to production workload, I switched some queries to use RocksDB. I am concerned with its memory usage though. Here is sample output from my streaming query: "stateOperators" : [ { "operatorName" : "dedupeWithinWatermark", "...
Thank you for the input. Is there any particular reason why deduplication watermark makes it store everything and not just the key needed for deduplication? The 1st record has to be written to the table anyway, and its content is irrelevant as it jus...
Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...
Hi,I need to install the below maven coordinates on the clusters using databricks init scripts.1. coordinate: com.microsoft.azure:synapseml_2.12:0.11.2 with repo https://mmlspark.azureedge.net/maven2. coordinate: com.microsoft.azure:spark-mssql-conne...
Our Databricks workspace was created by a personal account. Now the person has left the organization. We would like to change the owner to a Service account(preferably, else to an Admin account).Questions:Is it possible to change the owner of the wor...
Are you in aws or Azure.When you say workspace admin, that could be many. So, you can have multiple workspace admin.
Hi,At our organization, we have added front end privatelink connection to a Databricks workspace in Azure, and public access to the workspace is disabled. I am able to access the workspace UI with the private IP (in the browser), and able to call the...
Hi @Retired_mod ,Thank you for the support.Really appreciate it.Thanks
Given the size of a Workflow may become too big to manage in a single Terraform project, what would be your recommendation as a best practice to manage and deploy the workflows via code to maintainer a predictable result between environments?Would it...
While configuring databricks, we've set the "default_catalog_name", which sets the default schema when users connect via an ODBC connection. While the naming isn't consistent, this does have one desired effect, that is, when users connect, it default...
I want to alter the schema and basically point it to a new path in the data lake #UnityCatalog
don't think so.You can alter the owner and dbproperties using the alter schema command, but not the location.https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-alter-schema
I am trying to setup my volumes and give them paths in the data lake but I keep getting this message:Input path url 'abfss://my-container@my-storage-account.dfs.core.windows.net/' overlaps with managed storage within 'CreateVolume' callThere WAS some...
I'm trying to use terraform to assign a cluster policy to an account-level group (sync'd from AAD via SCIM)My provider is configured like thisprovider "databricks" {alias = "azure_account"host = "accounts.azuredatabricks.net"account_id = "%DATABRICKS...
Our production workspace has several Repos integrated with GitHub. These repos aways point to master and should never be modified manually by a human directly in the workspace as the pulls are triggered by a GitHub Actions workflow. This workflow cal...
Hi,I am provisioning databricks workspaces using terraform and want to add specific github repo url that can be used. In UI there is an option for that but when it comes to terraform there is nothing specific. I came across custom_config option here ...
Hello,This can normally be achieved using this terraform resource:resource "databricks_workspace_conf" "this" { custom_config = { "enableProjectsAllowList": true, "projectsAllowList": "url1,url2,url3", } }Cheers
We have setup a Databricks workspace in our own Azure VNET, including a private endpoint. Connecting to the WS works fine (through the private ip address). However, when creating my first cluster, I run into this problem:"ADD_NODES_FAILED...Failed to...
I want to use Data Markerplace but only as private / local mode, so don't want to publish any products outside my organization.I know I can create private listing , but it can be done only from provider console.I'm added to marketplace role but not s...
