cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

ksenija
by Contributor
  • 1087 Views
  • 3 replies
  • 0 kudos

Connecting Databricks and Azure Devops

Hi everyone,When I tried to create new Databricks job that is using a notebook from a repo, it asked me to set up Azure DevOps Services (Personal access token) in Linked Accounts under my username. And now every time I want to create a new branch or ...

ksenija_0-1717148736222.png ksenija_1-1717148797484.png
  • 1087 Views
  • 3 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @ksenija , Looks like the Git credentials for the job uses a different account and are missing.  The job is configured to use {a particular user} but this account has credentials for {another configured user}. So you need to update the git details...

  • 0 kudos
2 More Replies
746837
by New Contributor II
  • 4550 Views
  • 3 replies
  • 0 kudos

Databricks and SMTP

Using databricks as aws partner trying to run python script to validate email addresses.  Whenever it gets to the smtp portion it times out.  I am able to telnet from python to the POP servers and get a response, I can ping domains and get replies, b...

  • 4550 Views
  • 3 replies
  • 0 kudos
Latest Reply
Babu_Krishnan
Contributor
  • 0 kudos

@746837 , Did you resolve this issue ? 

  • 0 kudos
2 More Replies
Kutbuddin
by New Contributor II
  • 3254 Views
  • 5 replies
  • 1 kudos

Resolved! Stream Query termination using available now trigger and toTable.

We are running a streaming job in databricks with custom streaming logic which consumes a CDC stream from mongo and appends to a delta table, at the end of the streaming job we have a internal checkpointing logic which creates an entry into a table w...

  • 3254 Views
  • 5 replies
  • 1 kudos
Latest Reply
Kutbuddin
New Contributor II
  • 1 kudos

I was expecting spark.sql(f"insert into table {internal_tab_name} values({dt})") to execute at the end after the streaming query was written to the table. What I observed:The spark sql query spark.sql(f"insert into table {internal_tab_name} values({d...

  • 1 kudos
4 More Replies
phguk
by New Contributor III
  • 1171 Views
  • 2 replies
  • 0 kudos

Efficient methods to make a temporary copy of a table

I'm using a tool (SAS) that doesn't inherently support time travel - that's to say it doesn't generate SQL including Timestamp or Version (for example). An obvious work-around could be to first copy/clone the version of the table, which SAS can then ...

  • 1171 Views
  • 2 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@phguk I think that Shallow Clone would be the best solution here.

  • 0 kudos
1 More Replies
radothede
by Contributor II
  • 1855 Views
  • 3 replies
  • 3 kudos

Monitoring VM costs using cluster pools

Hello,With ref to docs https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags cluster tags are not propagated to VM when created within a pool.Is there any workaround for monitoring VM costs using cluster pools (j...

Administration & Architecture
cluster pools
costs
vm
  • 1855 Views
  • 3 replies
  • 3 kudos
Latest Reply
radothede
Contributor II
  • 3 kudos

Dear @Retired_mod ,as You mentioned, Databricks does not provide out of the box support for VM usage monitoring for job clusters created from cluster pool.If we really want to use cluster pool, I would consider:1) splitting the pool into separate poo...

  • 3 kudos
2 More Replies
radothede
by Contributor II
  • 3368 Views
  • 1 replies
  • 0 kudos

Databricks DBU pre-purchase

Hello there,are pre-purchased DBU still valid? Can we use it?https://learn.microsoft.com/en-us/azure/cost-management-billing/reservations/reservation-discount-databricksCan someone please explain how it works in practice, by example?What if I pre-puc...

Administration & Architecture
DBCU
optimize costs
Pre-purchase
reservation
  • 3368 Views
  • 1 replies
  • 0 kudos
Latest Reply
radothede
Contributor II
  • 0 kudos

@Retired_mod could you please kindly look at this one? Thank You in advance.

  • 0 kudos
Snoonan
by Contributor
  • 3408 Views
  • 3 replies
  • 0 kudos

Resolved! Secrete management

Hi all, I am trying to use secrets to connect to my Azure storage account. I want to be able to read the data form the storage account using a pyspark notebook.Has anyone experience setting up such a connection or has good documentation to do so?I ha...

  • 3408 Views
  • 3 replies
  • 0 kudos
Latest Reply
DonatienTessier
Contributor
  • 0 kudos

Hi Sean,There are two ways to handle secret scopes:databricks-backed scopes: scope is related to a workspace. You will have to handle the update of the secrets.Azure Key Vault-backed scopes: scope is related to a Key Vault. It means than you configur...

  • 0 kudos
2 More Replies
Aiden-Z
by New Contributor III
  • 2824 Views
  • 3 replies
  • 1 kudos

Resolved! Init script failure after workspace upload

We have a pipeline in Azure Devops that deploys init scripts to the workspace folder on an Azure Databricks resource using the workspace API (/api/2.0/workspace/import), we use format "AUTO" and overwrite "true" to achieve this. After being uploaded ...

  • 2824 Views
  • 3 replies
  • 1 kudos
Latest Reply
Aiden-Z
New Contributor III
  • 1 kudos

If anyone else comes across this problem, the issue was a deployment powershell script was changing LF to CRLF before upload in the init script. The solution was to upload with LF line endings in the pipeline. 

  • 1 kudos
2 More Replies
pawelmitrus
by Contributor
  • 2622 Views
  • 4 replies
  • 0 kudos

Azure Databricks account api can't auth

Hi,does anyone know about any existing issue with azure databricks account api? I cannot do below:1. login with cli `databricks auth login --account-id <acc_id>, this is what I get https://adb-MY_ID.azuredatabricks.net/oidc/accounts/MY_ACC_ID/v1/auth...

  • 2622 Views
  • 4 replies
  • 0 kudos
Latest Reply
pawelmitrus
Contributor
  • 0 kudos

UPDATE Solved, the very same solution started to work today from running a pipeline with tf - M2M auth. with a service principal with fed auth. That's the 2 from my above post.When trying to follow these steps https://learn.microsoft.com/en-us/azure/...

  • 0 kudos
3 More Replies
NadithK
by Contributor
  • 2157 Views
  • 1 replies
  • 0 kudos

Public exposure for clusters in SCC enabled workspaces

Hi,We are facing a requirement where we need to somehow expose one of our Databricks clusters to an external service. Our organization's cyber team is running a security audit of all of the resource we use and they have some tools which they use to r...

  • 2157 Views
  • 1 replies
  • 0 kudos
Latest Reply
NadithK
Contributor
  • 0 kudos

Hi @Retired_mod ,Thank you very much for the reply. But I don't think this actually resolves our concern.All these solutions talk about utilizing the databricks cluster to access/read data in Databricks. They focus on getting to the Databricks data t...

  • 0 kudos
leungi
by Contributor
  • 4056 Views
  • 2 replies
  • 1 kudos

Resolved! Install system libraries on the cluster

The `Library` option in cluster config allows installation of language-specific libraries - e.g., PyPi for Python, CRAN for R.Some of these libraries - e.g., `sf` - require system libraries - e.g., `libudunits2-dev`, `libgdal-dev`.How may one install...

  • 4056 Views
  • 2 replies
  • 1 kudos
Latest Reply
feiyun0112
Honored Contributor
  • 1 kudos

you can install in init scripthttps://docs.databricks.com/en/init-scripts/index.html 

  • 1 kudos
1 More Replies
otydos
by New Contributor II
  • 2852 Views
  • 2 replies
  • 0 kudos

Authenticate with Terraform to Databricks Account level using Azure MSI(System assigned)

Hello, I want to authenticate with terraform to databricks account level with : Azure Managed Identity(System-assigned) of my Azure VMto perform operation like create group. I followed differents tutorial and the documentation on Azure and Databricks...

  • 2852 Views
  • 2 replies
  • 0 kudos
Latest Reply
DonatienTessier
Contributor
  • 0 kudos

Hello,On my side, I always have to add the provider in each resource block.You can try that:  resource "databricks_group" "xxxxx" { provider = databricks.accounts display_name = "xxxxx" }  About authentication, you can also try to add:auth_type  ...

  • 0 kudos
1 More Replies
Flask
by New Contributor II
  • 981 Views
  • 2 replies
  • 0 kudos

Problem loading catalog data from multi node cluster after changing Vnet IP range in AzureDatabricks

We've changed address range for Vnet and subnet that the Azure Databricks workspace(standard sku) was using, after that when we try to access the catalog data, we're getting socket closed error. This error is only with Multi node cluster, for single ...

  • 981 Views
  • 2 replies
  • 0 kudos
Latest Reply
Flask
New Contributor II
  • 0 kudos

Yes, it is mentioned that we cannot change the Vnet. I've changed the range in the same vnet but not the Vnet. Is there any troubleshooting that I can do to find this issue. The problem is, I don't want to recreate the workspace. It is a worst case s...

  • 0 kudos
1 More Replies
807326
by New Contributor II
  • 5134 Views
  • 2 replies
  • 0 kudos

Enable automatic schema evolution for Delta Lake merge for an SQL warehouse

Hello! We tried to update our integration scripts and use SQL warehouses instead of general compute clusters to fetch and update data, but we faced a problem. We use automatic schema evolution when we merge tables, but with SQL warehouse, when we try...

  • 5134 Views
  • 2 replies
  • 0 kudos
Latest Reply
TheKnightCoder
New Contributor II
  • 0 kudos

why can we not enable autoMerge in SQL warehouse when my tables are delta tables?

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels