- 10394 Views
- 13 replies
- 6 kudos
Resolved! Install python packages from Azure DevOps feed with service principal authentication
At the moment I install python packages from our Azure DevOps feed with a PAT token as authentication mechanism. This works well, but I want to use a service principal instead of the PAT token.I have created an Azure service principal and assigned it...
- 10394 Views
- 13 replies
- 6 kudos
- 6 kudos
I'm kinda late to the party but what is the suggested way of retriving the access token rn? Using some #bash or python code stored in global init script or cluster scoped init scripts? I don't want to stored this code in the notebook.Idea is to block...
- 6 kudos
- 883 Views
- 1 replies
- 0 kudos
Resolved! Can anyone share Databricks security model documentation or best-practice references
Can anyone share Databricks security model documentation or best-practice references
- 883 Views
- 1 replies
- 0 kudos
- 0 kudos
Here is the official documentation of Databricks: https://docs.databricks.com/aws/en/security/ Do you need to dive deeper into any specific area?
- 0 kudos
- 894 Views
- 1 replies
- 1 kudos
Moving Databricks Metastore Storage Account Between Azure Subscriptions
I have two Azure subscriptions: one for Prod and another for Non-Prod. During the initial setup of the Non-Production Databricks Workspace, I configured the metastore storage account in the Non-Prod subscription. However, I now want to move this meta...
- 894 Views
- 1 replies
- 1 kudos
- 1 kudos
Assuming the metastore is the same for your DEV and PROD environments and what you want is just to use the same storage account + container to place managed tables, volumes, etc. in theory you just need to copy all content from your source storage ac...
- 1 kudos
- 785 Views
- 1 replies
- 1 kudos
A single DLT for Ingest - feedback on this architecture
What are your thoughts on this Databricks pipeline design?Different facilities will send me backups of a proprietary transactional database containing tens of thousands of tables. Each facility may have differences in how these tables are populated o...
- 785 Views
- 1 replies
- 1 kudos
- 1 kudos
Your design shows strong alignment with the Medallion Architecture principles and addresses schema variability well, but there are some scalability and governance considerations worth discussing. Also Pre-Bronze, Building a schema registry early is e...
- 1 kudos
- 2774 Views
- 3 replies
- 0 kudos
Looking for experiences with DABS CLI Deployment, Terraform and Security
Hi Community,I hope my topic finds you well. Within our Databricks landscape we decided to use DABS (Databricks Asset Bundles) however we found out (the hard way) that it uses Terraform for Deployment purposes. This is a concern now for Security and ...
- 2774 Views
- 3 replies
- 0 kudos
- 0 kudos
Try to use always service principals to deploy your asset bundles. If desired take a look here: https://www.youtube.com/watch?v=5WreXn0zbt8 Concerning terraform state, it is indeed generated, take a look at this picture extracted from one of my deplo...
- 0 kudos
- 2604 Views
- 1 replies
- 1 kudos
Resolved! Restricting Catalog and External Location Visibility Across Databricks Workspaces
Restricting Catalog and External Location Visibility Across Databricks Workspaces I am facing exact similar issue, But I don't want to create separate metastore. and I have added environment name as a prefix to all external locations. All the locatio...
- 2604 Views
- 1 replies
- 1 kudos
- 1 kudos
You can hide or scope external locations and catalogs so they are only visible within their respective Databricks workspaces—even when using a shared metastore—by using "workspace binding" (also called isolation mode or workspace-catalog/workspace-ex...
- 1 kudos
- 581 Views
- 1 replies
- 0 kudos
Inquire the location of creating a metastore resource
I created Databricks on AWS today.Of course, we're planning to switch to paid.By the way, the Metastore is in the US Region after I was born.I'm a Korea APJ, and is the Metastore all only run in the US?Does the Metastore have no impact if I query or...
- 581 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @LEE_SUKJUN , I think the general principle should be to keep all components (metastore, workspace, and cloud storage) in the same region in order to avoid cross-region latency, data egress costs, and compliance issues.Concerning number of metasto...
- 0 kudos
- 1560 Views
- 2 replies
- 0 kudos
Using or integrating SIlver or Gold Zone data in an Operational API
I am looking to understand what sort of approach we can take to use Silver or Gold zone data in an Operational style API, or even if we should. We have data that makes it's way to the Silver and Gold zones in our Medallion Architecture and it kind of...
- 1560 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks is an analytics system and isn't optimized to perform as an OLTP system. Additionally, Databricks compute can scale to zero if you set it to do so. This means if you want to use gold/silver data in a real-time way you need to keep a clus...
- 0 kudos
- 1551 Views
- 1 replies
- 1 kudos
Resolved! Trying to Backup Dashboards and Queries from our Workspace.
We are using a databricks workspace and our IT team is decommissioning it as our time with it is being done. I have many queries and dashboards developed. I want to copy these, unfortunately when i download using zip or .dbc these queries or dashboar...
- 1551 Views
- 1 replies
- 1 kudos
- 1 kudos
NotebooksThese are the easiest assets to back up.You can export them individually or in bulk as:.dbc – Databricks archive format (can re-import directly into a new workspace).source or .py – raw code export (ideal for version control)To download in b...
- 1 kudos
- 3166 Views
- 5 replies
- 1 kudos
Problem with Metastore
Hello community.We are facing an issue when deploying and configuring metastore using terraform. We are using Azure Devops pipeline for deployment. The identity running the pipeline is a managed identity and it's set as account admin in Account porta...
- 3166 Views
- 5 replies
- 1 kudos
- 1 kudos
@jzu , is this a new error or is it the same as before. I need more details please. Louis.
- 1 kudos
- 5363 Views
- 2 replies
- 0 kudos
R-studio on Dedicated Cluster Invalid Access Token
Hello!! Currently I have an R-studio installed on a Dedicated Cluster over Azure Databricks, here are the specs:I must to make enfasis over the Access mode: Manual and Dedicated to a Group.Here, we install R-studio using a notebook with the following...
- 5363 Views
- 2 replies
- 0 kudos
- 0 kudos
You’re seeing two key issues with your RStudio Server on Azure Databricks: RStudio stops working after 1–2 days. You get permission errors using sparklyr and can’t update the Connections pane. Let’s address each: 1. RStudio Server Stops Working A...
- 0 kudos
- 920 Views
- 1 replies
- 1 kudos
Resolved! Signing up BAA requiring Compliance Security Profile activation
Hello folks — I need help with enabling compliance security profile for my account. I need this to execute/sign a HIPPA BAA on my account. For this, I need an enhanced security & compliance add-on for this. I first reached out to help@databricks.com...
- 920 Views
- 1 replies
- 1 kudos
- 1 kudos
I was able to turn this feature on by upgrading the plan to "Enterprise". Thanks!
- 1 kudos
- 5218 Views
- 1 replies
- 0 kudos
Resolved! Getting "Data too long for column session_data'" creating a CACHE table
Hi, I'm trying to leverage CACHE TABLE to create temporary tables that are cleaned up at the end of the session.In creating one of these, I'm getting Data too long for column 'session_data'. The query I'm using isn't referencing a session_data colu...
- 5218 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for sharing the details—this is a common point of confusion with caching versus temporary objects in Databricks. What’s likely happening The error message “Data too long for column 'session_data'” is emitted by the metastore/metadata persiste...
- 0 kudos
- 4913 Views
- 1 replies
- 0 kudos
Azure Network Connectivity Configurations API failing
It seems like since yesterday evening (Europe time) there's a platform-side issue with Network Connectivity Configurations API on Azure Databricks Accounts.API calls are being redirected to a login page, causing multiple different tools, such as Terr...
- 4913 Views
- 1 replies
- 0 kudos
- 0 kudos
I can't speak to this specific incident as it was over a year ago, but for future reference: Monitoring Databricks service health Please subscribe to the Azure Databricks Status Page for your region(s) and the specific components you use. You can opt...
- 0 kudos
- 4638 Views
- 1 replies
- 0 kudos
Databricks Managed MLFlow with Different Unity Catalog for Multi-tenant Production Tracing
Is the Databricks Managed MLFlow only trace LLM traffic through Serving Endpoint? Does it support manual tracing in my LLM application with decorator @mlflow.trace ?Also, How can Databricks Managed MLFlow support multi-tenant cases where traces need ...
- 4638 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeffreym9 , Managed MLflow Tracing captures traces in notebooks, local IDEs, jobs, and Serving Endpoints—via autologging or manual instrumentation, so it’s not limited to Serving Endpoints-only. Set your tracking URI to databricks and traces are...
- 0 kudos
