cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

edmundsecho
by New Contributor II
  • 12532 Views
  • 3 replies
  • 0 kudos

how register a published app

Is there a way to register a web app with Databricks to enable it to access a user's data using OAuth2? i.e., be included in the list of published apps with a pre-negotiated scope? 

  • 12532 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
2 More Replies
Carsten03
by New Contributor III
  • 3058 Views
  • 7 replies
  • 2 kudos

Resolved! CloudWatch Agent Init Script Fails

Hi,I am trying to install the CloudWatch log agent on my cluster, using this tutorial from AWS https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/They provide an init script there but when I try to start my cluster I get...

  • 3058 Views
  • 7 replies
  • 2 kudos
Latest Reply
alexbishop
New Contributor II
  • 2 kudos

Hi @Yeshwanth I have the exact same issue here. I have tried to update my cluster with the init script that you kindly shared in this thread. However, my error is still:Init script failure:Cluster scoped init script s3://databricks-init-scripts-caden...

  • 2 kudos
6 More Replies
VJ3
by New Contributor III
  • 7305 Views
  • 1 replies
  • 0 kudos

Resolved! workspace local group once workspace has been migrated/Associated to Unity Catalog

Hello,Once we associate/migrate Databricks workspace to Unity Catalog, all the workspaces group will sync with Unity Catalog and workspace group would be renamed to workspace-local group. Databricks recommends to remove workspace-local group. Do we k...

  • 7305 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @VJ3, When you associate or migrate a Databricks workspace to the Unity Catalog, all workspace groups will synchronize with the Unity Catalog, and the workspace groups will be renamed as workspace-local groups. Databricks indeed recommends removin...

  • 0 kudos
Datazilla
by New Contributor III
  • 7822 Views
  • 6 replies
  • 1 kudos

Is a central UC Catalog management a Good Practice?

I am working at large company with many more or less independent divisions and we are currently working on the roll out of Unity Catalog in Azure. The idea was to have a central infrastructure repository (deployed via Terraform) to manage all central...

  • 7822 Views
  • 6 replies
  • 1 kudos
Latest Reply
Datazilla
New Contributor III
  • 1 kudos

Workspace Admins: Consider configuring permissions for workspace admins in the Account console to strike a balance between autonomy and governance.@Kaniz_Fatma Do you have any information about this configuration? I cannot find such thing in the Acco...

  • 1 kudos
5 More Replies
chrisf_sts
by New Contributor II
  • 13616 Views
  • 1 replies
  • 1 kudos

Resolved! Guidance for creating DLT pipeline with raw json call logs

Hi I am looking for some support on how to handle the following situation.  I have a call center that generates call log files in json format that are sent to an s3 bucket.  Some of the raw json files contain more than one call log object and they ar...

  • 13616 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @chrisf_sts, Handling call log files in JSON format and creating a Delta Live data processing pipeline involves several steps.    Let’s break it down:   Ingestion: You can ingest the raw JSON files directly into a bronze table without separating t...

  • 1 kudos
sheridan06
by New Contributor III
  • 914 Views
  • 2 replies
  • 0 kudos

Resolved! Architecture With the Enhanced Security and Compliance Add-On

Morning everyone!  Wondering if someone has a high-level reference architecture of Databricks on AWS that includes the Enhanced Security and Compliance Add-On??thanks,Brad

  • 914 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yeshwanth
Honored Contributor
  • 0 kudos

@sheridan06 please check this documentation: https://www.databricks.com/trust/security-features/protect-your-data-with-enhanced-security-and-compliance Let us know if it helps.

  • 0 kudos
1 More Replies
prashantjjain33
by New Contributor II
  • 1085 Views
  • 2 replies
  • 0 kudos

Disabled S3 bucket version, Next steps?

We have been using Databricks for some time and didn't knew that S3 bucket versioning was not recommended. We have disabled it now. What are the next steps that we need to take to clean up the data, should we create a lifecycle rule to delete older v...

  • 1085 Views
  • 2 replies
  • 0 kudos
Latest Reply
prashantjjain33
New Contributor II
  • 0 kudos

Adding lifecycle policies to databricks s3 bucket does not impact existing data, is that correct @Yeshwanth .

  • 0 kudos
1 More Replies
Krubug
by New Contributor
  • 576 Views
  • 1 replies
  • 0 kudos

Resolved! Set Deafult Cluster

HelloIs there a way to set one of the clusters in my workspace to be the default cluster when choosing from list of available clusters when running my Code?Thanks in Advanced.

  • 576 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Honored Contributor
  • 0 kudos

Hi @Krubug ,Thanks for bringing up your concerns, always happy to help  Upon checking internally, it seems that we do not currently support setting a default cluster for the workspace. But we do have a Feature request titled "Workspace - Settings - S...

  • 0 kudos
jannikj
by New Contributor
  • 2263 Views
  • 1 replies
  • 0 kudos

Notebook cells stuck on "waiting to run" when using Cluster Libraries

Hey,we're observing the following problem when trying to run a notebook on a cluster with libraries installed:All notebook cells are stuck in "Waiting to run" (also ones containing only a '1+1' statement).When running the cluster without installing t...

  • 2263 Views
  • 1 replies
  • 0 kudos
Latest Reply
nkraj
New Contributor III
  • 0 kudos

This can happen if Metastore client fails to initialize. With python libraries/Jars added, the REPL creation step involves adding installed libraries with Spark addJar operation. This would initialize metastore client and can get stuck if there is an...

  • 0 kudos
Matt4
by New Contributor III
  • 1489 Views
  • 5 replies
  • 1 kudos

Resolved! Azure Databricks Assistant and Private Link

Hi,I was wondering if anyone is using the Databricks Assistant using the standard private link configuration of their network.Kind of curious how data is passed from the Databricks Workspace, in a private link configuration, to the Azure OpenAI servi...

Administration & Architecture
assistant
privatelink
  • 1489 Views
  • 5 replies
  • 1 kudos
Latest Reply
Matt4
New Contributor III
  • 1 kudos

I asked our SA at Databricks. AWS is different if anyone is looking for that answerAzure Open AI connects to the control planeIt doe not use private endpoints, though since you are an Azure customer, the traffic stays on the Azure global network (no ...

  • 1 kudos
4 More Replies
pawelzak
by New Contributor III
  • 888 Views
  • 2 replies
  • 0 kudos

Databricks dashboard programatically

Hi,How can I create a databricks dashboard, filters and visuals programatically (api, terraform, sdk, cli...)?Thanks,Pawel

  • 888 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
1 More Replies
tototox
by New Contributor III
  • 1104 Views
  • 2 replies
  • 0 kudos

i want file upload in volume using api

how to upload file in volume using api??

  • 1104 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
1 More Replies
Kenthos
by New Contributor II
  • 1430 Views
  • 2 replies
  • 2 kudos

Resolved! Guide to Migrating from AWS to Azure

Hello, I would like to know if we have any documents, checklists, or step-by-step instructions for migrating from Databricks on AWS to Azure.

  • 1430 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 2 kudos
1 More Replies
cesarv
by New Contributor
  • 606 Views
  • 2 replies
  • 0 kudos

Error upgrading Databricks workspace to premium

I tried to update a databricks workspace from Standard to Premium in the Azure portal clicking "Upgrade to Premium" but even when it was supposedly "upgraded", I still can't restrict the use of clusters or modify permissions (I'm an administrator on ...

  • 606 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the response – you'll find some great info. Your input is valuable, so pick the best solution ...

  • 0 kudos
1 More Replies
GS24
by New Contributor II
  • 1330 Views
  • 3 replies
  • 0 kudos

Connecting to Azure Databricks deployed using Vnet injection over Public Internet

I'm trying to connect to Azure Databricks (deployed using Vnet injection method) from a 3rd party service running on Azure in the same region. When I try to connect using the Databricks hostname directly in my connection the host name always resolves...

Administration & Architecture
Azure Private Link
Databricks
Vnet Injection
  • 1330 Views
  • 3 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels