- 13043 Views
- 3 replies
- 1 kudos
Hey,I am using Bitbucket Cloud and I want to connect my repository to Databricks. I am able to connect with my personal app password but what I am looking for is an authentication of a technical user.I need the integration to point to my dbt repo, wh...
- 13043 Views
- 3 replies
- 1 kudos
Latest Reply
Hi @Kaniz,thank you for your response! With this link you provided, I was able to authenticate with Bitbucket Cloud. The solution was to use x-token-auth as a username. I have tried with the generated email address before which didn't work. Thank you...
2 More Replies
- 494 Views
- 1 replies
- 0 kudos
We want to build monitoring and Alerting solution for Azure Databricks that should capture Resource Utilization details (like Aggregated CPU%, Memory% etc.) and Cost consumption at the Account Level.We have Unity Catalog Enabled and there are multipl...
- 494 Views
- 1 replies
- 0 kudos
Latest Reply
@smehta_0908 Greetings!
You can utilize Datadog for monitoring CPU and memory of clusters.
https://docs.datadoghq.com/integrations/databricks/?tab=driveronly
For Cost consumption at accounts level you can make use of billable usage logs using the Acc...
- 766 Views
- 2 replies
- 0 kudos
When experimenting with LLMs on Databricks clusters, I have become interested in knowing if the LLM (Llama2 or otherwise), tries to make calls to the internet (i.e., the settings for use_remote_code=True in Huggingface models, as just one example).Mo...
- 766 Views
- 2 replies
- 0 kudos
Latest Reply
Thanks! I see the general direction there.
1 More Replies
- 773 Views
- 1 replies
- 1 kudos
When I`m trying to install xml package, I`m getting error PERMISSION_DENIED
- 773 Views
- 1 replies
- 1 kudos
Latest Reply
Hey @FlavioSM The error message indicates that the library you're trying to install (com.databricks:spark-xml_2.13:0.17.0) is not on the allowlist for shared clusters in your Databricks workspace. Shared clusters are clusters that multiple users can ...
- 856 Views
- 5 replies
- 0 kudos
Hi everyone,I would like to know if the following behavior is expected or if it is a misconfiguration in SCIMWe are going through a change in the email of some users. So we did a test, changing the email of one of them, but the result was not OK beca...
- 856 Views
- 5 replies
- 0 kudos
Latest Reply
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
4 More Replies
- 3107 Views
- 4 replies
- 0 kudos
Hi team,Greetings.Do you know if there is a way to update the creator of a databricks_job using the API? the Documentation does not show "the creator" property and when I tried setting the creator, this property is not updated in the workspace UI.The...
- 3107 Views
- 4 replies
- 0 kudos
Latest Reply
Hi @danmlopsmaz , Thanks for bringing up your concerns, always happy to help
I understand the customer wanted to change the creator of the job but at this moment we could change the owner but not the creator. But you should be able to clone an exist...
3 More Replies
- 458 Views
- 3 replies
- 0 kudos
Is there a way to register a web app with Databricks to enable it to access a user's data using OAuth2? i.e., be included in the list of published apps with a pre-negotiated scope?
- 458 Views
- 3 replies
- 0 kudos
Latest Reply
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
2 More Replies
- 1546 Views
- 7 replies
- 2 kudos
Hi,I am trying to install the CloudWatch log agent on my cluster, using this tutorial from AWS https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/They provide an init script there but when I try to start my cluster I get...
- 1546 Views
- 7 replies
- 2 kudos
Latest Reply
Hi @Yeshwanth I have the exact same issue here. I have tried to update my cluster with the init script that you kindly shared in this thread. However, my error is still:Init script failure:Cluster scoped init script s3://databricks-init-scripts-caden...
6 More Replies
by
VJ3
• New Contributor III
- 551 Views
- 1 replies
- 0 kudos
Hello,Once we associate/migrate Databricks workspace to Unity Catalog, all the workspaces group will sync with Unity Catalog and workspace group would be renamed to workspace-local group. Databricks recommends to remove workspace-local group. Do we k...
- 551 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @VJ3, When you associate or migrate a Databricks workspace to the Unity Catalog, all workspace groups will synchronize with the Unity Catalog, and the workspace groups will be renamed as workspace-local groups. Databricks indeed recommends removin...
- 1517 Views
- 6 replies
- 1 kudos
I am working at large company with many more or less independent divisions and we are currently working on the roll out of Unity Catalog in Azure. The idea was to have a central infrastructure repository (deployed via Terraform) to manage all central...
- 1517 Views
- 6 replies
- 1 kudos
Latest Reply
Workspace Admins: Consider configuring permissions for workspace admins in the Account console to strike a balance between autonomy and governance.@Kaniz Do you have any information about this configuration? I cannot find such thing in the Account Co...
5 More Replies
- 720 Views
- 1 replies
- 1 kudos
Hi I am looking for some support on how to handle the following situation. I have a call center that generates call log files in json format that are sent to an s3 bucket. Some of the raw json files contain more than one call log object and they ar...
- 720 Views
- 1 replies
- 1 kudos
Latest Reply
Hi @chrisf_sts, Handling call log files in JSON format and creating a Delta Live data processing pipeline involves several steps.
Let’s break it down:
Ingestion:
You can ingest the raw JSON files directly into a bronze table without separating t...
- 477 Views
- 2 replies
- 0 kudos
Morning everyone! Wondering if someone has a high-level reference architecture of Databricks on AWS that includes the Enhanced Security and Compliance Add-On??thanks,Brad
- 477 Views
- 2 replies
- 0 kudos
Latest Reply
@sheridan06 please check this documentation: https://www.databricks.com/trust/security-features/protect-your-data-with-enhanced-security-and-compliance
Let us know if it helps.
1 More Replies
- 498 Views
- 2 replies
- 0 kudos
We have been using Databricks for some time and didn't knew that S3 bucket versioning was not recommended. We have disabled it now. What are the next steps that we need to take to clean up the data, should we create a lifecycle rule to delete older v...
- 498 Views
- 2 replies
- 0 kudos
Latest Reply
Adding lifecycle policies to databricks s3 bucket does not impact existing data, is that correct @Yeshwanth .
1 More Replies
- 395 Views
- 1 replies
- 0 kudos
HelloIs there a way to set one of the clusters in my workspace to be the default cluster when choosing from list of available clusters when running my Code?Thanks in Advanced.
- 395 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Krubug ,Thanks for bringing up your concerns, always happy to help
Upon checking internally, it seems that we do not currently support setting a default cluster for the workspace. But we do have a Feature request titled "Workspace - Settings - S...
- 1426 Views
- 1 replies
- 0 kudos
Hey,we're observing the following problem when trying to run a notebook on a cluster with libraries installed:All notebook cells are stuck in "Waiting to run" (also ones containing only a '1+1' statement).When running the cluster without installing t...
- 1426 Views
- 1 replies
- 0 kudos
Latest Reply
nkraj
New Contributor III
This can happen if Metastore client fails to initialize. With python libraries/Jars added, the REPL creation step involves adding installed libraries with Spark addJar operation. This would initialize metastore client and can get stuck if there is an...
