cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RohithChippa
by New Contributor III
  • 856 Views
  • 7 replies
  • 1 kudos

Databricks cleanroom functionality and billing

I'm new to Databricks and have been tasked with exploring Databricks Clean Rooms. I'm a bit confused about how billing works for Clean Rooms and their overall functionality. Specifically, I'm curious about the following:Environment Hosting: Are Clean...

  • 856 Views
  • 7 replies
  • 1 kudos
Latest Reply
RohithChippa
New Contributor III
  • 1 kudos

Is it adding foreign catalog table in datacleanroom feature also available after GA release, cause I tried it but was not able to see the foreign catalog in add data assets tab

  • 1 kudos
6 More Replies
Kasen
by New Contributor III
  • 2540 Views
  • 2 replies
  • 0 kudos

Unable to grant catalog access to service principal

Hi everyone,I created a service principals called TestServicePrincipal. I tried to grant the catalog access to the service principals, but the error mentioned that it could not find principal with name TestServicePrincipal. If I grant the access to s...

Kasen_0-1715058248230.png Kasen_1-1715058284642.png
Community Platform Discussions
grant access
service principals
  • 2540 Views
  • 2 replies
  • 0 kudos
Latest Reply
OYESPEEDY
New Contributor II
  • 0 kudos

The issue could be related to how the service principal is being resolved in your system. Unlike users, service principals are often registered in a directory (like Azure AD), and their names might not match what you’re using. Instead of using TestSe...

  • 0 kudos
1 More Replies
DataYoga
by New Contributor
  • 3607 Views
  • 3 replies
  • 0 kudos

Informatica ETLs

I'm delving into the challenges of ETL transformations, particularly moving from traditional platforms like Informatica to Databricks. Given the complexity of legacy ETLs, I'm curious about the approaches others have taken to integrate these with Dat...

  • 3607 Views
  • 3 replies
  • 0 kudos
Latest Reply
thelogicplus
Contributor
  • 0 kudos

@DataYoga , you may explore the tool and services from Travinto Technologies . They have very good tools. We had explored their tool for our code coversion from  Informatica, Datastage and abi initio to DATABRICKS , pyspark. Also we used for SQL quer...

  • 0 kudos
2 More Replies
fabiogiehl2024
by New Contributor III
  • 530 Views
  • 4 replies
  • 0 kudos

Login comunity

Na tela de login, eu coloco o código de verificação recebido por email e ao clicar volta para a mesma tela sem mensagem de erro e não loga.

  • 530 Views
  • 4 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Is this happening with any browser in Windows? 

  • 0 kudos
3 More Replies
385653
by New Contributor II
  • 16653 Views
  • 6 replies
  • 0 kudos

bigquery in notebook failing with unity catalog enabled cluster

bigquery(reading data from google cloud) failing with unity catalog enabled cluster. Same working fine without unity cluster. Any help is appreciated!Thanks,Sai

  • 16653 Views
  • 6 replies
  • 0 kudos
Latest Reply
GeoPer
New Contributor III
  • 0 kudos

Did anyone manage to solve this?Currently we can save data using a Unity Catalog enabled Access mode: Single User cluster, but we CANNOT with a Access mode: Shared.We have tried the steps described in (https://docs.databricks.com/en/connect/external-...

  • 0 kudos
5 More Replies
AlexVB
by New Contributor III
  • 3957 Views
  • 3 replies
  • 4 kudos

Metabase support

Databricks x MetabaseHi, as someone who previously used Metabase as their self-service BI tool in their org, I was disappointed to see that Databricks is not supported officially: https://www.metabase.com/data_sources/The community drivers project is...

  • 3957 Views
  • 3 replies
  • 4 kudos
Latest Reply
ramiro
New Contributor II
  • 4 kudos

It happened! With the release of Metabase 51, Databricks is now officially supported as a driver 拾- Databricks and Metabase docs- Video guide

  • 4 kudos
2 More Replies
Loki466
by New Contributor
  • 1757 Views
  • 1 replies
  • 1 kudos

Unable to list a folder with square bracket in name

I am trying to iterate over a container(adls gen2) in azure databricks using pyspark. basically I am using dbutils.fs.ls to list the contents for folder using recursive function.this logic works perfectly fine for all folder except for the folders eh...

  • 1757 Views
  • 1 replies
  • 1 kudos
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

[] comes under valid characters, so it is an issue with dbfs.fs. dbutils.fs.ls("/Workspace/Users/user@databricks.com/qwe[rt]") fails with java.net.URISyntaxException: Illegal character in path at index I tested a workaround meanwhile that can help ...

  • 1 kudos
tim-mcwilliams
by New Contributor III
  • 7955 Views
  • 10 replies
  • 1 kudos

Notebook cell gets hung up but code completes

Have been running into an issue when running a pymc-marketing model in a Databricks notebook. The cell that fits the model gets hung up and the progress bar stops moving, however the code completes and dumps all needed output into a folder. After the...

  • 7955 Views
  • 10 replies
  • 1 kudos
Latest Reply
Lingesh
Databricks Employee
  • 1 kudos

@tim-mcwilliams I'm not sure if you found a workaround or a fix for this issue. We have recently found another issue (Integration between PyMC and Databricks Kernel does not go well. Specifically, the rendering logic of the progress bar in PyMC) that...

  • 1 kudos
9 More Replies
NhanNguyen
by Contributor III
  • 246 Views
  • 1 replies
  • 0 kudos

[Error reached the limit for number of private messages]

Dear team,Today I want to reply my friend on databricks private message but failed with the error "You have reached the limit for number of private messages that you can send for now. Please try again later."Could you help me check on this?Thanks!

  • 246 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

This is likely a rate limit imposed to prevent spam or excessive messaging, can you try later and advise if the problem persists?

  • 0 kudos
samarth_solanki
by New Contributor II
  • 3786 Views
  • 4 replies
  • 0 kudos

Creating a python package that uses dbutils.secrets

Hello Databricks,I wanted to create python package which has a python script which has a class , this class is where we give scope and key and we het the secret. This package will be used inside a databricks notebook I want to use dbutils.secret for ...

  • 3786 Views
  • 4 replies
  • 0 kudos
Latest Reply
krishnab
New Contributor II
  • 0 kudos

from pyspark.dbutils import DBUtils from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() dbutils = DBUtils(spark)Adding this to the module file solves the problem.  

  • 0 kudos
3 More Replies
NIK251
by New Contributor III
  • 667 Views
  • 4 replies
  • 0 kudos

Delta Live Table Pipeline to EventHub

i want to read and load the data to eventhub. And there is an error message:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 138.0 failed 4 times, most recent failure: Lost task 0.3 in stage 138.0 (TID 177) (10.139.6...

metrics-infor.png metrics-infor1.png
  • 667 Views
  • 4 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi, The error originates from the eventhub connector. Kindly check this with the EventHub Spark connector team. Please use the latest connector. https://github.com/Azure/azure-event-hubs-spark There are known issues with the Event hub connector like ...

  • 0 kudos
3 More Replies
naatyraja
by New Contributor
  • 216 Views
  • 1 replies
  • 0 kudos

%run working different in different taks of same job

I am using %run "/Workspace/Shared/SCPO_POC/Tredence-Dev-Phase3/Publix Weekly Sales POC - Main Folder/Weekly_Workflow_Parametrized/Get_Widgets_Values" before all my notebooks of a single job which has bunch of child jobs and child tasks... The % run ...

  • 216 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi, Can you please share the complete error?Can you try restarting the cluster, that might helo to mitigate the issue.

  • 0 kudos
NIK251
by New Contributor III
  • 1147 Views
  • 2 replies
  • 1 kudos

Resolved! Delta Live Table Pipeline

I have the error message when try to create a delta live table pipeline.My error is: com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 1207-112912-8e84v9h5: Encountered Quota Exhaustion issue in ...

  • 1147 Views
  • 2 replies
  • 1 kudos
Latest Reply
NIK251
New Contributor III
  • 1 kudos

Thanks sir, I solved it.

  • 1 kudos
1 More Replies
chethankumar
by New Contributor III
  • 288 Views
  • 1 replies
  • 0 kudos

How do I set up Databricks observability using AWS cloudwatch

How do I set up Databricks observability, including metrics and logging? I am using AWS-based Databricks and want to monitor it. I plan to use CloudWatch as the observability tool but couldn't find proper documentation to configure it.

  • 288 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @chethankumar, Setting up CloudWatch might required customer setup, by setting clusters using init script. You might want to consider contacting your account team to guide you through the process. Additional please find the post which could be use...

  • 0 kudos
HeckGunderson
by New Contributor
  • 246 Views
  • 1 replies
  • 0 kudos

Where can I quickly test custom metric queries?

I'm working on adapting some custom metrics to add to the tables' dashboard. Right now when I throw in a test query, I need to refresh metrics and it takes 5-10 minutes to let me know I've probably forgotten a parenthesis somewhere.Where can I test t...

  • 246 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @HeckGunderson, Have you tried running the code using notebook with your Databricks workspace? https://docs.databricks.com/en/notebooks/index.html Also you can do it via CLI using SQL connector. https://docs.databricks.com/ja/dev-tools/python-sql-...

  • 0 kudos