cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Discussions
Engage in dynamic conversations covering diverse topics within the Databricks Community. Explore discussions on data engineering, machine learning, and more. Join the conversation and expand your knowledge base with insights from experts and peers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Community Discussions

Engage in vibrant discussions covering diverse learning topics within the Databricks Community. Expl...

2784 Posts

Activity in Discussions

Start a conversation
lizou8
by • New Contributor
  • 4 Views
  • 0 replies
  • 0 kudos

delta sharing issue after enable predictive optimization

Some of our delta sharing tables are not working May be related to this, or maybe not, we enabled predictive optimization on all tables a few days agoes not working any morebut any new tables created works fine after setting thisSET TBLPROPERTIES (de...

Administration & Architecture
Reply
  • 4 Views
  • 0 replies
  • 0 kudos
Reply
Angus-Dawson
by • Visitor
  • 7 Views
  • 0 replies
  • 0 kudos

PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect

I'm trying to use Databricks Connect to run queries on Delta Tables locally. However, SQL queries using spark.sql don't seem to work properly, even though spark.read.table works.>>> from databricks.connect import DatabricksSession>>> spark = Databric...

Data Engineering
Reply
  • 7 Views
  • 0 replies
  • 0 kudos
Reply
Chinu
by • New Contributor III
  • 1718 Views
  • 3 replies
  • 0 kudos

Tableau Desktop connection error from Mac M1

Hi, Im getting the below error while connecting SQL Warehouse from the tableau desktop. I installed the latest ODBC drivers (2.7.5) but I can confirm that the driver name is different. From the error message I see libsparkodbc_sbu.dylib but in my lap...

Get Started Discussions
Reply
  • 1718 Views
  • 3 replies
  • 0 kudos
Reply
Latest Reply
jefflipkowitz
New Contributor III
  • 0 kudos

Have you referred to this document?https://help.tableau.com/current/pro/desktop/en-us/examples_databricks.html https://help.tableau.com/current/pro/desktop/en-us/examples_databricks.htm

  • 0 kudos
2 More Replies
Sameekshaji
by • Visitor
  • 41 Views
  • 1 replies
  • 0 kudos

Memory leak

I created a databricks JDBC connection class following the code ##https://docs.databricks.com/en/integrations/jdbc/authentication.htm I observed that after a number of execute SQL calls , there were 27000 instances of com.databricks.client.jdbc42.int...

Community Discussions
Reply
  • 41 Views
  • 1 replies
  • 0 kudos
Reply
Latest Reply
Sameekshaji
Visitor
  • 0 kudos

PLEASE revert ASAP

  • 0 kudos
NickLee
by • Visitor
  • 22 Views
  • 0 replies
  • 0 kudos

How to update num_workers dynamically in a job cluster

I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's defau...

NickLee_0-1722018584496.png
Data Engineering
Reply
  • 22 Views
  • 0 replies
  • 0 kudos
Reply
VedranZoricic
by • Visitor
  • 18 Views
  • 0 replies
  • 0 kudos

Exam Suspended

Hello I just had an exam for a Databricks data engineering professional certification and I failed the exam due to some violation and they didn't give me much information saying that I need to contact you.I got an alert about taking my eyes off the s...

Certifications
Reply
  • 18 Views
  • 0 replies
  • 0 kudos
Reply
JonLaRose
by • New Contributor III
  • 2272 Views
  • 4 replies
  • 1 kudos

Delta Lake S3 multi-cluster writes - DynamoDB

Hi there!I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in...

Administration & Architecture
Reply
  • 2272 Views
  • 4 replies
  • 1 kudos
Reply
Latest Reply
prem14f
Visitor
  • 1 kudos

Hi, could you please help me here? How can i use this configuration in DataBricks? So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3Dynam...

  • 1 kudos
3 More Replies
Miguel_Salas
by • Visitor
  • 21 Views
  • 0 replies
  • 1 kudos

Last file in S3 folder using autoloader

Nowadays we already use the autoloader with checkpoint location, but I still wanted to know if it is possible to read only the last updated file within a folder. I know it somewhat loses the purpose of checkpoint locatioAnother question is it possibl...

Data Engineering
Reply
  • 21 Views
  • 0 replies
  • 1 kudos
Reply
deane_h_s
by • New Contributor
  • 24 Views
  • 0 replies
  • 0 kudos

UCX Installation Error

I'm getting the following error when trying to install UCX on my local machine using the databricks CLI v0.212.1. I've authenticated the CLI with the required workspace using databricks config with the host and PAT generated. I also have admin privil...

Community Discussions
Reply
  • 24 Views
  • 0 replies
  • 0 kudos
Reply
YS1
by • New Contributor III
  • 100 Views
  • 2 replies
  • 0 kudos

Delta Live Tables and Pivoting

Hello,I'm trying to create a DLT pipeline where I read data as a streaming dataset from a Kafka source, save it in a table, and then filter, transform, and pivot the data. However, I've encountered an issue: DLT doesn't support pivoting, and using fo...

Data Engineering
dlt
streaming
Reply
  • 100 Views
  • 2 replies
  • 0 kudos
Reply
Latest Reply
Slash
New Contributor II
  • 0 kudos

Hi @YS1 ,As a workaround you can rewrite pivot to sql with case statements.Below Pivot:data = [ ("ProductA", "North", 100), ("ProductA", "South", 150), ("ProductA", "East", 200), ("ProductA", "West", 250), ("ProductB", "North", 30...

  • 0 kudos
1 More Replies
Takao
by • Visitor
  • 48 Views
  • 0 replies
  • 0 kudos

How to run OPTIMIZE to too big data set which has 11TB and more ?

Sorry for my very poor English and low Databricks Skill.At work, my boss asked me to perform liquid clustering on four columns for a Delta Lake table with an 11TB capacity and over 80 columns, and I was estimating the resources and costs required to ...

Data Engineering
Reply
  • 48 Views
  • 0 replies
  • 0 kudos
Reply
prasadvaze
by • Valued Contributor II
  • 34 Views
  • 0 replies
  • 0 kudos

Grant permission on catalog but revoke from schema for the same user

I have a catalog ( in unity catalog) containing multiple schemas.  I need an AD group to have select permission on all the schemas so at catalog level I granted Select to AD grp.  Then, I need to revoke permission on one particular schema in this cat...

Data Engineering
Reply
  • 34 Views
  • 0 replies
  • 0 kudos
Reply
janelleglover
by • New Contributor III
  • 1288 Views
  • 4 replies
  • 9 kudos

Resolved! Submit your feedback and win a $25 gift card!

Your feedback is crucial to us and directly influences how we innovate and improve our customer experience.  To share your success and feedback with Databricks, please  visit this link.  The first 20 people to submit a completed survey response will ...

2023-08-instructor-led-training-community-post-image-358x250-2x (1).png
Community Discussions
Reply
  • 1288 Views
  • 4 replies
  • 9 kudos
Reply
Latest Reply
janelleglover
New Contributor III
  • 9 kudos

Hi everyone!  Thank you all for the incredible response to this post! We are so grateful to hear your feedback.  This post is now closed, but stay tuned for similar opportunities in the future.  Thanks again! 

  • 9 kudos
3 More Replies
Schofield
by • Visitor
  • 31 Views
  • 0 replies
  • 0 kudos

List deployed Databricks asset bundles (DABs)?

Is there a databricks cli command or REST API to list all the DABs that have been deployed to a workspace?

Administration & Architecture
Reply
  • 31 Views
  • 0 replies
  • 0 kudos
Reply
BenDataBricks
by • New Contributor II
  • 934 Views
  • 8 replies
  • 4 kudos

OAuth U2M Manual token generation failing

I am writing a frontend webpage that will log into DataBricks and allow the user to select datasets.I am new to front end development, so there may be some things I am missing here, but I know that the DataBricks SQL connector for javascript only wor...

Data Engineering
Reply
  • 934 Views
  • 8 replies
  • 4 kudos
Reply
Latest Reply
MaheshMandlik
Visitor
  • 4 kudos

@Slash Thank you for your help. Your solution has worked very well for me.

  • 4 kudos
7 More Replies
Load more

Featured Posts

View all