cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

elgeo
by Valued Contributor II
  • 6063 Views
  • 1 replies
  • 3 kudos

Resolved! Generate new token error

Hello. I need to install Databricks CLI. While I am trying to generate new access token (User Settings->Generate new token) I get the following error:Could not create token with comment "cli" and lifetime (seconds) of 86400.I tried with different com...

  • 6063 Views
  • 1 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Please check in the Admin console that tokens are enabled and that you can manage them.

  • 3 kudos
elgeo
by Valued Contributor II
  • 5863 Views
  • 3 replies
  • 5 kudos

Resolved! Delta Table - Reduce time travel storage size

Hello! I am trying to understand time travel feature. I see with "DESCRIBE HISTORY" command that all the transaction history on a specific table is recorded by version and timestamp. However, I understand that this occupies a lot of storage especiall...

  • 5863 Views
  • 3 replies
  • 5 kudos
Latest Reply
elgeo
Valued Contributor II
  • 5 kudos

Thank you @Werner Stinckens​ for your reply. However I still haven't managed to delete history even after setting the below. The number of history rows remains the same when running "DESCRIBE HISTORY".SET spark.databricks.delta.retentionDurationCheck...

  • 5 kudos
2 More Replies
jerry747847
by New Contributor III
  • 3905 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks Associate Practice Exam -query

Dear Experts, Can anyone please let me know how option "C" is the answer to Question 31 for PracticeExam-DataEngineerAssociate. https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DataEngineerAssociate.pdf?_ga=2.185796329.11...

  • 3905 Views
  • 5 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Question 17 is even worse. "A data engineer is overwriting data" vs "should simply be overwritten instead"One situation I assume is DROP and CREATE and another is INSERT INTO OVERWRITE but here both are called the same.A data engineer is overwriting ...

  • 1 kudos
4 More Replies
archanarddy
by New Contributor
  • 1389 Views
  • 0 replies
  • 0 kudos

metastore is down

I am trying to run a scala notebook, but my job just spins and says Metastore is down. Can someone help me. Thanks in advance.

  • 1389 Views
  • 0 replies
  • 0 kudos
Ken1
by New Contributor III
  • 2568 Views
  • 2 replies
  • 7 kudos

PySpark Error in Azure Databricks

I have this error - com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILEDwhen i ran dbutils.notebook.run(.......)

  • 2568 Views
  • 2 replies
  • 7 kudos
Latest Reply
Debayan
Databricks Employee
  • 7 kudos

Hi @Godswill Mbata​ ,this looks like issue seems to be with High Concurrency cluster. Could you please confirm if you are using high concurrency cluster?Please refer: https://community.databricks.com/s/question/0D53f00001cx3ybCAA/strange-error-with-d...

  • 7 kudos
1 More Replies
plynton
by New Contributor II
  • 2052 Views
  • 1 replies
  • 2 kudos

Resolved! Dataframe to update subset of fields in table...

I have a table that I'll update with multiple inputs (csv). Is there a simple way to update my target when the source fields won't be a 1:1 match? Another challenge I've run into is that my sources don't have a header field, though I guess I could ...

  • 2052 Views
  • 1 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Read your CSV as a dataframe and than make update using Merge (upsert).

  • 2 kudos
witnessthee
by New Contributor II
  • 7566 Views
  • 3 replies
  • 2 kudos

Resolved! Error when using pyflink on databricks, An error occurred while trying to connect to the Java server

Hi, right now I am trying to run a pyflink script that can connect to a kafka server. When I run that script, I got an error "An error occurred while trying to connect to the Java server 127.0.0.1:35529". Do I need to install a extra jdk for that? er...

  • 7566 Views
  • 3 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

did you get Flink running on the Databricks cluster? Because that seems to be the issue here.

  • 2 kudos
2 More Replies
bayu
by New Contributor III
  • 37389 Views
  • 8 replies
  • 4 kudos

Can't login back to Databricks

Hi,I'm trying the Databricks Premium. Got registered etc. No problem.Then I setup AWS and workspace etc. No problem.Back to Databricks and now can't login. Trying to reset password but get nothing in the email.Trying other browser also doesn't work.C...

  • 37389 Views
  • 8 replies
  • 4 kudos
Latest Reply
bayu
New Contributor III
  • 4 kudos

Hi @Debayan Mukherjee​ could you please help me out?Is there any way I can login to my Databricks account?

  • 4 kudos
7 More Replies
FerArribas
by Contributor
  • 5677 Views
  • 2 replies
  • 8 kudos

Resolved! Error cluster launch: Security Daemon Registration

I have created a workspace in AWS with private link. When we launch a cluster we get the following error: "Security Daemon Registration Exception: Failed to set up the spark container due to an error when registering the container to security daemon"...

  • 5677 Views
  • 2 replies
  • 8 kudos
Latest Reply
FerArribas
Contributor
  • 8 kudos

Thanks @Sivaprasad C S​  It's already working. The problem was that the AWS STS private link was wrongly associated with the correct subnet.

  • 8 kudos
1 More Replies
iajay543
by New Contributor
  • 3164 Views
  • 2 replies
  • 0 kudos

Unable to sign it at customer-academy.databricks.com but can't do the same at partner-academy.databricks.com

Hi thereI have used my company email to register an account for https://customer-academy.databricks.com/learn/signin a while back.Now what I need to do is create an account with https://partner-academy.databricks.com/learn/signin using my company ema...

  • 3164 Views
  • 2 replies
  • 0 kudos
Latest Reply
Nadia1
Databricks Employee
  • 0 kudos

Hello Ajay,Thank you for reaching out. Please submit a ticket to https://help.databricks.com/s/contact-us?ReqType=training . Thank you!

  • 0 kudos
1 More Replies
ivanychev
by Contributor II
  • 5743 Views
  • 7 replies
  • 2 kudos

How do I enable new Visualization tools in my Databricks notebooks?

Databricks released new visualization toolshttps://docs.databricks.com/notebooks/visualizations/index.htmlBut I don't see them in my notebook (DBR 10.4) nor I can find out how to enable them. How do I use the new tools?

Screenshot 2022-07-21 at 21.59.19
  • 5743 Views
  • 7 replies
  • 2 kudos
Latest Reply
cweymouth
New Contributor II
  • 2 kudos

@Atanu Sarkar​ , I am still not seeing this. I am running a cluster on runtime 10.4 LTS ML. The "Legacy Visualizations" are the only option available. This feature should be fully deployed, yes? 

  • 2 kudos
6 More Replies
qanut
by New Contributor II
  • 1524 Views
  • 1 replies
  • 1 kudos

DBFS Delete Bug on Community Edition

I uploaded some files in Community Edition. But when I try to delete them I get the popup that say do you want to delete, and when I select delete, the popup goes away but the file doesn't delete. Seems the the GUI is bugged. However, when I use d...

  • 1524 Views
  • 1 replies
  • 1 kudos
Latest Reply
Simranarora
Databricks Employee
  • 1 kudos

Thanks for reporting. We will look into it!!

  • 1 kudos
Harsh1
by New Contributor II
  • 1238 Views
  • 0 replies
  • 0 kudos

Issues in Metastore Migration using Databricks Migration Tool

Hi Team,As I'm performing the Databricks workspace migration, during Metastore migration I'm facing below issue.As we found differences in the Metastore table count between Legacy and Target workspace, we checked error logs.After going through Failed...

  • 1238 Views
  • 0 replies
  • 0 kudos
swetha
by New Contributor III
  • 8033 Views
  • 2 replies
  • 2 kudos

Databricks job cluster logs

I am using databricks job cluster for multitask jobs, when my job failed/succeeded I couldn't see any logs, Do I need to add any location in advanced options, cluster logging to see the logs for the failed/succeeded jobs or what it is and how it work...

  • 8033 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
1 More Replies
bmat
by New Contributor II
  • 2890 Views
  • 1 replies
  • 1 kudos

How are you currently analyzing A/B tests data in Databricks?

Super curious about people's workflows for analyzing A/B tests. Trying to learn how to optimize my approach as it is very manual right now.

  • 2890 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Ben Mathew​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels