cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
k9
by • New Contributor II
  • 4515 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks CLI v0.17.6 issue

I do have multiple groups created in my databricks account and I have databricks cli installed on my mac. Some of the cli functions return errors that i cannot find solution for. databricks groups listReturns:Error: b'{"error_code":"INTERNAL_ERROR","...

Data Engineering
Reply
  • 4515 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@kenan hasanov​ which version python you have installed on your machine please, you need to have 3-3.6 or 2-2.7.9 above , please try to go with latest one as you are only seeing issues with few functions. please raise issue in case if you are still f...

  • 1 kudos
2 More Replies
206734
by • New Contributor
  • 939 Views
  • 2 replies
  • 0 kudos

Need help recovering password

guys, i cannot recover my password at my databricks workspace. I tried to reset my password by using the reset my password buttons in both login and admin console, but i am not getting the reset email. I already checked my spam box and i'm getting em...

Data Engineering
Reply
  • 939 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Victor Fernandes​ Just a friendly follow-up, Are you still looking for help regarding this issue? please let us know

  • 0 kudos
1 More Replies
KuldeepChitraka
by • New Contributor III
  • 2165 Views
  • 2 replies
  • 0 kudos

Lakehouse Architecture : How notebooks are organized and executed

We are implementing a lakehouse architecture and using Notebook to transform data from object storage. Most of the time our source is database for which there one folder per table in object storage.We have structure like below for various notebooksGO...

Data Engineering
Reply
  • 2165 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with your request

  • 0 kudos
1 More Replies
sher
by • Valued Contributor II
  • 2270 Views
  • 5 replies
  • 1 kudos

I have completed Data Engineer Associate (Version 3) Certificate. when we will get the Certificate in default?

I have completed Data Engineer Associate (Version 3) Certificate.when we will get the Certificate in default?

Data Engineering
Reply
  • 2270 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @sherbin w​ Just a friendly follow-up, Are you still looking for help regarding this issue or you were able to received your voucher? please let us know

  • 1 kudos
4 More Replies
renuka
by • New Contributor II
  • 2644 Views
  • 2 replies
  • 1 kudos

Unable to login databricks

I am not able to login databricks academy, getting this error please help.

Data Engineering
Reply
  • 2644 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @Renuka Bhutada​ Just a friendly follow-up, Are you still looking for help regarding this issue? please let us know

  • 1 kudos
1 More Replies
chanansh
by • Contributor
  • 2169 Views
  • 3 replies
  • 0 kudos

Delta table cannot be previewed in the Data UI

I have saved a delta table which I can read using `spark.table(table_name)`. However, when I go to the "Data" panel in databricks I get an error when I choose that table.An error occurred while fetching table: <table_name>com.databricks.backend.commo...

Data Engineering
Reply
  • 2169 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
chanansh
Contributor
  • 0 kudos

I don't know. The way I have saved the table was with autoloader as follows:​I am saving a structure stream into a table using:``` .writeStream .format("delta") # <----------- .option("checkpointLocation", checkpoint_path) .option("path", ou...

  • 0 kudos
2 More Replies
thaarantes
by • New Contributor
  • 1460 Views
  • 3 replies
  • 2 kudos

Voucher not sent

I have concluded the Databricks Platform Administrator - ILT Training, that offered a $200 dollars voucher after completion. I answered the questions of the survey and sent my certificate. How can I get the code?

Data Engineering
Reply
  • 1460 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Hi @Thais Arantes​,Just a friendly follow-up, Are you still looking for help regarding this issue or you were able to received your voucher? please let us know

  • 2 kudos
2 More Replies
jonathan-dufaul
by • Valued Contributor
  • 4339 Views
  • 4 replies
  • 2 kudos

How can I store timestamps using the local (not offset) timestamp instead of storing as UTC-Offset?

I have data that looks like this:2021-11-25T19:00:00.000-0500 2021-03-03T13:00:00.000-0500 2021-03-09T15:00:00.000-0500 2021-03-13T16:00:00.000-0500 2021-03-19T03:00:00.000-0400 2021-05-28T03:00:00.000-0400which is accurate, except I'm pulling the da...

Data Engineering
Reply
  • 4339 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Jonathan Dufault​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...

  • 2 kudos
3 More Replies
sika
by • New Contributor II
  • 11416 Views
  • 2 replies
  • 0 kudos

ignoreDeletes in DLT pipeline

Hi all,I have a DLT pipeline as so:raw -> cleansed (SCD2) -> curated. 'Raw' is utilizing autoloader, to continously read file from a datalake. These files can contain tons of duplicate, which causes our raw table to become quite large. Therefore, we ...

Data Engineering
Reply
  • 11416 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sika
New Contributor II
  • 0 kudos

Ok, i'll try an add additional details. Firstly: The diagram below shows our current dataflow: Our raw table is defined as such: TABLES = ['table1','table2']   def generate_tables(table_name): @dlt.table( name=f'raw_{table_name}', table_pro...

  • 0 kudos
1 More Replies
JordanYaker
by • Contributor
  • 4904 Views
  • 8 replies
  • 1 kudos

Why is Delta Lake creating a 238.0TiB shuffle on merge?

I'm frankly at a loss here. I have a task that is consistently performing just awfully. I took some time this morning to try and debug it and the physical plan is showing a 238TiB shuffle:== Physical Plan == AdaptiveSparkPlan (40) +- == Current Plan...

image
Data Engineering
Reply
  • 4904 Views
  • 8 replies
  • 1 kudos
View Replies
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hi @Jordan Yaker​,Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
7 More Replies
Abel_Martinez
by • Contributor
  • 20040 Views
  • 10 replies
  • 39 kudos

Why Python logs shows [REDACTED] literal in spaces when I use dbutils.secrets.get in my code?

When I use  dbutils.secrets.get in my code, spaces in the log are replaced by "[REDACTED]" literal. This is very annoying and makes the log reading difficult. Any idea how to avoid this?See my screenshot...

Data Engineering
Reply
  • 20040 Views
  • 10 replies
  • 39 kudos
View Replies
Latest Reply
jlb0001
New Contributor III
  • 39 kudos

I ran into the same issue and found that the reason was that the notebook included some test keys with values of "A" and "B" for simple testing. I noticed that any string with a substring of "A" or "B" was "[REDACTED]".​So, in my case, it was an eas...

  • 39 kudos
9 More Replies
ShellyXiao
by • New Contributor II
  • 12383 Views
  • 1 replies
  • 0 kudos

Azure Databricks cluster driver config

Hi there,I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute...

Data Engineering
Reply
  • 12383 Views
  • 1 replies
  • 0 kudos
View Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 2010 Views
  • 1 replies
  • 7 kudos

Databricks recently added SQL alerts feature that enables users to create notifications based on various conditions and trigger them within their job ...

Databricks recently added SQL alerts feature that enables users to create notifications based on various conditions and trigger them within their job workflows.SQL alerts inform users about potential issues and easily ensure critical data availabilit...

ezgif-3-324d970c83
Data Engineering
Reply
  • 2010 Views
  • 1 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Thanks for sharing this @Hubert Dudek​ .

  • 7 kudos
sintsan
by • New Contributor II
  • 1621 Views
  • 1 replies
  • 1 kudos

Resolved! spark.sparkContext.setCheckpointDir - External Azure Storage

Is it possible to direct spark.sparkContext.setCheckpointDir to an external Azure Storage Container location (instead of DBFS) & if so how, there's very little documentation on that.

Data Engineering
Reply
  • 1621 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

yes,the directory must be an HDFS path if running on a cluster.All you need to do is provide the correct path.

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels