cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

zll_0091
by New Contributor III
  • 1017 Views
  • 3 replies
  • 4 kudos

Resolved! How does AutoLoader works when triggered via Azure Data Factory?

Hi,I am currently creating an AutoLoader in databricks and will be using ADF as an orchestrator.I am quite confused how this will handle my data so please clarify if I misunderstood it.First, I will run my ADF pipeline which includes an activity to c...

zll_0091_0-1721789223632.png
  • 1017 Views
  • 3 replies
  • 4 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 4 kudos

Auto loader will process files incrementally. Let's say you have a files in existing directory called /input_filesFirst time you run autoloader, it will read all files in that directory (unless you set an option includeExsistingFiles to false, like y...

  • 4 kudos
2 More Replies
zll_0091
by New Contributor III
  • 1024 Views
  • 3 replies
  • 2 kudos

How can I deduplicate data from my stream?

Hi,I'm new to databricks and I'm trying to use stream for my incremental data. This data has duplicates which can be solved using a window function. Can you check where my code goes wrong?1-------#Using Auto Loader to read new files schema = df1.sche...

zll_0091_1-1721743629228.png
  • 1024 Views
  • 3 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 2 kudos

Hi @zll_0091 ,Change the output mode to update. Other than that, your code looks fine, but I would rename variable microdf to windowSpec, because now it's little confusing.

  • 2 kudos
2 More Replies
Mudasirfiyaz20
by New Contributor II
  • 1354 Views
  • 1 replies
  • 0 kudos

Easy GIF Animator 7.4.8 Crack + License Key 2024

Easy GIF Animator 7.4.8 Crack + License Key 2024Easy GIF Animator Crack is popular ever for animation creation and editing purpose there are sufficient modules that works in moderate session and also upgrade the further styles which available here to...

  • 1354 Views
  • 1 replies
  • 0 kudos
Latest Reply
Mudasirfiyaz20
New Contributor II
  • 0 kudos

Download Easy GIF Animator 

  • 0 kudos
ani2409
by New Contributor II
  • 875 Views
  • 3 replies
  • 0 kudos

Error Creating Primary Key Constraint in DLT

Hello There!Greetings!!I am getting the following error when trying to Create a DLT table in my Gold Layer..com.databricks.sql.managedcatalog.PrimaryKeyColumnsNullableException: Cannot create the primary key `x_key` because its child column(s) `x_key...

  • 875 Views
  • 3 replies
  • 0 kudos
Latest Reply
ani2409
New Contributor II
  • 0 kudos

Thank you @szymon_dybczak for the response.As you can see I have already defined the not null constraint in my definition for the primary key x_keyCONSTRAINT pk_key_not_null EXPECT (x_key IS NOT NULL) But still I am getting the same error.Also I chec...

  • 0 kudos
2 More Replies
arijit_sani
by New Contributor II
  • 3802 Views
  • 2 replies
  • 1 kudos

Oracle datawarehous Replacement With Databricks Using DeltaLake

I am new to Spark and DataBricks and exploring these to understand to replace Oracle DataWarehouse by DataBricks(deltalake) and to use Spark to improve the ELT/ETL performance of existing DW.Now, I have done some lookups in databricks blogs, spark do...

Community Platform Discussions
datawarehouse oracle-databricks Migration data
  • 3802 Views
  • 2 replies
  • 1 kudos
Latest Reply
arijit_sani
New Contributor II
  • 1 kudos

Hi, Could you please advise on this

  • 1 kudos
1 More Replies
Rajaniesh
by New Contributor III
  • 15834 Views
  • 6 replies
  • 0 kudos

Error "Root storage credential for metastore does not exist"While creating the Databricks Volume in

Hi, I tried to create the databricks volume in unity catalog but it threw this error:Root storage credential for metastore XXXXXX does not exist. Please contact your Databricks representative or consider updating the metastore with a valid storage cr...

  • 15834 Views
  • 6 replies
  • 0 kudos
Latest Reply
KrzysztofPrzyso
New Contributor III
  • 0 kudos

I had a similar issue, that was resolved by assigning storage credential to the metastore using REST API call.- make sure that you are metastore or account admin- make sure that the storage credential is correctly configured, its access corrector ide...

  • 0 kudos
5 More Replies
Juanitoc10
by New Contributor II
  • 479 Views
  • 0 replies
  • 0 kudos

DBFS problem

Hi All, When trying to get to my DBFS in Databricks comunity edition I get the next errorI have already disabled and Enabled de DBFS file browser but it did not help

imagen.png
  • 479 Views
  • 0 replies
  • 0 kudos
danmlopsmaz
by New Contributor II
  • 896 Views
  • 1 replies
  • 0 kudos

Issue with VSCode Extension and Databricks Cluster Using Docker Image

I've encountered a significant issue while using the VSCode extension for Databricks, particularly when working with a cluster configured with a Docker image. Here's a detailed description of the problem:Problem DescriptionWhen attempting to upload a...

danmlopsmaz_0-1721482196211.png danmlopsmaz_1-1721482508346.png
  • 896 Views
  • 1 replies
  • 0 kudos
Latest Reply
danmlopsmaz
New Contributor II
  • 0 kudos

Hi @Retired_mod thanks for a such quick response.Actually, I am using the Dockerfile from the Databricks runtime example here: https://github.com/databricks/containers/blob/master/ubuntu/minimal/Dockerfile . The configuration with the VSCode extensio...

  • 0 kudos
sambgp
by New Contributor
  • 747 Views
  • 2 replies
  • 0 kudos

RDD Issue in Unity Catalog

 ################################################ New Code as per UC ################################################ def parse_json(df, *cols, clean=True        res = df    for i in cols:           if clean:            res = (                res.wit...

  • 747 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rishabh_Tiwari
Databricks Employee
  • 0 kudos

Hi @sambgp , Thank you for reaching out to our community! We're here to help you.  To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...

  • 0 kudos
1 More Replies
Krishna2110
by New Contributor II
  • 911 Views
  • 4 replies
  • 2 kudos

Databricks All Delta Tables Data Read

If we want to read all the data of the databricks tables at single time how can we able to do it.

  • 911 Views
  • 4 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 2 kudos

Hi @Krishna2110 ,Here it is, it should work now tables = spark.sql("SHOW TABLES IN ewt_edp_prod.crm_raw").collect() for row in tables: table_name = f"ewt_edp_prod.{row[0]}.{row[1]}" try: df = spark.table(table_name) count = df...

  • 2 kudos
3 More Replies
Dp15
by Contributor
  • 1977 Views
  • 2 replies
  • 2 kudos

Resolved! Convert Date String to Date type Returning null

Hi,I am using Databricks SQL and I am converting a integer field which is of format ('20240719' or 'yyyyMMdd'),now I am able to convert it to date type using to_date(forecastedHorizonPeriodDate,'yyyyMMdd')I then tried to change the format of this dat...

  • 1977 Views
  • 2 replies
  • 2 kudos
Latest Reply
Dp15
Contributor
  • 2 kudos

Hey @szymon_dybczak Thank you for your answer

  • 2 kudos
1 More Replies
virendra1212
by New Contributor
  • 3655 Views
  • 1 replies
  • 0 kudos

Login issue

Hi,Getting below error message.We were not able to find a Community Edition workspace with this email. Please login to non-community-edition workspaces you may have access to.

  • 3655 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @virendra1212 ,Take a look at below thread. Maybe some of the suggestions will help in your case.Community Edition Login Issues Below is a list of ... - Page 4 - Databricks Community - 26926You signed up before May 14, 2022 -Please confirm that th...

  • 0 kudos
payalbhatia
by New Contributor II
  • 535 Views
  • 0 replies
  • 0 kudos

Shuffle Partitions

What if I have lot of empty shuffled partitions due to data skewness Secondly , if the shuffle partition size is 128 MB and if the size of the key's partition is 700 MB 

  • 535 Views
  • 0 replies
  • 0 kudos
Junda
by New Contributor II
  • 836 Views
  • 3 replies
  • 4 kudos

The way to visualize logging of Spark Job graphically

Is there any way to visualize logging such as execution runtime or memory usage of Spark Job graphically just like the image below by utilizing Databricks Partner Connect for free?Also, I'd appreciate to know any other ways to visualize the logging o...

  • 836 Views
  • 3 replies
  • 4 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 4 kudos

Pleasure to help

  • 4 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors