cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

TMNGB
by New Contributor II
  • 1381 Views
  • 2 replies
  • 2 kudos

Resolved! Does MERGE statement preserve order? (Slowly Changing Dimensions)

In the case of processing multiple source files - with potentially, one or multiple entity versions per source - being able to use the MERGE statement whilst preserving the order is key to ensure the correct versioning of entity versions (aka, versio...

  • 1381 Views
  • 2 replies
  • 2 kudos
Latest Reply
Noopur_Nigam
Valued Contributor II
  • 2 kudos

Hi @Guilherme Banhudo​ I hope that werners answer would have helped you. Please let me know if you still have doubts or queries.

  • 2 kudos
1 More Replies
77796
by New Contributor II
  • 3336 Views
  • 4 replies
  • 0 kudos

Databricks S3A error - java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory not found

We are getting the below error for runtime 10.x and 11.x when writing to s3 via saveAsNewAPIHadoopFile function. The same jobs are running fine on runtime 9.x and 7.x. The difference betwen 9.x and 10.x is the former has hadoop 2.7 bindings with sp...

  • 3336 Views
  • 4 replies
  • 0 kudos
Latest Reply
77796
New Contributor II
  • 0 kudos

We have resolved this issue by using s3 scheme instead of s3a i.e. pairRDD.saveAsNewAPIHadoopFile("s3://bucket/testout.dat",

  • 0 kudos
3 More Replies
zyang
by Contributor
  • 2435 Views
  • 6 replies
  • 2 kudos

azure databricks notebook cannot load the difference

I am trying to commit and push my change to the branch, I cannot load the difference. I haven't changed many cells and each cells doesn't exceed the 500 lines in the notebook file. I am wondering why this happens and how to solve it?

Screenshot 2022-06-26 101907
  • 2435 Views
  • 6 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hey there @z yang​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
5 More Replies
OldDogNewTrix
by New Contributor
  • 590 Views
  • 3 replies
  • 0 kudos
  • 590 Views
  • 3 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Jim Carlson​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 0 kudos
2 More Replies
Yagao
by New Contributor
  • 664 Views
  • 2 replies
  • 0 kudos

How to do python within sql query in Databricks ?

How to do python within sql query in Databricks ?

  • 664 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Ya Gao​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 0 kudos
1 More Replies
Mapajr
by New Contributor III
  • 1941 Views
  • 3 replies
  • 3 kudos

Issues pushing repos on Gitlab with Databricks

Our company uses Gitlab enterprise edition and we link our repos up to databricks through this. Randomly we will get errors when trying to push the repo and we have to spend hours debugging trying to figure out what is causing the push error on datab...

  • 1941 Views
  • 3 replies
  • 3 kudos
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hey there @Mark Patrick​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 3 kudos
2 More Replies
ViktorWolf
by New Contributor
  • 971 Views
  • 3 replies
  • 0 kudos

Why sometimes autoloader lose the checkpoint path and break the streaming?

Why sometimes autoloader lose the checkpoint path and break the streaming?

  • 971 Views
  • 3 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Vittorio Antonacci​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 0 kudos
2 More Replies
al_joe
by Contributor
  • 605 Views
  • 0 replies
  • 3 kudos

Why is this simple numerical operation not precise?

I was experimenting with beginner tutorial and saw this strange output ...Why is this so ? And why is the behavior not consistent for ALL rows updated by the same statement?8.8 - 1 = 7.800000000000001See screenshot ...

20220827_180902_msedge_DE_2.1_-_Managing_Delta_Tables_-_Databricks_-_Pers
  • 605 Views
  • 0 replies
  • 3 kudos
RiyazAli
by Valued Contributor
  • 3618 Views
  • 9 replies
  • 5 kudos

Issue with .dbc in the Advanced Data Engineering course in Databricks Academy

The very first notebook of the dbc notebook which is a setup cell fails.

image
  • 3618 Views
  • 9 replies
  • 5 kudos
Latest Reply
Niha1
New Contributor III
  • 5 kudos

Hi Riyaz,Please find the snippet of the error below--:"AnalysisException: Path does not exist: dbfs:/user/nniha9188@gmail.com/dbacademy/machine_learning/datasets/airbnb/sf-listings/sf-listings-2019-03-06-clean.parquet"Source-The source for this datas...

  • 5 kudos
8 More Replies
Shakzz
by New Contributor III
  • 4215 Views
  • 3 replies
  • 13 kudos
  • 4215 Views
  • 3 replies
  • 13 kudos
Latest Reply
Vidula
Honored Contributor
  • 13 kudos

Hey there @Shakti Chand​ Hope all is well! Just wanted to check in if you were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...

  • 13 kudos
2 More Replies
ankit_k
by New Contributor
  • 1028 Views
  • 3 replies
  • 0 kudos

Move a GCP Project with Databricks in it to new Organization

We are trying to move a GCP project to a Newly created Org and new billing account. We have a Databricks instance from GCP Marketplace with licensing As per the docs when we change a billing account for a Project the license on the first billing acco...

  • 1028 Views
  • 3 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Ankit K​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best?  Else please let us know if you need more help. We'd love to hear from you.Cheers!

  • 0 kudos
2 More Replies
alan_cooper
by New Contributor II
  • 1598 Views
  • 3 replies
  • 2 kudos

Can you manage Workspaces using Terraform on an AWS Standard Tier account.

Background: I am looking to migrate a small PoC Databricks workspace in Azure over to a productionised deployment in AWS.I need to deploy all the Workspaces and all the notebooks etc code in those workspaces using a CI/CD pipeline, using Terraform to...

  • 1598 Views
  • 3 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @ALAN COOPER​ I hope all is well! I just wanted to check in if you were able to resolve your issue would you be happy to mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
2 More Replies
tafmuko
by New Contributor II
  • 593 Views
  • 0 replies
  • 2 kudos

Azure Databricks Tenant/Directory Swicth

HelloWe are looking to move a PAYG subscription to a new tenant. We've got Azure Databricks deployed. There are a few questions:Will Databricks continue to function properly (aside from the RBAC list which we will re-apply to all resources)Since Data...

  • 593 Views
  • 0 replies
  • 2 kudos
ABose8
by Contributor
  • 3833 Views
  • 14 replies
  • 8 kudos

Resolved! Attended 27th July 2022 webinar but have not recieved voucher,even uploaded Lakehouse certificate

@Kaniz Fatma​ @Samantha Menot​ This is Arindam Bose.Actually I attended Databricks webinar on 27th July for (Databricks Certification Exam Overview Training: Databricks Certified Data Analyst Associate).I was expecting vouchers for Databricks Certifi...

  • 3833 Views
  • 14 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @abose8 and @Shashank Tiwari​, Let us fix this for you asap. Please don't worry. It will be taken care of as soon as possible.

  • 8 kudos
13 More Replies
Rita
by New Contributor III
  • 5572 Views
  • 12 replies
  • 6 kudos

How to connect Cognos 11.1.7 to Azure Databricks

We are trying to connect Cognos 11.1.7 to Azure Databricks, but no success.Can you please help or guide us how to connect Cognos 11.1.7 to Azure Databricks.This is very critical to our user community. Can you please help or guide us how to connect Co...

  • 5572 Views
  • 12 replies
  • 6 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 6 kudos

Hi @Rita Mordukhay​ , Just a friendly follow-up. Do you still need help? Please let us know.

  • 6 kudos
11 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels