cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

plynton
by New Contributor II
  • 992 Views
  • 3 replies
  • 1 kudos

Incorrect results with df.query()

I have tried pulling a single row from a .csv using df.query()However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:df = spark.read.option("header",True).csv(data_fldr + "config...

  • 992 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Peter Ott​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
bozhu
by Contributor
  • 1400 Views
  • 2 replies
  • 1 kudos

Multiple DLT Pipelines Sharing a Single Cluster

When you use Workflows to orchestrate standard notebooks, they can share a single cluster. It will be awesome if we can achieve the same for DLT pipelines orchestrated in a Workflows Job.I understand DLT pipelines utilising their own special clusters...

  • 1400 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Bo Zhu​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Lars_J
by New Contributor
  • 1272 Views
  • 2 replies
  • 0 kudos

Databricks-jdbc and vulnerabilities CVE-2022-42004, CVE-2022-42003

The latest version of Databricks-jdbc available through Maven (2.6.29) now has these two vulnerabilities:https://nvd.nist.gov/vuln/detail/CVE-2022-42004https://nvd.nist.gov/vuln/detail/CVE-2022-42003All due to depending on and including in the jar th...

  • 1272 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Lars Joreteg​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
AJDJ
by New Contributor III
  • 1622 Views
  • 5 replies
  • 1 kudos

Pipeline in Community edition

Hi there,I'm learning Databricks using the community edition. I noticed I don't have way to practice the pipeline in community edition. (The icon below compute). Says I need to upgrade. Is there any way to practice pipeline and follow learning lesson...

  • 1622 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @AJ DJ​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
4 More Replies
gbradley145
by New Contributor III
  • 6862 Views
  • 6 replies
  • 5 kudos

SQL CTE in Databricks or something similar?

%sqlWITH genCTE AS (SELECT MAX(PredID) + 1 AS PredID, 145 AS SystemIDFROM TableAUNION ALLSELECT PredID + 1FROM genCTE)SELECT * FROM genCTEWhen I attempt this, I get an error that genCTE does not exists.There may be a better way to what I am trying to...

  • 6862 Views
  • 6 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Greg Bradley​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 5 kudos
5 More Replies
William_Scardua
by Valued Contributor
  • 4090 Views
  • 7 replies
  • 3 kudos

uuid in Merge

Hi guys,I'm trying to use uuid in the merge but I always get an error...import uuid   ( df_events.alias("events").merge( source = df_updates.alias("updates"), condition = "events.cod = updates.cod and events.num = updates.num" ).whenMatch...

  • 4090 Views
  • 7 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @William Scardua​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 3 kudos
6 More Replies
sm1
by New Contributor III
  • 2525 Views
  • 6 replies
  • 3 kudos

New Visualization Tools

How do I add new visualization tool option to my databricks? I don't see a plus sign that will let you choose "Visualization" in my display command results :(. Please help.

Capture
  • 2525 Views
  • 6 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Suky Muliadikara​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 3 kudos
5 More Replies
ramankr48
by Contributor II
  • 2132 Views
  • 2 replies
  • 3 kudos

Issue with identity key column in databricks?

For the identity key I've used both GENERATED ALWAYS AS IDENTITY(start with 1 increment by 1) andGENERATED BY DEFAULT AS IDENTITY(start with 1 increment by 1)but in both cases, if I'm running my script once then it is fine (identity key is working as...

  • 2132 Views
  • 2 replies
  • 3 kudos
Latest Reply
lizou
Contributor II
  • 3 kudos

yes, by default option allow duplicated values per design.I will avoid this option and use only use GENERATED ALWAYS AS IDENTITY Using BY DEFAULT option is worse than not using it at all in BY Default option, If I forget to set starting value, the ID...

  • 3 kudos
1 More Replies
bozhu
by Contributor
  • 1080 Views
  • 1 replies
  • 4 kudos

DLT DataPlaneException

I created an Azure Databricks workspace with my Visual Studio Subsciption, so far everything has been working as expected although I have requested to increase CPU core limit once.I am now getting this "DataPlaneException" error in the DTL during "Wa...

  • 1080 Views
  • 1 replies
  • 4 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 4 kudos

@Bo Zhu​ can we get more error log, looks quota limit exceeded. did you get a chance to check quota in azure portal and see if cores exists for config that you selected. try to select another cluster config and validate

  • 4 kudos
AmineHY
by Contributor
  • 1795 Views
  • 2 replies
  • 5 kudos

Resolved! My DLT pipeline return ACL Verification Failed

Python Commanddf = spark.read.format('csv').option('sep', ';').option("recursiveFileLookup", "true").load('dbfs:/***/data_files/PREVISIONS/')Here is the content of the folder  Each folder contain the following files: Full logorg.apache.spark.sql.stre...

image image.png
  • 1795 Views
  • 2 replies
  • 5 kudos
Latest Reply
AmineHY
Contributor
  • 5 kudos

Yes some of the files I don't have the right to access (mistakenly) In this case, how do you think I can tell DTL to handle this exception and ignore the file, since I can read some files but not all?

  • 5 kudos
1 More Replies
Retko
by Contributor
  • 780 Views
  • 1 replies
  • 2 kudos

How to jump back to latest positions in the Notebook

Hi,when developing I often need to jump around the Notebook to fix and run things. It would be really helpful so I can jump back to several latest positions (cells), similarly, like in Office Word by SHIFT+F5 key. Is here a way now in Databricks?Than...

  • 780 Views
  • 1 replies
  • 2 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 2 kudos

@Retko Okter​ go to any of notebook and click on help-->keyboard shortcuts, they will show all possibilities that you need

  • 2 kudos
db-avengers2rul
by Contributor II
  • 1457 Views
  • 4 replies
  • 4 kudos

course code - 'ACAD-INTRO-DELTALAKE' Notebook has errors

Dear DB Team,While following a course from DB Academy course code - 'ACAD-INTRO-DELTALAKE' noticed the notebooks has errors can you please check i have also attached the notebookRegards,Rakesh

  • 1457 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rakesh Reddy Gopidi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 4 kudos
3 More Replies
hbenzineb
by New Contributor II
  • 1768 Views
  • 2 replies
  • 4 kudos

How to restore a dbfs directory ?

Hello,Can I restore a dbfs directory ?I dropped a database using an important directory Someone can help me please ?

  • 1768 Views
  • 2 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @HASSEN BEN ZINEB​ , Unfortunately, there is no possible way to restore the deleted files from DBFS. Only possible way is to have a backup and restore it to the DBFS. Please go through the DBFS document for more details.

  • 4 kudos
1 More Replies
elgeo
by Valued Contributor II
  • 3184 Views
  • 0 replies
  • 2 kudos

SQL While do loops

Hello. Could you please suggest a workaround for a while do loop in Databricks SQL?WHILE LSTART>0 DO SET LSTRING=CONCAT(LSTRING, VSTRING2)Thank you in advance

  • 3184 Views
  • 0 replies
  • 2 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels