cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RohitKulkarni
by Contributor
  • 2726 Views
  • 6 replies
  • 4 kudos

External table issue

Hello Team,I am using df.write command and the table is getting created. If you refer the below screenshot the table got created in Tables folder in dedicated sql pool. But i required in the External Tables folder. RegardsRK

image image
  • 2726 Views
  • 6 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Rohit Kulkarni​ , We haven't heard from you on the last response from @Werner Stinckens​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to...

  • 4 kudos
5 More Replies
PriyaTech
by New Contributor
  • 2535 Views
  • 2 replies
  • 2 kudos

Resolved! Converting Dataframe into Nested xml

e.g.dataframe is having firstname,lastname,middlename,id,salaryI need to convert dataframe in xml file but in nested format.output as nested xml<Name>    <firatname> <middlename>    <lastname>    </Name><id></id><salary></salary>Anyone has ides ho...

  • 2535 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Priyanka Jain​, We haven't heard from you on the last response from @Werner Stinckens​​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to o...

  • 2 kudos
1 More Replies
19582
by New Contributor II
  • 629 Views
  • 2 replies
  • 2 kudos

Run a simple spark-scala jar (hello-world) on existing running cluster

I have created a simple hello-world jar that I would like to run as a job. I also have an existing cluster. Now when I create a job to run on existing cluster, it fails for some unknown reason (don't see much in the errors), while if I run the same j...

  • 629 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Meet Anadkat​, We haven’t heard from you on the last response from @Hubert Dudek​​. Or else, If you have any solution, please share it with the community as it can be helpful to others.

  • 2 kudos
1 More Replies
SujitOjha
by New Contributor
  • 1229 Views
  • 1 replies
  • 1 kudos

What is the way to do DEEP CLONE and copy the checkpoints folder also?

When I use Deep Clone, I dont see the checkpoint folder to be copied?Is there a possibilities to copy the checkpoint folder also, as I have to resume the streaming job to updated location.

  • 1229 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16753725469
Contributor II
  • 1 kudos

Delta clones are recommended for disaster recovery it doesn't exactly replicate table history in the context of specific snapshots, but it does ensure that the changes are replicatedBut we can't use a clone table with the copy of the source checkpoin...

  • 1 kudos
Phani1
by Valued Contributor
  • 1099 Views
  • 2 replies
  • 5 kudos

Delta table Concurrent Updates for Non-partitioned tables

When we implemented the concurrent updates on a table which do not have a partition column we ran into ConcurrentAppendException [ensured where the condition is different for each concurrent update statement]So do we need to go by partition approach ...

  • 1099 Views
  • 2 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

Please check that both streaming queries don't use the same checkpoint,Auto increment id can also make problems as it is kept in schemaSchema evolution also can make problems

  • 5 kudos
1 More Replies
User16826992666
by Valued Contributor
  • 2994 Views
  • 3 replies
  • 0 kudos

How do I make a parameter in a Databricks SQL dashboard apply to multiple visuals?

I have created a few queries and visualizations in Databricks SQL which use parameters. Each query has the same parameter. But when I pin the visualizations to a dashboard, each of the visuals keeps it's own parameter drop down.I want to have one dro...

  • 2994 Views
  • 3 replies
  • 0 kudos
Latest Reply
User16826992666
Valued Contributor
  • 0 kudos

To achieve this, you can edit the source of the parameters on each of the visuals on the dashboard. The source for each visual can be changed to read from a shared dashboard parameter. These are the steps to do this- 1.) First click on the three dots...

  • 0 kudos
2 More Replies
kfoster
by Contributor
  • 2194 Views
  • 4 replies
  • 0 kudos

Resolved! DLT Pipeline Retries

Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?

  • 2194 Views
  • 4 replies
  • 0 kudos
Latest Reply
Cedric
Valued Contributor
  • 0 kudos

Hi @Kristian Foster​,Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures...

  • 0 kudos
3 More Replies
jowziph
by New Contributor III
  • 2919 Views
  • 2 replies
  • 7 kudos

How can I fix strange magic command behaviour in Databricks notebooks?

Hi,I have encountered a strange issue when using magic commands on my Azure Databricks workspace.When I create a cell and type the magic command for a language which is not default in the notebook, the code is instead interpreted as the default noteb...

magic command manually typed magic command manually typed 2 image
  • 2919 Views
  • 2 replies
  • 7 kudos
Latest Reply
jowziph
New Contributor III
  • 7 kudos

Hi,Yeah this is happening when magic command is used in between two cells of the default language. But this is not the only case.This is happening when magic command is manually typed out (e.g. %sql or %scala in a python notebook). OR when magic comm...

  • 7 kudos
1 More Replies
Gim
by Contributor
  • 5423 Views
  • 7 replies
  • 10 kudos

Resolved! Delta Table storage best practices

Hi!We have a project where we do some Data Engineering for a client. I implemented a scheduled batch processing of Databricks' autoloader (stream w/ availableNow) since they primarily have numerous file exports from several sources. We wanted to foll...

  • 5423 Views
  • 7 replies
  • 10 kudos
Latest Reply
jose_gonzalez
Moderator
  • 10 kudos

Hi @Gimwell Young​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 10 kudos
6 More Replies
kpendergast
by Contributor
  • 3456 Views
  • 6 replies
  • 4 kudos

Resolved! Hyperleaup to push data to Tableau Server

Would any care to share how they got the Hyperleaup library working I am currently stuck at an error at publish and cannot seem to find a solution. TypeError: publish() got an unexpected keyword argument 'file_path'I am %pip installing all the requir...

  • 3456 Views
  • 6 replies
  • 4 kudos
Latest Reply
badnishant79
New Contributor II
  • 4 kudos

Hi. Yes dashboard includes multiple filters but only uploaded dashboard on server without any other sheets. I am looking into the extract that other users have suggested. Thanks.

  • 4 kudos
5 More Replies
HariharaSam
by Contributor
  • 1160 Views
  • 3 replies
  • 0 kudos

DRIVER Garbage Collection

Does anyone know how to fix this ..??

image
  • 1160 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Hariharan Sambath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
2 More Replies
KrishZ
by Contributor
  • 1399 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

  • 1399 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
RohitKulkarni
by Contributor
  • 1723 Views
  • 2 replies
  • 2 kudos

Get file from SharePoint to copy into Azure blob storage

Hello Team,I am trying to copy the xlx files from sharepoint and move to the Azure blob storageUSERNAME = app_config_client.get_configuration_setting(key='BIAppConfig:SharepointUsername',label='BIApp').valuePASSWORD = app_config_client.get_configura...

  • 1723 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
shanakap
by New Contributor II
  • 1239 Views
  • 2 replies
  • 0 kudos

Create H3 hexagons Mosaic

I am new to using `Mosaic` on Databricks. The task is to create a Heatmap with the counts within Hexagons of H3 resolution. Since the dataset is quite large I am opting for Mosaic. As the first step I am trying to create the hexagons covering the giv...

2ATLt
  • 1239 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Shanaka Pererea​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
1 More Replies
Ryan_Chynoweth
by Honored Contributor III
  • 873 Views
  • 3 replies
  • 2 kudos

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

  • 873 Views
  • 3 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Great article, thank you for sharing

  • 2 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels