cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Panna
by • New Contributor II
  • 1957 Views
  • 1 replies
  • 3 kudos

Is there only one element type option for an array?

I'm creating an array which contains both string and double, just wondering if I can have multiple element type options for one array column? Thanks

Data Engineering
Reply
  • 1957 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

Elements of any type that share a least common type can be used, https://docs.databricks.com/sql/language-manual/functions/array.html#arguments.Please correct me if I misunderstood to understand the requirement.

  • 3 kudos
kfoster
by • Contributor
  • 4071 Views
  • 4 replies
  • 0 kudos

Resolved! DLT Pipeline Retries

Is there a way to limit how many retries DLT Pipelines run when in "Production" mode? What is the key value pair I use in the configuration?

Data Engineering
Reply
  • 4071 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Cedric
Databricks Employee
  • 0 kudos

Hi @Kristian Foster​,Yes. We have two spark configuration that can be set. pipelines.numStreamRetryAttempts and pipelines.numUpdateRetryAttempts. The former configures how many times we retry each flow before failing the update. The latter configures...

  • 0 kudos
3 More Replies
jowziph
by • New Contributor III
  • 4842 Views
  • 2 replies
  • 7 kudos

How can I fix strange magic command behaviour in Databricks notebooks?

Hi,I have encountered a strange issue when using magic commands on my Azure Databricks workspace.When I create a cell and type the magic command for a language which is not default in the notebook, the code is instead interpreted as the default noteb...

magic command manually typed magic command manually typed 2 image
Data Engineering
Reply
  • 4842 Views
  • 2 replies
  • 7 kudos
View Replies
Latest Reply
jowziph
New Contributor III
  • 7 kudos

Hi,Yeah this is happening when magic command is used in between two cells of the default language. But this is not the only case.This is happening when magic command is manually typed out (e.g. %sql or %scala in a python notebook). OR when magic comm...

  • 7 kudos
1 More Replies
Gim
by • Contributor
  • 12774 Views
  • 7 replies
  • 10 kudos

Resolved! Delta Table storage best practices

Hi!We have a project where we do some Data Engineering for a client. I implemented a scheduled batch processing of Databricks' autoloader (stream w/ availableNow) since they primarily have numerous file exports from several sources. We wanted to foll...

Data Engineering
Reply
  • 12774 Views
  • 7 replies
  • 10 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 10 kudos

Hi @Gimwell Young​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 10 kudos
6 More Replies
kpendergast
by • Contributor
  • 5236 Views
  • 6 replies
  • 4 kudos

Resolved! Hyperleaup to push data to Tableau Server

Would any care to share how they got the Hyperleaup library working I am currently stuck at an error at publish and cannot seem to find a solution. TypeError: publish() got an unexpected keyword argument 'file_path'I am %pip installing all the requir...

Data Engineering
Reply
  • 5236 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
badnishant79
New Contributor II
  • 4 kudos

Hi. Yes dashboard includes multiple filters but only uploaded dashboard on server without any other sheets. I am looking into the extract that other users have suggested. Thanks.

  • 4 kudos
5 More Replies
HariharaSam
by • Contributor
  • 2082 Views
  • 3 replies
  • 0 kudos

DRIVER Garbage Collection

Does anyone know how to fix this ..??

image
Data Engineering
Reply
  • 2082 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Hariharan Sambath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
2 More Replies
KrishZ
by • Contributor
  • 2372 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

Data Engineering
Reply
  • 2372 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
RohitKulkarni
by • Contributor II
  • 2986 Views
  • 2 replies
  • 2 kudos

Get file from SharePoint to copy into Azure blob storage

Hello Team,I am trying to copy the xlx files from sharepoint and move to the Azure blob storageUSERNAME = app_config_client.get_configuration_setting(key='BIAppConfig:SharepointUsername',label='BIApp').valuePASSWORD = app_config_client.get_configura...

Data Engineering
Reply
  • 2986 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
shanakap
by • New Contributor II
  • 2248 Views
  • 2 replies
  • 0 kudos

Create H3 hexagons Mosaic

I am new to using `Mosaic` on Databricks. The task is to create a Heatmap with the counts within Hexagons of H3 resolution. Since the dataset is quite large I am opting for Mosaic. As the first step I am trying to create the hexagons covering the giv...

2ATLt
Data Engineering
Reply
  • 2248 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Shanaka Pererea​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
1 More Replies
Ryan_Chynoweth
by • Esteemed Contributor
  • 1605 Views
  • 3 replies
  • 2 kudos

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

Data Engineering
Reply
  • 1605 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Great article, thank you for sharing

  • 2 kudos
2 More Replies
KrishZ
by • Contributor
  • 1536 Views
  • 2 replies
  • 1 kudos

Rowlock in databricks sql?

I am trying to get the commands for xlock, holdlock, rowlock, readpast in databricks sql.Looking to implement this

Data Engineering
Reply
  • 1536 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @Krishna Zanwar​,We dont currently support these features, but you can create a feature request if you would like to have these features added to Databricks SQL. Please follow the instructions from here https://docs.databricks.com/resources/ideas....

  • 1 kudos
1 More Replies
syedmuhammedmeh
by • New Contributor III
  • 3134 Views
  • 2 replies
  • 6 kudos

Resolved! Databricks Kafka Read Not connecting

I'm trying to read data from GCP kafka through azure databricks but getting below warning and notebook is simply not completing. Any suggestion please? WARN NetworkClient: Consumer groupId Bootstrap broker rack disconnectedPlease note I've properly c...

Data Engineering
Reply
  • 3134 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Could you share the full error stack trace from your driver's logs? This is a Warning message, we need to take a look at the error level messages.

  • 6 kudos
1 More Replies
antoniok
by • New Contributor II
  • 3762 Views
  • 1 replies
  • 3 kudos

dbutils.fs.ls is giving "null uri host This can be caused by unencoded / in the password string"

I'm trying to list number of files in s3 bucket. I've initially used "aws s3 ls <s3://>" to list the files and it worked. However, when trying to do the same using dbutils.fs.ls, I'm getting java.lang.NullPointerException: null uri host. This can be ...

Data Engineering
Reply
  • 3762 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
marcus1
New Contributor III
  • 3 kudos

You might be encountering an issue with bucket naming. Which I'm also getting with a bucket named something.[0-9]https://issues.apache.org/jira/browse/HADOOP-17241

  • 3 kudos
Lizzz
by • New Contributor II
  • 3629 Views
  • 1 replies
  • 3 kudos

Resolved! Forward Spark structured streaming metrics to Datadog

We have a spark streaming application written in Pyspark that we'd like to monitor with Datadog. By default, datadog collects a couple of streaming metrics like 'spark.structured_streaming.processing_rate' and 'spark.structured_streaming.latency'. Ho...

Data Engineering
Reply
  • 3629 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 3 kudos

@Liz Zhang​ , Please refer to the below documentation contain pyspark implementation of streamingQueryListener https://www.databricks.com/blog/2022/05/27/how-to-monitor-streaming-queries-in-pyspark.html

  • 3 kudos
fhte
by • New Contributor
  • 1890 Views
  • 2 replies
  • 0 kudos

How to install R GeoLift library on Databrickts

Hi, I am having problems installing the GeoLift library. I am proceeding as per the official instructions: https://facebookincubator.github.io/GeoLift/docs/GettingStarted/InstallingRThis is what I run in the notebook:1) I install this particular vers...

Screenshot 2022-09-14 at 08.59.09
Data Engineering
Reply
  • 1890 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Ludmila Kuncarova​,I would like to share the following link to our docs https://docs.databricks.com/libraries/notebooks-r-libraries.html in this link you will be able to find more details on how to install R libraries.

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels