cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
rocky5
by • New Contributor III
  • 2077 Views
  • 2 replies
  • 2 kudos

DLT UDF and c#

Hello, can I create spark function in .net and use it in DLT table? I would like to encrypt some data, in documentation scala code is being used as an example, but would it be possible to write decryption/encryption function using C# and use it withi...

Data Engineering
Reply
  • 2077 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Meghala
Valued Contributor II
  • 2 kudos

It's not possible. SQL Server 2008 contains SQL CLR runtime that runs .NET languages.

  • 2 kudos
1 More Replies
Aravind_P04
by • New Contributor II
  • 3339 Views
  • 3 replies
  • 4 kudos

Clarification on merging multiple notebooks and other

1. Do we have any feature like merge the cells from one or more notebooks into other notebook.2. Do we have any feature like multiple cells from excel is copied it into multiple cells in a notebook . Generally all excel data is copied it into one cel...

Data Engineering
Reply
  • 3339 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 4 kudos

1) We can't merge cells right now2)We don't have this feature as well3) We don't have multiple editing right now4)You will know only if you face an error. A Notification will pop up5)You can"t keep running the execution because the cells can be linke...

  • 4 kudos
2 More Replies
Aviral-Bhardwaj
by • Esteemed Contributor III
  • 2757 Views
  • 6 replies
  • 30 kudos

DLT PipeLine Understanding

Hey, guys, I hope you are doing very well today I was going through some databricks documentation and I found dlt documentation but when I am trying to implement it, it is not working very well can anyone can share with me whole code step by step and...

Data Engineering
Reply
  • 2757 Views
  • 6 replies
  • 30 kudos
View Replies
Latest Reply
Meghala
Valued Contributor II
  • 30 kudos

even Im also going through some databricks documentation

  • 30 kudos
5 More Replies
hello_world
by • New Contributor III
  • 3366 Views
  • 3 replies
  • 2 kudos

What exact difference does Auto Loader make?

New to Databricks and here is one thing that confuses me.Since Spark Streaming is already capable of incremental loading by checkpointing. What difference does it make by enabling Auto Loader?

Data Engineering
Reply
  • 3366 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Meghala
Valued Contributor II
  • 2 kudos

Auto Loader provides a Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files i...

  • 2 kudos
2 More Replies
KuldeepChitraka
by • New Contributor III
  • 7470 Views
  • 4 replies
  • 6 kudos

Error handling/exception handling in NOtebook

What is a common practice to to write notebook which includes error handling/exception handling.Is there any example which depicts how notebook should be written to include error handling etc.

Data Engineering
Reply
  • 7470 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
Meghala
Valued Contributor II
  • 6 kudos

runtime looks for handlers (try-catch) that are registered to handle such exceptions

  • 6 kudos
3 More Replies
Aviral-Bhardwaj
by • Esteemed Contributor III
  • 9060 Views
  • 3 replies
  • 25 kudos

Understanding Joins in PySpark/Databricks In PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you ...

Understanding Joins in PySpark/DatabricksIn PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you to merge data from different sources into a single dataset and potentially perform transformations on...

Data Engineering
Reply
  • 9060 Views
  • 3 replies
  • 25 kudos
View Replies
Latest Reply
Meghala
Valued Contributor II
  • 25 kudos

very informative

  • 25 kudos
2 More Replies
FranPérez
by • New Contributor III
  • 10301 Views
  • 7 replies
  • 4 kudos

set PYTHONPATH when executing workflows

I set up a workflow using 2 tasks. Just for demo purposes, I'm using an interactive cluster for running the workflow. { "task_key": "prepare", "spark_python_task": { "python_file": "file...

Data Engineering
Reply
  • 10301 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 4 kudos

Hi @Fran Pérez​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 4 kudos
6 More Replies
Jessica1
by • New Contributor
  • 537 Views
  • 0 replies
  • 0 kudos

There are a number of risks associated with using social networking sites for Dating services, including the possible exploitation of minors, the pote...

There are a number of risks associated with using social networking sites for Dating services, including the possible exploitation of minors, the potential for human trafficking, and the possibility of illegal activities such as money laundering traf...

Data Engineering
Reply
  • 537 Views
  • 0 replies
  • 0 kudos
SaraGHn
by • New Contributor III
  • 1264 Views
  • 1 replies
  • 4 kudos

Error for sparkdl.xgboost import XgboostRegressor

I get the error :cannot import name 'resnet50' from 'keras.applications' (/local_disk0/.ephemeral_nfs/envs/pythonEnv-a3e7b0cc-064d-4585-abfd-6473ed1c1a5b/lib/python3.8/site-packages/keras/applications/__init__.py) It looks like the Keras.applications...

image
Data Engineering
Reply
  • 1264 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 4 kudos

try to install these libraries via init script some time this happen due to spark version in databricks , libraries can make conflict with Runtime version

  • 4 kudos
georgian2133
by • New Contributor
  • 1712 Views
  • 0 replies
  • 0 kudos

Getting error [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES]

[DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(DocDate AND orderedhl)" due to data type mismatch: the left and right operands of the binary operator have incompatible types ("STRING" and "DECIMAL(38,6)").; line 67, pos 066. group by 67. or...

Data Engineering
Reply
  • 1712 Views
  • 0 replies
  • 0 kudos
joakon
by • New Contributor III
  • 2959 Views
  • 5 replies
  • 1 kudos

Resolved! slow running query

Hi All, I would you to get some ideas on how to improve performance on a data frame with around 10M rows. adls- gen2df1 =source1 , format , parquet ( 10 m)df2 =source2 , format , parquet ( 10 m)df = join df1 and df2 type =inner join df.count() is ...

Data Engineering
Reply
  • 2959 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution

  • 1 kudos
4 More Replies
test_user
by • New Contributor II
  • 28087 Views
  • 3 replies
  • 1 kudos

How to explode an array column and repack the distinct values into one array in DB SQL?

Hi, I am new to DB SQL. I have a table where the array column (cities) contains multiple arrays and some have multiple duplicate values. I need to unpack the array values into rows so I can list the distinct values. The following query works for this...

Data Engineering
Reply
  • 28087 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

try to use SQL windows functions here

  • 1 kudos
2 More Replies
Aviral-Bhardwaj
by • Esteemed Contributor III
  • 6531 Views
  • 6 replies
  • 33 kudos

Resolved! Timezone understanding

Today I was working in Timezone kind of data but my Singapore user want to see their time in the Data and USA user want to see their time in the datainstead of both, we all are getting UTC time,how to solve this issuePlease guide Data can be anything...

Data Engineering
Reply
  • 6531 Views
  • 6 replies
  • 33 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 33 kudos

I got it guys it was happening due to a library conflict now your answers are really helpful I tried all things

  • 33 kudos
5 More Replies
Ruby8376
by • Valued Contributor
  • 2693 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks authentication

Hi there!!we are planning to use databricks -tableau on prem integration for reporting. Data would reside in delta lake and using ta leau-databricks connector, user would be able to generate reports from that data .question is: a private end point wi...

Data Engineering
Reply
  • 2693 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

and make sure that you are going with SPARK SQL connection , else it will always fail

  • 1 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels