cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
ThomasKastl
by • Contributor
  • 4945 Views
  • 6 replies
  • 5 kudos

Resolved! Databricks runs cell, but stops output and hangs afterwards.

tl;dr: A cell that executes purely on the head node stops printed output during execution, but output still shows up in the cluster logs. After execution of the cell, Databricks does not notice the cell is finished and gets stuck. When trying to canc...

Data Engineering
Reply
  • 4945 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

As that library work on pandas problem can be that it doesn't support pandas on spark. On the local version, you probably use non-distributed pandas. You can check behavior by switching between:import pandas as pd import pyspark.pandas as pd

  • 5 kudos
5 More Replies
165036
by • New Contributor III
  • 1954 Views
  • 1 replies
  • 1 kudos

Resolved! Mounting of S3 bucket via Terraform is frequently timing out

Summary of the problemWhen mounting an S3 bucket via Terraform the creation process is frequently timing out (running beyond 10 minutes). When I check the Log4j logs in the GP cluster I see the following error message repeated:```22/07/26 05:54:43 ER...

Data Engineering
Reply
  • 1954 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
165036
New Contributor III
  • 1 kudos

Solved. See here: https://github.com/databricks/terraform-provider-databricks/issues/1500

  • 1 kudos
dataAllMyLife
by • New Contributor
  • 1075 Views
  • 1 replies
  • 0 kudos

JDBC Connection closes between 'stmt.execute( ... ) and stmt.executeQuery( ... )

I'm running a Java application that registers a CSV table with HIVE and then checks the number of rows imported. Its done in several steps.:Statement stmt = con.createStatement();....stmt.execute( "CREATE TABLE ( <definition> < > );.....ResultSet rs...

Data Engineering
Reply
  • 1075 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

@Reto Matter​  Are you running a jar job or using dbconnect to run java code? Please provide how are you trying to make a connection and full exception stack trace.

  • 0 kudos
624398
by • New Contributor III
  • 2196 Views
  • 3 replies
  • 2 kudos

Resolved! Making py connector to raise an error for wrong SQL when asking to plan a query

Hey all,My aim is to validate a given SQL string without actually running it.I thought I could use the `EXPLAIN` statement to do so.So I tried using the `databricks-sql-connector` for python to explain a query, and so determine whether it's valid or ...

Data Engineering
Reply
  • 2196 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Please try in pyspark:try: spark.sql("SELECT BAD-QUERY AS FOO")._jdf.queryExecution().toString() except: print("incorrect query")or just:try: spark.sql("SELECT BAD-QUERY AS FOO").explain() except: print("incorrect query")

  • 2 kudos
2 More Replies
jayallenmn
by • New Contributor III
  • 2198 Views
  • 4 replies
  • 3 kudos

Resolved! Couple of Delta Lake questions

Hey guys,We're considering Delta Lake as the storage for our project and have a couple questions. The first one is what's the pricing for Delta Lake - can't seem to find a page that says x amount costs y.The second question is more technical - if we...

Data Engineering
Reply
  • 2198 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

delta lake itself is free. It is a file format. But you will have to pay for storage and compute of course.If you want to use Databricks with delta lake, it will not be free unless you use the community edition.Depending on what you are planning to...

  • 3 kudos
3 More Replies
karthikM
by • New Contributor
  • 1670 Views
  • 2 replies
  • 0 kudos

Delta Live Tables

is DLT supported for Scala? Any reference implementations or wikis to get started?

Data Engineering
Reply
  • 1670 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
is DLT supported for Scala? Any reference implementations or wikis to get started?

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
Daps022
by • New Contributor
  • 2084 Views
  • 3 replies
  • 1 kudos

How Streaming pieline handle by delta lake house

Data Engineering
Reply
  • 2084 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Rheiman
Contributor II
  • 1 kudos

Try looking into the structured streaming API. There you will learn about how to join streams and static data, how to set triggers for the streams, minibatching and other things that are important to the reliability of your application.Structured Str...

  • 1 kudos
2 More Replies
Yagao
by • New Contributor
  • 3376 Views
  • 5 replies
  • 2 kudos

How to do python within sql query in Databricks ?

Can anyone show me one use case how to do python within sql query ?

Data Engineering
Reply
  • 3376 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
tomasz
Databricks Employee
  • 2 kudos

To run Python within a SQL query you have to first define a Python function and then register it as a UDF. Once that is done you are able to call that UDF within a SQL query. Please take a look at this documentation here:https://docs.databricks.com/s...

  • 2 kudos
4 More Replies
Komal7
by • New Contributor
  • 975 Views
  • 2 replies
  • 0 kudos

When is AQE becoming GA? Is it coming in older spark versions?

Data Engineering
Reply
  • 975 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Komal Gyanani​,AQE was a major improvement added to Spark 3.0. It was added since Databricks runtime 7.3 LT (Spark 3.0) https://docs.databricks.com/release-notes/runtime/releases.html and here is docs on AQE https://docs.databricks.com/spark/late...

  • 0 kudos
1 More Replies
Marcosan
by • New Contributor II
  • 1339 Views
  • 2 replies
  • 3 kudos

What’s the best way to pass dependency versions dynamically to a cluster

I am using init scripts and would like to be able to control the version of a component that we release internally and frequently. We are now manually updating a dbfs requirement.txt file but I think that this problem may have been encountered befor...

Data Engineering
Reply
  • 1339 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

You can programmatically create cluster templates in JSON files and include config JSON files with libraries needed. Cluster deployment in that scenario needs to be controlled via API https://docs.databricks.com/dev-tools/api/latest/clusters.html

  • 3 kudos
1 More Replies
Edel
by • New Contributor II
  • 1151 Views
  • 1 replies
  • 2 kudos

Have you compared the performance between ADWC and Delta Lake for data warehousing?

Just want to know if you have a benchmark or some tests comparing Oracle ADWC vs Delta lake for data warehousing

Data Engineering
Reply
  • 1151 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 2 kudos

This widget could not be displayed.
Just want to know if you have a benchmark or some tests comparing Oracle ADWC vs Delta lake for data warehousing

This widget could not be displayed.
  • 2 kudos
This widget could not be displayed.
HowardZ
by • New Contributor
  • 1961 Views
  • 1 replies
  • 0 kudos

How do I create an athena table (instead of hive table) in databricks?

My dashboard uses Athena as data source for its availability (I don't need to fire up the cluster and manually refresh the data), but it requires me to create the tables manually. Wondering if there is a similar method like the .saveAsTable() to crea...

Data Engineering
Reply
  • 1961 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
My dashboard uses Athena as data source for its availability (I don't need to fire up the cluster and manually refresh the data), but it requires me to create the tables manually. Wondering if there is a similar method like the .saveAsTable() to crea...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
xsn
by • New Contributor II
  • 1168 Views
  • 1 replies
  • 1 kudos

How unity catalog run agnostic to the underlined system

Data Engineering
Reply
  • 1168 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
avinash_goje
by • New Contributor II
  • 2549 Views
  • 2 replies
  • 2 kudos

How to send metrics from GCP Databricks to Grafana Cloud through Prometheus?

While connecting the Databricks and Grafana, I have gone through the following approach.Install Grafna Agent in Databrics Clusters from Databricks console --> Not working since the system is not booted with systemd as init systemSince Spark 3 has Pro...

Data Engineering
Reply
  • 2549 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

There is a repo with Prometheus gateway https://gist.github.com/Lowess/3a71792d2d09e38bf8f524644bbf8349. In the community, we usually use DataDog as both plays nicely https://docs.datadoghq.com/integrations/databricks/?tabs=driveronly

  • 2 kudos
1 More Replies
MMMM
by • New Contributor III
  • 1076 Views
  • 1 replies
  • 0 kudos

missing notebook from workshop

Hi,I was going through this sessionhttps://tinyurl.com/databrickshcarebut on slides there is link to notebook which is broken. can you guys fix and share the link so I could try these notebooks ?this is mentioned in the slides for notebook linkhttps:...

Data Engineering
Reply
  • 1076 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi,I was going through this sessionhttps://tinyurl.com/databrickshcarebut on slides there is link to notebook which is broken. can you guys fix and share the link so I could try these notebooks ?this is mentioned in the slides for notebook linkhttps:...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels