cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
KevinXu
by • New Contributor III
  • 5825 Views
  • 1 replies
  • 3 kudos

pyspark SQL cannot resolve 'explode()' due to data type mismatch

Running Pyspark script getting the following error depending on which xml I query:cannot resolve 'explode(...)' due to data type mismatchThe pyspark code:from pyspark.sql import SparkSession   JOB_NAME = "Complex file to delimeted files transformer" ...

Data Engineering
Reply
  • 5825 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 3 kudos

This widget could not be displayed.
Running Pyspark script getting the following error depending on which xml I query:cannot resolve 'explode(...)' due to data type mismatchThe pyspark code:from pyspark.sql import SparkSession   JOB_NAME = "Complex file to delimeted files transformer" ...

This widget could not be displayed.
  • 3 kudos
This widget could not be displayed.
MariusC
by • New Contributor III
  • 9000 Views
  • 5 replies
  • 5 kudos

Resolved! Power BI with Databricks SQL Endpoint

Hello,We are trying to load a Delta table from an Azure Data Lake Storage container into Power BI using the Databricks SQL Endpoint.We configured the SQL Workspace data to have access to the ADLS Delta table and created a view; we are able to query t...

error screenshot sample_error sample_query ADLS_delta_query
Data Engineering
Reply
  • 9000 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 5 kudos

@Marius Condescu​ Could you please include below spark config and try-spark.hadoop.fs.azure.account.oauth.provider.type.ariaprime.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProviderspark.hadoop.fs.azure.account.auth.typ...

  • 5 kudos
4 More Replies
Anonymous
by • Not applicable
  • 1301 Views
  • 0 replies
  • 0 kudos

Data Visualized | Lego style

Something fun for your Friday! If you are a visual person like me, you may like this image that was recently shared in our internal Databricks slack instance. Who else 李s Legos? If you have seen data all 6 ways with Databricks, give this a 李 !!!

Image
Data Engineering
Reply
  • 1301 Views
  • 0 replies
  • 0 kudos
Raymond_Garcia
by • Contributor II
  • 1243 Views
  • 0 replies
  • 1 kudos

EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[elasticsearch_server:80]]

Hi, I wondered if some of you have had this issue before and how it can be solved. In a Databricks Job, we have a UBQ with a Painless script for ES. these are the options. Staging and prod are the same configurations, but Staging is failing with the ...

Data Engineering
Reply
  • 1243 Views
  • 0 replies
  • 1 kudos
alexa
by • New Contributor III
  • 6448 Views
  • 2 replies
  • 0 kudos

Resolved! How to use dateadd in databricks sql ?

I am trying to something like this but getting error like :Error in SQL statement: AnalysisException: Undefined function: 'DATEADD'. This function is neither a registered temporary function nor a permanent function registered in the database 'default...

Data Engineering
Reply
  • 6448 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
SergeRielau
Databricks Employee
  • 0 kudos

Dateadd was added in DBR 10.4 and is in DBSQL current.SELECT DATEADD(HOUR,IFNULL(100, 0),current_date) AS Date_Created_Local=> 2022-05-31T04:00:00.000+0000.You can also use one of these casts to turn any wellformed string into an interval:SELECT curr...

  • 0 kudos
1 More Replies
thushar
by • Contributor
  • 6089 Views
  • 0 replies
  • 0 kudos

How to install wheel package from git repo

Using VS code for development and a wheel package is created for shipment.We put this wheel package in Azure data lake storage and ADB notebook accessed this wheel package and installed it in the cluster. It is working fine. But instead of keeping th...

image.png
Data Engineering
Reply
  • 6089 Views
  • 0 replies
  • 0 kudos
junaid
by • New Contributor II
  • 1120 Views
  • 0 replies
  • 0 kudos

Issue in creating workspace - Custom AWS Configuration

Hello,We have tried to created new Databricks Workspace using "Custom AWS Configuration" and we have given our own VPC (Customer managed VPC) and tried but workspace failed to launch. There are Documents Available for resolve this error , According t...

Data Engineering
Reply
  • 1120 Views
  • 0 replies
  • 0 kudos
alejandrofm
by • Valued Contributor
  • 994 Views
  • 1 replies
  • 0 kudos

How to run sparkStream for earlier (not future messages)

Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:-Save money by don't have a cluster up 24/7-Have bigger files saved for each readThe stream is constant so...

Data Engineering
Reply
  • 994 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi, I'm listening to a stream for kinesis, don't need the data in real-time, so I could run it on an hourly basis looking to achieve two things:-Save money by don't have a cluster up 24/7-Have bigger files saved for each readThe stream is constant so...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
sriramkumar
by • New Contributor II
  • 3083 Views
  • 3 replies
  • 0 kudos

New Databricks Driver gives SQLNonTransientConnectionException when trying to connect to Databricks Instance

import com.databricks.client.jdbc.DataSource;   import java.sql.*;   public class testDatabricks { public static void main(String[] args) throws SQLException { String dbUrl = "jdbc:databricks://<hostname>:443;HttpPath=<HttpPath>;"; // Cop...

Data Engineering
Reply
  • 3083 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

This looks like due to maintenance on US . Are you still facing the issue @Sriramkumar Thamizharasan​ Is your workspace on eastus and eastus2 ?

  • 0 kudos
2 More Replies
repcak
by • New Contributor III
  • 5251 Views
  • 4 replies
  • 3 kudos

Resolved! Delta Live Tables with EventHub

Hello,I would like to integrate Databricks Delta Live Tables with Eventhub, but i cannot install com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.17 on delta live cluster.I tried installed in using Init script (by adding it in Json cluster settings...

image
Data Engineering
Reply
  • 5251 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

I think this has some details https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-spark-tutorial @Kacper Mucha​ is the issue resolved ?

  • 3 kudos
3 More Replies
Soma
by • Valued Contributor
  • 3307 Views
  • 5 replies
  • 6 kudos

Resolved! Start Up Notebook in Databricks

Hi Team,Is it possible to have a startup notebook in databricks similar to init sql script ( This need to run on start of every cluster )

Data Engineering
Reply
  • 3307 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Soma
Valued Contributor
  • 6 kudos

@Kaniz Fatma​ sure will do

  • 6 kudos
4 More Replies
Soma
by • Valued Contributor
  • 3527 Views
  • 0 replies
  • 0 kudos

Spark Failure Error Unable to download spark docker Image

Cluster terminated. Reason: Spark Image Download Failure  "reason": { "code": "SPARK_IMAGE_DOWNLOAD_FAILURE", "type": "SERVICE_FAULT", "parameters": { "instance_id": "6565aa39b0ae4fe69c7fe6f313e3ca2a", "databricks_error_message": "Failed to set up th...

Data Engineering
Reply
  • 3527 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels