cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Bagger
by New Contributor II
  • 1634 Views
  • 1 replies
  • 0 kudos

Databricks Connect - driver error

Hi We are experiencing instability when executing queries using Databricks Connect.Sometimes we are unable to receive the full result set without encountering an error with the message "Driver is up but is not responsive..."When we run the same query...

Data Engineering
databricks connect
driver
  • 1634 Views
  • 1 replies
  • 0 kudos
vgupta
by New Contributor II
  • 6578 Views
  • 5 replies
  • 4 kudos

DLT | Cluster terminated by System-User | INTERNAL_ERROR: Communication lost with driver. Cluster 0312-140502-k9monrjc was not reachable for 120 seconds

Dear Community, Hope you are doing well.For the last couple of days I am seeing very strange issues with my DLT pipeline, So every 60-70 mins it is getting failed in continuous mode, with the ERROR; INTERNAL_ERROR: Communication lost with driver. Clu...

DLT_ERROR DLT_Cluster_events
  • 6578 Views
  • 5 replies
  • 4 kudos
Latest Reply
Reddy-24
New Contributor II
  • 4 kudos

Hello @Debayan , I am facing same issue, while running Delta live table, This job is running in produtcuion, but it's not working in dev, i have tried to increae the worker nodes but no use. Can you please help on this. 

  • 4 kudos
4 More Replies
RohitKulkarni
by Contributor II
  • 1886 Views
  • 2 replies
  • 3 kudos

Databricks Libaries

Hello Team,I am trying to use the below libraries in data bricks .But they are not supporting.import com.microsoft.spark.sqlanalyticsfrom com.microsoft.spark.sqlanalytics.Constants import ConstantsPlease advise the correct Libraries nameRegardsRohit

  • 1886 Views
  • 2 replies
  • 3 kudos
Latest Reply
User16752242622
Valued Contributor
  • 3 kudos

Hi @Rohit Kulkarni​ Yes, This module is not supported in databricks. May I know the use case behind using this library in databricks?FYI: To access Azure Synapse from Databricks using the Azure Synapse connector.Check the below doc: https://docs.data...

  • 3 kudos
1 More Replies
Soma
by Valued Contributor
  • 5979 Views
  • 11 replies
  • 2 kudos

spark streaming listener is lagging

We use pyspark streaming listener and it is lagging for 10 hrsThe data streamed in 10 am IST is logged at 10 PM IstCan someone explain how logging listener interface work

  • 5979 Views
  • 11 replies
  • 2 kudos
Latest Reply
michalforgusion
New Contributor II
  • 2 kudos

If you're experiencing lag in a Spark Streaming application, there are several potential reasons and corresponding solutions you can try:1. **Resource Allocation**:- **Insufficient Resources**: Make sure that you have allocated enough resources (CPU,...

  • 2 kudos
10 More Replies
Gilg
by Contributor II
  • 2675 Views
  • 2 replies
  • 0 kudos

APPLY_CHANGES with json data

Hi Team,I am building a DLT pipeline and planning to use APPLY_CHANGES from Bronze to Silver. In the bronze table, a column has a json value. This value contains questions and answers as key, value pair and can change depending on list of questions h...

  • 2675 Views
  • 2 replies
  • 0 kudos
SaraCorralLou
by New Contributor III
  • 27144 Views
  • 5 replies
  • 2 kudos

Resolved! Error: The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

What is the problem?I am getting this error every time I run a python notebook on my Repo in Databricks.BackgroundThe notebook where I am getting the error is a notebook that creates a dataframe and the last step is to write the dataframe to a Delta ...

  • 27144 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Sara Corral​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 2 kudos
4 More Replies
Kratik
by New Contributor III
  • 1988 Views
  • 0 replies
  • 0 kudos

Spark submit job running python file

I have a spark submit job which is running one python file called main.py.The other file is alert.py which is being imported in main.py.Also main.py is using multiple config files.Alert.py is passed in --py-files and other config files are passed as ...

Data Engineering
pyfiles
spark
submit
  • 1988 Views
  • 0 replies
  • 0 kudos
Anonymous
by Not applicable
  • 4091 Views
  • 3 replies
  • 2 kudos
  • 4091 Views
  • 3 replies
  • 2 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 2 kudos

User sessions automatically timeout after six hours of idle time. This is not configurable like @Kunal Gaurav​  mentioned. Please raise a feature request if you have a requirement to configure this.Now, in Azure you could configure AAD refresh token ...

  • 2 kudos
2 More Replies
alexkit
by New Contributor II
  • 2485 Views
  • 4 replies
  • 3 kudos

ASP1.2 Error create database in Spark Programming with Databricks training

I'm on Demo and Lab in Dataframes section. I've imported the dbc into my company cluster and has run "%run ./Includes/Classroom-Setup" successfully. When i run the 1st sql command %sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/m...

  • 2485 Views
  • 4 replies
  • 3 kudos
Latest Reply
KDOCKX
New Contributor II
  • 3 kudos

I had the same issue and solved it like this:In the includes folder, there is a reset notebook, run the first command, this unmounts all mounted databases.Go back to the ASP 1.2 notebook and run the %run ./Includes/Classroom-Setup codeblock.Then run ...

  • 3 kudos
3 More Replies
MonishKumar
by New Contributor
  • 2520 Views
  • 1 replies
  • 0 kudos

SFTP - JSchException: Algorithm negotiation fail

When I tried to read the SFTP (CSV file) in Databricks I'm getting the below error"JSchException: Algorithm negotiation fail"Code:var df = spark.read.options(Map("header"->"true","host"->"20.118.190.30","username"->"user","password"->"pass","fileForm...

Data Engineering
SFTP Spark SCALA Databricks CSV JSCH
  • 2520 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16752239289
Databricks Employee
  • 0 kudos

@MonishKumar Could you provide the entire exception ?From the one line error message, I suspect this is due to the SSL cipher suites required by the SFTP server is not available on cluster. You can run below to get the cipher suites that sftp require...

  • 0 kudos
Manjula_Ganesap
by Contributor
  • 1710 Views
  • 1 replies
  • 0 kudos

Delta Live Table Graph different with no change in Notebook code

I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not under...

Manjula_Ganesap_0-1694001824330.png Manjula_Ganesap_1-1694001865082.png
  • 1710 Views
  • 1 replies
  • 0 kudos
Latest Reply
Manjula_Ganesap
Contributor
  • 0 kudos

@Retired_mod  - Thank you for your response. There is no change in the table dependencies. The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I c...

  • 0 kudos
FabriceDeseyn
by Contributor
  • 1313 Views
  • 0 replies
  • 0 kudos

merge breaking persistance of dataframe

Hi allIn the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen?   from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initializ...

FabriceDeseyn_1-1694011507567.png
  • 1313 Views
  • 0 replies
  • 0 kudos
RP2007
by New Contributor
  • 2148 Views
  • 2 replies
  • 1 kudos

I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamen

 I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamentals. I can't access the quiz page. Can you please help on this?Getting below error:-403FORBIDDENYou don't have permission to access this page2023-08-...

  • 2148 Views
  • 2 replies
  • 1 kudos
Latest Reply
APadmanabhan
Databricks Employee
  • 1 kudos

Hello Both, This link would be of help.

  • 1 kudos
1 More Replies
guostong
by New Contributor III
  • 6847 Views
  • 2 replies
  • 0 kudos

Resolved! how to set jobs permission with rest api

create job with cli, but can not set the permission with cli,have to use rest api to set permission:https://docs.databricks.com/api/workspace/permissions/setbelow is my command in windows to set permission:curl -X PUT https://my-workspace-url.azureda...

  • 6847 Views
  • 2 replies
  • 0 kudos
Latest Reply
guostong
New Contributor III
  • 0 kudos

thank you, the new permission list should be a whole list, not the new permission

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels