- 18755 Views
- 3 replies
- 1 kudos
Troubleshooting Spill
I am trying to troubleshoot why spill occurred during DeltaOptimizeWrite. I am running a 64-core cluster with 256 GB RAM, which I expect to be handle this amount data (see attached DAG).
- 18755 Views
- 3 replies
- 1 kudos
- 1 kudos
You can resolver the Spill to memory by increasing the shuffle partitions, but 16 GB of spill memory should not create a major impact of your job execution. Could you share more details on the actual source code that you are running?
- 1 kudos
- 8138 Views
- 4 replies
- 1 kudos
Resolved! Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error
Got below failure on scheduled job on interactive cluster and the next scheduled run executed fine.I want to know why this error occurred and how can I prevent it to happen again.And how to debug these errors in future ? com.databricks.backend.commo...
- 8138 Views
- 4 replies
- 1 kudos
- 1 kudos
@JKR Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 1 kudos
- 3222 Views
- 1 replies
- 1 kudos
Resolved! Cloning content of Repos into shared Workspace
Hello, I have a git repository on Databricks with notebooks that are meant to be shared with other users. The reason these notebooks are in git as opposed to the "shared" workspace already is because they are to be continuously improved and need sepa...
- 3222 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello, Thanks for contacting Databricks Support. I presume you're looking to transfer files from external repositories to Databricks workspace. I'm afraid currently there is no direct support on it. You may consider to use REST API which allows for...
- 1 kudos
- 2881 Views
- 1 replies
- 0 kudos
Databricks Connect - driver error
Hi We are experiencing instability when executing queries using Databricks Connect.Sometimes we are unable to receive the full result set without encountering an error with the message "Driver is up but is not responsive..."When we run the same query...
- 2881 Views
- 1 replies
- 0 kudos
- 3482 Views
- 2 replies
- 3 kudos
Databricks Libaries
Hello Team,I am trying to use the below libraries in data bricks .But they are not supporting.import com.microsoft.spark.sqlanalyticsfrom com.microsoft.spark.sqlanalytics.Constants import ConstantsPlease advise the correct Libraries nameRegardsRohit
- 3482 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Rohit Kulkarni​ Yes, This module is not supported in databricks. May I know the use case behind using this library in databricks?FYI: To access Azure Synapse from Databricks using the Azure Synapse connector.Check the below doc: https://docs.data...
- 3 kudos
- 12196 Views
- 11 replies
- 2 kudos
spark streaming listener is lagging
We use pyspark streaming listener and it is lagging for 10 hrsThe data streamed in 10 am IST is logged at 10 PM IstCan someone explain how logging listener interface work
- 12196 Views
- 11 replies
- 2 kudos
- 2 kudos
If you're experiencing lag in a Spark Streaming application, there are several potential reasons and corresponding solutions you can try:1. **Resource Allocation**:- **Insufficient Resources**: Make sure that you have allocated enough resources (CPU,...
- 2 kudos
- 3875 Views
- 2 replies
- 0 kudos
APPLY_CHANGES with json data
Hi Team,I am building a DLT pipeline and planning to use APPLY_CHANGES from Bronze to Silver. In the bronze table, a column has a json value. This value contains questions and answers as key, value pair and can change depending on list of questions h...
- 3875 Views
- 2 replies
- 0 kudos
- 47117 Views
- 5 replies
- 2 kudos
Resolved! Error: The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.
What is the problem?I am getting this error every time I run a python notebook on my Repo in Databricks.BackgroundThe notebook where I am getting the error is a notebook that creates a dataframe and the last step is to write the dataframe to a Delta ...
- 47117 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Sara Corral​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 2 kudos
- 3051 Views
- 0 replies
- 0 kudos
Spark submit job running python file
I have a spark submit job which is running one python file called main.py.The other file is alert.py which is being imported in main.py.Also main.py is using multiple config files.Alert.py is passed in --py-files and other config files are passed as ...
- 3051 Views
- 0 replies
- 0 kudos
- 7735 Views
- 3 replies
- 2 kudos
- 7735 Views
- 3 replies
- 2 kudos
- 2 kudos
User sessions automatically timeout after six hours of idle time. This is not configurable like @Kunal Gaurav​ mentioned. Please raise a feature request if you have a requirement to configure this.Now, in Azure you could configure AAD refresh token ...
- 2 kudos
- 5204 Views
- 4 replies
- 3 kudos
ASP1.2 Error create database in Spark Programming with Databricks training
I'm on Demo and Lab in Dataframes section. I've imported the dbc into my company cluster and has run "%run ./Includes/Classroom-Setup" successfully. When i run the 1st sql command %sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/m...
- 5204 Views
- 4 replies
- 3 kudos
- 3 kudos
I had the same issue and solved it like this:In the includes folder, there is a reset notebook, run the first command, this unmounts all mounted databases.Go back to the ASP 1.2 notebook and run the %run ./Includes/Classroom-Setup codeblock.Then run ...
- 3 kudos
- 3810 Views
- 1 replies
- 0 kudos
SFTP - JSchException: Algorithm negotiation fail
When I tried to read the SFTP (CSV file) in Databricks I'm getting the below error"JSchException: Algorithm negotiation fail"Code:var df = spark.read.options(Map("header"->"true","host"->"20.118.190.30","username"->"user","password"->"pass","fileForm...
- 3810 Views
- 1 replies
- 0 kudos
- 0 kudos
@MonishKumar Could you provide the entire exception ?From the one line error message, I suspect this is due to the SSL cipher suites required by the SFTP server is not available on cluster. You can run below to get the cipher suites that sftp require...
- 0 kudos
- 2568 Views
- 1 replies
- 0 kudos
Delta Live Table Graph different with no change in Notebook code
I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not under...
- 2568 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod - Thank you for your response. There is no change in the table dependencies. The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I c...
- 0 kudos
- 1951 Views
- 0 replies
- 0 kudos
merge breaking persistance of dataframe
Hi allIn the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen? from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initializ...
- 1951 Views
- 0 replies
- 0 kudos
- 4208 Views
- 2 replies
- 1 kudos
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamen
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamentals. I can't access the quiz page. Can you please help on this?Getting below error:-403FORBIDDENYou don't have permission to access this page2023-08-...
- 4208 Views
- 2 replies
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
