Databricks Community

Dicer · 05-24-2024

I am using the distributed Pandas on Spark, not the single node Pandas.But when I try to run the following code to transform a data frame with 652 x 729803 data points df_ps_pct = df.pandas_api().pct_change().to_spark() , it returns me this error: ...

Dicer · 10-31-2023

I want to import the ibapi python module in Azure Databricks Notebook.Before this, I downloaded the the TWS API folder from https://interactivebrokers.github.io/# I need to go through the following steps to install the API:Download and install TWS Ga...

Dicer · 11-02-2022

I tried to VACUUM a delta table, but there is a Syntax error.Here is the code:%sql set spark.databricks.delta.retentionDurationCheck.enabled = False VACUUM test_deltatable

Dicer · 10-24-2022

I only have 1000 columns. Each column has 252 rows, so there are only 252000 data points.How come it can route tasks for the best-cached locality for 7 hours?

Dicer · 08-12-2022

Data type:AAPL_Time: timestampAAPL_Close: floatRaw Data:AAPL_Time AAPL_Close 2015-05-11T08:00:00.000+0000 29.0344 2015-05-11T08:30:00.000+0000 29.0187 2015-05-11T09:00:00.000+0000 29.0346 2015-05-11T09:3...

Dicer · 05-27-2024

Hi @anardinelli Thank your for your reply. This is indeed one of the solutions.However, my driver used memory size is always not always constant. When it is doing specific code operation, it triggers the above error: Driver is up but is not responsiv...

Dicer · 11-14-2022

@orian hindi Not only using pandas_api(), using spark also causes high used memory.

Dicer · 11-13-2022

I think deploying a larger VM with more memory may work. In the ML case, deploying one big VM is better. But Spark workers will also spill the data on disk if the dataset is larger than the memory size. So why is there a memory problem? Is there any ...

Dicer · 11-12-2022

@lizou I also think so. Every time, when the memory use% reaches 50%+, Databricks will autoscale more workers. 3 days ago, I used pandas to transpose the data, but it seems super slow and failed due to GC. Yesterday, I tried to transpose the data on...

Dicer · 11-11-2022

@lizou Today, I have the same problem when I spark transpose 1000 columns x 4284 rows structured data matrix. The data size is about 2GB.Here is the code:https://github.com/NikhilSuthar/TransposeDataFramefrom pyspark.sql.functions import * from pys...

Databricks Community

User Stats

User Activity

Why Pandas on Spark can trigger `Driver is up but is not responsive, likely due to GC` ?

Import folder (no .whl or .jar files) and run the `python3 setup.py bdist_wheel` for lib install

PARSE_SYNTAX_ERROR: Syntax error at or near 'VACUUM'

Is it reasonable for the process "Determining the location of DBIO file fragments." to take me 7 hours?

Azure Databricks: Failed to extract data which is between two timestamps within those same dates using Pyspark

Re: Why Pandas on Spark can trigger `Driver is up but is not responsive, likely due to GC` ?

Re: Fatal error: Python kernel is unresponsive

Re: Fatal error: Python kernel is unresponsive

Re: Fatal error: Python kernel is unresponsive

Re: Fatal error: Python kernel is unresponsive