cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Vik1
New Contributor II
since ‎01-21-2022
‎06-26-2023

User Stats

  • 3 Posts
  • 0 Solutions
  • 0 Kudos given
  • 3 Kudos received

User Activity

Some very simple functions in Pandas on Spark are very slow

06-22-2022
I have a pandas on spark dataframe with 8 million rows and 20 columns. It took 3.48 minutes to run df.shape and it takes. It also takes a long time to run df.head took 4.55 minutes . By contrast df.var1.value_counts().reset_index() took only 0.18 sec...

Data persistence, Dataframe, and Delta

01-21-2022
I am new to databricks platform. what is the best way to keep data persistent so that once I restart the cluster I don't need to run all the codes again?So that I can simply continue developing my notebook with the cached data.I have created many dat...

Cluster setup for ML work for Pandas in Spark, and vanilla Python.

01-21-2022
My setup:Worker type: Standard_D32d_v4, 128 GB Memory, 32 Cores, Min Workers: 2, Max Workers: 8Driver type: Standard_D32ds_v4, 128 GB Memory, 32 CoresDatabricks Runtime Version: 10.2 ML (includes Apache Spark 3.2.0, Scala 2.12)I ran a snowflake quer...