Hi,I am facing the same issue and I choose All Purpose cluster.In my case, I need some jobs to run every 10 minutes.- With All Purpose cluster, it take 1 minute- With Job cluster, it takes 5 minutes (4 minutes for setting up + 1 minutes to perform t...
Hi,On the top right corner, there is a circle icon which is next to the bell.Click on it, then choose 'Profile'.When you go to Profile page, choose 'Edit' on the top right corner.
Hi,I think you can follow these steps:1. Use window function to create a new column by shifting, then your df will look like thisid value lag1 A-B-C-D-E-F null2 A-B-G-C-D-E-F A-B-C-D-E-F3 A-B-G-D-E-F ...
Hi,I agree with Werners, try to avoid loop with Pyspark Dataframe.If your dataframe is small, as you said, only about 1000 rows, you may consider to use Pandas.Thanks.
