Databricks

maartenvr · 09-05-2023

Hi all,We recently upgraded our databricks compute cluster from runtime version 10.4 LST, to 12.2 LST.After the upgrade one of our python scripts suddenly fails with a module not found error; indicating that our customly created module "xml_parser" i...

maartenvr · 02-28-2023

Hi all,I am using a persist call on a spark dataframe inside an application to speed-up computations. The dataframe is used throughout my application and at the end of the application I am trying to clear the cache of the whole spark session by calli...

maartenvr · 02-10-2023

Hi,I am using an (Azure) Databricks Compute cluster in a Jupyter notebook using the Databricks connect Python package. Our cluster is on Databrick runtime 10.4 LST and coherently I am using the databricks-connect==10.4.18.In my notebook I am able to ...

maartenvr · 09-28-2023

FYI: For now we have found a workaround.We are adding the package as ZIP file to the current spark session with .addyFiles.So after creating a spark session using Databricks-connect we run the following:spark.sparkContext.addPyFile("C:/path/to/custom...

maartenvr · 09-14-2023

Going to 13.3 (LTS) unfortunately requires quite some extra work for our team.We would need to start using / configure Unity Catalog.For now I have opened a ticket with the Databricks support team.If I find any solution I will post it here.

maartenvr · 09-12-2023

Hi @Debayan ,We were already on the latest databricks-connect version (12.2.12) compatibly with LTS 12.2I tried to run the code with a downgraded version as well after your comment (12.2.10) but that didn't do the trick.@Kaniz, thanks for the suggest...

maartenvr · 03-13-2023

That might clarify, since I do use the MEMORY_AND_DISK option.The .getRDDStorageInfo() method is actually also not supperted for me, but I have enough info to continue now. Thanks for the help!

maartenvr · 03-13-2023

Hi Suteja, Thanks for the quick reply.I have already tried the ```spark.catalog.clearCache()``` method but it doesn't work and was actually the reason for me posting the question.In your code example, you are calling unperist on the dataframe after w...

Databricks

User Stats

User Activity

Installed Library / Module not found through Databricks connect LST 12.2

Unable to clear cache using a pyspark session

Errors on Python API for DeltaTables on Databricks Spark 10.4 LST

Re: Installed Library / Module not found through Databricks connect LST 12.2

Re: Installed Library / Module not found through Databricks connect LST 12.2

Re: Installed Library / Module not found through Databricks connect LST 12.2

Re: Unable to clear cache using a pyspark session

Re: Unable to clear cache using a pyspark session