Databricks

User16752246494 · 05-28-2021

Does anyone know how to debug notebook code using IntelliJ or is there any other tool for it?like debugging in Spark cluster using export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005are there any similar sett...

User16752246494 · 11-26-2021

Hi ,You can also use the following.import org.apache.commons.io.IOUtils // jar will be already there in spark cluster no need to worry import java.net.URL val urlfile=new URL("https://people.sc.fsu.edu/~jburkardt/data/csv/airtravel.csv") val tes...

User16752246494 · 11-26-2021

Hi ,We can also read CSV directly without writing it to DBFS.Scala spark Approachimport org.apache.commons.io.IOUtils // jar will be already there in spark cluster no need to worry import java.net.URL val urlfile=new URL("https://people.sc.fsu.edu...

User16752246494 · 10-11-2021

Hi,We tried a simulate the question on our end and what we did was packaged a module inside a whl file.Now to access the wheel file we created another python file test_whl_locally.py. Inside test_whl_locally.py to access the content of the wheel file...

User16752246494 · 10-11-2021

Hi @D M ,Good day.Can you please try with --py-files /dbfs/path/to/.egg job_main.pyThe above will invoke the fuse driver.If the spark-submit still fails, Can you please provide the full stack trace?

User16752246494 · 09-27-2021

Hi @Gerhard Brueckl ,VACUUM bigtable DRY RUNwill just list the files for deletion but not actually deleting the physical file. As we see there are around 2M files listed. Once the files are actually vacuumed then you will not see this much time take...

Databricks

User Stats

User Activity

How do you connect databricks notebook with IntelliJ remote debug mode?

Re: read csv directly from url with pyspark

Re: reading data from url using spark

Re: Run Pyspark job of Python egg package using spark submit on databricks

Re: Run Pyspark job of Python egg package using spark submit on databricks

Re: Slow performance of VACUUM on Azure Data Lake Store Gen2