Databricks Community

@ashutosh0710 you can inspect the libraries available in a cluster at /databricks/jars to understand which jars and versions are available, or similarly simply inspect the Spark UI Environment Classpath list.

Hello @DBUser2, The COPY INTO command does not have a specific documented limit on the size of the data or the number of files that can be ingested at a time. Timeout errors can occur due to network issues, resource limitations, or long-running opera...

Actually this would if I am using it in a native notebook environment, however I am trying to create a UDF because I want these queries to be executed from an external, JDBC connection, and I dont wish to wait for the cluster to spin up for a noteboo...

@skumarrm Please try the below: Set Up Task Parameters: In the job configuration, you can set up task parameters to pass values from one task to another.For TASK1 (DLT), ensure it outputs the PipelineID or PipelineName. Use Task Parameters in TASK2:...

@PKD28  The error indicates that the driver memory is not enough to handle the load.Please refer to this doc for more info and on how to fix thishttps://kb.databricks.com/en_US/jobs/driver-unavailable

Hi @bhanuteja_1  The scala.Product is a core class in the Scala std library used for tuples, case classes, etc. There seem to be a classloading problem or more likely a jar conflict. Are you deploying a job using custom jars, uber jars, and having de...

Hi @bhanuteja_1 , Where are you running this from? Based on the short output, it looks like from a Databricks Notebook, but it would be a weird error unless you're having some classpath overrides or jar conflicts, leading to this error; it is simply ...

Hi @Nathant93 , Does it come with a "Caused by" in the error stacktrace? If there isn't any in the Spark logs, perhaps you can provide a reproducer code leading to this exception. Stacktrace, DBR version and repro code would help. The (java.util.conc...

any help here , thanks