Databricks Community

source2sea · 05-12-2023

object OurMainObject extends LazyLogging with IOApp { def run(args: List[String]): IO[ExitCode] = { logger.info("Started the application") val conf = defaultOverrides.withFallback(defaultApplication).withFallback(defaultReference) val...

source2sea · 04-26-2023

ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)should one create it in the work space manually via UI? and how?would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.

source2sea · 04-25-2023

i copied my question from an very old question/post that i reponded. and decided to move it to here:context:I have jar (scala), using scala pureconfig (wrapper of typesafe config)uploaded an application.conf file to a path which is mounted to the wor...

source2sea · 04-25-2023

https://spark.apache.org/docs/latest/submitting-applications.htmlmainly want to know if extra class path could be used or not when i submit a job

source2sea

1. change to spark native catalog approach (not hive metadata store) works. Syntax is essentially: CREATE TABLE IF NOT EXISTS dbName.tableName (columns names and types ) USING parquet PARTITIONED BY ( runAt STRING ) LOCA...

source2sea

Changing it to a CLUSTER level for OAuth authentication helped me solve the problem.I wish the notebook AI bot could tell me the solution.before the changes, my configraiotn was at the notebook leve.and it has below errorsAnalysisException: org.apac...

source2sea · 02-10-2024

I see db runtime 14 is out, but still 2.12, when would databricks plan to support 2.13 or 3 thank you

source2sea · 02-02-2024

Hi, even though i set the conf to be true, on writing to disk it had exceptions complaining it has duplicate columns.below is the error message org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data to save: branchavailablity....

source2sea · 05-16-2023

my workaround now is to make the code like below, so the databricks jobs becomes failure. case Left(ex) => { IO(logger.error("Glue failure", ex)).map(_ => ExitCode.Error) IO.raiseError(ex) }

Databricks Community

User Stats

User Activity

how to make databricks job to fail when the application has already given "exit code 1"?

ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)

pass application.conf file into databricks jobs

what mode is the deploy-mode when calling spark in databricks/

Re: Hive Table Creation - Parquet does not support Timestamp Datatype?

Re: Database creation error

Re: When will databricks runtime be released for Scala 2.13?

Re: spark is case sensitive? Spark is not case sensitive by default. If you have same column name in

Re: how to make databricks job to fail when the application has already given "exit code 1"?