Databricks Community

I've created a UDF in Scala (using Databricks notebook) which checks if the value of a field in a table is greater than 0, then sum those values. UDF is working as expected. The problem is, that this UDF will be used by multiple SQLs across different...

We have a process that will write spark sql to a file, this process will generate thousands of spark sql files in the production environment.These files will be created in the ADLS Gen2 directory.sample spark file---val 2023_I = spark.sql("select rm....