- 6718 Views
- 7 replies
- 4 kudos
Resolved! Retrieve job id and run id from scala
I need to retrieve job id and run id of the job from a jar file in Scala.When I try to compile below code in IntelliJ, below error is shown.import com.databricks.dbutils_v1.DBUtilsHolder.dbutils object MainSNL { @throws(classOf[Exception]) de...
- 6718 Views
- 7 replies
- 4 kudos
- 4 kudos
Maybe its worth going through the Task Parameter variables section of the below dochttps://docs.databricks.com/data-engineering/jobs/jobs.html#task-parameter-variables
- 4 kudos
- 13865 Views
- 12 replies
- 11 kudos
Resolved! Is there any difference between performance of Python and SQL ?
I read somewhere that Python code is converted to SQL at the end. So is it true or there is any difference in performance while working with Scala, Python or SQL ?
- 13865 Views
- 12 replies
- 11 kudos
- 11 kudos
To add on the consideration of UDFs, try to consider using HOFs (Higher Order Functions) whenever possible first as there is a signifcant performance benefit as seen here.
- 11 kudos
- 5397 Views
- 5 replies
- 2 kudos
Get current date as string in databricks using scala
I want to get current date in scala as a string for example today current date is 3rd jan, want to store it as a new variable dynamically as below, how to get it.val currdate : String = “20220103”when I am using val currdate = Calendar.getInstance.ge...
- 5397 Views
- 5 replies
- 2 kudos
- 2 kudos
Hey @Vibhor Sethi Hope you are well!Thank you for posting your question and letting us know that you were able to resolve the issue. Would you be happy to mark it as the best solution? It would be really helpful for the other members too.Cheers!
- 2 kudos
- 1516 Views
- 2 replies
- 1 kudos
Table access control cluster with R language
Hello,I want to have a high concurrency cluster with table access control and I want to use R language on it.I know that the documentation says that R and Scala is not available with table access control.But maybe you have some tricks or best practic...
- 1516 Views
- 2 replies
- 1 kudos
- 4487 Views
- 4 replies
- 1 kudos
Resolved! rdd.foreachPartition() does nothing?
I expected the code below to print "hello" for each partition, and "world" for each record. But when I ran it the code ran but had no print outs of any kind. No errors either. What is happening here?%scala val rdd = spark.sparkContext.parallelize(S...
- 4487 Views
- 4 replies
- 1 kudos
- 3350 Views
- 3 replies
- 3 kudos
Resolved! Turn spark.sql query into scala function
Hello,I'm learning Scala / Spark and try to understand what's wrong with my function:I have a spark.sql query, stored in a variable:val uViewName = spark.sql(""" SELECT v.Data_View_Name FROM apoHierarchy AS h INNER JOIN apoView AS v ON h.View_N...
- 3350 Views
- 3 replies
- 3 kudos
- 3 kudos
try add .first()(0) it will return only value from first row/column as currently you are returning Dataset: var uViewName = spark.sql(s""" SELECT v.Data_View_Name FROM apoHierarchy AS h INNER JOIN apoView AS v ON h.View_Name = v.Context_View_N...
- 3 kudos
- 3497 Views
- 0 replies
- 0 kudos
How to resolve "java.lang.ClassNotFoundException: com.databricks.spark.util.RegexBasedAWSSecretKeyRedactor" when running Scala Spark project using databricks-connect ?
Currently I am learning how to use databricks-connect to develop Scala code using IDE (VS Code) locally. The set-up of the databricks-connect as described here https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect was succues...
- 3497 Views
- 0 replies
- 0 kudos
- 2960 Views
- 5 replies
- 8 kudos
Resolved! Trigger.AvailableNow on scala - compile issue
Hi everybody,Trigger.AvailableNow is released within the databricks 10.1 runtime and we would like to use this new feature with autoloader.We write all our data pipeline in scala and our projects import spark as a provided dependency. If we try to sw...
- 2960 Views
- 5 replies
- 8 kudos
- 8 kudos
You can switch to python. Depending on what you're doing and if you're using UDFs, there shouldn't be any difference at all in terms of performance.
- 8 kudos
- 4416 Views
- 5 replies
- 1 kudos
Resolved! Problems with HiveMetastoreClient and internal Databricks Metastore.
I've been trying to use the HiveMetastoreClient class in Scala to extract some metadata from Databricks internal Metastore, without success. I'm currently using the 7.3 LTS runtime.The error seems to be related to some kind of inconsistency between...
- 4416 Views
- 5 replies
- 1 kudos
- 1 kudos
Thanks for the reference, @Atanu Sarkar .Seems a little odd to me that I'd need to change the internal Databricks Metastore table to add a column expected by the client default Scala client. I'm afraid this could cause issues with other users/jobs ...
- 1 kudos
- 3819 Views
- 4 replies
- 0 kudos
Resolved! SQLServerException: deadlock
I'm using databricks to connect to a SQL managed instance via JDBC. SQL operations I need to perform include DELETE, UPDATE, and simple read and write. Since spark syntax only handles simple read and write, I had to open SQL connection using Scala an...
- 3819 Views
- 4 replies
- 0 kudos
- 0 kudos
this is not a spark error but purely the database.There are tons of articles online on how to prevent deadlocks, but there is no single solution for this.
- 0 kudos
- 2668 Views
- 0 replies
- 0 kudos
Multithreading in SCALA DATABRICKS
Hi Team, I was trying to call/run multiple notebooks in one notebook concurrent. But the caller notebooks are getting executing one by one whereas I need to run all the caller notebooks concurrently. I have also tried using Threading in Scala Databri...
- 2668 Views
- 0 replies
- 0 kudos
- 715 Views
- 0 replies
- 1 kudos
vimeo.com
Auto Loader provides Python and Scala methods to ingest new data from a folder location into a Delta Lake table by using directory listing or file notifications. Here's a quick video (7:00) on how to use Auto Loader for Databricks on AWS with Databri...
- 715 Views
- 0 replies
- 1 kudos
- 1402 Views
- 3 replies
- 1 kudos
python dataframe or hiveSql update based on predecessor value?
I have a million in rows that I need to update which looks for the highest count of the predecessor from the same source data and replaces the same value on a different row. For example.Original DF.sno Object Name shape rating1 Fruit apple round ...
- 1402 Views
- 3 replies
- 1 kudos
- 1 kudos
basically you have to create a dataframe (or use a window function, that will also work) which gives you the group combination with the most occurances. So a window/groupby on object, name, shape with a count().Then you have to determine which shape...
- 1 kudos
- 5689 Views
- 9 replies
- 8 kudos
Resolved! Getting Null values at the place of data which was removed manually from excel file( solved )
I was reading an excel file with one column,country india India india India indiadataframe i got from this data : df.show()+-------+ |country| +-------+ | india | | India | | india | | India | | india | +-------+In the next step i removed last value ...
- 5689 Views
- 9 replies
- 8 kudos
- 8 kudos
@sarvesh singh - Thank you for letting us know. Would you be happy to mark the best answer so others can find the solution easily?
- 8 kudos
- 3352 Views
- 1 replies
- 1 kudos
How to provide custom class extending SparkPlugin/ExecutorPlugin in Databricks 7.3?
How to properly configure the jar containing the class and spark plugin in Databricks?During DBR 7.3 cluster creation, I tried by setting the spark.plugins, spark.driver.extraClassPath and spark.executor.extraClassPath Spark configs by copying the ja...
- 3352 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Krishna Kashiv - I don't know if we've met yet. My name is Piper and I'm a community moderator here. Thank you for your new question. It looks thorough! Let's give it a while to see what our members have to say. Otherwise, we will circle back...
- 1 kudos