- 11532 Views
- 3 replies
- 3 kudos
Azure Databricks User Alerts: query_result_table
Hi,How can I set up the notification email to show all of rows from query_result_table, not only first 10?
- 11532 Views
- 3 replies
- 3 kudos
- 3 kudos
I see that my previous message has been cut. So, I just wanted to check where can I change email setting using the code above, because when I open User > Account Settings > Notifications I can see only option like this: there is no place to configur...
- 3 kudos
- 5990 Views
- 1 replies
- 0 kudos
Unable to use job cluster for task in workflows
Hi,I have a workflow setup in Databricks using 12.2 LTS ML.I am trying to use a job cluster for the task but i am getting the following error: Spark Conf: ‘spark.databricks.acl.enabled’ is not allowed when choosing an access modeAs a result I have to...
- 5990 Views
- 1 replies
- 0 kudos
- 19398 Views
- 1 replies
- 1 kudos
Resolved! Some streams terminated before this command could finish! -> java.lang.NoClassDefFoundError: scala/c
HelloI do face:Some streams terminated before this command could finish!java.lang.NoClassDefFoundError: scala/compat/java8/FutureConverters$Running some very simple query on eventhub :df = spark \.readStream \.format("eventhubs") \.options(**ehConf) ...
- 19398 Views
- 1 replies
- 1 kudos
- 1 kudos
Of course just after writing that post I did realized how dummy this question is .... after adding scala_java8_compat_2_12_1_0_2.jar it works as expected
- 1 kudos
- 2126 Views
- 0 replies
- 0 kudos
Thoughts on how to improve string search queries
Please see sample code I am running below. What options can I explore to improve speed of query execution in such a scenario? Current full code takes about 4 hrs to run on 1.5 billion rows. Thanks!SELECT fullVisitorId ,VisitId ,EventDate ,PagePath ,d...
- 2126 Views
- 0 replies
- 0 kudos
- 4486 Views
- 1 replies
- 0 kudos
API for Databricks code functionality
I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...
- 4486 Views
- 1 replies
- 0 kudos
- 2041 Views
- 1 replies
- 0 kudos
Error ingesting files with databricks jobs
The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.
- 2041 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz
- 0 kudos
- 9885 Views
- 0 replies
- 0 kudos
Deleted the s3 bucket assocated with metastore
I deleted the aws s3 bucket for the databricks metastore by mistake.How to fix this? can I re-create the s3 bucket? Or can I delete the metastore (I don't have much data in it), and re-generate one? Thank you!
- 9885 Views
- 0 replies
- 0 kudos
- 15289 Views
- 2 replies
- 1 kudos
Resolved! Not able to access Account console page
Hello All,I'm not be able to access the account console page. I'm a portal admin, my workspace is premium, and yet the Databricks portal stays in a loop, always returning to the Workspaces Overview page and not going to the accounts console page so t...
- 15289 Views
- 2 replies
- 1 kudos
- 1 kudos
Guys, the problem is that I was not signed in as a Global Administrator in Azure AD. After that, I reset all the Browser settings and I was able to do that. Here's the tip.
- 1 kudos
- 2271 Views
- 1 replies
- 1 kudos
Performance issue while calling Sagemaker Endpoint in pyspark udf
Hi,I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.Please suggest how to call sagemaker endpoint from pyspark.Regards,S...
- 2271 Views
- 1 replies
- 1 kudos
- 2301 Views
- 0 replies
- 0 kudos
org.apache.spark.SparkException - FileReadException
Sometimes getting this kind of error "org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 12224.0 failed 4 times, most recent failure: Lost task 1.5 in stage 12224.0 (TID ) (12.xxx.x.xxx executor 1): com.datab...
- 2301 Views
- 0 replies
- 0 kudos
- 4055 Views
- 2 replies
- 0 kudos
how to edit or delete the post in this community
how to edit or delete the post in this community
- 4055 Views
- 2 replies
- 0 kudos
- 4940 Views
- 3 replies
- 1 kudos
Add Oracle Jar to Databricks cluster policy
I created a policy for users to use when they create their own Job clusters. When I'm editing the policy, I don't have the UI options for adding library (I can only see Definitions and Permissions tabs). I need to add via JSON the option to allows th...
- 4940 Views
- 3 replies
- 1 kudos
- 1 kudos
@adrianhernandez are you admin to workspace, if not you might be missing permissions, if you have policies enabled, admin can allow you.https://docs.databricks.com/en/administration-guide/clusters/policies.html#librariesif your workspace is Unity cat...
- 1 kudos
- 2490 Views
- 2 replies
- 1 kudos
Web terminal and clusters
Hi, I have come across this piece of documentation:Databricks does not support running Spark jobs from the web terminal. In addition, Databricks web terminal is not available in the following cluster types:Job clustersClusters launched with the DISAB...
- 2490 Views
- 2 replies
- 1 kudos
- 4910 Views
- 0 replies
- 0 kudos
Creating java UDF for Spark SQL
Hello, I have created a sample java UDF which masks few characters of a string. However I facing couple of issues when uploading and using it.First I could only import it, which for now is OK. But when do the following,create function udf_mask as 'ba...
- 4910 Views
- 0 replies
- 0 kudos
- 3176 Views
- 0 replies
- 2 kudos
dbutils.fs.ls MAX_LIST_SIZE_EXCEEDED
Hi!I'm experiencing different behaviours between two DBX Workspaces when trying to list file contents from an abfss: location.In workspace A running len(dbutils.fs.ls('abfss://~~@~~~~.dfs.core.windows.net/~~/')) results in "Out[1]: 1551", while runni...
- 3176 Views
- 0 replies
- 2 kudos
