- 3216 Views
- 1 replies
- 0 kudos
Can the same Databricks account be used for both AWS and Azure
I am testing Databricks with non-AWS S3 object storage and want to test it with Databricks on Azure and Databricks on AWS. My Databricks account is currently using Databricks on AWS with metadata and a single node compute running. Can the same accou...
- 3216 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you Kaniz for posting the link. Looking at that, I believe the answer is:This is not possible in Databricks for now.
- 0 kudos
- 2142 Views
- 0 replies
- 0 kudos
Databricks data engineer associate got paused
Hi team,I've faced a disappointing experience during my first certification attempt and need help in resolving the issue.While attending the certification - Databricks data engineer associate on each 2-3 questions I kept receiving a message that the ...
- 2142 Views
- 0 replies
- 0 kudos
- 3700 Views
- 3 replies
- 0 kudos
Resolved! Databricks data engineer associate Exam got suspended.
@Cert-Team I have registered for Databricks certified data engineer associate exam and I have done all biometric and other prerequisites required and launched my exam. While writing the exam, it's exited twice with some technical issue although I don...
- 3700 Views
- 3 replies
- 0 kudos
- 2352 Views
- 0 replies
- 0 kudos
dataframe column structtype metadata not getting saved to unitycatalog
I have the below schemaschema = StructType([StructField(name="Test",dataType=StringType(),nullable=False,metadata={"description": "This is to test metadata description."})])data = [('Test1',), ('Test2',), ('Test3',)] df = spark.createDataFrame(data, ...
- 2352 Views
- 0 replies
- 0 kudos
- 4887 Views
- 1 replies
- 0 kudos
Connecting to Databricks using Python(VS Code)
I'm trying to connect to tables/views in Databricks using Python(via VS Code). However, I'm getting the following error:- File "C:\Users\XXXXXXXX\AppData\Roaming\Python\Python311\site-packages\urllib3\util\retry.py", line 592, in incrementraise MaxRe...
- 4887 Views
- 1 replies
- 0 kudos
- 7300 Views
- 0 replies
- 0 kudos
Get Network usage read and write from a job cluster on databricks
Hi ! I am writing because I am trying to get the data from network usage from a databricks job cluster. I know that the data is on spark-ui / storage but when the job cluster is terminate I cannot acces to the information , and I trying to make some ...
- 7300 Views
- 0 replies
- 0 kudos
- 11040 Views
- 3 replies
- 3 kudos
Azure Databricks User Alerts: query_result_table
Hi,How can I set up the notification email to show all of rows from query_result_table, not only first 10?
- 11040 Views
- 3 replies
- 3 kudos
- 3 kudos
I see that my previous message has been cut. So, I just wanted to check where can I change email setting using the code above, because when I open User > Account Settings > Notifications I can see only option like this: there is no place to configur...
- 3 kudos
- 5640 Views
- 1 replies
- 0 kudos
Unable to use job cluster for task in workflows
Hi,I have a workflow setup in Databricks using 12.2 LTS ML.I am trying to use a job cluster for the task but i am getting the following error: Spark Conf: ‘spark.databricks.acl.enabled’ is not allowed when choosing an access modeAs a result I have to...
- 5640 Views
- 1 replies
- 0 kudos
- 18990 Views
- 1 replies
- 1 kudos
Resolved! Some streams terminated before this command could finish! -> java.lang.NoClassDefFoundError: scala/c
HelloI do face:Some streams terminated before this command could finish!java.lang.NoClassDefFoundError: scala/compat/java8/FutureConverters$Running some very simple query on eventhub :df = spark \.readStream \.format("eventhubs") \.options(**ehConf) ...
- 18990 Views
- 1 replies
- 1 kudos
- 1 kudos
Of course just after writing that post I did realized how dummy this question is .... after adding scala_java8_compat_2_12_1_0_2.jar it works as expected
- 1 kudos
- 1977 Views
- 0 replies
- 0 kudos
Thoughts on how to improve string search queries
Please see sample code I am running below. What options can I explore to improve speed of query execution in such a scenario? Current full code takes about 4 hrs to run on 1.5 billion rows. Thanks!SELECT fullVisitorId ,VisitId ,EventDate ,PagePath ,d...
- 1977 Views
- 0 replies
- 0 kudos
- 4175 Views
- 1 replies
- 0 kudos
API for Databricks code functionality
I have a Databricks notebook for which I want to create an API. From that API I will have to call the notebook and perform certain operations. Result will be sent back to API. I dont want to do via Postman, as someone has to install Postman at their ...
- 4175 Views
- 1 replies
- 0 kudos
- 1755 Views
- 1 replies
- 0 kudos
Error ingesting files with databricks jobs
The source path that i want to ingest files with is:"gs://bucket-name/folder1/folder2/*/*.json"I have a file in this path that ends with ".json.gz" and the databricks job ingests this file even though it doesn't suppose to.How can i fix it?Thanks.
- 1755 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz
- 0 kudos
- 9720 Views
- 0 replies
- 0 kudos
Deleted the s3 bucket assocated with metastore
I deleted the aws s3 bucket for the databricks metastore by mistake.How to fix this? can I re-create the s3 bucket? Or can I delete the metastore (I don't have much data in it), and re-generate one? Thank you!
- 9720 Views
- 0 replies
- 0 kudos
- 14638 Views
- 2 replies
- 1 kudos
Resolved! Not able to access Account console page
Hello All,I'm not be able to access the account console page. I'm a portal admin, my workspace is premium, and yet the Databricks portal stays in a loop, always returning to the Workspaces Overview page and not going to the accounts console page so t...
- 14638 Views
- 2 replies
- 1 kudos
- 1 kudos
Guys, the problem is that I was not signed in as a Global Administrator in Azure AD. After that, I reset all the Browser settings and I was able to do that. Here's the tip.
- 1 kudos
- 2103 Views
- 1 replies
- 1 kudos
Performance issue while calling Sagemaker Endpoint in pyspark udf
Hi,I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.Please suggest how to call sagemaker endpoint from pyspark.Regards,S...
- 2103 Views
- 1 replies
- 1 kudos
