- 1215 Views
- 0 replies
- 0 kudos
Suggestions for python its not working
Tab and Shift tab its not suggestin nothing in python codeTab only work as indentation and shift + tab do nothing.What can I do?
- 1215 Views
- 0 replies
- 0 kudos
Forum Posts
- 7155 Views
- 0 replies
- 0 kudos
How to retrieve a Job Name from the SparkContext
We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...
- 7155 Views
- 0 replies
- 0 kudos
- 4929 Views
- 6 replies
- 1 kudos
Long run time with %run command
My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...
- 4929 Views
- 6 replies
- 1 kudos
Latest Reply
" src="" />
- 1 kudos
My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...
- 1 kudos
- 948 Views
- 0 replies
- 0 kudos
case statements return same value
I have these 4 case statements count(*) as Total_claim_reciepts,count(case when claim_id like '%M%' and receipt_flag = 1 and is_firstpassclaim = 1 then 0 else claim_id end) as Total_claim_reciepts,count(case when claim_status ='DENIED' and claim_repa...
- 948 Views
- 0 replies
- 0 kudos
- 8661 Views
- 2 replies
- 0 kudos
NATIVE_XML_DATA_SOURCE_NOT_ENABLED
I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...
- 8661 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
i've tried already spark.read.format('com.databricks.spark.xml'). it receives the same error.
- 0 kudos
- 1825 Views
- 0 replies
- 0 kudos
Do we need to Request Databrikcs to Enable MOSIAC ML
HI Team,I am not seeing any specific articles/guides to use MOSIAC ML on Databricks. After Acquiring MOSIAC ML does anything got changed in terms of MOSIAC ML Use or just use just regular function
- 1825 Views
- 0 replies
- 0 kudos
- 2031 Views
- 3 replies
- 0 kudos
Databricks Delta table Insert Data Error
When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...
- 2031 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
seems ok to me, have you tried to display the data from table A and also the B/C join?
- 0 kudos
- 1065 Views
- 1 replies
- 0 kudos
how to make distributed predictions with sklearn model?
So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...
- 1065 Views
- 1 replies
- 0 kudos
Latest Reply
" src="" />
- 0 kudos
So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...
- 0 kudos
- 2395 Views
- 1 replies
- 0 kudos
problem with workspace after metastore deleted
I am completely new to Databricks AWS and start working on it a week ago. Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...
- 2395 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage. I can access the non-A...
- 0 kudos
- 2397 Views
- 3 replies
- 1 kudos
drop duplicates within watermark
Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...
- 2397 Views
- 3 replies
- 1 kudos
Latest Reply
- 3308 Views
- 2 replies
- 1 kudos
Resolved! Read zstd file from Databricks
I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0I've set PySpark commands as followspath = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd" schema = "some schema" df = spark.read...
- 3308 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
The available compression types are format dependent.For json, zstd is not (yet) available, whereas for parquet it is.
- 1 kudos
- 2692 Views
- 1 replies
- 1 kudos
How to schedule/refresh databricks alerts using REST API?
Hi, I am deploying Databricks SQL alerts using REST API. But I can't seem to figure out how to schedule their refresh task.I went through the documentation it says "Alerts can be scheduled using the sql_task type of the Jobs API, e.g. Jobs/Create"How...
- 2692 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
What they mention in the API docs is that you can create a job with sql_task of type Alert. To make it easier you can try creating the job first in the UI first and downloading the JSON config. Here is an example with the main parameters that should ...
- 1 kudos
- 8036 Views
- 1 replies
- 1 kudos
Shared access vs Single user access mode
I am running a notebook to get secret value from GCP Secret Manager. This is working well with Single user Access Mode, however it fail when i use a cluster with Shared Access mode. I have specified the same GCP service account on both of these clust...
- 8036 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Thanks for your response.I am using a cloud service account (same account that was used to create the workspace) on the cluster properties in case of both the single user cluster and on the shared user cluster. This service account has all the necess...
- 1 kudos
- 12083 Views
- 6 replies
- 0 kudos
Specify bottleneck for databricks cluster
Hi, Im trying to find out what is bottleneck on cluster when running loading process.Scenario: Loading CDC changes from sql server to Raw zone and merge changes into Bronze zone and then merge Bronze to Silver. All is orchestrated in data factory as ...
- 12083 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
stdout and stderr looks okay, do you have the log4j to share? You can make a doc out of it and share the doc here.
- 0 kudos
- 3037 Views
- 2 replies
- 0 kudos
Resolved! Unable to edit Catalog Owner
I created a Catalog and ownership was assigned to meI created databricks account-group on UC, added my user to this account-group, Assigned ownership of the catalog to this account-group.I deleted the account-groupNow, the catalog ownership is showin...
- 3037 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi, In addition to the previous message, you can refer to https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html#assign-a-metastore-admin to get more information on metastore and etc.
- 0 kudos
