cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
AdamStra2
by • New Contributor III
  • 15610 Views
  • 1 replies
  • 3 kudos

Resolved! Schema owned by Service Principal shows error in PBI

Background info:1. We have unity catalog enabled. 2. All of our jobs are run by Service Principal that has all necessary access it needs.Issue:One of the jobs checks existing schemas against the ones it is supposed to create in that given run and if ...

pic.png
Get Started Discussions
Reply
  • 15610 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @AdamStra2, This may be related to ownership chaining in SQL Server. Ownership chaining is a security feature in SQL Server that's designed to allow users to access objects in a database without requiring explicit permissions on the object itself....

  • 3 kudos
HHol
by • New Contributor
  • 2657 Views
  • 0 replies
  • 0 kudos

How to retrieve a Job Name from the SparkContext

We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...

Get Started Discussions
Reply
  • 2657 Views
  • 0 replies
  • 0 kudos
cmilligan
by • Contributor II
  • 3377 Views
  • 7 replies
  • 1 kudos

Long run time with %run command

My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

Get Started Discussions
Reply
  • 3377 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @cmilligan ,  - Long run times with %run command could be due to notebook size and complexity, Databricks cluster load, and network latency.- %run command executes another notebook immediately, making its functions and variables available in the c...

  • 1 kudos
6 More Replies
Ajbi
by • New Contributor II
  • 3896 Views
  • 2 replies
  • 0 kudos

NATIVE_XML_DATA_SOURCE_NOT_ENABLED

I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...

Get Started Discussions
Reply
  • 3896 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Ajbi
New Contributor II
  • 0 kudos

i've tried already  spark.read.format('com.databricks.spark.xml'). it receives the same error.  

  • 0 kudos
1 More Replies
liv1
by • New Contributor II
  • 1314 Views
  • 2 replies
  • 1 kudos

Structured Streaming from a delta table that is a dump of kafka and get the latest record per key

I'm trying to use Structured Streaming in scala to stream from a delta table that is a dump of a kafka topic where each record/message is an update of attributes for the key and no messages from kafka are dropped from the dump, but the value is flatt...

Get Started Discussions
Reply
  • 1314 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @liv1 ,  To get the latest message per key in your streaming job and perform stream-stream joins, you can use Databricks Delta's time travel feature in combination with foreachBatch(). You can use Delta's time travel feature to maintain the latest...

  • 1 kudos
1 More Replies
eric2
by • New Contributor II
  • 1447 Views
  • 3 replies
  • 0 kudos

Databricks Delta table Insert Data Error

When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...

Get Started Discussions
Reply
  • 1447 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

seems ok to me, have you tried to display the data from table A and also the B/C join?

  • 0 kudos
2 More Replies
ChaseM
by • New Contributor II
  • 731 Views
  • 2 replies
  • 0 kudos

how to make distributed predictions with sklearn model?

So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...

Get Started Discussions
Reply
  • 731 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @ChaseM, You can chunk a Spark DataFrame, convert each chunk to a Pandas DataFrame, and predict each chunk in parallel using worker nodes in Databricks. 

  • 0 kudos
1 More Replies
sg-vtc
by • New Contributor III
  • 1768 Views
  • 1 replies
  • 0 kudos

problem with workspace after metastore deleted

I am completely new to Databricks AWS and start working on it a week ago.  Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...

Get Started Discussions
Reply
  • 1768 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sg-vtc
New Contributor III
  • 0 kudos

I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage.  I can access the non-A...

  • 0 kudos
aerofish
by • New Contributor III
  • 1438 Views
  • 3 replies
  • 1 kudos

drop duplicates within watermark

Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...

Get Started Discussions
Reply
  • 1438 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
aerofish
New Contributor III
  • 1 kudos

Any maintainer can help me on this question??

  • 1 kudos
2 More Replies
bigt23
by • New Contributor II
  • 1940 Views
  • 2 replies
  • 1 kudos

Resolved! Read zstd file from Databricks

I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0I've set PySpark commands as followspath = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd" schema = "some schema" df = spark.read...

Get Started Discussions
Reply
  • 1940 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

The available compression types are format dependent.For json, zstd is not (yet) available, whereas for parquet it is.

  • 1 kudos
1 More Replies
Abhiqa
by • New Contributor
  • 1432 Views
  • 1 replies
  • 0 kudos

How to schedule/refresh databricks alerts using REST API?

Hi, I am deploying Databricks SQL alerts using REST API. But I can't seem to figure out how to schedule their refresh task.I went through the documentation it says "Alerts can be scheduled using the sql_task type of the Jobs API, e.g. Jobs/Create"How...

Abhiqa_0-1697550139434.png Abhiqa_1-1697550638337.png
Get Started Discussions
Alerts
REST API
sql query
sql_task
Reply
  • 1432 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
btafur
Contributor III
  • 0 kudos

What they mention in the API docs is that you can create a job with sql_task of type Alert. To make it easier you can try creating the job first in the UI first and downloading the JSON config. Here is an example with the main parameters that should ...

  • 0 kudos
naga_databricks
by • Contributor
  • 4401 Views
  • 2 replies
  • 1 kudos

Shared access vs Single user access mode

I am running a notebook to get secret value from GCP Secret Manager. This is working well with Single user Access Mode, however it fail when i use a cluster with Shared Access mode. I have specified the same GCP service account on both of these clust...

Get Started Discussions
Reply
  • 4401 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
naga_databricks
Contributor
  • 1 kudos

Thanks for your response.I am using a cloud service account (same account that was used to create the workspace) on the cluster properties in case of both the single user cluster and on the shared user cluster. This service account has all the necess...

  • 1 kudos
1 More Replies
alesventus
by • New Contributor III
  • 6776 Views
  • 6 replies
  • 0 kudos

Specify bottleneck for databricks cluster

Hi, Im trying to find out what is bottleneck on cluster when running loading process.Scenario: Loading CDC changes from sql server to Raw zone and merge changes into Bronze zone and then merge Bronze to Silver. All is orchestrated in data factory as ...

Get Started Discussions
Reply
  • 6776 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

stdout and stderr looks okay, do you have the log4j to share? You can make a doc out of it and share the doc here. 

  • 0 kudos
5 More Replies
Ankita1
by • New Contributor
  • 819 Views
  • 1 replies
  • 0 kudos

Deleting external table takes 8 hrs

Hi,I am trying to delete the data from the external partitioned table, it has around 3 years of data, and the partition is created on the date column.I am trying to delete each partition first and then the schema of the table, which takes around 8hrs...

Get Started Discussions
Reply
  • 819 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Ankita1 ,  If you need to delete a large amount of data from an external partitioned table, there are a few things you can do to try to reduce the time it takes: Deleting a large amount of data from an external partitioned table can take signific...

  • 0 kudos
smehta_0908
by • New Contributor II
  • 1638 Views
  • 2 replies
  • 0 kudos

Resolved! Unable to edit Catalog Owner

I created a Catalog and ownership was assigned to meI created databricks account-group on UC, added my user to this account-group, Assigned ownership of the catalog to this account-group.I deleted the account-groupNow, the catalog ownership is showin...

Get Started Discussions
Reply
  • 1638 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, In addition to the previous message, you can refer to https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html#assign-a-metastore-admin to get more information on metastore and etc.

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all