cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

s_park
by Databricks Employee
  • 21401 Views
  • 3 replies
  • 4 kudos

Training @ Data & AI World Tour 2023

Join your peers at the Data + AI World Tour 2023! Explore the latest advancements, hear real-world case studies and discover best practices that deliver data and AI transformation. From the Databricks Lakehouse Platform to open source technologies in...

Screenshot 2023-10-09 at 10.42.55 AM.png
Get Started Discussions
DAIWT
DAIWT_2023
Training
User_Group
  • 21401 Views
  • 3 replies
  • 4 kudos
Latest Reply
VjGian15
New Contributor II
  • 4 kudos

Introducing Mini Flush: Your Ticket to Ultimate Casino Thrills!Are you ready to embark on an electrifying journey into the world of online gambling? If so, look no further than Vijaybet Online Casino! Our state-of-the-art platform is your gateway to ...

  • 4 kudos
2 More Replies
sg-vtc
by New Contributor III
  • 1635 Views
  • 1 replies
  • 1 kudos

Resolved! Problem creating external delta table on non-AWS s3 bucket

I am testing Databricks with non-AWS S3 object storage.  I can access the non-AWS S3 bucket by setting these parameters:sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key...

sgvtc_0-1697817308224.png sgvtc_1-1697817308223.png sgvtc_2-1697817308221.png
Get Started Discussions
external delta table
  • 1635 Views
  • 1 replies
  • 1 kudos
Latest Reply
sg-vtc
New Contributor III
  • 1 kudos

Found the solution to disable it.  Can close this question.

  • 1 kudos
AdamStra2
by New Contributor III
  • 28102 Views
  • 0 replies
  • 3 kudos

Schema owned by Service Principal shows error in PBI

Background info:1. We have unity catalog enabled. 2. All of our jobs are run by Service Principal that has all necessary access it needs.Issue:One of the jobs checks existing schemas against the ones it is supposed to create in that given run and if ...

pic.png
  • 28102 Views
  • 0 replies
  • 3 kudos
HHol
by New Contributor
  • 7243 Views
  • 0 replies
  • 0 kudos

How to retrieve a Job Name from the SparkContext

We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...

  • 7243 Views
  • 0 replies
  • 0 kudos
cmilligan
by Contributor II
  • 5112 Views
  • 6 replies
  • 1 kudos

Long run time with %run command

My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

  • 5112 Views
  • 6 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
5 More Replies
hexoffender
by New Contributor
  • 998 Views
  • 0 replies
  • 0 kudos

case statements return same value

I have these 4 case statements count(*) as Total_claim_reciepts,count(case when claim_id like '%M%' and receipt_flag = 1 and is_firstpassclaim = 1 then 0 else claim_id end) as Total_claim_reciepts,count(case when claim_status ='DENIED' and claim_repa...

  • 998 Views
  • 0 replies
  • 0 kudos
Ajbi
by New Contributor II
  • 8717 Views
  • 2 replies
  • 0 kudos

NATIVE_XML_DATA_SOURCE_NOT_ENABLED

I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...

  • 8717 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ajbi
New Contributor II
  • 0 kudos

i've tried already  spark.read.format('com.databricks.spark.xml'). it receives the same error.  

  • 0 kudos
1 More Replies
eric2
by New Contributor II
  • 2106 Views
  • 3 replies
  • 0 kudos

Databricks Delta table Insert Data Error

When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...

  • 2106 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

seems ok to me, have you tried to display the data from table A and also the B/C join?

  • 0 kudos
2 More Replies
ChaseM
by New Contributor II
  • 1093 Views
  • 1 replies
  • 0 kudos

how to make distributed predictions with sklearn model?

So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...

  • 1093 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
So I have a sklearn style model which predicts on a pandas df. The data to predict on is a spark df. Simply converting the whole thing at once to pandas and predicting is not an option due to time and memory constraints.Is there a way to chunk a spar...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
sg-vtc
by New Contributor III
  • 2526 Views
  • 1 replies
  • 0 kudos

problem with workspace after metastore deleted

I am completely new to Databricks AWS and start working on it a week ago.  Pls excuse me if I ask or did something silly.I created a workspace and a single node cluster for testing. A metastore was created from Databricks quickstart and it was automa...

  • 2526 Views
  • 1 replies
  • 0 kudos
Latest Reply
sg-vtc
New Contributor III
  • 0 kudos

I restarted the compute node and this problem went away.ErrorClass=METASTORE_DOES_NOT_EXIST] Metastore 'b11fb1a0-a462-4dfb-b91b-e0795fde10b0' does not exist.New question: I am testing Databricks with non-AWS S3 object storage.  I can access the non-A...

  • 0 kudos
aerofish
by New Contributor III
  • 2527 Views
  • 3 replies
  • 1 kudos

drop duplicates within watermark

Recently we are using structured streaming to ingest data. We want to use watermark to drop duplicated event. But We encountered some wired behavior and unexpected exception. Anyone can help me to explain what is the expected behavior and how should ...

  • 2527 Views
  • 3 replies
  • 1 kudos
Latest Reply
aerofish
New Contributor III
  • 1 kudos

Any maintainer can help me on this question??

  • 1 kudos
2 More Replies
bigt23
by New Contributor II
  • 3580 Views
  • 2 replies
  • 1 kudos

Resolved! Read zstd file from Databricks

I just started to read `zstd` compressed file in Databricks on Azure, Runtime 14.1 on Spark 3.5.0I've set PySpark commands as followspath = f"wasbs://{container}@{storageaccount}.blob.core.windows.net/test-zstd" schema = "some schema" df = spark.read...

  • 3580 Views
  • 2 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

The available compression types are format dependent.For json, zstd is not (yet) available, whereas for parquet it is.

  • 1 kudos
1 More Replies
Abhiqa
by New Contributor II
  • 3053 Views
  • 1 replies
  • 1 kudos

How to schedule/refresh databricks alerts using REST API?

Hi, I am deploying Databricks SQL alerts using REST API. But I can't seem to figure out how to schedule their refresh task.I went through the documentation it says "Alerts can be scheduled using the sql_task type of the Jobs API, e.g. Jobs/Create"How...

Abhiqa_0-1697550139434.png Abhiqa_1-1697550638337.png
Get Started Discussions
Alerts
REST API
sql query
sql_task
  • 3053 Views
  • 1 replies
  • 1 kudos
Latest Reply
btafur
Databricks Employee
  • 1 kudos

What they mention in the API docs is that you can create a job with sql_task of type Alert. To make it easier you can try creating the job first in the UI first and downloading the JSON config. Here is an example with the main parameters that should ...

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors