cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
sanjay
by • Valued Contributor II
  • 1442 Views
  • 1 replies
  • 1 kudos

Performance issue while calling Sagemaker Endpoint in pyspark udf

Hi,I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.Please suggest how to call sagemaker endpoint from pyspark.Regards,S...

Get Started Discussions
Reply
  • 1442 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hi,I have pyspark dataframe which calls pyspark udf which in turn calls sagemaker endpoint. But when dataframe has more rows, endpoint start failing. Also it takes longer to process.Please suggest how to call sagemaker endpoint from pyspark.Regards,S...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
databicky
by • Contributor II
  • 2866 Views
  • 2 replies
  • 0 kudos

how to edit or delete the post in this community

how to edit or delete the post in this community

Get Started Discussions
Reply
  • 2866 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
how to edit or delete the post in this community

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
crvkumar
by • New Contributor II
  • 3265 Views
  • 1 replies
  • 0 kudos

Databricks Notebook says "Connecting.." for some users

For some users, after clicking on a notebook the screen says "connecting..." and the notebook does not open.The users are using Chrome browser and the same happens with Edge as well.What could be the reason?

Get Started Discussions
Reply
  • 3265 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
crvkumar
New Contributor II
  • 0 kudos

Thanks for the reply. But it does not happen to all users. It happens to some users in a user group. The clusters are fine. No errors in the cluster logs. The notebook is in fact not attached to any cluster.

  • 0 kudos
AdamStra2
by • New Contributor III
  • 1340 Views
  • 2 replies
  • 1 kudos

Web terminal and clusters

Hi, I have come across this piece of documentation:Databricks does not support running Spark jobs from the web terminal. In addition, Databricks web terminal is not available in the following cluster types:Job clustersClusters launched with the DISAB...

Get Started Discussions
Reply
  • 1340 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
AdamStra2
New Contributor III
  • 1 kudos

Hi @Retired_mod ,any update on my question? Thanks.

  • 1 kudos
1 More Replies
sm1274
by • New Contributor
  • 2096 Views
  • 0 replies
  • 0 kudos

Creating java UDF for Spark SQL

Hello, I have created a sample java UDF which masks few characters of a string. However I facing couple of issues when uploading and using it.First I could only import it, which for now is OK. But when do the following,create function udf_mask as 'ba...

Get Started Discussions
Reply
  • 2096 Views
  • 0 replies
  • 0 kudos
s_park
by • Honored Contributor
  • 21039 Views
  • 3 replies
  • 4 kudos

Training @ Data & AI World Tour 2023

Join your peers at the Data + AI World Tour 2023! Explore the latest advancements, hear real-world case studies and discover best practices that deliver data and AI transformation. From the Databricks Lakehouse Platform to open source technologies in...

Screenshot 2023-10-09 at 10.42.55 AM.png
Get Started Discussions
DAIWT
DAIWT_2023
Training
User_Group
Reply
  • 21039 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
VjGian15
New Contributor II
  • 4 kudos

Introducing Mini Flush: Your Ticket to Ultimate Casino Thrills!Are you ready to embark on an electrifying journey into the world of online gambling? If so, look no further than Vijaybet Online Casino! Our state-of-the-art platform is your gateway to ...

  • 4 kudos
2 More Replies
sg-vtc
by • New Contributor III
  • 1465 Views
  • 1 replies
  • 1 kudos

Resolved! Problem creating external delta table on non-AWS s3 bucket

I am testing Databricks with non-AWS S3 object storage.  I can access the non-AWS S3 bucket by setting these parameters:sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXX")sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key...

sgvtc_0-1697817308224.png sgvtc_1-1697817308223.png sgvtc_2-1697817308221.png
Get Started Discussions
external delta table
Reply
  • 1465 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sg-vtc
New Contributor III
  • 1 kudos

Found the solution to disable it.  Can close this question.

  • 1 kudos
AdamStra2
by • New Contributor III
  • 28018 Views
  • 0 replies
  • 3 kudos

Schema owned by Service Principal shows error in PBI

Background info:1. We have unity catalog enabled. 2. All of our jobs are run by Service Principal that has all necessary access it needs.Issue:One of the jobs checks existing schemas against the ones it is supposed to create in that given run and if ...

pic.png
Get Started Discussions
Reply
  • 28018 Views
  • 0 replies
  • 3 kudos
HHol
by • New Contributor
  • 7021 Views
  • 0 replies
  • 0 kudos

How to retrieve a Job Name from the SparkContext

We are currently starting to build certain data pipelines using Databricks.For this we use Jobs and the steps in these Jobs are implemented in Python Wheels.We are able to retrieve the Job ID, Job Run ID and Task Run Id in our Python Wheels from the ...

Get Started Discussions
Reply
  • 7021 Views
  • 0 replies
  • 0 kudos
cmilligan
by • Contributor II
  • 4518 Views
  • 6 replies
  • 1 kudos

Long run time with %run command

My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

Get Started Discussions
Reply
  • 4518 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in ...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
5 More Replies
hexoffender
by • New Contributor
  • 889 Views
  • 0 replies
  • 0 kudos

case statements return same value

I have these 4 case statements count(*) as Total_claim_reciepts,count(case when claim_id like '%M%' and receipt_flag = 1 and is_firstpassclaim = 1 then 0 else claim_id end) as Total_claim_reciepts,count(case when claim_status ='DENIED' and claim_repa...

Get Started Discussions
case statement
Reply
  • 889 Views
  • 0 replies
  • 0 kudos
Ajbi
by • New Contributor II
  • 8568 Views
  • 2 replies
  • 0 kudos

NATIVE_XML_DATA_SOURCE_NOT_ENABLED

I'm trying to read an xml file and receiving the following error. I've installed the maven library spark xml to the cluster, however I'm receiving the error. is there anything i'm missing?ErrorAnalysisException: [NATIVE_XML_DATA_SOURCE_NOT_ENABLED] N...

Get Started Discussions
Reply
  • 8568 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Ajbi
New Contributor II
  • 0 kudos

i've tried already  spark.read.format('com.databricks.spark.xml'). it receives the same error.  

  • 0 kudos
1 More Replies
eric2
by • New Contributor II
  • 1829 Views
  • 3 replies
  • 0 kudos

Databricks Delta table Insert Data Error

When trying to insert data into the Delta table in databricks, an error occurs as shown below. [TASK_WRITE_FAILED] Task failed while writing rows to abfss://cont-01@dlsgolfzon001.dfs.core.windows.net/dir-db999_test/D_RGN_INFO_TMP.In SQL, the results ...

Get Started Discussions
Reply
  • 1829 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

seems ok to me, have you tried to display the data from table A and also the B/C join?

  • 0 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels